Scripts et taches planifiees » Historique » Révision 25
Révision 24 (Florent Torregrosa, 04/08/2013 16:27) → Révision 25/118 (Florent Torregrosa, 04/08/2013 17:16)
Afin de gagner du temps et d'éviter les erreurs humaines, des scripts ont été écrits tout au long du projet. Certains sont même exécutés automatiquement toutes les semaines. {{>toc}} h1. Les tâches planifiées Pour exécuter ces tâches régulièrement, on utilise le "crontab":http://fr.wikipedia.org/wiki/Crontab. Il s'agit d'un programme installé sur notre serveur. Pour voir et modifier la liste des actions : # se connecter au serveur : <code>ssh assos@sas1.centrale-marseille</code> # taper la commande pour voir le crontab <code>crontab -l</code> # taper la commande pour modifier le crontab <code>crontab -e</code>. /!\ Ne pas effectuer cette opération avant de s'être renseigné sur "vi":http://fr.wikipedia.org/wiki/Vi (l'éditeur de texte utilisé qui n'est pas vraiment intuitif :p) et sur "la syntaxe du crontab":http://fr.wikipedia.org/wiki/Crontab#Modification ! {{tip(Pour utiliser un autre editeur que vi taper : export EDITOR=editor_name exemple : vim, nano)}} Voici la liste des tâches effectuées régulièrement. h2. La réinitialisation des variables dangeureuses Sur l'installation d7 principalement : une fois par semaine Cette tâche consiste à réinitialiser certaines variables qui donnent des droits considérés comme trop permissifs donc dangereux aux administrateurs des sites ou nécessaire pour d'autres scripts tels que piwik ou security_review. sites. En voici quelques exemples la liste : <pre> <code class="bash"> drush -y drushall_atest vset error_level 0 --yes </code> </pre> Cette commande permet de ne pas afficher les messages d'erreurs. d'erreurs aux utilisateurs autre que les administrateurs. En effet, ils contiennent parfois des informations sensibles sur l'installation et ne doivent donc pas être divulgués à n'importe qui. <pre> <code class="php"> drush -y drushall_atest php-eval variable_set\(\'allow_authorize_operations\',FALSE\)\; </code> </pre> Cette commande permet de ne pas autoriser les utilisateurs à installer et mettre à jour des modules via l'interface du site (fonctionnalité introduite dans drupal7). En effet, seul le club Drupal maintient les codes des projet, afin d'en garantir la pérennité. <pre <code class="php"> drushall_atest vset --always-set reverse_proxy TRUE drush -y drushall_atest vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]' </code> </pre> Ces commandes permettent de déclarer à Drupal drupal les serveurs proxy du CRI afin d'éviter qu'il ne répertorie tous les visiteurs comme ayant l'adresse des serveurs sus-cités. sus-cités serveurs. **Pas fini : voir http://assos.centrale-marseille.fr/content/t%C3%A2che/d%C3%A9clarer-les-proxy-du-cri-%C3%A0-drupal** h3. Comment le lancer ? N'importe où, taper <code>reinit_var.sh</code>. h2. La sauvegarde des bases de données Sur les installations d6 et d7 : une fois par semaine On utilise les [[Scripts_et_taches_planifiees#Tout-sauvegarder|scripts de sauvegarde créés par le club Drupal]]. h2. La purge des sauvegardes de bdd Sur les installations d6 et d7 : toutes les semaines Cette tâche utilise le script de [[Scripts_et_taches_planifiees#purge_des_sauvegardes.sh|purge des sauvegardes]] afin de libérer de l'espace disque en supprimant les sauvegardes de bdd les plus vieilles. h1. Liste des scripts à disposition Les scripts utilisés sont hébergés dans le répertoire bin du compte assos. h2. all-reset-permissions.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # This script puts the correct permissions to sites folders, settings.php and scripts. ######### drupal 6 for dir in `find $d6_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all ! -name images ! -name languages` ; do chmod 755 $dir chmod 400 $dir/settings.php done ######### drupal 7 for dir in `find $d7_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all` ; do chmod 755 $dir chmod 400 $dir/settings.php done ####### bin chmod -R 700 $dir_scripts </code> </pre> h2. contrib-usage.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: Drupal_version, project_status, project_name # List sites that have the project_name with the corresponding project_status. check_arguments $# 3 if [ $1 = d6 ] ; then cd $d6_dir_sites else cd $d7_dir_sites fi for dir in `find . -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do # List non-core projects that correspond to the status. # Keep project_name if listed. # Count line result. 0 if not listed or 1 if listed. # Print site_dir if listed. cd $dir; if [ 1 -eq `drush pml --no-core --status=$2 | grep $3 | wc -l` ] ; then echo $dir; fi done </code> </pre> h2. d6-all-drush.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: drush_command check_arguments $# 1 for dir in `find $d6_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do cd $dir echo $dir drush $* done </code> </pre> h2. d6-all-dump-full.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` # Dump D6 database with webassos user. mysqldump --single-transaction webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY > $d6_dir_full_backup/webassos/$current_date.webassos.sql # Dump D6 form database with forum user. mysqldump forum -h myweb.serv.int -u forum --password=dtcAltF12 > $d6_dir_full_backup/forum/$current_date.forum.sql </code> </pre> h2. d6-all-update-contrib.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh d6-all-drush.sh -y en update # Launch cron to check for updates. d6-all-drush.sh -q cron d6-all-drush.sh cc all d6-all-dump-full.sh d6-all-drush.sh -y upc --no-core d6-all-drush.sh -y updb d6-all-drush.sh -y dis update # Cron report. d6-all-drush.sh cron </code> </pre> h2. d6-dump-individual.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: auto or manual, site_prefix # WARNING : backup are regularly flushed, put your backup in a safe place. cd $dir_individual_backup #récupération des tables du site dans le fichier liste_tables.temp tables='_%' liste="$1$tables" mysql -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY -BNe "show tables like '"$liste"'" webassos | tr '\r\n' ' ' > liste_tables.temp #transformation de cette liste en une variable var=$(cat liste_tables.temp) #sauvegarde de toutes ces tables current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` suffixe="_dump$current_date.sql" fichier="$1$suffixe" if [ $1 = 'auto' ] ; then mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_auto_backup/$fichier else mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_manual_backup/$fichier fi #suppression du fichier temporaire utilisé rm liste_tables.temp </code> </pre> h2. d6-reset-variables.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh cd $d6_dir_sites d6-all-drush.sh vset --always-set reverse_proxy TRUE d6-all-drush.sh vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]' drush @sites ev "variable_set('update_notify_emails', array($email_multi_assos));" --yes </code> </pre> h2. d7-all-drush-logged.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: drush_command check_arguments $# 1 for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir current_date=`date "+%Y-%m-%d_%Hh:%Mm"` echo "======$current_date======" >> $d7_dir_log/$x.log echo "Arguments are: $* " >> $d7_dir_log/$x.log drush $* >> $d7_dir_log/$x.log #To have the same caracters limit we needed 'end' that have 4 caracters, we used the Croate language for this ! LOL echo -e "=============KRAJ=============\n" >> $d7_dir_log/$x.log done </code> </pre> h2. d7-all-drush.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: drush_command check_arguments $# 1 for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir echo $dir drush $* done </code> </pre> h2. d7-all-dump-full.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` # Dump D7 databases at once. mysqldump -h $db_server -u $db_user -pNoNo82jJ --all-databases > $d7_dir_full_backup/myassos/$current_date.d7_full.sql </code> </pre> h2. d7-all-dump-individual.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: auto or manual current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir drush cc all if [ $1 = 'auto' ] ; then drush sql-dump --result-file="$d7_dir_individual_auto_backup/$dir/$current_date.$dir.sql" else drush sql-dump --result-file="$d7_dir_individual_manual_backup/$dir/$current_date.$dir.sql" fi done </code> </pre> h2. d7-all-releases-info.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # List version of a module or theme that exists in sites/all/* # and all versions that are available for same project on drupal.org # modules for dir in `find $d7_dir_sites/all/modules -maxdepth 1 -mindepth 1 -type d` ; do drush pm-releases $dir done # themes for dir in `find $d7_dir_sites/all/themes -maxdepth 1 -mindepth 1 -type d` ; do drush pm-releases $dir done </code> </pre> h2. d7-all-reset-variables.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir echo $dir d7-reset-variables.sh done </code> </pre> h2. d7-all-security-review.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # grep -v success to keep site_name and error lines. bash -c 'd7-all-drush.sh security-review |& grep -v success | mail -s "Security review" $email_multi_assos' </code> </pre> h2. d7-all-update-contrib.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh d7-all-drush.sh -y en update # Launch cron to check for updates. d7-all-drush.sh -q cron d7-all-drush.sh cc all d7-all-dump-individual.sh d7-all-drush-logged.sh -y upc --no-core d7-all-drush.sh -y updb d7-all-drush.sh -y dis update # Cron report. d7-all-drush.sh cron | mail -s "Cron report: d7-all-update-contrib.sh" $email_multi_assos </code> </pre> h2. d7-all-update-localisation.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh d7-all-drush.sh -y en l10n_update d7-all-drush.sh l10n-update-refresh d7-all-drush.sh l10n-update d7-all-drush.sh -y dis l10n_update </code> </pre> h2. d7-create-site.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-config-site.sh $1 . scripts-utils.sh # ARGS: site name ######## Exceptions check_arguments $# 1 # "-" is forbidden because it provokes database error. if `echo $1 | grep -` ; then echo '"-" is forbidden in the site name' exit 1 fi # Site name length must be lower or equal to 16 due to database limitations. if [ $(echo $1 | wc -n) -le 16 ] ; then echo "site name can't have more than 16 characters" exit 1 fi ###### Initialisation cd $d7_dir db_password=`ask_password_db $db_server $db_user` site_password=`gen_password` # Check if site database already exists. if mysql -h $db_server -u $db_user -e "USE $d7_site_name" -p$db_password 2>/dev/null ; then echo "Database $d7_site_name already exists" exit 1 fi # Check if site folder already exists. if [ -d $d7_site_dir ] ; then echo "Folder $d7_site_dir already exists" exit 1 fi ###### Main mkdir $d7_site_dir # Backup requirements mkdir $d7_dir_individual_auto_backup/$d7_site_name mkdir $d7_dir_individual_manual_backup/$d7_site_name current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql # Create and grant privileges on database mysql -h $db_server -u $db_user -e "CREATE DATABASE $d7_site_name" -p$db_password mysql -h $db_server -u $db_user -e "GRANT ALL PRIVILEGES ON $d7_site_name.* TO '$d7_site_name'@'%' IDENTIFIED BY '$site_password'" -p$db_password # Create settings.php sed "s/\%\%DBUSER\%\%/$d7_site_name/ ; s/\%\%DBNAME\%\%/$d7_site_name/ ; s/\%\%DBPASS\%\%/$site_password/ ; s/\%\%nomsite\%\%/$d7_site_name/" < $d7_settings_template > $d7_site_settings # Create symbolic link cd $d7_dir ln -s . $d7_site_name # Next Instructions echo "Go to http://assos.centrale-marseille.fr/$d7_site_name/install.php to continue." echo "Press enter when ready to go on." read key # Init variables d7-reset-variables.sh $d7_site_name # Permissions chmod -R 755 $d7_site_dir chmod 400 $d7_site_settings # Last instructions echo "Last instructions:" echo "- Advice the webmaster to close account creation on the website" echo "- Give the webmaster a link to the club Drupal's tutorials " echo "- Create a node of type \"Site\" on default" echo "- Register the webmaster on webmasters@listes.centrale-marseille.fr" </code> </pre> h2. d7-delete-site.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-config-site.sh $1 . scripts-utils.sh # ARGS: site name. check_arguments $# 1 db_password=`ask_password_db $db_server $db_user` # Delete database. mysql -h $db_server -u $db_user -p$db_password -e "IF EXISTS DROP DATABASE $d7_site_name" # Delete MYSQL user. mysql -h $db_server -u $db_user -p$db_password -e "DROP USER '$d7_site_name'@'%'" # Delete site's folder. chmod -R 700 $d7_site_dir rm -r $d7_site_dir # Delete symbolic link. rm $d7_dir/$d7_site_name # Delete database backups. rm -r $d7_dir_individual_auto_backup/assos.centrale-marseille.fr.$d7_site_name rm -r $d7_dir_individual_manual_backup/assos.centrale-marseille.fr.$d7_site_name echo "Don't forget to:" echo "- Refresh node site on default" echo "- Check particular behavior (normally detailed on site node)" </code> </pre> h2. d7-reset-variables.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: [site_name] if [ ! -z $1 ] ; then . scripts-config-site.sh $1 cd $d7_site_dir fi current_timestamp=`date "+%s"` ####### General variables drush -y vset --always-set reverse_proxy TRUE drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]' drush -y ev "variable_set('update_notify_emails', array($email_multi_assos));" drush -y vset error_level 0 ###### Hidden captcha drush -y en hidden_captcha # Log wrong answers. drush -y vset captcha_log_wrong_responses 1 # Use hidden captcha for all forms. drush -y sqlq --db-prefix "UPDATE {captcha_points} SET module = 'hidden_captcha', captcha_type = 'Hidden CAPTCHA' WHERE module is NULL;" # Flush captcha cache. drush -y vdel captcha_placement_map_cache # Randomely generate a math question as the label of the hidden captcha field. drush -y vset hidden_captcha_label "$RANDOM + $RANDOM" ####### Piwik drush -y en piwik drush -y vset piwik_site_id "101" drush -y vset piwik_url_http "http://piwik.centrale-marseille.fr/" drush -y vset piwik_url_https "https://piwik.centrale-marseille.fr/" # Piwik cache. drush -y vset piwik_cache 1 drush -y vset piwik_visibility_roles "1" # Anonymous and authenticated statistics. drush -y vset --format=json piwik_roles '{"1":0,"2":0}' drush -y vset piwik_page_title_hierarchy 1 # Activate local search. drush -y vset piwik_site_search 1 ###### Security review # For untrusted roles: # 1 : anonymous user # 2 : authenticated user # 3 : administrator drush -y en security_review drush -y ev "variable_set('security_review_untrusted_roles', array('1'));" # Security review can't check for files permissions on multi_assos. drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '1', skiptime = $current_timestamp, skipuid= '1' WHERE reviewcheck IN ('file_perms');" drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '0', skiptime = 0, skipuid= NULL WHERE reviewcheck NOT IN ('file_perms');" </code> </pre> h2. delete-temporary-files.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh find $dir_multi_assos -name '*~' -delete </code> </pre> h2. disk-usage.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh echo -e "Quotas: 9.1 GB\n" echo "Assos:" du -hcs $dir_multi_assos | grep total echo -e "\n" echo "D6:" cd $d6_dir_sites du -h -d 1 echo -e "\n" echo "D7:" cd $d7_dir_sites du -h -d 1 echo -e "\n" echo "Logs:" du -hcs $dir_log | grep total echo -e "\n" echo "Backups:" cd $dir_backup du -h -d 2 </code> </pre> h2. flush-files.sh <pre> <code class="php"> #!/bin/sh # ARGS: number of files to keep, email address # usage: flush files from a folder except a number equal to argument. # Check if scripts-config.sh is imported. if [ -z $scripts_utils ] ; then echo "Import of scripts-utils.sh required." exit 1 fi check_arguments $# 2 backups_number=`ls | wc -l` let "number_of_backups_to_delete = backups_number - $1" if [ $number_of_backups_to_delete -gt 0 ] ; then ls | head -$number_of_backups_to_delete | xargs rm else dir=`pwd` echo "There are not enough files in $dir to Flush it. Check if backup script works fine." | mail -s "[db] $dir has a backup problem" $2 fi </code> </pre> h2. flush-full-databases-backups.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: Drupal version cd $dir_full_backup/$1 for dir in `ls` ; do cd $dir flush-files.sh $db_full_backup_number $email_multi_assos done </code> </pre> h2. flush-individual-databases-backups.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: manual or auto, Drupal version cd $dir_individual_backup/$1/$2 for dir in `ls` ; do cd $dir if [ $1 = 'auto' ] ; then flush-files.sh $db_individual_auto_backup_number $email_multi_assos else flush-files.sh $db_individual_manual_backup_number $email_multi_assos fi done </code> </pre> h2. scripts-config-site.sh <pre> <code class="php"> #!/bin/sh # Sets useful variables of a site. # ARGS: site_name # Requires scripts-config.sh # Check if scripts-config.sh is imported. if [ -z $scripts_config ] ; then echo "Import of scripts-config.sh required." exit 1 fi if [ -z $1 ] ; then echo "This script needs a site name as argument." exit 1 fi scripts_config_site='imported' d7_site_name=$1 d7_site_dir="$d7_dir_sites/assos.centrale-marseille.fr.$d7_site_name" d7_site_settings="$d7_site_dir/settings.php" </code> </pre> h2. scripts-config.sh <pre> <code class="php"> #!/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/users/guest/assos/bin PATH=$PATH:/usr/guest/assos/bin scripts_config='imported' email_multi_assos="assos@centrale-marseille.fr" dir_multi_assos='/users/guest/assos' dir_scripts="$dir_multi_assos/bin" dir_template="$dir_multi_assos/Desktop" ###### Database db_server="myassos.serv.int" db_user="assos" dir_log="$dir_multi_assos/log" ###### Backup db_full_backup_number=3 db_individual_manual_backup_number=3 db_individual_auto_backup_number=3 dir_backup="$dir_multi_assos/backup" dir_full_backup="$dir_backup/full" dir_individual_backup="$dir_backup/individual" dir_individual_manual_backup="$dir_individual_backup/manual" dir_individual_auto_backup="$dir_individual_backup/auto" ###### D7 variables d7_dir="$dir_multi_assos/htmltest" d7_dir_sites="$d7_dir/sites" ### Log d7_dir_log="$dir_log/d7" ### Backup d7_dir_full_backup="$dir_full_backup/d7" d7_dir_individual_manual_backup="$dir_individual_manual_backup/d7" d7_dir_individual_auto_backup="$dir_individual_auto_backup/d7" ### Template d7_settings_template_name="settings-D7-bddinde-template.php" d7_settings_template="$dir_template/$d7_settings_template_name" ###### D6 variables d6_dir="$dir_multi_assos/html" d6_dir_sites="$d6_dir/sites" ### Backup d6_dir_full_backup="$dir_full_backup/d6" d6_dir_individual_manual_backup="$dir_individual_manual_backup/d6" d6_dir_individual_auto_backup="$dir_individual_auto_backup/d6" </code> </pre> h2. scripts-utils.sh <pre> <code class="php"> #!/bin/sh # This script contains useful functions for other scripts. # Check if scripts-config.sh is imported. if [ -z $scripts_config ] ; then echo "Import of scripts-config.sh required." exit 1 fi scripts_utils='imported' ask_password() { # read -s doesn't work with sh. # usage: pass=`ask_password "password please:"` echo $1 >&2 echo -n ">" >&2 stty_avant=`stty -g` stty -echo read p stty $stty_avant echo "$p" unset p } ask_password_db() { # ARGS: server_name, user_name local db_password="pour_boucler" # empty db request to validate password while ! mysql -h $1 -u $2 -p$db_password -e "" 2>/dev/null ; do db_password=`ask_password "database password:"` echo "" done } generate_password(){ # ARGS: [password_length] # The password contains special characters. '/' must be excluded to avoid sed malfunction. local site_password='/' if [ -z $1 ] ; then local password_length=20 else local password_length=$1 fi while echo "$site_password" | grep -Fq '/' ; do site_password=`dd if=/dev/urandom count=1 | uuencode -m - | head -n 2 | tail -n 1 | cut -c-$password_length` done echo $site_password } count_d7_sites(){ find $d7_dir_sites -type d ! -name all -maxdepth 1 | wc -l } check_arguments() { # ARGS: number of arguments passed to script, number of arguments required if [ $1 -le $2 ] ; then echo "Number of arguments insuffisant." exit 1 fi } </code> </pre>