Scripts et taches planifiees » History » Revision 26
Revision 25 (Florent Torregrosa, 08/04/2013 05:16 PM) → Revision 26/118 (Florent Torregrosa, 08/04/2013 05:19 PM)
Afin de gagner du temps et d'éviter les erreurs humaines, des scripts ont été écrits tout au long du projet. Certains sont même exécutés automatiquement toutes les semaines. {{>toc}} h1. Les tâches planifiées h2. Crontab Pour exécuter ces tâches régulièrement, on utilise le "crontab":http://fr.wikipedia.org/wiki/Crontab. Il s'agit d'un programme installé sur notre serveur. Pour voir et modifier la liste des actions : # se connecter au serveur : <code>ssh assos@sas1.centrale-marseille</code> # taper la commande pour voir le crontab <code>crontab -l</code> # taper la commande pour modifier le crontab <code>crontab -e</code>. /!\ Ne pas effectuer cette opération avant de s'être renseigné sur "vi":http://fr.wikipedia.org/wiki/Vi (l'éditeur de texte utilisé qui n'est pas vraiment intuitif :p) et sur "la syntaxe du crontab":http://fr.wikipedia.org/wiki/Crontab#Modification ! {{tip(Pour utiliser un autre editeur que vi taper : export EDITOR=editor_name exemple : vim, nano)}} <pre> <code class="php"> Ici bientôt le crontab </code> </pre> h3. Voici la liste des tâches effectuées régulièrement. h2. La réinitialisation des variables Cette tâche consiste à réinitialiser certaines variables qui donnent des droits considérés comme trop permissifs donc dangereux aux administrateurs des sites ou nécessaire pour d'autres scripts tels que piwik ou security_review. En voici quelques exemples : <pre> <code class="bash"> drush -y vset error_level 0 </code> </pre> Cette commande permet de ne pas afficher les messages d'erreurs. En effet, ils contiennent parfois des informations sensibles sur l'installation et ne doivent donc pas être divulgués à n'importe qui. <pre> <code class="php"> drush -y vset --always-set reverse_proxy TRUE drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]' </code> </pre> Ces commandes permettent de déclarer à Drupal les serveurs proxy du CRI afin d'éviter qu'il ne répertorie tous les visiteurs comme ayant l'adresse des serveurs sus-cités. **Pas fini : voir http://assos.centrale-marseille.fr/content/t%C3%A2che/d%C3%A9clarer-les-proxy-du-cri-%C3%A0-drupal** h2. La sauvegarde des bases de données Sur les installations d6 et d7 : une fois par semaine On utilise les [[Scripts_et_taches_planifiees#Tout-sauvegarder|scripts de sauvegarde créés par le club Drupal]]. h2. La purge des sauvegardes de bdd Sur les installations d6 et d7 : toutes les semaines Cette tâche utilise le script de [[Scripts_et_taches_planifiees#purge_des_sauvegardes.sh|purge des sauvegardes]] afin de libérer de l'espace disque en supprimant les sauvegardes de bdd les plus vieilles. h1. Liste des scripts à disposition Les scripts utilisés sont hébergés dans le répertoire bin du compte assos. h2. all-reset-permissions.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # This script puts the correct permissions to sites folders, settings.php and scripts. ######### drupal 6 for dir in `find $d6_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all ! -name images ! -name languages` ; do chmod 755 $dir chmod 400 $dir/settings.php done ######### drupal 7 for dir in `find $d7_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all` ; do chmod 755 $dir chmod 400 $dir/settings.php done ####### bin chmod -R 700 $dir_scripts </code> </pre> h2. contrib-usage.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: Drupal_version, project_status, project_name # List sites that have the project_name with the corresponding project_status. check_arguments $# 3 if [ $1 = d6 ] ; then cd $d6_dir_sites else cd $d7_dir_sites fi for dir in `find . -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do # List non-core projects that correspond to the status. # Keep project_name if listed. # Count line result. 0 if not listed or 1 if listed. # Print site_dir if listed. cd $dir; if [ 1 -eq `drush pml --no-core --status=$2 | grep $3 | wc -l` ] ; then echo $dir; fi done </code> </pre> h2. d6-all-drush.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: drush_command check_arguments $# 1 for dir in `find $d6_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do cd $dir echo $dir drush $* done </code> </pre> h2. d6-all-dump-full.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` # Dump D6 database with webassos user. mysqldump --single-transaction webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY > $d6_dir_full_backup/webassos/$current_date.webassos.sql # Dump D6 form database with forum user. mysqldump forum -h myweb.serv.int -u forum --password=dtcAltF12 > $d6_dir_full_backup/forum/$current_date.forum.sql </code> </pre> h2. d6-all-update-contrib.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh d6-all-drush.sh -y en update # Launch cron to check for updates. d6-all-drush.sh -q cron d6-all-drush.sh cc all d6-all-dump-full.sh d6-all-drush.sh -y upc --no-core d6-all-drush.sh -y updb d6-all-drush.sh -y dis update # Cron report. d6-all-drush.sh cron </code> </pre> h2. d6-dump-individual.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: auto or manual, site_prefix # WARNING : backup are regularly flushed, put your backup in a safe place. cd $dir_individual_backup #récupération des tables du site dans le fichier liste_tables.temp tables='_%' liste="$1$tables" mysql -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY -BNe "show tables like '"$liste"'" webassos | tr '\r\n' ' ' > liste_tables.temp #transformation de cette liste en une variable var=$(cat liste_tables.temp) #sauvegarde de toutes ces tables current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` suffixe="_dump$current_date.sql" fichier="$1$suffixe" if [ $1 = 'auto' ] ; then mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_auto_backup/$fichier else mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_manual_backup/$fichier fi #suppression du fichier temporaire utilisé rm liste_tables.temp </code> </pre> h2. d6-reset-variables.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh cd $d6_dir_sites d6-all-drush.sh vset --always-set reverse_proxy TRUE d6-all-drush.sh vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]' drush @sites ev "variable_set('update_notify_emails', array($email_multi_assos));" --yes </code> </pre> h2. d7-all-drush-logged.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: drush_command check_arguments $# 1 for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir current_date=`date "+%Y-%m-%d_%Hh:%Mm"` echo "======$current_date======" >> $d7_dir_log/$x.log echo "Arguments are: $* " >> $d7_dir_log/$x.log drush $* >> $d7_dir_log/$x.log #To have the same caracters limit we needed 'end' that have 4 caracters, we used the Croate language for this ! LOL echo -e "=============KRAJ=============\n" >> $d7_dir_log/$x.log done </code> </pre> h2. d7-all-drush.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-utils.sh # ARGS: drush_command check_arguments $# 1 for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir echo $dir drush $* done </code> </pre> h2. d7-all-dump-full.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` # Dump D7 databases at once. mysqldump -h $db_server -u $db_user -pNoNo82jJ --all-databases > $d7_dir_full_backup/myassos/$current_date.d7_full.sql </code> </pre> h2. d7-all-dump-individual.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: auto or manual current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir drush cc all if [ $1 = 'auto' ] ; then drush sql-dump --result-file="$d7_dir_individual_auto_backup/$dir/$current_date.$dir.sql" else drush sql-dump --result-file="$d7_dir_individual_manual_backup/$dir/$current_date.$dir.sql" fi done </code> </pre> h2. d7-all-releases-info.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # List version of a module or theme that exists in sites/all/* # and all versions that are available for same project on drupal.org # modules for dir in `find $d7_dir_sites/all/modules -maxdepth 1 -mindepth 1 -type d` ; do drush pm-releases $dir done # themes for dir in `find $d7_dir_sites/all/themes -maxdepth 1 -mindepth 1 -type d` ; do drush pm-releases $dir done </code> </pre> h2. d7-all-reset-variables.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do cd $dir echo $dir d7-reset-variables.sh done </code> </pre> h2. d7-all-security-review.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # grep -v success to keep site_name and error lines. bash -c 'd7-all-drush.sh security-review |& grep -v success | mail -s "Security review" $email_multi_assos' </code> </pre> h2. d7-all-update-contrib.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh d7-all-drush.sh -y en update # Launch cron to check for updates. d7-all-drush.sh -q cron d7-all-drush.sh cc all d7-all-dump-individual.sh d7-all-drush-logged.sh -y upc --no-core d7-all-drush.sh -y updb d7-all-drush.sh -y dis update # Cron report. d7-all-drush.sh cron | mail -s "Cron report: d7-all-update-contrib.sh" $email_multi_assos </code> </pre> h2. d7-all-update-localisation.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh d7-all-drush.sh -y en l10n_update d7-all-drush.sh l10n-update-refresh d7-all-drush.sh l10n-update d7-all-drush.sh -y dis l10n_update </code> </pre> h2. d7-create-site.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-config-site.sh $1 . scripts-utils.sh # ARGS: site name ######## Exceptions check_arguments $# 1 # "-" is forbidden because it provokes database error. if `echo $1 | grep -` ; then echo '"-" is forbidden in the site name' exit 1 fi # Site name length must be lower or equal to 16 due to database limitations. if [ $(echo $1 | wc -n) -le 16 ] ; then echo "site name can't have more than 16 characters" exit 1 fi ###### Initialisation cd $d7_dir db_password=`ask_password_db $db_server $db_user` site_password=`gen_password` # Check if site database already exists. if mysql -h $db_server -u $db_user -e "USE $d7_site_name" -p$db_password 2>/dev/null ; then echo "Database $d7_site_name already exists" exit 1 fi # Check if site folder already exists. if [ -d $d7_site_dir ] ; then echo "Folder $d7_site_dir already exists" exit 1 fi ###### Main mkdir $d7_site_dir # Backup requirements mkdir $d7_dir_individual_auto_backup/$d7_site_name mkdir $d7_dir_individual_manual_backup/$d7_site_name current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"` touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql # Create and grant privileges on database mysql -h $db_server -u $db_user -e "CREATE DATABASE $d7_site_name" -p$db_password mysql -h $db_server -u $db_user -e "GRANT ALL PRIVILEGES ON $d7_site_name.* TO '$d7_site_name'@'%' IDENTIFIED BY '$site_password'" -p$db_password # Create settings.php sed "s/\%\%DBUSER\%\%/$d7_site_name/ ; s/\%\%DBNAME\%\%/$d7_site_name/ ; s/\%\%DBPASS\%\%/$site_password/ ; s/\%\%nomsite\%\%/$d7_site_name/" < $d7_settings_template > $d7_site_settings # Create symbolic link cd $d7_dir ln -s . $d7_site_name # Next Instructions echo "Go to http://assos.centrale-marseille.fr/$d7_site_name/install.php to continue." echo "Press enter when ready to go on." read key # Init variables d7-reset-variables.sh $d7_site_name # Permissions chmod -R 755 $d7_site_dir chmod 400 $d7_site_settings # Last instructions echo "Last instructions:" echo "- Advice the webmaster to close account creation on the website" echo "- Give the webmaster a link to the club Drupal's tutorials " echo "- Create a node of type \"Site\" on default" echo "- Register the webmaster on webmasters@listes.centrale-marseille.fr" </code> </pre> h2. d7-delete-site.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh . scripts-config-site.sh $1 . scripts-utils.sh # ARGS: site name. check_arguments $# 1 db_password=`ask_password_db $db_server $db_user` # Delete database. mysql -h $db_server -u $db_user -p$db_password -e "IF EXISTS DROP DATABASE $d7_site_name" # Delete MYSQL user. mysql -h $db_server -u $db_user -p$db_password -e "DROP USER '$d7_site_name'@'%'" # Delete site's folder. chmod -R 700 $d7_site_dir rm -r $d7_site_dir # Delete symbolic link. rm $d7_dir/$d7_site_name # Delete database backups. rm -r $d7_dir_individual_auto_backup/assos.centrale-marseille.fr.$d7_site_name rm -r $d7_dir_individual_manual_backup/assos.centrale-marseille.fr.$d7_site_name echo "Don't forget to:" echo "- Refresh node site on default" echo "- Check particular behavior (normally detailed on site node)" </code> </pre> h2. d7-reset-variables.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: [site_name] if [ ! -z $1 ] ; then . scripts-config-site.sh $1 cd $d7_site_dir fi current_timestamp=`date "+%s"` ####### General variables drush -y vset --always-set reverse_proxy TRUE drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]' drush -y ev "variable_set('update_notify_emails', array($email_multi_assos));" drush -y vset error_level 0 ###### Hidden captcha drush -y en hidden_captcha # Log wrong answers. drush -y vset captcha_log_wrong_responses 1 # Use hidden captcha for all forms. drush -y sqlq --db-prefix "UPDATE {captcha_points} SET module = 'hidden_captcha', captcha_type = 'Hidden CAPTCHA' WHERE module is NULL;" # Flush captcha cache. drush -y vdel captcha_placement_map_cache # Randomely generate a math question as the label of the hidden captcha field. drush -y vset hidden_captcha_label "$RANDOM + $RANDOM" ####### Piwik drush -y en piwik drush -y vset piwik_site_id "101" drush -y vset piwik_url_http "http://piwik.centrale-marseille.fr/" drush -y vset piwik_url_https "https://piwik.centrale-marseille.fr/" # Piwik cache. drush -y vset piwik_cache 1 drush -y vset piwik_visibility_roles "1" # Anonymous and authenticated statistics. drush -y vset --format=json piwik_roles '{"1":0,"2":0}' drush -y vset piwik_page_title_hierarchy 1 # Activate local search. drush -y vset piwik_site_search 1 ###### Security review # For untrusted roles: # 1 : anonymous user # 2 : authenticated user # 3 : administrator drush -y en security_review drush -y ev "variable_set('security_review_untrusted_roles', array('1'));" # Security review can't check for files permissions on multi_assos. drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '1', skiptime = $current_timestamp, skipuid= '1' WHERE reviewcheck IN ('file_perms');" drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '0', skiptime = 0, skipuid= NULL WHERE reviewcheck NOT IN ('file_perms');" </code> </pre> h2. delete-temporary-files.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh find $dir_multi_assos -name '*~' -delete </code> </pre> h2. disk-usage.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh echo -e "Quotas: 9.1 GB\n" echo "Assos:" du -hcs $dir_multi_assos | grep total echo -e "\n" echo "D6:" cd $d6_dir_sites du -h -d 1 echo -e "\n" echo "D7:" cd $d7_dir_sites du -h -d 1 echo -e "\n" echo "Logs:" du -hcs $dir_log | grep total echo -e "\n" echo "Backups:" cd $dir_backup du -h -d 2 </code> </pre> h2. flush-files.sh <pre> <code class="php"> #!/bin/sh # ARGS: number of files to keep, email address # usage: flush files from a folder except a number equal to argument. # Check if scripts-config.sh is imported. if [ -z $scripts_utils ] ; then echo "Import of scripts-utils.sh required." exit 1 fi check_arguments $# 2 backups_number=`ls | wc -l` let "number_of_backups_to_delete = backups_number - $1" if [ $number_of_backups_to_delete -gt 0 ] ; then ls | head -$number_of_backups_to_delete | xargs rm else dir=`pwd` echo "There are not enough files in $dir to Flush it. Check if backup script works fine." | mail -s "[db] $dir has a backup problem" $2 fi </code> </pre> h2. flush-full-databases-backups.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: Drupal version cd $dir_full_backup/$1 for dir in `ls` ; do cd $dir flush-files.sh $db_full_backup_number $email_multi_assos done </code> </pre> h2. flush-individual-databases-backups.sh <pre> <code class="php"> #!/bin/sh . scripts-config.sh # ARGS: manual or auto, Drupal version cd $dir_individual_backup/$1/$2 for dir in `ls` ; do cd $dir if [ $1 = 'auto' ] ; then flush-files.sh $db_individual_auto_backup_number $email_multi_assos else flush-files.sh $db_individual_manual_backup_number $email_multi_assos fi done </code> </pre> h2. scripts-config-site.sh <pre> <code class="php"> #!/bin/sh # Sets useful variables of a site. # ARGS: site_name # Requires scripts-config.sh # Check if scripts-config.sh is imported. if [ -z $scripts_config ] ; then echo "Import of scripts-config.sh required." exit 1 fi if [ -z $1 ] ; then echo "This script needs a site name as argument." exit 1 fi scripts_config_site='imported' d7_site_name=$1 d7_site_dir="$d7_dir_sites/assos.centrale-marseille.fr.$d7_site_name" d7_site_settings="$d7_site_dir/settings.php" </code> </pre> h2. scripts-config.sh <pre> <code class="php"> #!/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/users/guest/assos/bin PATH=$PATH:/usr/guest/assos/bin scripts_config='imported' email_multi_assos="assos@centrale-marseille.fr" dir_multi_assos='/users/guest/assos' dir_scripts="$dir_multi_assos/bin" dir_template="$dir_multi_assos/Desktop" ###### Database db_server="myassos.serv.int" db_user="assos" dir_log="$dir_multi_assos/log" ###### Backup db_full_backup_number=3 db_individual_manual_backup_number=3 db_individual_auto_backup_number=3 dir_backup="$dir_multi_assos/backup" dir_full_backup="$dir_backup/full" dir_individual_backup="$dir_backup/individual" dir_individual_manual_backup="$dir_individual_backup/manual" dir_individual_auto_backup="$dir_individual_backup/auto" ###### D7 variables d7_dir="$dir_multi_assos/htmltest" d7_dir_sites="$d7_dir/sites" ### Log d7_dir_log="$dir_log/d7" ### Backup d7_dir_full_backup="$dir_full_backup/d7" d7_dir_individual_manual_backup="$dir_individual_manual_backup/d7" d7_dir_individual_auto_backup="$dir_individual_auto_backup/d7" ### Template d7_settings_template_name="settings-D7-bddinde-template.php" d7_settings_template="$dir_template/$d7_settings_template_name" ###### D6 variables d6_dir="$dir_multi_assos/html" d6_dir_sites="$d6_dir/sites" ### Backup d6_dir_full_backup="$dir_full_backup/d6" d6_dir_individual_manual_backup="$dir_individual_manual_backup/d6" d6_dir_individual_auto_backup="$dir_individual_auto_backup/d6" </code> </pre> h2. scripts-utils.sh <pre> <code class="php"> #!/bin/sh # This script contains useful functions for other scripts. # Check if scripts-config.sh is imported. if [ -z $scripts_config ] ; then echo "Import of scripts-config.sh required." exit 1 fi scripts_utils='imported' ask_password() { # read -s doesn't work with sh. # usage: pass=`ask_password "password please:"` echo $1 >&2 echo -n ">" >&2 stty_avant=`stty -g` stty -echo read p stty $stty_avant echo "$p" unset p } ask_password_db() { # ARGS: server_name, user_name local db_password="pour_boucler" # empty db request to validate password while ! mysql -h $1 -u $2 -p$db_password -e "" 2>/dev/null ; do db_password=`ask_password "database password:"` echo "" done } generate_password(){ # ARGS: [password_length] # The password contains special characters. '/' must be excluded to avoid sed malfunction. local site_password='/' if [ -z $1 ] ; then local password_length=20 else local password_length=$1 fi while echo "$site_password" | grep -Fq '/' ; do site_password=`dd if=/dev/urandom count=1 | uuencode -m - | head -n 2 | tail -n 1 | cut -c-$password_length` done echo $site_password } count_d7_sites(){ find $d7_dir_sites -type d ! -name all -maxdepth 1 | wc -l } check_arguments() { # ARGS: number of arguments passed to script, number of arguments required if [ $1 -le $2 ] ; then echo "Number of arguments insuffisant." exit 1 fi } </code> </pre>