Actions

Historique

Scripts et taches planifiees » Historique » Révision 27

« Précédent | Révision 27/118 (diff) | Suivant »
Florent Torregrosa, 04/08/2013 17:20
remove h3

Afin de gagner du temps et d'éviter les erreurs humaines, des scripts ont été écrits tout au long du projet. Certains sont même exécutés automatiquement toutes les semaines.

Contenu
taper la commande pour modifier le crontab crontab -e. /!\ Ne pas effectuer cette opération avant de s'être renseigné sur "vi":http://fr.wikipedia.org/wiki/Vi (l'éditeur de texte utilisé qui n'est pas vraiment intuitif :p) et sur "la syntaxe du crontab":http://fr.wikipedia.org/wiki/Crontab#Modification !

h1. Les tâches planifiées

h2. Crontab

Pour exécuter ces tâches régulièrement, on utilise le "crontab":http://fr.wikipedia.org/wiki/Crontab. Il s'agit d'un programme installé sur notre serveur.

Pour voir et modifier la liste des actions :

se connecter au serveur : `ssh assos@sas1.centrale-marseille`

taper la commande pour voir le crontab `crontab -l`

taper la commande pour modifier le crontab `crontab -e`. /!\ Ne pas effectuer cette opération avant de s'être renseigné sur "vi":http://fr.wikipedia.org/wiki/Vi (l'éditeur de texte utilisé qui n'est pas vraiment intuitif :p) et sur "la syntaxe du crontab":http://fr.wikipedia.org/wiki/Crontab#Modification !¶

{{tip(Pour utiliser un autre editeur que vi taper : export EDITOR=editor_name exemple : vim, nano)}}


Ici bientôt le crontab

Voici la liste des tâches effectuées régulièrement.

h2. La réinitialisation des variables

Cette tâche consiste à réinitialiser certaines variables qui donnent des droits considérés comme trop permissifs donc dangereux aux administrateurs des sites ou nécessaire pour d'autres scripts tels que piwik ou security_review.

En voici quelques exemples :


drush -y vset error_level 0

Cette commande permet de ne pas afficher les messages d'erreurs. En effet, ils contiennent parfois des informations sensibles sur l'installation et ne doivent donc pas être divulgués à n'importe qui.


drush -y vset --always-set reverse_proxy TRUE
drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]'

Ces commandes permettent de déclarer à Drupal les serveurs proxy du CRI afin d'éviter qu'il ne répertorie tous les visiteurs comme ayant l'adresse des serveurs sus-cités. Pas fini : voir http://assos.centrale-marseille.fr/content/t%C3%A2che/d%C3%A9clarer-les-proxy-du-cri-%C3%A0-drupal

h2. La sauvegarde des bases de données

Sur les installations d6 et d7 : une fois par semaine

On utilise les scripts de sauvegarde créés par le club Drupal.

h2. La purge des sauvegardes de bdd

Sur les installations d6 et d7 : toutes les semaines

Cette tâche utilise le script de purge des sauvegardes afin de libérer de l'espace disque en supprimant les sauvegardes de bdd les plus vieilles.

h1. Liste des scripts à disposition

Les scripts utilisés sont hébergés dans le répertoire bin du compte assos.

h2. all-reset-permissions.sh


#!/bin/sh

. scripts-config.sh

# This script puts the correct permissions to sites folders, settings.php and scripts.

######### drupal 6
for dir in `find $d6_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all ! -name images ! -name languages` ; do
    chmod 755 $dir
    chmod 400 $dir/settings.php
done

######### drupal 7
for dir in `find $d7_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all` ; do
    chmod 755 $dir
    chmod 400 $dir/settings.php
done

####### bin
chmod -R 700 $dir_scripts

h2. contrib-usage.sh


#!/bin/sh

. scripts-config.sh
. scripts-utils.sh

# ARGS: Drupal_version, project_status, project_name
# List sites that have the project_name with the corresponding project_status.

check_arguments $# 3

if [ $1 = d6 ] ; then
    cd $d6_dir_sites
else
    cd $d7_dir_sites
fi

for dir in `find . -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do
    # List non-core projects that correspond to the status.
    # Keep project_name if listed.
    # Count line result. 0 if not listed or 1 if listed.
	# Print site_dir if listed.

    cd $dir;
	if [ 1 -eq `drush pml --no-core --status=$2 | grep $3 | wc -l` ] ; then
        echo $dir; 
    fi
done

h2. d6-all-drush.sh


#!/bin/sh

. scripts-config.sh
. scripts-utils.sh

# ARGS: drush_command

check_arguments $# 1

for dir in `find $d6_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do
    cd $dir
    echo $dir
    drush $*
done

h2. d6-all-dump-full.sh


#!/bin/sh

. scripts-config.sh

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

# Dump D6 database with webassos user.
mysqldump --single-transaction webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY > $d6_dir_full_backup/webassos/$current_date.webassos.sql

# Dump D6 form database with forum user.
mysqldump forum -h myweb.serv.int -u forum --password=dtcAltF12 > $d6_dir_full_backup/forum/$current_date.forum.sql

h2. d6-all-update-contrib.sh


#!/bin/sh

. scripts-config.sh

d6-all-drush.sh -y en update
# Launch cron to check for updates.
d6-all-drush.sh -q cron

d6-all-drush.sh cc all
d6-all-dump-full.sh

d6-all-drush.sh -y upc --no-core
d6-all-drush.sh -y updb

d6-all-drush.sh -y dis update

# Cron report.
d6-all-drush.sh cron

h2. d6-dump-individual.sh


#!/bin/sh

. scripts-config.sh

# ARGS: auto or manual, site_prefix
# WARNING : backup are regularly flushed, put your backup in a safe place.

cd $dir_individual_backup

#récupération des tables du site dans le fichier liste_tables.temp
tables='_%'
liste="$1$tables"

mysql -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY -BNe "show tables like '"$liste"'" webassos | tr '\r\n' ' ' > liste_tables.temp

#transformation de cette liste en une variable
var=$(cat liste_tables.temp)

#sauvegarde de toutes ces tables
current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`
suffixe="_dump$current_date.sql"
fichier="$1$suffixe"

if [ $1 = 'auto' ] ; then
        mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_auto_backup/$fichier
    else
        mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_manual_backup/$fichier
    fi

#suppression du fichier temporaire utilisé
rm liste_tables.temp

h2. d6-reset-variables.sh


#!/bin/sh

. scripts-config.sh

cd $d6_dir_sites
d6-all-drush.sh vset --always-set reverse_proxy TRUE
d6-all-drush.sh vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]'
drush @sites ev "variable_set('update_notify_emails', array($email_multi_assos));" --yes

h2. d7-all-drush-logged.sh


#!/bin/sh

. scripts-config.sh
. scripts-utils.sh

# ARGS: drush_command

check_arguments $# 1

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do
    cd $dir
    current_date=`date "+%Y-%m-%d_%Hh:%Mm"`
    echo "======$current_date======" >> $d7_dir_log/$x.log
    echo "Arguments are: $* " >> $d7_dir_log/$x.log
    drush $* >> $d7_dir_log/$x.log
    #To have the same caracters limit we needed 'end' that have 4 caracters, we used the Croate language for this ! LOL 
    echo -e "=============KRAJ=============\n" >> $d7_dir_log/$x.log
done

h2. d7-all-drush.sh


#!/bin/sh

. scripts-config.sh
. scripts-utils.sh

# ARGS: drush_command

check_arguments $# 1

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do
    cd $dir
    echo $dir
    drush $*
done

h2. d7-all-dump-full.sh


#!/bin/sh

. scripts-config.sh

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

# Dump D7 databases at once.
mysqldump -h $db_server -u $db_user -pNoNo82jJ --all-databases > $d7_dir_full_backup/myassos/$current_date.d7_full.sql

h2. d7-all-dump-individual.sh


#!/bin/sh

. scripts-config.sh

# ARGS: auto or manual

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do
    cd $dir
    drush cc all
    if [ $1 = 'auto' ] ; then
        drush sql-dump --result-file="$d7_dir_individual_auto_backup/$dir/$current_date.$dir.sql"
    else
        drush sql-dump --result-file="$d7_dir_individual_manual_backup/$dir/$current_date.$dir.sql"
    fi
done

h2. d7-all-releases-info.sh


#!/bin/sh

. scripts-config.sh

# List version of a module or theme that exists in sites/all/*
# and all versions that are available for same project on drupal.org

# modules
for dir in `find $d7_dir_sites/all/modules -maxdepth 1 -mindepth 1 -type d` ; do
    drush pm-releases $dir
done

# themes
for dir in `find $d7_dir_sites/all/themes -maxdepth 1 -mindepth 1 -type d` ; do
    drush pm-releases $dir
done

h2. d7-all-reset-variables.sh


#!/bin/sh

. scripts-config.sh

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do
    cd $dir
    echo $dir
    d7-reset-variables.sh
done

h2. d7-all-security-review.sh


#!/bin/sh

. scripts-config.sh

# grep -v success to keep site_name and error lines.
bash -c 'd7-all-drush.sh security-review |& grep -v success | mail -s "Security review" $email_multi_assos'

h2. d7-all-update-contrib.sh


#!/bin/sh

. scripts-config.sh

d7-all-drush.sh -y en update
# Launch cron to check for updates.
d7-all-drush.sh -q cron

d7-all-drush.sh cc all
d7-all-dump-individual.sh

d7-all-drush-logged.sh -y upc --no-core
d7-all-drush.sh -y updb

d7-all-drush.sh -y dis update

# Cron report.
d7-all-drush.sh cron | mail -s "Cron report: d7-all-update-contrib.sh" $email_multi_assos

h2. d7-all-update-localisation.sh


#!/bin/sh

. scripts-config.sh

d7-all-drush.sh -y en l10n_update
d7-all-drush.sh l10n-update-refresh
d7-all-drush.sh l10n-update
d7-all-drush.sh -y dis l10n_update

h2. d7-create-site.sh


#!/bin/sh

. scripts-config.sh
. scripts-config-site.sh $1
. scripts-utils.sh

# ARGS: site name

######## Exceptions
check_arguments $# 1

# "-" is forbidden because it provokes database error.
if `echo $1 | grep -` ; then
    echo '"-" is forbidden in the site name'
    exit 1
fi

# Site name length must be lower or equal to 16 due to database limitations.
if [ $(echo $1 | wc -n) -le 16 ] ; then
    echo "site name can't have more than 16 characters"
    exit 1
fi

###### Initialisation
cd $d7_dir
db_password=`ask_password_db $db_server $db_user`
site_password=`gen_password`

# Check if site database already exists.
if mysql -h $db_server -u $db_user -e "USE $d7_site_name" -p$db_password 2>/dev/null ; then
    echo "Database $d7_site_name already exists"
    exit 1
fi

# Check if site folder already exists.
if [ -d $d7_site_dir ] ; then
    echo "Folder $d7_site_dir already exists"
    exit 1
fi

###### Main
mkdir $d7_site_dir

# Backup requirements
mkdir $d7_dir_individual_auto_backup/$d7_site_name
mkdir $d7_dir_individual_manual_backup/$d7_site_name
current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql
touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql
touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql

# Create and grant privileges on database
mysql -h $db_server -u $db_user -e "CREATE DATABASE $d7_site_name" -p$db_password
mysql -h $db_server -u $db_user -e "GRANT ALL PRIVILEGES ON $d7_site_name.* TO '$d7_site_name'@'%' IDENTIFIED BY '$site_password'" -p$db_password

# Create settings.php
sed "s/\%\%DBUSER\%\%/$d7_site_name/ ; s/\%\%DBNAME\%\%/$d7_site_name/ ; s/\%\%DBPASS\%\%/$site_password/ ; s/\%\%nomsite\%\%/$d7_site_name/" < $d7_settings_template > $d7_site_settings

# Create symbolic link
cd $d7_dir
ln -s . $d7_site_name

# Next Instructions
echo "Go to http://assos.centrale-marseille.fr/$d7_site_name/install.php to continue."
echo "Press enter when ready to go on."
read key

# Init variables
d7-reset-variables.sh $d7_site_name

# Permissions
chmod -R 755 $d7_site_dir
chmod 400 $d7_site_settings

# Last instructions
echo "Last instructions:"
echo "- Advice the webmaster to close account creation on the website"
echo "- Give the webmaster a link to the club Drupal's tutorials "
echo "- Create a node of type \"Site\" on default"
echo "- Register the webmaster on webmasters@listes.centrale-marseille.fr"

h2. d7-delete-site.sh


#!/bin/sh

. scripts-config.sh
. scripts-config-site.sh $1
. scripts-utils.sh

# ARGS: site name.

check_arguments $# 1

db_password=`ask_password_db $db_server $db_user`

# Delete database.
mysql -h $db_server -u $db_user -p$db_password -e "IF EXISTS DROP DATABASE $d7_site_name"

# Delete MYSQL user.
mysql -h $db_server -u $db_user -p$db_password -e "DROP USER '$d7_site_name'@'%'"

# Delete site's folder.
chmod -R 700 $d7_site_dir
rm -r $d7_site_dir

# Delete symbolic link.
rm $d7_dir/$d7_site_name

# Delete database backups.
rm -r $d7_dir_individual_auto_backup/assos.centrale-marseille.fr.$d7_site_name
rm -r $d7_dir_individual_manual_backup/assos.centrale-marseille.fr.$d7_site_name

echo "Don't forget to:"
echo "- Refresh node site on default"
echo "- Check particular behavior (normally detailed on site node)"

h2. d7-reset-variables.sh


#!/bin/sh

. scripts-config.sh

# ARGS: [site_name]

if [ ! -z $1 ] ; then
    . scripts-config-site.sh $1
    cd $d7_site_dir
fi

current_timestamp=`date "+%s"`

####### General variables
drush -y vset --always-set reverse_proxy TRUE
drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]'
drush -y ev "variable_set('update_notify_emails', array($email_multi_assos));"
drush -y vset error_level 0

###### Hidden captcha
drush -y en hidden_captcha
# Log wrong answers.
drush -y vset captcha_log_wrong_responses 1
# Use hidden captcha for all forms.
drush -y sqlq --db-prefix "UPDATE {captcha_points} SET module = 'hidden_captcha', captcha_type = 'Hidden CAPTCHA' WHERE module is NULL;"
# Flush captcha cache.
drush -y vdel captcha_placement_map_cache
# Randomely generate a math question as the label of the hidden captcha field.
drush -y vset hidden_captcha_label "$RANDOM + $RANDOM"


####### Piwik
drush -y en piwik
drush -y vset piwik_site_id "101"
drush -y vset piwik_url_http "http://piwik.centrale-marseille.fr/"
drush -y vset piwik_url_https "https://piwik.centrale-marseille.fr/"
# Piwik cache.
drush -y vset piwik_cache 1
drush -y vset piwik_visibility_roles "1"
# Anonymous and authenticated statistics.
drush -y vset --format=json piwik_roles '{"1":0,"2":0}'
drush -y vset piwik_page_title_hierarchy 1
# Activate local search.
drush -y vset piwik_site_search 1


###### Security review
# For untrusted roles:
# 1 : anonymous user
# 2 : authenticated user
# 3 : administrator
drush -y en security_review
drush -y ev "variable_set('security_review_untrusted_roles', array('1'));"
# Security review can't check for files permissions on multi_assos.
drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '1', skiptime = $current_timestamp, skipuid= '1' WHERE reviewcheck IN ('file_perms');"
drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '0', skiptime = 0, skipuid= NULL WHERE reviewcheck NOT IN ('file_perms');"

h2. delete-temporary-files.sh


#!/bin/sh

. scripts-config.sh

find $dir_multi_assos -name '*~' -delete

h2. disk-usage.sh


#!/bin/sh

. scripts-config.sh

echo -e "Quotas: 9.1 GB\n"

echo "Assos:"
du -hcs $dir_multi_assos | grep total
echo -e "\n"

echo "D6:"
cd $d6_dir_sites
du -h -d 1
echo -e "\n"

echo "D7:"
cd $d7_dir_sites
du -h -d 1
echo -e "\n"

echo "Logs:"
du -hcs $dir_log | grep total
echo -e "\n"

echo "Backups:"
cd $dir_backup
du -h -d 2

h2. flush-files.sh


#!/bin/sh

# ARGS: number of files to keep, email address
# usage: flush files from a folder except a number equal to argument.

# Check if scripts-config.sh is imported.
if [ -z $scripts_utils ] ; then
    echo "Import of scripts-utils.sh required."
    exit 1
fi

check_arguments $# 2

backups_number=`ls | wc -l`
let "number_of_backups_to_delete = backups_number - $1"

if [ $number_of_backups_to_delete -gt 0 ] ; then
    ls | head -$number_of_backups_to_delete | xargs rm
else
    dir=`pwd`
    echo "There are not enough files in $dir to Flush it. Check if backup script works fine." | mail -s "[db] $dir has a backup problem" $2
fi

h2. flush-full-databases-backups.sh


#!/bin/sh

. scripts-config.sh

# ARGS: Drupal version

cd $dir_full_backup/$1

for dir in `ls` ; do
    cd $dir
    flush-files.sh $db_full_backup_number $email_multi_assos
done

h2. flush-individual-databases-backups.sh


#!/bin/sh

. scripts-config.sh

# ARGS: manual or auto, Drupal version

cd $dir_individual_backup/$1/$2

for dir in `ls` ; do
    cd $dir
    if [ $1 = 'auto' ] ; then
        flush-files.sh $db_individual_auto_backup_number $email_multi_assos
    else
        flush-files.sh $db_individual_manual_backup_number $email_multi_assos
    fi
done

h2. scripts-config-site.sh


#!/bin/sh

# Sets useful variables of a site.
# ARGS: site_name
# Requires scripts-config.sh

# Check if scripts-config.sh is imported.
if [ -z $scripts_config ] ; then
    echo "Import of scripts-config.sh required."
    exit 1
fi

if [ -z $1 ] ; then
    echo "This script needs a site name as argument."
    exit 1
fi

scripts_config_site='imported'

d7_site_name=$1
d7_site_dir="$d7_dir_sites/assos.centrale-marseille.fr.$d7_site_name"
d7_site_settings="$d7_site_dir/settings.php"

h2. scripts-config.sh


#!/bin/sh

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/users/guest/assos/bin
PATH=$PATH:/usr/guest/assos/bin

scripts_config='imported'

email_multi_assos="assos@centrale-marseille.fr"

dir_multi_assos='/users/guest/assos'
dir_scripts="$dir_multi_assos/bin"
dir_template="$dir_multi_assos/Desktop"

###### Database
db_server="myassos.serv.int"
db_user="assos"
dir_log="$dir_multi_assos/log"

###### Backup
db_full_backup_number=3
db_individual_manual_backup_number=3
db_individual_auto_backup_number=3

dir_backup="$dir_multi_assos/backup"
dir_full_backup="$dir_backup/full"
dir_individual_backup="$dir_backup/individual"
dir_individual_manual_backup="$dir_individual_backup/manual"
dir_individual_auto_backup="$dir_individual_backup/auto"


###### D7 variables
d7_dir="$dir_multi_assos/htmltest"
d7_dir_sites="$d7_dir/sites"

### Log
d7_dir_log="$dir_log/d7"

### Backup
d7_dir_full_backup="$dir_full_backup/d7"
d7_dir_individual_manual_backup="$dir_individual_manual_backup/d7"
d7_dir_individual_auto_backup="$dir_individual_auto_backup/d7"

### Template
d7_settings_template_name="settings-D7-bddinde-template.php"
d7_settings_template="$dir_template/$d7_settings_template_name"


###### D6 variables
d6_dir="$dir_multi_assos/html"
d6_dir_sites="$d6_dir/sites"

### Backup
d6_dir_full_backup="$dir_full_backup/d6"
d6_dir_individual_manual_backup="$dir_individual_manual_backup/d6"
d6_dir_individual_auto_backup="$dir_individual_auto_backup/d6"

h2. scripts-utils.sh


#!/bin/sh

# This script contains useful functions for other scripts.

# Check if scripts-config.sh is imported.
if [ -z $scripts_config ] ; then
    echo "Import of scripts-config.sh required."
    exit 1
fi

scripts_utils='imported'

ask_password() {
    # read -s doesn't work with sh.
    # usage: pass=`ask_password "password please:"`
    echo $1 >&2
    echo -n ">" >&2
    stty_avant=`stty -g`
    stty -echo
    read p
    stty $stty_avant
    echo "$p"
    unset p
}

ask_password_db() {
    # ARGS: server_name, user_name
    local db_password="pour_boucler"
    # empty db request to validate password
    while ! mysql -h $1 -u $2 -p$db_password -e "" 2>/dev/null ; do
        db_password=`ask_password "database password:"`
        echo ""
    done
}

generate_password(){
    # ARGS: [password_length]
    # The password contains special characters. '/' must be excluded to avoid sed malfunction.

    local site_password='/'

    if [ -z $1 ] ; then
        local password_length=20
    else
        local password_length=$1
    fi
    
    while echo "$site_password" | grep -Fq '/' ; do
        site_password=`dd if=/dev/urandom count=1 | uuencode -m - | head -n 2 | tail -n 1 | cut -c-$password_length`
    done

    echo $site_password
}

count_d7_sites(){
    find $d7_dir_sites -type d ! -name all -maxdepth 1 | wc -l
}

check_arguments() {
    # ARGS: number of arguments passed to script, number of arguments required
    if [ $1 -le $2  ] ; then
        echo "Number of arguments insuffisant."
        exit 1
    fi
}

Fichiers (0)

Mis à jour par Florent Torregrosa il y a plus de 11 ans · 118 révisions

Projet

Général

Profil

Club Drupal

Wiki

Scripts et taches planifiees » Historique » Révision 27

se connecter au serveur : `ssh assos@sas1.centrale-marseille`

taper la commande pour voir le crontab `crontab -l`

Projet

Général

Profil

Club Drupal

Wiki

Scripts et taches planifiees » Historique » Révision 27

se connecter au serveur : ssh assos@sas1.centrale-marseille

taper la commande pour voir le crontab crontab -l

se connecter au serveur : `ssh assos@sas1.centrale-marseille`

taper la commande pour voir le crontab `crontab -l`