# uname -a - Mot-clé - debian

Redémarrer une machine lorsqu'un programme l'étouffe inopinément (Debian 10)

Mathieu — Wed, 14 Oct 2020 12:40:00 +0200

Admettons que vous ayez une machine avec un programme bugué. Typiquement, un logiciel métier avec une fuite de mémoire, qui va pour une raison inconnue saturer la mémoire vive et provoquer son propre crash ou le crash des autres services sur la machine.

Vous pourriez lui mettre une limite de mémoire via /etc/security/limits.conf mais ça ne fera que planter le programme (segmentation fault) pour protéger les autres, et dans le cas d'un logiciel métier cela veut dire prendre ensuite d'autres actions, manuelles ou automatiques, via un système de monitoring approprié.

C'est vrai, le monitoring est la bonne solution quand on suit un logiciel aussi critique qu'un logiciel métier, mais dans le cas d'un logiciel qui doit "juste tourner", d'une équipe de maintenance réduite ou n'ayant pas la possibilité d'astreintes, une solution simple pour s'assurer que la machine tourne même en cas d'incident, c'est de redémarrer la machine lors d'un incident. Pour cela, on peut utiliser le programme watchdog.

Cela doit être fait avec motivations et circonspection, une analyse "post-mortem" doit suivre chaque reboot, dans le cas d'une tentative d'élévation de privilèges par Buffer Overflow, rien ne dit que l'attaque n'a pas réussie même si le système a redémarré.

Watchdog : Configuration

Watchdog s'installe via les paquets :

apt install watchdog

Il y a ensuite deux choses à configurer : les règles pour déclencher le reboot, et le mode de chargement du programme ("no actions" ou actif).

Dans mon cas, je voulais redémarrer si le serveur avait moins de 20% de RAM disponible, ce serveur tourne en permanence à 50% de consommation de RAM et n'est jamais sensé dépasser, ou alors c'est signe d'un problème, d'où le reboot à déclencher.

La configuration de watchdog se trouve dans /etc/watchdog.conf et fournit deux valeurs à régler : min-memory et allocatable-memory . Attention, ces valeurs sont à renseigner en taille de page RAM.

Pour identifier la taille des pages sur le système, utilisez la commande :

getconf PAGESIZE

Ensuite, un simple calcul permet d'identifier la valeur à mettre dans min-memory et allocatable-memory. Par exemple, pour un PAGESIZE de 4096 (4 kB), 2 GB (2048 MB) de RAM et 20% de ce total :

( 20 % * 2048 * 1000 ) / 4096

Une fois min-memory et allocatable-memory configurés, il convient de tester le trigger avant de le démarrer;

Les options de lancement se trouvent dans /etc/default/watchdog . Modifier watchdog_options pour y renseigner :

watchdog_options="-v --no-action"

Désactivez Watchdog du démarrage automatique, puis démarrez-le :

systemctl disable watchdog
service watchdog start

Vous en verrez la trace dans les logs via service watchdog status ou journalctl -f .

Watchdog : Activation

Si les essais sont concluants, retournez dans /etc/default/watchdog pour activer le module noyau. Cela évitera que Watchdog se fasse tuer par un processus quelconque (oom-killer par exemple) :

watchdog_module="softdog"

Si vous ne voulez laisser aucune chance pour éviter le reboot (par exemple si oom-killer a réussi à baisser la consommation de ram sous le seuil acceptable et que le reboot n'est plus nécessaire), rajoutez nowayout :

watchdog_module="softdog nowayout"

Attention, une fois chargé en nowayout, le module noyau restera actif même si watchdog est stoppé (légitimement ou non), le seul moyen de le retirer est de retirer softdog des modules noyau (via rmmod), ou de rebooter.

Activez ensuite watchdog :

systemctl enable watchdog
service watchdog start

watchdog doit être visible dans les modules noyau chargés :

lsmod | grep softdog

Sources

Maman, j'ai patché Debian

Mathieu — Thu, 19 Mar 2020 15:33:00 +0100

Un billet en forme de note à moi-même sur ce qu'il faut faire pour correctement :

Récupérer un package depuis upstream
Appliquer un ou plusieurs patches
Le signer et le re-déployer en production

Préparer l'environnement

Pour compiler et signer un paquet simplement il vous faut :

Un utilisateur (non root)
Une clé GPG
Les build-essentials et les devscripts (parce qu'on est feignasse)

Créez un utilisateur non-root et loguez-vous avec. N'utilisez pas "su" à partir de root, parce que sinon GPG ne pourra pas vous demander la phrase de passe (des histoires de droits sur les TTY).

Créez une clé GPG, et paramétrez-la :

gpg --full-generate-key

Récupérez l'identifiant de clé à la fin de la procédure, ou avec gpg --list-keys si vous l'avez loupé.

Modifiez ou créez le fichier ~/.devscripts et ajoutez :

DEBUILD_SET_ENVVAR_DEBSIGN_KEYID=xxxxxxxx

Avec le xxxxxx qui correspond à votre identifiant de clé.

Récupérer le paquet et les dépendances de compilation

Le plus simple c'est quand le paquet existe déjà et qu'il faut simplement patcher. S'il n'existe aucun paquet, il faut créer un nouveau paquet, éventuellement debianizer la configuration, et c'est une autre paire de manches (et c'est pas le sujet ici).

Pour récupérer le paquet upstream :

apt-get source nomdupaquet

Si le paquet est introuvable, ajoutez les dépôts src à votre sources.list :

deb-src http://deb.debian.org/debian/ buster main contrib
deb-src http://security.debian.org/debian-security buster/updates main contrib
deb-src http://deb.debian.org/debian/ buster-updates main contrib

Il faut ensuite récupérer les paquets nécessaires à la compilation. Coup de bol, si vous avez pu avoir le paquet source à l'étape précédente, c'est facile :

apt-get build-dep nomdupaquet

Patcher le paquet

Le format dpatch est obsolète, en principe votre package utilise quilt comme tout paquet récent. Il suffit de télécharger le patch depuis git et le placer dans le dossier debian/patches.

Ensuite, ajoutez le nom du fichier que vous avez ajouté au fichier debian/patches/series . Attention, l'ordre dans series est important.

Déclarer les changements

Ce n'est pas nécessaire la première fois, mais si vous re-compilez un paquet, il faut ajouter un commentaire dans le Changelog. Le plus simple : utilisez la commande dch -i et modifiez la ligne de changelog, en changeant bien la version du paquet pour qu'elle soit consécutive à la précédente.

Compiler le paquet

Rendez-vous dans le dossier du paquet, et lancez la commande debuild . C'est tout. Rentrez votre phrase de passe pour la clé à la fin de la procédure.

Déployer en production

Pour déployer un paquet, deux solutions :

Envoyer le paquet puis l'installer avec dpkg -i lefichier.deb ou un outil d'orchestration
Installer un DPA (Debian Private Repository) et l'ajouter au sources.list

L'installation du serveur DPA fera l'objet d'un autre billet (un jour).

Sources

Debian 8 : Configure Nginx and Passenger to supercharge your PuppetMaster

Mathieu — Wed, 22 Jun 2016 21:16:00 +0200

The Puppet master comes by default with a basic WEBrick server. It allow a quick start for those that are not familiar with Puppet, but when the number of Puppet nodes grows, the performances of the default WEBrick server are going down quickly.

The Puppet documentation show how to configure Apache and Passenger to replace the default WEBrick server, but what if you have a lot of nodes ? What if you want to apply your configuration within minutes, instead of the default half-hour threshold before the agent asks the master if something changed ?

Or you may just want a fancy Nginx instead of your plain-old-reliable Apache.

Here is how.

Check your hostname

Your hostname is the base configuration for your node, you should check that it's correct, otherwise you will run into problems after Puppet installation.

# hostname -f

If everything is okay, check your hosts file

# cat /etc/hosts

If your hostname is inside your host file, carry on. Otherwise, set it.

Install Puppet and Puppetmaster

I suppose that you also need the puppet agent installed on the Puppetmaster server.

Install Puppet and Puppetmaster :

# apt-get install puppet puppetmaster

Stop the Puppetmaster :

# service puppetmaster stop

Prevent the puppetmaster from starting. Nginx will spawn on the right port instead of the WEBrick server, previously spawn by Puppetmaster service. Edit the file /etc/defaults/puppetmaster :

# Start puppetmaster on boot?
START=no

Configure the Puppet agent : edit the file /etc/puppet/puppet.conf to point your agent on the master (for instance puppetmaster.example.com).

Also, comment the two lines that are "needed for passenger", our configuration don't need them. Actually, if you keep it, it will not work.

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post
server=puppetmaster.example.com

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
#ssl_client_header = SSL_CLIENT_S_DN
#ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]
report = true

Enable your puppet agent :

# puppet agent --enable

Install Nginx and Passenger

We will install the bundle Nginx+Passenger shipped by Phusion repositories.

Add the key to your keyring :

# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7

The Phusion repository uses HTTPS, add HTTPS transport to APT :

# apt-get install apt-transport-https ca-certificates

Finally, install Nginx and Passenger :

# apt-get update
# apt-get install nginx-extras passenger

Configure Nginx and Puppetmaster application

Edit /etc/nginx/nginx.conf and uncomment the reference to passenger config :

    ##
    # Phusion Passenger config
    ##
    # Uncomment it if you installed passenger or passenger-enterprise
    ##

    include /etc/nginx/passenger.conf;

Create the file /etc/nginx/nginx/sites-available/puppet.conf with the following content :

server {
    listen                     8140 ssl;
    server_name                puppet puppetmaster puppetmaster.example.com;

    passenger_enabled          on;
    passenger_app_env          production;

    passenger_set_header       X-Client-Verify  $ssl_client_verify;
    passenger_set_header       X-Client-DN $ssl_client_s_dn;
    passenger_set_header       X-SSL-Subject    $ssl_client_s_dn;
    passenger_set_header       X-SSL-Issuer     $ssl_client_i_dn;

    access_log                 /var/log/nginx/puppet_access.log;
    error_log                  /var/log/nginx/puppet_error.log;

    root                       /etc/puppet/rack/public;

    ssl_certificate            /var/lib/puppet/ssl/certs/puppetmaster.example.com.pem;
    ssl_certificate_key        /var/lib/puppet/ssl/private_keys/puppetmaster.example.com.pem;
    ssl_crl                    /var/lib/puppet/ssl/ca/ca_crl.pem;
    ssl_client_certificate     /var/lib/puppet/ssl/certs/ca.pem;
    ssl_ciphers                'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    ssl_prefer_server_ciphers  on;
    ssl_verify_client          optional;
    ssl_verify_depth           1;
    ssl_session_cache          shared:SSL:128m;
    ssl_session_timeout        5m;
}

Remove the default virtual host from Nginx as we don't need it :

# rm /etc/nginx/sites-enabled/default

And enable your newly created server :

# ln -s /etc/nginx/sites-available/puppet.conf /etc/nginx/sites-enabled/puppet.conf

Before restarting Nginx, we will configure the Ruby application for Puppetmaster.

Create the directory /etc/puppet/rack and its subdirectories /etc/puppet/rack/public and /etc/puppet/rack/tmp

# mkdir -p /etc/puppet/rack/public /etc/puppet/rack/tmp

Create the file /etc/puppet/rack/config.ru with the following content :

# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $LOAD_PATH.unshift('/opt/puppet/lib')

$0 = "master"

# Set the PATH in environment variable
ENV['PATH'] = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"

# Rack applications typically don't start as root.  Set --confdir, --vardir,
# --logdir, --rundir to prevent reading configuration from
# ~/ based pathing.
ARGV << "--confdir" << "/etc/puppet"
ARGV << "--vardir"  << "/var/lib/puppet"
ARGV << "--logdir"  << "/var/log/puppet"
ARGV << "--rundir"  << "/var/run/puppet"
#ARGV << "--codedir"  << "/etc/puppet/code"

# always_cache_features is a performance improvement and safe for a master to
# apply. This is intended to allow agents to recognize new features that may be
# delivered during catalog compilation.
ARGV << "--always_cache_features"

# NOTE: it's unfortunate that we have to use the "CommandLine" class
#  here to launch the app, but it contains some initialization logic
#  (such as triggering the parsing of the config file) that is very
#  important.  We should do something less nasty here when we've
#  gotten our API and settings initialization logic cleaned up.
#
# Also note that the "$0 = master" line up near the top here is
#  the magic that allows the CommandLine class to know that it's
#  supposed to be running master.
#
# --cprice 2012-05-22

require 'puppet/util/command_line'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Util::CommandLine.new.execute

Chown the file for Puppet user :

# chown puppet:puppet /etc/puppet/rack/config.ru

And finally, restart Nginx :

# service nginx restart

Then, test you configuration by running the agent :

# puppet agent --test

Troubleshooting and errors

Error 500

Warning: Error 500 on SERVER: Internal Server Error

Read the logs at /var/log/nginx/error.log and /etc/nginx/puppet_error.log

Error 403

Warning: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /node/puppetmaster.example.com [find] at :119

Check that your hostname resolves, and that your host file is clean. In particular, you should have the host name of your server on the same line than localhost :

127.0.0.1    localhost puppetmaster puppetmaster.example.com

Also check that your Puppet configuration is correct, in particular check that the two lines "required for Passenger" are commented.

Sources

"SQLSTATE[HY000] [2002] No such file or directory" for compiled PHP

Mathieu — Tue, 19 Apr 2016 22:52:00 +0200

Let's say you connect to MySQL using "localhost"

Let's say you compiled PHP

Let's say you didn't specify the --with-mysql-sock= parameter in your configure command when you built mysql

And let's suppose you cannot connect to MySQL using PHP. CLI works fine, but not CGI.

Solution : fix the default sockets in php.ini (use your own working socket paths) :

pdo_mysql.default_socket=/var/run/mysqld/mysqld.sock
mysqli.default_socket = /var/run/mysqld/mysqld.sock

# You shouldn't use mysql_ extension, but if you did:
mysql.default_socket = /var/run/mysqld/mysqld.sock

I suppose that automatically converting "localhost" to an unix socket is done for performance reason on unix systems.

Debian 8 : Limit SSH users to SFTP

Mathieu — Thu, 18 Feb 2016 10:38:00 +0100

Let’s say you want to configure a secure remote file access for you users, but you can’t use FTPS for some reasons (problems with passive mode and commercial firewalls ? Yes !). Your only secure solution is either a VPN, or a SFTP access.

SFTP is great, but it may implies giving full command line access to your end users. In order to prevent that, you could set-up a jailed SSH access with Jailkit and some bind mount, but it’s not that trivial to configure and to maintain ; and it may not work with software virtualization (Docker, LXCs…). There is a simpler solution.

The solution is : use the native chroot and limitations abilities of OpenSSH. Here is how.

Warning!

You should not configure this on your primary SSH access. By doing so, you will simply lock you out of your server.

In this article, we will set up a completely new instance of OpennSSH server, running next to the original, and handling SFTP only.

1. Setup the secondary SSH access (SFTP-only)

Create a new configuration file by copying the primary configuration :

cp /etc/ssh/sshd_config /etc/ssh/sftp_config

Now edit the file /etc/ssh/sftp_config and change the listening port (for instance 10022) :

Port 10022

Change the PID file for this new instance, set something meaningful :

PidFile /var/run/sftp.pid

Then add these lines to /etc/ssh/sftp_config :

ChrootDirectory %h
ForceCommand internal-sftp
AllowTcpForwarding no

Here is a sample of a full configuration :

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 10022
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

PidFile /var/run/sftp.pid

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin no
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile    %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server
ChrootDirectory %h
ForceCommand internal-sftp
AllowTcpForwarding no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

Now, let’s configure autostart. Copy /lib/systemd/system/ssh.service to /lib/systemd/system/sftp.service and adjust settings :

[Unit]
Description=OpenBSD Secure Shell server (SFTP only)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS -f /etc/ssh/sftp.conf
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=sftp.service

And enable your service :

systemctl enable sftp.service

Make sure the symlink /etc/systemd/system/sftp.service is created.

And try to start it :

service sftp start

2. Reconfigure the primary SSH access

In order to prevent normal users to log into a full shell, we have to change the primary configuration.

The configuration file should be located in /etc/ssh/sshd_config . Add an AllowUsers or AllowGroups directive to this file :

# One or the other but not both!
AllowUsers root admin
#AllowGroups sudo

Here is a sample of a full configuration :

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin without-password
StrictModes yes
AllowUsers root admin

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile    %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

UsePAM yes

Restart your primary SSH access, but don’t close your terminal afterwards :

service ssh restart

Now open a new terminal and check that your primary SSH is still working. If not, rollback your configuration.

3. Conclusion

Now you should have two SSH sockets listening : one for everyone using exclusively SFTP, and the other with full SSH access for authorized accounts.

Don’t hesitate to reply in comments if you encounter problems.

Sources

https://wiki.archlinux.org/index.php/SFTP_chroot

Add Drush to Jailkit

Mathieu — Tue, 16 Feb 2016 10:16:00 +0100

Here is the configuration I use to make Drush working inside a Jailkit chrooted shell :

/etc/jailkit/jk_init.ini :

[php]
comment = the PHP interpreter and libraries
executables = /usr/bin/php5, /usr/bin/php
directories = /usr/lib/php5, /usr/share/php, /usr/share/php5, /etc/php5, /usr/share/php-geshi, [B]/usr/share/zoneinfo[/B]
includesections = env

[env]
comment = environment variables
executables = /usr/bin/env

[mysql-client]
comment = mysql client
executables = /usr/bin/mysql
paths = /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18

[drush]
comment = drush (drupal command line)
includesections = php, mysql-client, uidbasics, netbasics
directories = /usr/share/zoneinfo, /etc/ssl/certs, /usr/share/ca-certificates

Once the jailed shell works, add the Drush dependencies to the jail :

jk_init -v -c /etc/jailkit/jk_init.ini -f -k -j /absolute/path/to/jail/ drush

Set-up SQL quarantine with Amavisd-new and ISPConfig

Mathieu — Sun, 06 Dec 2015 16:56:00 +0100

It's documented, but it took me two days to do it correctly, so here is how to reconfigure an ISPConfig installation of Amavis to store quarantined mail in SQL database, in order to install a quarantine viewer like Mailzu.

1. Prerequisites

A working Postfix+Amavis stack with ISPConfig
A working SQL (PostgreSQL, MySQL...) database
Optional : a working mail server with PHP (for Mailzu)

Jump to Installation if you know what you are doing.

2. Off-subject generic explanations

2.1 Amavis and ISPConfig policies

ISPConfig policies, Tag-Levels

In a default ISPConfig installation per-user ISPConfig policies are loaded. The configuration file for Amavis, written by ISPConfig contains :

@lookup_sql_dsn =
   ( ['DBI:mysql:database=dbispconfig;host=127.0.0.1;port=3306', 'ispconfig', 'xxxx'] );
$sql_select_policy =
   'SELECT *,spamfilter_users.id'.
   ' FROM spamfilter_users LEFT JOIN spamfilter_policy ON spamfilter_users.policy_id=spamfilter_policy.id'.
   ' WHERE spamfilter_users.email IN (%k) ORDER BY spamfilter_users.priority DESC';
$sql_select_white_black_list = 'SELECT wb FROM spamfilter_wblist'.
    ' WHERE (spamfilter_wblist.rid=?) AND (spamfilter_wblist.email IN (%k))' .
    ' ORDER BY spamfilter_wblist.priority DESC';

It means that whatever you would set as $sa_spam_subject_tag, $sa_tag_level_deflt, $sa_tag2_level_deflt, $sa_kill_level_deflt, $sa_dsn_cutoff_level, it will be overridden by per-user policies.

The ISPConfig policies can be changed in tab Email => Spamfilter => Policy in ISPConfig panel. If you struggle wondering why your message keeps getting smashed at level 4.5, look at the sa_tag_level in policies. We will have to change values in that policies, to make the SQL quarantine working.

2.2 Lookup DSN and Storage DSN

DSN (Data Source Name) are the connection strings with host, username, and password, used to connect to databases.

Amavis can set two DSN : one for Policies lookup (used to retrieve ISPConfig policies from Panel), and one for storage of mail meta informations and quarantine. We will use the Storage DSN to set up a secondary database for quarantine storage, to not mess with existing ISPConfig database.

2.3 Levels and cutoffs

Amavis uses Spamassassin to score the mail, in order to decide what to do with it. The category of test (spam test, antivirus, etc) and the score along with levels determines the actions Amavis will trigger, and the final destiny where the mail belongs.

Spamassassin levels are :

tag_level : a message above that score will be tagged with X-Spam-Status, X-Spam-Score and X-Spam-Level headers.
tag2_level : a message above that score will be marked as X-Spam-Status: Yes and the subject is changed if sa_spam_modifies_subj is set to true.
kill_level : a message above that score is taken to the final_spam_destiny, and quarantined, it will not be delivered unless D_PASS is set to final_spam_destiny.
dsn_cutoff_level : a message above that level will never trigger a bounce or a reject, whatever spam_destiny is.
quarantine_cutoff_level : a message above that level will not be quarantined.

2.4 Final destinations

Once the message is categorized by Amavis tests (through SpamAssassin, ClamAV, etc), Amavis decides if it should be delivered to user mailbox or not, and if a bounce will be issued.

This is the purpose of $final_virus_destiny, $final_spam_destiny, $final_banned_destiny, $final_bad_header_destiny.

They can take the following values :

D_PASS : mail will be delivered to inbox.
D_BOUNCE : mail will not be delivered, and a delivery status notification will be returned by Postifx to sender (except if the score exceeds the dsn_cutoff level)
D_REJECT : Postfix will answer REJECT to the distant mail server, and the distant mail server may produce a delivery status notification to the user
D_DISCARD : forgive and forget : the mail will not be delivered and the sender is not informed. The mail may be quarantined if the quarantine_cutoff level is not exceeded.

3. Installation

For Amavis : Nothing ! Amavis comes out-of-the-box with SQL storage.
For Mailzu : see Mailzu section.

4. Configuration

4.1 Database

Create an user and a database for quarantine storage :

# mysql -u root -p
mysql> CREATE DATABASE amavis_storage;
mysql> CREATE USER 'amavis_storage'@'localhost' IDENTIFIED BY 'xxxx';
mysql> GRANT ALL PRIVILEGES ON amavis_storage.* TO 'amavis_storage'@'localhost';
mysql> FLUSH PRIVILEGES;

Load the initial schema from Amavis docs (usually located in /usr/share/doc/amavisd-new/ ).

Delete unnecessary tables, as we will be using this database only for mail storage and not for lookups :

# mysql -u amavis_storage -p amavis_storage
mysql> DROP TABLE users;
mysql> DROP TABLE mailaddr;
mysql> DROP TABLE policy;
mysql> DROP TABLE wblist;

Nota Bene : while executing DROP TABLE users, don't be silly, and do not remove mysql users database.

4.2 Amavis

Update your Amavis configuration /etc/amavis/conf.d/50_user :

@storage_sql_dsn = ( ['DBI:mysql:database=amavis_storage;host=127.0.0.1;port=3306', 'amavis_storage', 'xxxx'] );  # none, same, or separate database

# Quarantine SPAM into SQL server.
$spam_quarantine_to = 'spam-quarantine';
$spam_quarantine_method = 'sql:';

# Quarantine VIRUS into SQL server.
$virus_quarantine_to = 'virus-quarantine';
$virus_quarantine_method = 'sql:';

# Quarantine BANNED message into SQL server.
$banned_quarantine_to = 'banned-quarantine';
$banned_files_quarantine_method = 'sql:';

# Quarantine Bad Header message into SQL server.
$bad_header_quarantine_method = 'sql:';
$bad_header_quarantine_to = 'badheader-quarantine';

# Do not store non-quarantined messages info
# You can set it to 1 (the default) to test if Amavis is filling correctly the tables maddr, msgs, and msgcrpt
$sql_store_info_for_all_msgs = 0;

#
# SQL Select statements
#

$sql_select_policy =
   'SELECT *,spamfilter_users.id'.
   ' FROM spamfilter_users LEFT JOIN spamfilter_policy ON spamfilter_users.policy_id=spamfilter_policy.id'.
   ' WHERE spamfilter_users.email IN (%k) ORDER BY spamfilter_users.priority DESC';

$sql_select_white_black_list = 'SELECT wb FROM spamfilter_wblist'.
    ' WHERE (spamfilter_wblist.rid=?) AND (spamfilter_wblist.email IN (%k))' .
    ' ORDER BY spamfilter_wblist.priority DESC';

#
# Quarantine settings
#

$final_virus_destiny = D_BOUNCE;
$final_spam_destiny = D_DISCARD;
$final_banned_destiny = D_BOUNCE;
$final_bad_header_destiny = D_PASS;

# Default settings, we st this very high to not filter aut emails accidently
$sa_spam_subject_tag = '[SPAM] ';
$sa_tag_level_deflt  = 20.0;  # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 60.0; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 60.0; # triggers spam evasive actions
$sa_dsn_cutoff_level = 100;   # spam level beyond which a DSN is not sent
#$sa_debug = 1;

#
# Disable spam and virus notifications for the admin user.
# Can be overridden by the policies in mysql
#

$virus_admin = undef;
$spam_admin = undef;

#
# Enable Logging
#

$DO_SYSLOG = 1;
$LOGFILE = "/var/log/amavis.log";  # (defaults to empty, no log)

# Set the log_level to 5 for debugging
$log_level = 0;                # (defaults to 0)

4.3 ISPConfig policies

Remember that ISPconfig policies are overriding a lot of our configuration in 50_user. In order to majke the quarantine work, you have to reconfigure all the available policies in ISPConfig Panel.

Look at your policies list, you have to change the quarantine settings for every policies :

ISPConfig Mail Spamfilter Policy

When editing a policy, on the Quarantine tab, set the destinations :

ISPConfig Mail Spamfilter Policy Quarantine destinations

If you do not fill something in these fields, Amavis will not store quarantined mails in SQL database, and will just discard it !

These fields correspond to the virus_quarantine_to, spam_quarantine_to, banned_quarantine_to, bad_header_quarantine_to variables in Amavis configuration, and an empty value is overriding those we set in Amavis configuration.

4.4 Test

Send some spam to your server, check if the tables are populated :

mysql> SELECT * FROM maddr;

Check if meta informations are populated :

mysql> SELECT * FROM msgs;
mysql> SELECT * FROM msgrcpt;

And if quarantine is filling :

mysql> SELECT * FROM quarantine;

5. Cleanup !

You should not "setup and forget" your quarantine SQL storage. Messages has to be deleted periodically, otherwise your database will grow forever. Look at the documentaion in /usr/share/docs/amavisd-new to make a cronjob like this :

#!/bin/bash

SQL_HOST="localhost";
SQL_LOGIN="amavis_storage"
SQL_PASSWORD="xxxx"
SQL_DB="amavis_storage"

mysql --user="$SQL_LOGIN" --password="$SQL_PASSWORD" --host="$SQL_HOST" $SQL_DB -e " \
  DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 30*24*3600; \
  DELETE FROM msgrcpt WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id); \
  DELETE FROM quarantine WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id); \
  DELETE FROM maddr WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id) AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id); \
"

6. Mailzu

I have to admit, Mailzu seems a bit obsolete as I had to patch to make it working with Amavis 3.3 tables. But it still works pretty well for a simple task like reading and releasing quarantine mails.

6.1 Installation

Download the source files at http://sourceforge.net/projects/mailzu/.

62. Patch

The existing Mailzu source code is quite old, and the schema of Amavis SQL tables changed. Download and apply this patch in to make Mailzu work.

6.3 Configuration

I suppose that you know how to spawn PHP with CGI to serve the Mailzu files.

Configure your database login and password in config/config.php :

$conf['db']['dbType'] = 'mysql';
$conf['db']['dbUser'] = 'amavis_storage';
$conf['db']['dbPass'] = 'xxxx';
$conf['db']['dbName'] = 'amavis_storage';
$conf['db']['hostSpec'] = 'localhost:3306';

I am using IMAP login to authenticate in Mailzu. Unfortunately, I had to turn off SSL authentication, as it wasn't working. Here is my configuration :

$conf['auth']['serverType'] = 'imap';
$conf['auth']['imap_hosts'] = array( 'localhost:143' );
$conf['auth']['imap_type'] = 'imaptls';
$conf['auth']['imap_domain_name'] = 'example.com';

Don't forget to set yourself "super" :

$conf['auth']['s_admins'] = array ('me@example.com');

And to set your web uri :

$conf['app']['weburi'] = 'https://example.com/mailzu';

6.4 Configure in-app release

Mailzu can also release quarantined mail. I did not implement this function, but you have to set up the amavisd-release internface on an inet socket on port 9998, instead of the existing unix socket located at /var/lib/amavis/amavisd.sock . Read more.

References

Configurer dibbler-client pour IPv6 sur une Dedibox (Online.net) avec Debian 8 (Jessie)

Mathieu — Wed, 21 Oct 2015 14:24:00 +0200

La documentation d’Online pour IPv6 ne traite pas le cas de Debian, Ubuntu, ou toute distribution utilisant systemd (CentOS6, etc). La documention doit être étendue pour ajouter le service sysetmd au démarrage, en lieu et place de l’init script.

Vérifiez que l’IPv6 est activé

Cela devrait normalement être le cas, puisque le noyau par défaut de Debian 8 inclut nativement IPv6 et ne peut pas être désactivé. Mais au cas où vous ne verriez pas le link-local sur vos interfaces :

Changer dans /etc/modprobe.d/local.conf :

options ipv6 disable=0

Ajouter dans /etc/modules :

ipv6

Il faudra sans doute redémarrer pour appliquer les changments.

Récupérez votre préfixe et votre DUID depuis la console d’Online

Vous devez demander l’activation de IPv6 au support et créer votre /64 dans la console d’Online pour obtenir votre DUID.

Créez un /64 et n’utilisez pas le /48 ou le 56 directement, vous pourriez le regretter si vous souhaitez redécouper le réseau. Vous n’avez qu’un seul /48 par compte, un /56 pour chaque serveur, et un /64 pour chaque failover souscrit.

Configurez l’interface réseau

Encore une fois, c’est sans doute facultatif puisque Dibbler reconfigure l’interface lorsqu’il se lance, donc explicitons ça dans le fichier /etc/network/interfaces, juste au cas où :

iface eth0 inet6 static
    address your_ipv6_address
    netmask 64
    accept_ra 2

Notes sur Proxmox et le forwarding

Sous Proxmox on travaille sur l’interface bridge, c’est vmbr0 et non pas eth0.

Si le forwarding est activé, vous devez forcer le accept_ra à 2, une valeur de 1 fera ignorer les router advertisements lorsque le forwarding est activé. Explications. Ajoutez dans sysctl.conf :

net.ipv6.conf.vmbr0.accept_ra = 2

Notez bien que dans le cas où vous adressez vos machines virtuelles en IPv6 vous ne devez pas brancher l’interface IPv6 de vos VM directement sur l’interface vmbr0. Voilà pourquoi.

Compilez et installez Dibbler

Téléchargez Dibbler 1.0.1 à partir du site officiel
Installez build-essential : apt-get install build-essential
Décompressez Dibbler : tar -xzf dibbler-1.0.1.tar.gz && cd dibbler-1.0.1
Compilez et installez Dibbler : ./configure && ./make

Configurez Dibbler

Créez les dossiers et fichiers de configuration comme le précise la documentation :

Configurez le DUID dans /var/lib/dibbler/client-duid :

mkdir /var/lib/dibbler/
touch /var/lib/dibbler/client-duid
chmod 640 /var/lib/client-duid
# Set up you duid in client-duid
vim /var/lib/client-duid

Configurez /etc/dibbler/client.conf :

mkdir /etc/dibbler
vim /etc/dibbler/client.conf

Voici le contenu de mon client.conf :

auth-protocol reconfigure-key
auth-replay monotonic
auth-methods digest-hmac-md5
duid-type duid-ll
inactive-mode
log-level 8
iface eth0 {
    pd
    ia
}

Encore une fois, c’est à adapter en fonction du nom de votre interface.

Démarrez le client pour tester la connectivité :

dibbler-client run

Pour vérifier que Dibbler a configuré l’interface, pressez CTRL+Z pour suspendre le processus et vérifiez que l’IP et les routes sont bien configurées. Tapez la commande “fg” pour retourner au processus en cours d’exécution, et tapez CTRL+C pour stopper Dibbler.

Si pendant l’opération les routes et les IPs sont incorrectes, vérifiez que votre pare feu accepte les connexions entrantes par le port 546 UDP.

Configurez Dibbler au démarrage

Cette section est différente de la documentation d’Online.

Avant systemd, le système d’init par dépendances se contentait d’un script dans /etc/init.d/. Avec systemd, il faut créer un fichier service dans /etc/systemd/system/ .

Créez le fichier suivant : /etc/systemd/system/dibbler.service :

[Unit]
Description=Dibbler
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/dibbler-client start
ExecStop=/usr/local/sbin/dibbler-client stop
PrivateTmp=true
NonBlocking=yes

[Install]
WantedBy=multi-user.target

Lancez la commande suivante pour que systemd lise le fichier :

systemctl daemon-reload

Et activez le service :

systemctl enable dibbler.service

Et essayez de le lancer pour la première fois :

systemctl start dibbler.service

Vérifions que tout est correct :

service dibbler status
ifconfig
route -6
ping6 whatever-you-want.com

Si tout va bien, redémarrez et l’IPv6 fonctionnera dès le démarrage !

Addendum : juste au cas où, n’utilisez pas resolvconf pour pousser les DNS IPv6 automatiquement, conservez autant que possible vos DNS IPv4. Ça serait tellement dommage que votre connectivité IPv6 saute et que vous vous retrouviez sans DNS (oui ça m’est arrivé…).

Sources

Configure dibbler-client for IPV6 networking on Dedibox (or any Online.net) servers with Debian 8 (Jessie)

Mathieu — Wed, 21 Oct 2015 12:50:00 +0200

The Online documentation for IPv6 is not dealing with the case of Debian 8, Ubuntu, or any distribution using systemd. Systemd replaces upstart, so the procedure has to be extended to add systemd service for startup, replacing the previoux behavior that was using init scripts.

Ensure that you are IPv6-proof

It should be the case as Debian 8 is shipping a kernel with native IPv6, but just to be sure :

In /etc/modprobe.d/local.conf :

options ipv6 disable=0

In /etc/modules :

ipv6

You may have to reboot in order to apply the changes.

Set up your IPv6 prefix and get your DUID on Online console

You have to request IPv6 activation to support and create your /64 on Online console before getting your DUID working.

Make a /64 and do not use your /48 or your /56 directly, as you may regret it. You can have only one /48 by account, one /56 by server, and one /64 by IP failover (the /48 is divided to make the /56 and so on).

Configure your network interface

It may not be mandatory as Dibbler will reconfigure your interface, but you have to ensure that you accept router advertisements. Add to /etc/network/interfaces :

iface eth0 inet6 static
    address your_ipv6_address
    netmask 64
    accept_ra 2

Notes about Proxmox and forwarding

On Proxmox, you are working on the bridge interface, it should be vmbr0 instead of eth0.

If you enabled forwarding on this interface (to give your VM an access to IPv6 network), you have to force the accept_ra to 2, while the default value of 1 wil make your Debian to ignore router advertisements when forwarding is enabled ! Read more. Add to sysctl.conf :

net.ipv6.conf.vmbr0.accept_ra = 2

Also do not set the IPv6 interface of your VM to vmbr0, as you can break your network access. Read more.

Compile and install Dibbler

Download Dibbler 1.0.1 from the offical website.
Install build-essential : apt-get install build-essential
Extract Dibbler : tar -xzf dibbler-1.0.1.tar.gz && cd dibbler-1.0.1
Compile and install Dibbler : ./configure && ./make

Configure Dibbler

Make the directories and set up according to the documentation :

Set up duid :

mkdir /var/lib/dibbler/
touch /var/lib/dibbler/client-duid
chmod 640 /var/lib/client-duid
# Set up you duid in client-duid
vim /var/lib/client-duid

Set up client.conf :

mkdir /etc/dibbler
vim /etc/dibbler/client.conf

The content of my client.conf :

auth-protocol reconfigure-key
auth-replay monotonic
auth-methods digest-hmac-md5
duid-type duid-ll
inactive-mode
log-level 8
iface eth0 {
    pd
    ia
}

Note that if your interface name is different you have to change it accordingly. For instance, on a Proxmox host server, it should be vmbr0 instead of eth0.

Start dibbler to try the connectivity :

dibbler-client run

Hit CTRL+Z to suspend the process and check that the IP and routes are configured. Type the command “fg” to get back to the running process and hit CTRL+C to stop it.

If it doesn’t work, check your firewall, dibbler needs to listen port 546 UDP.

Set Dibbler at startup

This section differs from Online documentation.

Before systemd, the dependency-based boot sequence only needed an init script, for instance /etc/init.d/dibbler . With systemd, you have to make a service file in /etc/systemd/system/ .

Make the following file : /etc/systemd/system/dibbler.service :

[Unit]
Description=Dibbler
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/dibbler-client start
ExecStop=/usr/local/sbin/dibbler-client stop
PrivateTmp=true
NonBlocking=yes

[Install]
WantedBy=multi-user.target

Run the following command to make systemd read file :

systemctl daemon-reload

Enable the service :

systemctl enable dibbler.service

And then, try to start it :

systemctl start dibbler.service

Check everything is fine :

service dibbler status
ifconfig
route -6
ping6 whatever-you-want.com

Reboot and the IPv6 networking should work at startup !

Addendum : just in case, don’t use resolvconf with IPv6 DNS pushing and if possible keep IPv4 DNS, it would be such a pity if the IPv6 networking crashes while your DNS servers are set to IPv6 addresses (yeah, it happened to to me)…

Sources

About ToBeHost : the last of my student's projects

Mathieu — Thu, 01 Oct 2015 23:55:00 +0200

Hello there, this is not your ordinary reading on this blog. This post is a sort of introduction for a series of posts about the making of a free web hosting provider, implementing a web panel from scratch using Drupal for front-end and Puppet as back-end.

But first, let me introduce you the reason why we started the project. I said “we” because I am not the only one in the boat, but it’s me who will make all the technical choices, and implement it. I am also the one who was saying “give up” to my colleague a few months ago, before going back with this new and exciting crazy idea.

Genesis

ToBeHost is a project we started around 2008~2009. Before that, I was an active member at a free hosting provider (“Chez Mémé .net”). When this hosting provider closed, due to the lack of time from its founders, me and a friend I met on this hosting provider stated that we had both the technical and time resources to make our own hosting provider. It was just loaning a dedicated server and configuring all the stuff you need to have your site working. We were not experts (yet), but be were confident in the fact that we had the capacities to learn, and to make it working. This is also the moment where I started hacking on GNU/Linux

We tried several web panels. Between love and hate, we finally choose VHFFS. VHFFS is a great piece of software. For the kiddie I was, it was the sort of “perfect-even-scary” piece of Linux engineering. Because mass web hosting involves dealing with a lot of “subsystems” to make it work, what VHFFS was doing was simply amazing to me. Using libnss and nscd to make system user from a database was for me the cleverest way to make system users without having to deal with system commands. Using unix permissions and system users and groups instead of hacky chroots, safe_mode, and open_basedir was definitely feeling right to me. I learned a lot about Linux and Debian, and my first (and actually unique) contributed patch made me feel so proud of me.

The long love (and hate) story

Dealing with a single server, on a student time, was not complicated. When something went wrong, I was looking for the answer by myself because I was (and I am still) afraid of what people can think or say to me on a public IRC channel. We had some downtimes, and a few fears while restoring backups, but things were globally going well.

Some things were a little bit cumbersome, but I was thinking that it was part of the job. Re-compiling PureFTPD ans MySQL package at each upgrade was not actually a problem, because I was eager to learn how to patch, compile, and hack into Debian. I even rewrote the SQL queries when we upgraded to Debian Lenny, which was using libnss2.

The years passed, I was growing and becoming older (hey, don’t laugh). My years at university was taking much and much time over my “internet time”, as I was getting involved in the students’ association and my final exams were approaching. Finally, the project was in an informal standby, we decided that we had to close it, because we were both very busy elsewhere.

We officially closed. We sent the mail to users, I was ready to backup my configuration files and delete the data. We didn’t do it. Because of our sloppy feelings, and the fact that “it worked”, without “touching anything”. It was such a pity to let it go.

The last breath (or not)

Some months ago, we realized it had been several years that the project was officially dead. I didn’t stop hacking Linux. I had left my full-time job because I was working too much on projects I didn’t like (and some other reasons), and was running a freelance activity. My colleague was running a successful IT company with his associates, and we were working together on some professional projects (we still are).

New tools and software have appeared since my last glance to ToBeHost, and I also learned how to use new tools. I asked my colleague to let the project go, but we finally paid another month for the server.

And then, I thought: web hosting is actually what I am into at that time. I know how to do it the right way, with the smallest administration effort. None of the alternative panels was fitting as well as VHFFS for the work we wanted. Let’s take all the VHFFS goods (strictly unix permissions, not be afraid of user access to shell, trust user isolation and quota), and add some bullshit (PHP panel, made on top of a CMS), with a little bit of magic (Puppet automation). Let’s put it together, and see what is coming from it.

Voilà. This is how this project started.

Addendum

The project started, and it is pratically ready for beta-test. I will try to write that series of posts and explain how we made it, in order to inspire other people, and to help me keep my ideas clear.

As you may think, it is and it will be highly experimental. I don’t know if the growing of Puppet objects will be sustainable in the long term. I don’t know if Drupal will drive me crazy to the point I would rage delete my code. I don’t know if the whole project will be clean enough to release it opensource. But we will try.

Wanna join ?

Postfix : configure postmaster, hostmaster, and abuse catchall for RFC compliance

Mathieu — Sat, 29 Aug 2015 18:02:00 +0200

This short howto will show you how to set up a catchall for common required email addresses. Some mail servers are testing if mail is accepted on this addresses to detect spammymail servers. Hostmaster address can also be used for domain Trading, to check the ownership of the domain.

1. Create a file named /etc/postfix/regexp-catchall.cf with the following content:

# Catchall to comply with RFC standards
/^postmaster@/    youshouldreadit@mydomain.com
/^hostmaster@/    youshouldreadit@mydomain.com
/^abuse@/         youshouldreadit@mydomain.com

Replace youshouldreadit@mydomain.com with a mail address you actually read.

2. Open /etc/postfix/main.cf and locate (or create) the line virtual_alias_maps, and add at the end regexp:/etc/postfix/regexp-catchall.cf, for instance:

virtual_alias_maps = proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, regexp:/etc/postfix/regexp-catchall.cf

3. Restart Postfix.

Warning : read comment #4 for issues with this setup.

Linux : déterminer la taille réelle d'un fichier creux

Mathieu — Wed, 10 Jun 2015 18:12:00 +0200

Pour référence, et parce que c’est parfois utile de connaître la taille réelle d’un fichier creux.

Pour afficher la taille déclarée :

$ ls -lh nomdufichier
-rw-r--r-- 1 root root 401G Jun 10 18:09 nomdufichier

Pour afficher la taille réelle :

$ du -h nomdufichier
60G    nomdufichier

Rappel : un fichier creux est un fichier dont la taille déclarée pour le système est différente de la taille effective sur le disque. C’est particulièrement utile pour créer des fichiers à écriture aléatoire (par exemple des fichiers téléchargés par P2P), ou des disques de machines virtuelles (qcow2, vmdk).

Un fichier creux s’obtient en déclarant un index de fin de fichier plus grand que la taille effectivement “remplie” par le fichier sur le disque.

Configurer un backup incrémental avec duplicity, rsync, et backupninja sous Debian

Mathieu — Tue, 09 Jun 2015 14:37:00 +0200

English version.

Introduction

Si vous savez ce qu'est un backup, vous devriez savoir qu'il y a plusieurs types de backups :

Un backup complet copie tous les fichiers, en espérant que votre disque de backup soit assez grand pour accueillir plus d'un backup.
Un backup incrémental commence par faire un backup complet, puis les fois suivantes n'enregistre que les "différences" sur les fichiers modifiés.

Évidemment, un backup complet est plus simple à restaurer, puisque ce sont les fichiers, et rien que les fichiers ; alors que le backup incrémental a un format de fichier spécial lui permettant de représenter les "diffs" à chaque sauvegarde. Mais considérant le gain d'un backup incrémental en bande passante, vitesse, et espace disque , votre solution pour un backup régulier devrait être le backup incrémental.

Les outils

Duplicity est un logiciel libre similaire à rdiff-backup. Duplicity permet de créer un backup incrémental. Duplicity peut également chiffrer les fichiers, pour pouvoir les envoyer en toute sécurité sur un service de stockage distant. Dans une configuration classique, Duplicity est utilisé conjointement avec rsyc pour optimiser les temps de transfert vers le stockage distant, mais vous pouvez très bien utiliser un disque de stockage local, un serveur FTP, ou un cloud Amazon E3 pour y envoyer vos sauvegardes. Comme le titre l'indique, dans cet exemple j'utiliserai rsync.

Et les bases de données ? Les bases de données ne peuvent pas être sauvegardées simplement en copiant les fichiers de données, car cela peut provoquer une corruptio de données, et vous aurez des bases inutilisables dans vos sauvegardes. Il nous faudrait donc un script qui dump toutes les bases de données avant de sauvegarder ces fichiers de dump via notre méthode de sauvegarde préférée.

Bonne nouvelle : backupninja sait faire tout cela. Backupninja est une sorte de "backup-master" : il est capable de récupérer des données depuis différentes sources (fichiers, bases de données...) et les envoyer à différentes destinations (backup simple, duplicity...), il suffit juste d'écrire les fichiers de configuration correspondants !

Dans cet exemple, nous utiliserons donc backupninja pour récupérer nos fichiers et nos bases de données, et les envoyer pour un backup incrémental Duplicity, stocké sur un serveur SSH distant, en utilisant rsync.

C'est parti !

Pré-requis

Une machine sous Debian que nous allons backuper, appelons-la production
- Un serveur de backup avec accès SSH (SSH est requis pour rsync mais vous pourriez avoir un simple serveur FTP), appelons-la backup
Savoir modifier des fichiers de configuration et relancer un service

Configurez le compte sur le serveur de backup

Pour pouvoir envoyer les sauvegardes vers le serveur backup via rsync, nous avons besoin de nous identifier sur ce serveur à partir de production sans spécifier de mot de passe.

Tout d'abord, travaillons en root sur production : sudo -s

Créez les clef SSH pour root : ssh-keygen -t rsa

Après avoir tapé la commande précédente, la clef publique va s'installer dans /root/.ssh/id_rsa.pub et la clef privée dans /root/.ssh/id_rsa

À présent, considérons backup, vérifiez que votre serveur SSH sur backup accepte l'authentification par clefs privées. Dans le fichier /etc/ssh/sshd_config, les lignes suivantes doivent apparaître (non commentées) :

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile     %h/.ssh/authorized_keys

Sinon, ajoutez-les et relancez le service ssh.

Maintenant, ajoutez l'utilisateur prod_server sur backup : useradd prod_server
Vous pourriez avoir envie changer son dossier home, je suppose que vous savez comment le faire.

Retour sur production, copiez la clef publique dans le fichier /home/prod_server/.ssh/authorized_keys de backup.
La méthode bourrin : scp /root/.ssh/id_rsa.pub root@backup:/home/prod_server/.ssh/authorized_keys
La méthode gentlemen : copiez-collez le contenu de production:/root/.ssh/id_rsa.pub à la fin du fichier backup:/home/prod_server/.ssh/authorized_keys

Si vous avez tout fait correctement, vous devriez pouvoir vous connecter en ssh depuis votre compte root de production sur le compte prod_server de backup sans spécifier de mot de passe.

Installez backninja (et ses amis)

Sur production, entrez la commande : sudo apt-get install backupninja duplicity rsync

Configurez le dump des bases de données

C'est parti, configurons backupninja !

Pour MySQL, copiez le fichier d'exemple à partir de /usr/share/doc/backupninja/examples/example.mysql dans le dossier /etc/backup.d/10-alldb.mysql et modifiez-le pour vos propres besoins (le fichier est simple et bien commenté).

databases   = all
backupdir   = /var/backups/mysql
hotcopy     = no
sqldump     = yes
compress    = yes
configfile = /etc/mysql/debian.cnf

Pour PostgreSQL, même chose ! Copiez le fichier /usr/share/doc/backupninja/examples/example.mysql dans le dossier /etc/backup.d/10-alldb.pgsql et modifiez les valeurs souhaitées.

backupdir = /var/backups/postgres
databases = all
compress = yes
format = plain

Pourquoi est-ce que j'utilise le préfixe 10- au début de mes fichiers ? Pour la même raison pour laquelle vous préfixez vos fichiers de configuration Nginx : pour gérer l'ordre de lecture. J'ai besoin de créer les dumps des bases de données avant de créer le backup des fichiers, sinon Duplicity enregistrera les fichiers de la veille, et les fichiers que nous venons de créer devront attendre le lendemain pour être sauvegardés ! Donc, les fichiers de configuration qui vont créer les dumps seront préfixé par 10- et les fichiers pour configurer le backup incrémental de Duplicity seront préfixés par 20-, par exemple.

Les dumps SQL seront stockés dans /var/backups/ selon ma configuration actuelle.

Configuration de Duplicity pour le backup incrémental

Comme tout à l'heure, copiez le fichier d'exemple : cp /usr/share/doc/backupninja/examples/example.dup /etc/backup.d/20-files.dup

Et modifiez les valeurs, en particulier la valeur desturl :

# Disable testconnect because we use desturl
testconnect = no

# You may change this if you are a partition-maniac
tmpdir = /tmp

[gpg]
# Using symetric encryption for archive files
# Note that an encryption method is mandaory, either with symetric or private keys
# Don't forget to note that password somewhere !
password = whateveryouwant

[source]
# Specify the paths to your files
include = /var/spool/cron/crontabs
include = /var/log
include = /var/mail
include = /var/www
include = /etc
include = /root
include = /home
include = /usr/local

# And to your database dumps !
include = /var/backups/mysql
include = /var/backups/postgresql

# There are some files we don't need
# Don't forget to add the tmpdir in the exclude list, if it was included in the previous paths !
exclude = /home/*/.gnupg
exclude = /var/cache/backupninja/duplicity
exclude = /tmp

[dest]
# Adjust these to your own taste
incremental = yes
increments = 15
keep = 30
keepincroffulls = all

# Specify the backup server crendentials
## desturl = file:///usr/local/backup
## desturl = rsync://user@other.host//var/backup/bla
## desturl = s3+http://
## desturl = ftp://myftpuser@ftp.example.org/remote/ftp/path
desturl = rsync://prod_server@backup//home/prod_server/data

# Only if you choose FTP
# ftp_password = whateveryouwant

Cette configuration utilise un chiffrement symétrique pour les archives sauvegardées. Si vous voulez mettre en place un chiffrement asymétrique, vous pouvez lire la documentation et adapter la configuration.

Configurez la fréquence des backups

Vous avez peut-être remarqué que backupninja a créé un cronjob dans /etc/cron.d/backupninja mais si vous ouvrez ce fichier, vous constaterez que le cronjob est lancé toutes les heures. Pourquoi ? Parce que le cronjob n'est utilisé que pour lancer le backup, qui vérifie si une mise à jour du backup est nécessaire avant de lancer réellement l'opération de backup. Pour modifier les heures et la fréquence des mises à jour, vous pouvez modifier le fichier /etc/backupninja.conf :

when = everyday at 01:00

Ajustez la valeur selon vos souhaits. Vous pouvez combiner plusieurs lignes, si vous voulez plusieurs dates de démarrage (en savoir plus).

Démarrez votre backup, et vérifiez son bon fonctionnement

C'est le moment de démarrer votre backup pour la première fois ! Si vous ne voulez pas attendre une heure pour que le cron se lance, vous pouvez lancer le backup immédiatement avec la commande backupninja -d -n

Si aucune erreur ne s'affiche pendant l'exécution, bravo ! Vous venez de configurer backupninja avec succès !

À présent, contrôlons que notre backup est bien arrivé : connectez-vous à backup et listez le fichiers dans le dossier /home/prod_server/data/

Vous devriez voir les fichiers d'index créés par Duplicity, vous ne pouvez pas les ouvrir sans utiliser quelques commandes :

Afficher le statut des "collections" : duplicity collection-status file:///home/prod_server/data
Lister les fichiers dans l'archive : duplicity list-current-files file:///home/prod_server/data
Restaurer le dernier backup dans un dossier précis duplicity restore file:///home/prod_server/data/ /home/prod_server/test-restore/ (si vous ne voyez des erreurs mais que les fichiers sont bien présents à la fin de l'exécution de la commande, c'est parce que Duplicity essaye de restaurer les atributs "prorpiétaire" des fichiers, et qu'il ne peut pas le faire si vous n'avez pas lancé la commande en root).

Vous devez avoir le programme Duplicity installé sur le serveur backup pour pouvoir utiliser ces commandes en mode shel sur backup.

Vous pouvez aussi lancer ces commandes sur le serveur production et accéder aux backups stockés sur backup en utilisant la desturl que vous avez configurée au lieu du préfixe file:// que vous utilisez quand vous lancez la commande sur backup.

Conclusion

Vous devriez maintenant être capables d'implémenter une solution complète de backup en utilisant backupninja.

J'espère que cet article vous a été utile, vous pouvez rédiger un commentaire via le formulaire ci-dessous si vous rencontrez des problèmes avec cette documentation.

Nagios : quick and dirty patch to enable (force) SSL on check_mysql_health

Mathieu — Sat, 06 Jun 2015 16:10:00 +0200

Sometimes you don’t want to set up a VPN just to safely monitor your MySQL servers. Because SSL should be implemented in check_mysql_health, here is a quick and dirty patch for SSL connexion. I assume you already configured your MySQL server to use SSL if client wants to (or if user requires ssl).

File /usr/lib/nagios/plugins/check_mysql_health at line 1863, after the following block :

    } else {
      $self->{dsn} .= sprintf ";host=%s", $self->{hostname};
      $self->{dsn} .= sprintf ";port=%s", $self->{port}
          unless $self->{socket} || $self->{hostname} eq 'localhost';
      $self->{dsn} .= sprintf ";mysql_socket=%s", $self->{socket}
          if $self->{socket};

Add these lines :

    $self->{dsn} .= ";mysql_ssl=1";
    $self->{dsn} .= ";mysql_ssl_client_key=/etc/ssl/mysql/client.key";
    $self->{dsn} .= ";mysql_ssl_client_cert=/etc/ssl/mysql/client.crt";
    $self->{dsn} .= ";mysql_ssl_ca_file=/etc/ssl/mysql/ca.crt";

Where /etc/ssl/mysql/client.key is the path to client key, /etc/ssl/mysql/client.crt the path to client certificate, and /etc/ssl/mysql/ca.crt the path to the CA certificate.

It should work, while there is still no “SSL switch” on that plugin.

EDIT : actually there is an undocumented param named “—mycnf” which should allow you to enable SSL for client connection in a prettier way.

Migrate an OpenVPN configuration to Debian 8 (Jessie) with systemd

Mathieu — Sat, 23 May 2015 10:32:00 +0200

This article could have been avoided if the Debian documentation was up-to-date. Actually it is not, and the solution came from Fedora documentation for OpenVPN.

Debian 8 uses systemd by default, and it implies several changes, in particular the way you start/stop your services.

The main topic : systemd

What changes

A new fancy command now manage the startup : systemctl (don't mess with the sysctl command used for network configuration !)
The startup dependencies are no longer in the LSB headers in startup scripts (way too simple, boy), the dependencies are stored as symlinks in subdirectories located in /etc/systemd/* . Note that /etc/systemd/ contains some static configuration files, and that the real services configuration files are stored in /lib/systemd/* (this is where the symlinks from /etc/systemd/* points to).
Some services have been split from monolithic startup to dynamic. It means that you potentially have to enable and run multiple "services" in order to actually start the full "service". For instance, OpenVPN no longer runs every available configuration in /etc/openvpn/*.conf , you have to explicitely activate each *.conf file as a service in systemd !

What doesn't change

You can still start your services with service servicename start.
The init scripts in /etc/init.d/* still exists, and some are still usable (monolithics services).

Run your OpenVPN configuration

While you can still use the command openvpn --config /etc/openvpn/yourconfigfile.conf, you should do it with systemd. If your configuration file is /etc/openvpn/sample.conf, you should start your VPN connexion with systemctl start openvpn@sample.service .

Note that service openvpn@sample start also works.

Start your VPN at boot

Again, the auto startup was too simple. You now have to enable every *.conf file at boot. Enable you newly sample.conf at startup with the command systemctl enable openvpn@sample.service

This actually creates a symlink in /etc/systemd/system/multi-user.target.wants/openvpn@sample.service pointing to /lib/systemd/system/openvpn@.service

Ok, it's simpler for dynamic loads, but who needs to dynamically enable and disable configuration at boot ? If I want a different configuration, I simply write different files in the right folder...

Meanwhile, in Debian Apache package

You can enable and disable VirtualHosts by using the /etc/apache2/sites-available/ and /etc/apache2/sites-enabled/ folders. No hidden features, no boot startup configuration, all configuration files in /etc/programname/ . Way too simple ?

Hey, what if we had to configure a startup script for every VirtualHost ? Should be fun, don't you think ?

Sources

Configure sender rate limits to prevent spam, using cluebringer (policyd) with Postfix

Mathieu — Fri, 13 Mar 2015 09:18:00 +0100

This small how-to will show you how to configure cluebringer (aka policyd) to set a per-hour/per-user limit for sent mails. Note that sending to multiple recipient will count like multiple mails were sent.

This how-to is Debian-oriented but should apply to any unix operating system.

Requirements

A mail server with Postfix installed.

Installation

Install a DBMS (MySQL for instance), cluebringer, and cluebringer-webui :

apt-get install mysql-server cluebringer cluebringer-mysql cluebringer-webui

Note that cluebringer-webui will install apache as a dependency if you don’t already have a webserver.

Set-up the Cluebringer database

Get the initial database schema that correspond to your DBMS, for instance mysql :

cp /usr/share/doc/postfix-cluebringer/database/policyd-db.mysql.gz ~/ && gunzip ~/policyd-db.mysql.gz

Create the database, and populate it with the initial dump :

# cd  ~/ && mysql -u root -p
mysql> CREATE DATABASE cluebringer;
mysql> CREATE USER 'cluebringer'@'localhost' IDENTIFIED BY 'mypassword';
mysql> GRANT ALL PRIVILEGES ON cluebringer.* TO 'cluebringer'@'localhost';
mysql> \. policyd-db.mysql
mysql> quit
mysql> Bye

Note that on Debian I had to modify the dump to make it work, TYPE=InnoDB was rejected by MySQL as an invalid syntax.

Configure Cluebringer

Add your DBMS credentials to the file /etc/cluebringer/cluebringer.conf :

DSN=DBI:mysql:dbname=cluebringer;host=localhost

DB_Type=mysql
DB_Host=localhost
DB_Port=3306
DB_Name=cluebringer
Username=cluebringer
Password=mypassword

And start it :

service postfix-cluebringer start

Configure Cluebringer webui

Configure the file /etc/cluebringer/cluebringer-webui.conf with your DBMS credentials :

<?php

$DB_DSN="mysql:host=localhost;dbname=cluebringer";
$DB_USER="cluebringer";
$DB_PASS="mypassword";

Cluebringer Webui needs a web server to run. Copy the sample configuration from the package documentation :

cp /usr/share/doc/postfix-cluebringer-webui/examples/httpd/cluebringer-httpd.conf /etc/apache2/conf.d/

Restart Apache :

service apache2 restart

You may need to adjust a few things to access it from the outside. If you a really lazy, just make a ssh tunnel to access the webserver from localhost :

ssh -L 8008:localhost:80 mylogin@mymailserver

Don’t forget : you have to make this tunnel from the outside, do not run this command on server, it won’t work.

You should now be able to open http://localhost:8080/ and see your fresh new Cluebinger Webui !

Configure Cluebringer using its webui

Add a policy

Under Policies -> Main, disable Test policy (select policy and choose Action -> Change and switch Disabled to yes, validate)

Add a new policy : Action -> Add, give it a name and a description

Activate your new policy : select policy and choose Action -> Change (switch Disabled to no)

Add a new member to your policy : select it and choose Action -> Members, and then Action -> Add. Specify any as source and any as destination.

Go back to your policy, choose Action -> Members, and the select your member, do Action -> Change, and activate your new member (switch Disabled to no).

Add a quota

Under Quotas -> Configure, disable Test quotas.

Add a new quota : Choose Action -> Add

Name : whatever you want
Track : user@domain
Period (seconds) : 3600
Link to policy : specify the policy you created here
Verdict : Defer
Data : set a custom error message here
Comment : whatever you want

Activate your quota : switch Disabled to no.

Add a limit to your quota : select your quota, and choose Action -> Limits, then Action -> Add.

Type : MessageCount
Counter Limit : 200

Activate your limit : switch Disabled to no.

Configure Postfix to call Cluebringer for each mail sent

Open /etc/postfix/main.cf and locate the line smtpd_sender_restrictions.

Add check_policy_service inet:127.0.0.1:10031 at the end of the line, for instance :

smtpd_sender_restrictions = check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf, check_policy_service inet:127.0.0.1:10031

If the line does not exists, simply add it.

Don’t forget to restart Postfix :

service postfix restart

Check your config

You can now send some mails to see what happens. To check if these mails are passed to Cluebringer, connect to MySQL as the cluebringer user :

# mysql -u cluebringer -p cluebringer

And execute the query :

mysql> SELECT * FROM quotas_tracking;

You should see the value LastUpdate and Counter updating when sending a mail. Note that sending to multiple recipient will count like multiple mails were sent.

Pitfalls, bleeding edges, etc

Cluebringer versions prior to 2.1.x does not support IPv6, your customers won’t be able to send any mail if they have an IPv6 connection.

Unfortunately, the Debian stable version (wheezy) provides Cluebringer 2.0.10 within its repositories, as well as the experimental release of Debian (sid). As an alternative, you should consider installing the 2.1.x experimental Cluebringer from official website instead of Debian packages from repositories.

References

Set up an incremental backup with duplicity, rsync, and backupninja on Debian

Mathieu — Fri, 20 Feb 2015 22:08:00 +0100

Version française.

This is a not-so-concise how-to about setting up an incremental backup, using Backupninja with Duplicity backend on Debian.

Abstract

If you know what a backup is, you should know there are several types of backups :

A full backup is when you just copy all your files, hoping that the hard drive on the backup server will not explode after 3 backups.
An incremental backup consist in a base full backup, and the next backups are just "diffs" sent to the backup server, to keep track of modified files.

Obviously, a full backup is easier to read and to restore because it's just plain files, whereas an incremental backup has a specific file format to represent diffs. But considering the gain in speed, bandwidth, and disk space, your choice for a long-term backup solution should be the incremental backup.

The tools

Duplicity is an opensource software similar to rdiff-backup. It creates incremental backups. Duplicity can also encrypt your backups, so they can be safely sent to any remote disk provider. A classic setup for Duplicity would be using rsync as a backend to send files faster to the remote backup server, but you can also use a local drive, a remote FTP server, or an Amazon E3 cloud server. As the title says, I will be using rsync for that setup.

But what about databases ? Databases can't be saved by simply copying files, it could lead to corrupted and unusable data in your backups, so you would use a backup script to fetch your databases before sending it to Duplicity.

Good news : backupninja is the global solution you need. Backupninja is a sort of "backup-master" : it can fetch different type of data (files, databases...) from different sources and sent it to different destinations (plain backup, duplicity, etc). you just have to write a specific config file for each source !

We will use backupninja to fetch our databases, we will add these SQL archives to our files backup, send this to Duplicity backend, and finally send it to our backup server with rsync. And with just 3 config files (one by SQL type, one for Duplicity and rsync).

Let's go !

Requirements

A Debian machine to backup, let's call it production
A backup server, with SSH access (SSH is required for rsync, but an FTP server can do the job), let's call it backup

Configure the backup user on the remote machine

In order to send the backed up files to backup, we will need a way to authenticate from production without specifying a password.

First, be root on production : sudo -s

Then, create ssh key pairs : ssh-keygen -t rsa

The public key will set up on /root/.ssh/id_rsa.pub and the private key in /root/.ssh/id_rsa

Now, let's consider backup, check if your SSH server on backup acccepts the public key authentications. In /etc/ssh/sshd_config, you should have the lines :

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile     %h/.ssh/authorized_keys

If not, simply add it (and restart ssh service).

Now, add the prod_server user on backup : useradd prod_server
You may want to change its home, I assume you know how to do it.

Now, back to production, simply copy the SSH public key to the backup file named /home/prod_server/.ssh/authorized_keys
The trashy way : scp /root/.ssh/id_rsa.pub root@backup:/home/prod_server/.ssh/authorized_keys
The classic way : copy/paste the content of production:/root/.ssh/id_rsa.pub at the end of the file backup:/home/prod_server/.ssh/authorized_keys

If you did it right, you should now be able to ssh from production to backup without password.

Install backupninja (and friends)

On production, use the Debian magic line : sudo apt-get install backupninja duplicity rsync

Configure databases dump

Okay, let's start configuring backupninja !

For MySQL, copy the sample file from /usr/share/doc/backupninja/examples/example.mysql to /etc/backup.d/10-alldb.mysql and change it for your own needs (the file is pretty trivial and well commented).

databases   = all
backupdir   = /var/backups/mysql
hotcopy     = no
sqldump     = yes
compress    = yes
configfile = /etc/mysql/debian.cnf

For PostgreSQL, all the same ! Copy /usr/share/doc/backupninja/examples/example.mysql to /etc/backup.d/10-alldb.pgsql and change the desired values.

backupdir = /var/backups/postgres
databases = all
compress = yes
format = plain

Why am I using that weird 10- at the beginning of the file ? For the same reason you have to prefix your files with number in Nginx configuration : for precedence. I need the database dumps to be done just before the files backup, otherwise, backupninja will send to duplicity the dumps of yesterday, and next create new dumps without including it in the Duplicity files for the current backup ! So, my database backup configuration files have to be like 10- and my Duplicity configuration file should be like 20-, for instance.

Note that the SQL dumps will be generated in /var/backups/ .

Configure Duplicity files backup

Again, copy the sample file : cp /usr/share/doc/backupninja/examples/example.dup /etc/backup.d/20-files.dup

And change its values :

# Disable testconnect because we use desturl
testconnect = no

# You may change this if you are a partition-maniac
tmpdir = /tmp

[gpg]
# Using symetric encryption for archive files
# Note that an encryption method is mandaory, either with symetric or private keys
# Don't forget to note that password somewhere !
password = whateveryouwant

[source]
# Specify the paths to your files
include = /var/spool/cron/crontabs
include = /var/log
include = /var/mail
include = /var/www
include = /etc
include = /root
include = /home
include = /usr/local

# And to your database dumps !
include = /var/backups/mysql
include = /var/backups/postgresql

# There are some files we don't need
# Don't forget to add the tmpdir in the exclude list, if it was included in the previous paths !
exclude = /home/*/.gnupg
exclude = /var/cache/backupninja/duplicity
exclude = /tmp

[dest]
# Adjust these to your own taste
incremental = yes
increments = 15
keep = 30
keepincroffulls = all

# Specify the backup server crendentials
## desturl = file:///usr/local/backup
## desturl = rsync://user@other.host//var/backup/bla
## desturl = s3+http://
## desturl = ftp://myftpuser@ftp.example.org/remote/ftp/path
desturl = rsync://prod_server@backup//home/prod_server/data

# Only if you choose FTP
# ftp_password = whateveryouwant

Note that this setup uses symetric encryption for duplicity archives, if you want an asymetric encryption to enforce the safety of your backups, you should read the docs and adapt this sample configuration.

Configure backupninja update frequency

You may have noticed that backupninja set up a cronjob in /etc/cron.d/backupninja but if you open that file, you will notice that this cronjob runs every hour. Why ? Because this cronjob is only used to start the backup manager, to check if a refresh is needed, not to actually run the backup every hour. You can set the backup frequency in the file /etc/backupninja.conf :

when = everyday at 01:00

Adjust it to your needs, and then save it. Note that you can combo multiple lines, if you want multiple start dates (read more).

Run you first backup, and check it

It's time for the first run ! If you don't want to wait for the cronjob, execute your first run with the command backupninja -d -n

If you see no errors, Bravo ! you have configured backupninja !

Now, let's check our backup : connect to backup and list files under /home/prod_server/data/

You should see the files created by duplicity, you cannot use it without some commands :

Show the collection status : duplicity collection-status file:///home/prod_server/data
List files in archive : duplicity list-current-files file:///home/prod_server/data
Restore the latest backup in a specific directory duplicity restore file:///home/prod_server/data/ /home/prod_server/test-restore/ (if you see errors but files are there after the command, it's because duplicity tries to chown files, while you were not root when starting the command).

Note that you must have Duplicity installed on backup on in order to use these commands on backup.

Also note that instead of using file:// on backup, you can run the command on production and use the ssh desturl.

Conclusion

You should now be able to implement a complete backup solution, using backupninja.

I hope this post was useful to you, you can write a comment below if you experienced problems with this how-to.

Conflict between broken Apache packages makes FCGI angry (mod_fcgid: can't lock process table)

Mathieu — Sat, 08 Feb 2014 17:37:00 +0100

Here is is a quick note on the fix I made today for a messed up Apache server.

The environment

Apache2 + Debian + mod_suexec + mod_fcgi

The problem

The server was down only for PHP-running websites. It triggered a connexion reset when trying to run PHP scripts. Static content and reverse proxies were working correctly, as if nothing was happening.

In the server logs, the following error was present :

mod_fcgid: can't lock process table

The suexec log was showing no anomaly.

The server wasn’t gracefully stopping anymore, and I had to kill him with the -9 signal to stop it.

The cause

Searching around the web was giving me very few clues on the problem. I couldn’t find why the CGI subprocess was broken, it is spawned by Apache and so it should not be missing permissions (there is no CGI daemon listening in Apache setup, contrary to Nginx)

What causes the Apache subprocesses to get spawned ? Either the Apache mpm-workers, or mpm-prefork, or mpm-event or mpm-itk. I knew I was using workers, but a glance in APT installed packages showed me broken packages on workers and ITK. Wait… ITK ?

Actually, I run Puppet, and Puppet was configured to keep Apache ot its latest version from repositories. With the release of a newer version of Apache, it had updated the apache2 package … and installed apache2-mpm-itk while apache2-mpm-workers was still present !

The result was a conflict while trying to spawn CGI processes, and a crash of the subprocess.

The fix

Doing a remove apache2-mpm-itk, and force-reinstall apache2-mpm-workers did the job.

Configure Postfix as standalone single-domain SMTP server using Unix users and PAM on Debian

Mathieu — Sat, 01 Feb 2014 22:05:00 +0100

Here is a quick setup to configure Postfix mail server, using existing Unix users.

The server will process mails for only one domain, and every existing user on the server will have a mail box inside his home directory.

Abstract

Postfix is an SMTP server, it receives incoming mail from other SMTP servers, and allows client to send mails to other SMTP servers.

What we don't want is an open mail relay. A mail relay is a SMTP server that take anything from any client, and send it to any SMTP server. We only want trusted users to send emails, to prevent anonymous clients from sending spam.

Incoming mail will be processed either if :

The domain name of one of the recipient matches the mail server domain, and the mail user name is also a system user (SMTP servers can send us incoming mails).
The client who tries to sends the mail has successfully authenticated.

Postfix authentication for clients can be handled by SASL. SASL is a standard protocol to provide an authentication layer. It can query PAM, or other authentication providers (MySQL users, etc).

Notes :

We will use PAM for Unix users SMTP authentication.
Unix users are stored in /etc/passwd and their passwords are stored in /etc/shadow.
Mails will be stored in the ~/Maildir/ of each users, in Maildir format.

Postfix : installation and configuration

Install Postfix : apt-get install postfix

Answer the questions during installation to setup your mail domain (the "example.com" in user@example.com).

Modify config files :

/etc/postfix/main.cf :

Configure TLS and Maildir :

# TLS parameters smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt smtpd_use_tls=yes smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for # information on enabling SSL in the smtp client. myhostname = mail.example.com alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases myorigin = /etc/mailname mydestination = example.com, localhost mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 mailbox_size_limit = 0 recipient_delimiter = + home_mailbox = Maildir/ # These are the "no relay" restrictions smtpd_recipient_restrictions = permit_mynetworks permit_inet_interfaces permit_sasl_authenticated reject_unauth_destination

/etc/postfix/master.cf :

Enable TLS and alternate (submission) ports :

submission inet n - - - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o milter_macro_daemon_name=ORIGINATING smtps inet n - - - - smtpd -o syslog_name=postfix/smtps -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o milter_macro_daemon_name=ORIGINATING

SASL : installation and configuration

SASL plugin for Postfix (Cyrus) is part of the dependencies of Postfix server.

Install SASL administration tools : apt-get install sasl2-bin

Enable SASL daemon at startup : edit /etc/default/saslauthd and switch START to yes.

Start it manually for the first time : service saslauthd start

Enable PAM authentication for SASL

Check that PAM is part of the MECHANISMS variable in /etc/default/saslauthd :

MECHANISMS="pam"

Create /etc/pam.d/smtp :

# # /etc/pam.d/smtp - specify PAM SMTP behavior # @include common-auth @include common-account @include common-password @include common-session

Enable SASL for Postfix

Add to /etc/postfix/main.cf :

smtpd_sasl_auth_enable = yes

Create /etc/postfix/sasl/smtpd.conf :

pwcheck_method: saslauthd mech_list: PLAIN LOGIN

Adjust OPTIONS in /etc/default/saslauthd :

OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Add postfix user to sasl group :

adduser postfix sasl

Configuration check

Restart all services (postfix, salsauthd).

Try authentication using SASL : testsaslauthd -u user -p password

Try authentication from command line, without mail client : https://qmail.jms1.net/test-auth.shtml

Try SMTP reception by sending mail to your domain (your MX fields in domain has to be configured accordingly).

Sources

Un serveur au régime : faites tourner un serveur complet dans un mouchoir de poche

Mathieu — Mon, 25 Nov 2013 07:15:00 +0100

Introduction

Ce n’est pas la période pour les régimes et pourtant aujourd’hui, dans un billet volontairement cryptique aux non-initiés, je vais vous dévoiler les quelques secrets qui permettent à des nerds invétérés de donner vie à des machines ridiculement pauvres en RAM. Pour la beauté du geste, mais pas que.

Je me souviens que l’on s’est souvent moqué de moi quand je disais qu’avec 512 mégas de RAM on pouvait faire tourner un serveur complet mail+ftp+web+sql. En remplaçant quelques programmes, vous pouvez parfaitement faire tourner votre application web standalone, un noeud de CDN performant, un miroir FTP, ou un relais mail.

Cet article va simplement vous présenter des alternatives, et terminer avec quelques astuces plus générales. Le but est de remplacer notre célèbre LAMP par des solutions moins gourmandes en mémoire.

Remplacer Apache

Pour remplacer Apache, nous avons le choix entre Nginx et Lighttpd (Lighty).

Lighttpd est le serveur le plus léger au monde. Sa structure est très simple, et l’impact d’une connexion en mémoire est très faible.

Nginx est un serveur qui consomme un peu plus au démarrage, mais qui utilise une gestion intelligente de ses connexions ouvertes pour optimiser son impact mémoire.

Inconvénients : oubliez les .htaccess, préparez-vous à faire un peu de configuration pour remplacer votre “a2enmod php5”, et changez vos habitudes pour redémarrer les services.

Remplacer MySQL

Ce n’est pas facile de remplacer MySQL, tellement le nombre de programme qui en dépendent est conséquent. Cependant, si c’est possible, n’hésitez pas une seconde, voici les deux alternatives “lowcost” possibles :

SQLite : une base SQLite est simplement un fichier qui est chargé en mémoire au moment où l’on veut lire dedans. Pas de démon résident, pas d’impact lorsque la base est inactive. Par contre, préparez vous à attendre un peu si votre base est conséquente : lire des données en n’utilisant que le cache disque, ça peut prendre du temps.

PostgreSQL : PostgreSQL est un peu le nouveau MySQL, mais en mieux. de faible impact mémoire au démarrage, doté de plus de fonctionnalités que MySQL et proposant de meilleures performances, il reste encore marginal mais a tout pour concurrencer les équivalents propriétaires comme Oracle. On peut l’utiliser dans un environnement limité, en configurant correctement les limites dans sa configuration.

Remplacer ProFTPd

Si vous proposez un hébergement, vous proposerez certainement un serveur FTP.

Si les utilisateurs du serveur sont des utilisateurs “de confiance” (ie : que vous connaissez), vous pouvez leur laisser utiliser le serveur SFTP fourni par le serveur SSH (à activer dans la configuration). De très faible impact mémoire, pour une meilleure sécurité et moins de configuration, le protocole SFTP est supporté par la majorité des clients FTP. C’est LA solution des flemmards pour mettre en place un serveur FTP.

Sinon, vous pouvez vous rabattre sur un VSFTPd, qui en plus de vous combler au niveau impact mémoire, vous comblera au niveau rapidité. Il gère aussi le FTPS.

Remplacer Postfix

Si vous n’utilisez pas la réception des mails et que vous ne faites que de l’envoi (avec ou sans smarthost), Exim vous comblera au moins autant que Postfix, vous proposant même sous Debian une interface de configuration “pour les nuls” : dpkg-reconfigure exim4-config.

Éviter la catastrophe

En cas de gonflement inattendu de la mémoire, et lorsque celle-ci commence à manquer, le système d’exploitation commence à sacrifier des programmes pour libérer de la RAM. C’est dommage, on aimerait éviter d’en arriver à de telles extrémités sur un serveur de production.

La solution est assez simple, bien qu’imparfaite : il suffit de rajouter de la SWAP. La SWAP est un fichier d’échange placé sur le disque dur, lorsque il n’y a plus de mémoire en RAM, une partie de la RAM est déchargée (ie copiée) sur la SWAP, pour laisser de la place aux programmes “actifs”. Bien entendu, cette mémoire déchargée devra être rechargée dès lors qu’un programme a besoin d’un bloc mémoire qui a été placé en SWAP, ralentissant le traitement à la vitesse des temps d’accès disque (pas cool).

Faire dormir tout le monde

Ce qui est pénible avec les serveurs, c’est la quantité de RAM qui est là, utilisée en permanence, pour finalement un seul pic d’utilisation. Si on pouvait récupérer la RAM des programmes qui n’interagissent pas avec l’extérieur, et ne les lancer que quand on en a besoin ? Inetd est fait pour ça !

Inetd c’est un “super démon” qui va écouter sur les ports demandés à la place des programmes, et qui va lancer les programmes pour traiter l’événement, dès lors qu’une connexion est ouverte sur un de ses ports d’écoute. On peut donc imaginer avoir un Lighttpd non lancé qui se fait réveiller lorsque Inetd reçoit une connexion sur le port 80. Lighttpd traite alors la requête et s’arrête immédiatement après, laissant la place pour d’autres programmes.

L’inconvénient ici vient des temps de traitement forcément plus longs qu’avec des démons résidents (démarrer le programme prend du temps).

Conclusion

Voilà, c’est à peu près tout, j’espère qu’avec ces petits changements dans votre configuration, vous aurez quelques pistes pour alléger votre serveur, si jamais l’expérience vous tente.