# uname -a - Mot-clé - spam

The war on SPAM: an review of the real world tools

Mathieu — Tue, 12 Sep 2017 02:54:00 +0200

Anti-spam techniques review: a few hints and tools review from my own experience.

Spam mesage are very common these days, but filtering them out is not as easy as it seems. The filtering techniques have evolved at the same rate than the spammers' evasive techniques, and the risk of filtering out a legitimate message is greater than ever.

It is also quite difficult to find good and up-to-date counter-measures list that anyone can implement.

Here is a summary of the anti-spam strategies I used / am still using. I hope it will help you understanding today's threats, and build your own solutions.

Definitions

MTA : Mail Transport Agent : this is the software that will actually do mail delivery. It is listening on port 25 and answers to SMTP commands. Some common MTAs : Postfix, Exim. More about MTA.
MX : MX records are DNS entries that are identifying the server responsible for mail delivery for the domain. More about MX records.
RBL : Realtime Blackhole List : a list of blacklisted IPs, that should be considered as spam sources. It is called Realtime because they are constantly updated. More about RBL.
RFC : Request For Comments, these are proposal for norms, some of them become norms. More about RFC.
IPS : Intrusion Protection System : these are "smart firewalls" that are blocking malicious requests, often based on behavioral rules. More about IPS.

The goals

My personal goals on SPAM war are pretty short:

Minimum false positive: having a spam is better than missing an important mail, try keeping the "permanent bashing" as low as possible
Hit harder on reoffending: coming-back spammers should be slapped harder
Internet neutrality: try not to encourage big mail farm, and let fair little providers doing their business

Available techniques

Blacklist / Blackhole Lists

Blacklists, Blackhole Lists (or RBLs) are the most ancient and most common measures for reducing spam. They are still pretty accurate, but:

It depends A LOT on the quality of the list, trashy lists are very common and they would end up sucking resources for no result, or worse, blocking legitimate emails
They are only accurate when updated often. I mean, very often (the R in RBL).
You should not use it directly on the MTA, these lists are an aggressive artefact of the past, where spam did not come from mail farms.
Very few are implementing IPv6, I agree that IPv6 spam is quite anecdotal, but it will probably change pretty soon (believe me)

Greylist

Greylisting is issuing a temporary REJECT code to force the foreign server to keep the message and send it back later. It aims at increasing the "cost per mail" for spam farms, as they cannot "hit and run" as fast as before.

The main culprits are that some providers are not implementing it well (hello Facebook) and so it needs an educated whitelist to work properly.

Some spammers are also re-sending the same mail several times in case of failure, looking like a legitimate mail server, and making Greylisting inefficient.

It is also hurting the fastness of the mail transmission, as the retry may occur several tens minutes after, slowing the mail delivery with little control on delays.

Also, some legitimate mail farms (hello OVH) are distributing their retry on several servers, making Greylisting inapplicable without fully whitelisting them.

RFC compliance

Spammers are often running special softwares for their crafted emails, tightening RFC compliance may be a good way to kick them out.

It can be as simple as forcing an HELO on SMTP protocol, or more tricky like checking the mail headers or enforcing a valid reverse.

In the majority of cases, it is very efficient, but:

Some home-made servers, especially Synology or Windows servers may be blocked while they are sending legitimate email. These servers are often ran by people who doesn't know or care on how to correctly setup a mail server. These buggy setups are more common than you think, and they are often legitimate senders, who have no clue of what is wrong, and are not willing to fix it (did I mentioned banking companies?).
In the vast majority of cases, IPv6-ready servers have no reverse on their IPv6 addresses and/or the IPv6 reverse is wrong.

SPF and DKIM

SPF and DKIM are anti-spoofing techniques. They does not guarantee that a mail is legitimate, but if the controls are showing an anomaly, it is very likely to be spam (or worse : scam or social engineering tentative).

The SPF technique is based on IP or sending domain whitelist: the sending email domain publishes a list of servers allowed to send email, along with a hint on what is expected if it does not pass (soft or hard reject).
DKIM is much more complex as the sending mail server has to cryptographically sign every message with a domain-specific key, which is then published in a special domain record.

SPF has been proven efficient at its beginnings, but today many spammers are just using stolen email accounts or custom domains that does not publish any SPF records, making it less pertinent.

DKIM is, like SPF, an anti spoofing technique, and spammers may be likely to sign mails from their custom domain if it becomes a necessity. Like SPF, a signed mail is not necessarily a clean mail. By the way, signature problems are pretty comon, and trashing an offending DKIM may not be the right behavior: Yahoo broke a lot of mailing lists when they enforced their DMARC policy.

To conclude, DKIM is relevant if you need to certify outgoing mails or if you are enforcing policies inside your company (to block spoofed email targeting your organisation) but it is definitely not an efficient anti-spam measure. And the recipient's servers may decide to simply ignore your painfully-configured DKIM headers.

Bayesian filters

Bayesian filters are frequency-based spam detection mechanisms. The idea is to sort out ham and spam for a short period of time, to let it "learn" what spam is made of, for efficient content-based detection. It has the benefit of being organisation-specific, as what is ham and what is spam may vary from one company to another (a company selling drugs may not be willing to filter out every message containing the word "pill").

But the learning process has to be taken seriously, and many end-users are just deleting spam instead of marking it for feeding the learning. By the way, the learning process needs IMAP folders to sort mails, and it will not work properly if all users are using POP mailboxes.

Spammers techniques

Or "the today's weapons of this war" .

Here is a short review of the spammers techniques I know, and some counter measures.

Address guessing

Some spammers are taking random web domains from their crawling, and then try to send their mails to commonly used addresses patterns. It can be webmaster@ ; ceo@ or whatever. These are easy to spot in log files, and a well configured MTA can take counter measures to lock out these guesses. Free (French ISP) is actually implementing this: postmaster.free.fr/index_en.html

Unfortunately, some legitimate email are sometimes sent to non existent addresses (typo, user deleted, etc) and legitimate MTA sometimes get blocked if the ban trigger is too low.

Moreover, nowadays' spammers are distributing their guesses trough zombie machines to stay under triggers, making it hard to spot.

Botnets

A lot of spammers are using zombies machines to send a high amount of mail in a short amount of time. Commercial ISP are taking the problem seriously, and many are blocking or filtering port 25 on their dynamic ranges, making impossible to have a custom mail server at home, but also preventing infected machines from sending direct-to-SMTP queries.

Direct to SMTP connections, ignoring MXes

Some spammers are just scanning IP ranges and directly talking to MTA, even if these MTAs are internal and pointed by no MX records. You may think that an authoritarian firewall is the neat solution, but it may be worth collecting these feisty IPs and feed them to an IPS, to protect the network from their guesses on the real MTAs.

Fake bounces and backsquatting

Sometimes you get an Undelivered Message notice (DSN) for a mail you never sent. This is probably backsquatting.
Backsquatting is sending email to a buggy address with a valid "from" address. An incorrectly configured mailserver would reply straight away to the from address to notify the failed delivery, instead of rejecting the mail and letting the foreign server doing the dirty job.

These configuration errors are pretty common, sometimes in defaut configurations or example configurations, but they can be easily avoided (look for documentation about backsquatting for your MTA, and test if your y server is vulnerable).

Real life advices

I am sorry, I don't have the magic wand to stop all Spam. A good spam fighting solution is always a combination of techniques, SpamAssassin for instance uses scoring from RBL as well as a bayesian filter and SPF checks.

I think that constant monitoring is important. Not just automated monitoring, but also clever log reading and mind openness on what can be a better solution for each problem. You never know what can happen in a spammer mind, and what works today may not work tomorrow. The Internet of Things is already a game changer.

I also advise you to be careful. Some decisions on our implementations may really hurt the Internet. Locking whole countries out is not without consequences, and the rise of IPv6 Internet has to be take into consideration from now.

Things are often not that pretty in mail servers, the temptation is great for a default blocking policy (hello MailInBlack). But as sysadmins, it is our responsability to not abuse and not hurting the smallest actors in the market (mails not coming from big farms). That may be a big word, but in my opinion, the freedom of the Internet also count on our neutrality on mail processing.

Thanks for reading.

C'est l'histoire d'un spam Partie 1: petit cours d'anatomie

Mathieu — Mon, 02 May 2016 15:48:00 +0200

Le SPAM par mail, c'est pénible. C'est du travail à temps plein si on veut s'en débarrasser en gênant le moins les utilisateurs. De la stricte application des RFC aux paramètres secrets de spamassassin, je vais essayer dans cette nouvelle série d'articles essayer de vous en faire mieux comprendre les techniques, et proposer à mon échelle les quelques solutions "qui marchent", du moins pour moi.

Voici donc pour cette première partie un cours d'autopsie. Le SPAM que nous décortiquer aujourd'hui est un mail est est resté coincé en queue sur le serveur mail. Bien que tous les mails en queue ne soient pas du SPAM, il est important de surveiller la taille de cette liste de mails "en transit" ; un grand nombre de mails dans cette liste indiquant souvent un problème d'envoi ou de réception.

Affichons donc la queue :

# mailq
1430281C33     8589 Thu Apr 28 14:51:58  MAILER-DAEMON
             (connect to serveurspamserveur.net.ae[86.96.xxx.xxx]:25: Connection refused)
                                         spammeur@serveurspamserveur.net.ae

Première constatation : l'adresse source ne correspond pas à une adresse gérée par le serveur, ce n'est donc pas un mail envoyé par un des utilisateurs du serveur. Mais alors comment ça se fait que je route des mails qui ne viennent pas de chez moi, et qui ne vont pas chez moi ? Mon serveur serait configuré en OpenRelay (ça serait inquiétant) ?

Examinons le contenu brut du mail à partir de son ID. La commande postcat permet de le faire pour Postfix, à adapter selon votre MTA.

# postcat -q 1430281C33

Ci-dessous la sortie de la commande postcat que j'ai colorisée :

*** ENVELOPE RECORDS deferred/1/1430281C33 ***
message_size:            8589             234               1               0            8589
message_arrival_time: Thu Apr 28 14:51:58 2016
create_time: Thu Apr 28 14:51:58 2016
named_attribute: log_message_origin=local
named_attribute: trace_flags=0
sender:
original_recipient: spammeur@spamserveur.net.ae
recipient: spammeur@spamserveur.net.ae
*** MESSAGE CONTENTS deferred/1/1430281C33 ***
Received: by monserveur.com (Postfix)
    id 1430281C33; Thu, 28 Apr 2016 14:51:58 +0200 (CEST)
Date: Thu, 28 Apr 2016 14:51:58 +0200 (CEST)
From: MAILER-DAEMON@monserveur.com (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: spammeur@spamserveur.net.ae
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="B25FA81C32.1461847918/monserveur.com"
Content-Transfer-Encoding: 8bit
Message-Id: <20160428125158.1430281C33@monserveur.com>

This is a MIME-encapsulated message.

--B25FA81C32.1461847918/monserveur.com
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host monserveur.com.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<monclient@gmail.com>: host
    gmail-smtp-in.l.google.com[2a00:1450:400c:xxxx::xxxx] said: 552-5.7.0 This
    message was blocked because its content presents a potential 552-5.7.0
    security issue. Please visit 552-5.7.0
    https://support.google.com/mail/answer/6590 to review our message 552 5.7.0
    content and attachment content guidelines. cw9si10769736wjb.20 - gsmtp (in
    reply to end of DATA command)

--B25FA81C32.1461847918/monserveur.com
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; monserveur.com
X-Postfix-Queue-ID: B25FA81C32
X-Postfix-Sender: rfc822; spammeur@spamserveur.net.ae
Arrival-Date: Thu, 28 Apr 2016 14:51:57 +0200 (CEST)

Final-Recipient: rfc822; monclient@gmail.com
Original-Recipient: rfc822;monclient@gmail.com
Action: failed
Status: 5.7.0
Remote-MTA: dns; gmail-smtp-in.l.google.com
Diagnostic-Code: smtp; 552-5.7.0 This message was blocked because its content
    presents a potential 552-5.7.0 security issue. Please visit 552-5.7.0
    https://support.google.com/mail/answer/6590 to review our message 552 5.7.0
    content and attachment content guidelines. cw9si10769736wjb.20 - gsmtp

--B25FA81C32.1461847918/monserveur.com
Content-Description: Undelivered Message
Content-Type: message/rfc822
Content-Transfer-Encoding: 8bit

Return-Path: <spammeur@spamserveur.net.ae>
Received: from localhost (localhost [127.0.0.1])
    by monserveur.com (Postfix) with ESMTP id B25FA81C32
    for <monclient@gmail.com>; Thu, 28 Apr 2016 14:51:57 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at monserveur.com
Received: from monserveur.com ([127.0.0.1])
    by localhost (monserveur.com [127.0.0.1]) (amavisd-new, port 10024)
    with ESMTP id vn_xZRLJ3EKq for <monclient@gmail.com>;
    Thu, 28 Apr 2016 14:51:51 +0200 (CEST)
Received: by monserveur.com (Postfix, from userid 5000)
    id BE51181C31; Thu, 28 Apr 2016 14:51:51 +0200 (CEST)
X-Sieve: Pigeonhole Sieve 0.4.2
X-Sieve-Redirected-From: contact@monclient.com
Delivered-To: contact@monclient.com
Received: from localhost (localhost [127.0.0.1])
    by monserveur.com (Postfix) with ESMTP id 95E2581C33
    for <contact@monclient.com>; Thu, 28 Apr 2016 14:51:51 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at monserveur.com
Received: from monserveur.com ([127.0.0.1])
    by localhost (monserveur.com [127.0.0.1]) (amavisd-new, port 10024)
    with ESMTP id 8agkFLPzF_vy for <contact@monclient.com>;
    Thu, 28 Apr 2016 14:51:44 +0200 (CEST)
Received: from bba148580.serveurspam.net.ae (bba148580.serveurspam.net.ae [217.165.xxx.xxx])
    by monserveur.com (Postfix) with ESMTP id 774C481C27
    for <contact@monclient.com>; Thu, 28 Apr 2016 14:51:44 +0200 (CEST)
Received: from www.serveurspam.net.ae ([127.0.0.1]) by AMAZONA-F00CA66 with Microsoft SMTPSVC(7.5.7601.17514);
     Thu, 28 Apr 2016 16:51:42 +0400
Date: Thu, 28 Apr 2016 16:51:42 +0400
To: "contact@monclient.com" <contact@monclient.com>
From: xxxx xxxx <spammeur@spamserveur.net.ae>
Subject: [SPAM] FW: Invoice
Message-ID: <be0fa65c2e77bf81d3fd1b0216e8e742@www.serveurspam.net.ae>
X-Priority: 3
X-Mailer: PHPMailer [version 1.73]
MIME-Version: 1.0
Content-Type: multipart/mixed;
    boundary="b1_be0fa65c2e77bf81d3fd1b0216e8e742"
X-OriginalArrivalTime: 28 Apr 2016 16:51:42.0538 (UTC) FILETIME=[40C0DFC0:01D1A068]

--b1_be0fa65c2e77bf81d3fd1b0216e8e742
Content-Type: multipart/alternative;
    boundary="b2_be0fa65c2e77bf81d3fd1b0216e8e742"

--b2_be0fa65c2e77bf81d3fd1b0216e8e742
Content-Type: text/plain; charset = "iso-8859-1"
Content-Transfer-Encoding: 8bit

Please find attached invoice #726990

Have a nice day

Rowena Pittman
Executive Director Sales Account Management Training Performance Support

--b2_be0fa65c2e77bf81d3fd1b0216e8e742
Content-Type: text/html; charset = "iso-8859-1"
Content-Transfer-Encoding: 8bit

<html>
<body>
<p>Please find attached invoice #726990<br><br><br>Have a nice day<br><br><br>Rowena Pittman<br>
Executive Director Sales Account Management Training Performance Support</p>
</body></html>

--b2_be0fa65c2e77bf81d3fd1b0216e8e742--
--b1_be0fa65c2e77bf81d3fd1b0216e8e742
Content-Type: application/zip; name="B1095_scan-invoice_0B0C3F.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="B1095_scan-invoice_0B0C3F.zip"

UEsDBBQAAAAIAIRenEiR5N/wYgcAAHQTAAAKAAAAMTk0MGMwOS5qc9VYaW/TMBj+DBL/IUQC
ElYGLacoCNJr7Wh6LKVrixBKHCdxDid17oz9d+ys45IzBN+YtmWLX7/v4+c93Uwnwgha4xSe

...

--b1_be0fa65c2e77bf81d3fd1b0216e8e742--

--B25FA81C32.1461847918/monserveur.com--
*** HEADER EXTRACTED deferred/1/1430281C33 ***
named_attribute: encoding=8bit
*** MESSAGE FILE END deferred/1/1430281C33 ***

Les lignes que j'ai mises en gris ne sont pas très intéressantes : il s'agit de l'enveloppe et des informations fournies par le système qui gère la queue, intéressons-nous plutôt au reste.

Les lignes en bleu sont les en-têtes, ce sont elles qui retracent à la fois le format du message, mais aussi leur trajet à travers les différents serveurs mails qui l'ont manipulé. Comme une liste de tampons sur un passeport. Ce sont en particulier les lignes commençant par Received, qui tracent le trajet des mails à travers les échanges entre les serveurs. La première ligne Received correspondant au dernier échange réalisé, et la dernière ligne Received au premier échange réalisé.

Dans l'en-tête on retrouve aussi l'ID du message, 1430281C33, et son parcours entre les seveurs mail. Chaque serveur par lequel le mail passe va rajouter une ligne, la première ligne correspondant au dernier serveur ayant reçu le message.

Les lignes en rouge sont ce que l'on appelle les boundaries (et leurs déclarations). Ce sont des lignes de séparation permettant de dissocier les pièces jointes du reste du message. Ici on en a plusieurs, parce qu'en fait à chaque échange, le mail s'est fait attacher à un autre mail. Ce n'est pas courant, à moins que le mail n'ai été enrichi d'informations supplémentaire après un "rejet" du serveur de destination. Ici, il s'est donc probablement fait "jeter" plusieurs fois !

Commençons notre analyse par la fin du message. Le mail original contenait visiblement lui aussi une pièce jointe, et à en juger par le texte invitant à ouvrir la pièce jointe, il est fort possible que ça soit un SCAM. Un message vous invitant à ouvrir un faux fichier PDF qui va infecter l'ordinateur. Heureusement donc que ce mail n'est pas arrivé jusqu'à sa destination.

Remontons un peu, on constate que le mail a été envoyé avec PHPMailer, et qu'il a été marqué en [SPAM] dans le sujet par mon filtre antispam. La première trace de l'arrivée du mail est la suivante :

Received: from bba148580.serveurspam.net.ae (bba148580.serveurspam.net.ae [217.165.xxx.xxx]) by monserveur.com (Postfix) with ESMTP id 774C481C27 for <contact@monclient.com>; Thu, 28 Apr 2016 14:51:44 +0200 (CEST)
Received: from www.serveurspam.net.ae ([127.0.0.1]) by AMAZONA-F00CA66 with Microsoft SMTPSVC(7.5.7601.17514); Thu, 28 Apr 2016 16:51:42 +0400

La ligne au-dessus, c'est la trace du passage à l'antispam et à l'antivirus :

Received: from localhost (localhost [127.0.0.1]) by monserveur.com (Postfix) with ESMTP id 95E2581C33 for <contact@monclient.com>; Thu, 28 Apr 2016 14:51:51 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at monserveur.com
Received: from monserveur.com ([127.0.0.1]) by localhost (monserveur.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8agkFLPzF_vy for <contact@monclient.com>; Thu, 28 Apr 2016 14:51:44 +0200 (CEST)

Rien de spécial, l'antivirus et l'antispam sont installés en local et ont fait leur job dès que le message a été reçu. Continuons.

X-Sieve: Pigeonhole Sieve 0.4.2
X-Sieve-Redirected-From: contact@monclient.com

Ça c'est le mécanisme qui gère les alias de mon serveur. Sieve va rediriger l'adresse contact@monclient.com vers la bonne adresse de destination, l'adresse contact@monclient.com étant un alias.

Received: from monserveur.com ([127.0.0.1]) by localhost (monserveur.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vn_xZRLJ3EKq for <monclient@gmail.com>; Thu, 28 Apr 2016 14:51:51 +0200 (CEST)
Received: by monserveur.com (Postfix, from userid 5000) id BE51181C31; Thu, 28 Apr 2016 14:51:51 +0200 (CEST)

On transfère tout ça au MTA local pour envoi à la destination finale :

Received: from localhost (localhost [127.0.0.1]) by monserveur.com (Postfix) with ESMTP id B25FA81C32 for <monclient@gmail.com>; Thu, 28 Apr 2016 14:51:57 +0200 (CEST)

Fin du premier message, on traverse la boundary pour voir ce qu'il s'est passé ensuite :

<monclient@gmail.com>: host gmail-smtp-in.l.google.com[2a00:1450:400c:xxxx::xxxx] said: 552-5.7.0 This message was blocked because its content presents a potential 552-5.7.0 security issue.

Ça c'est moins cool. En gros, gmail a jeté le message, parce que la pièce jointe était sans doute contaminée. Mon antivirus n'a rien vu, il va falloir chercher pourquoi (plus tard).

On continue de remonter, et au-dessus de la dernière boundary, on a :

Received: by monserveur.com (Postfix) id 1430281C33; Thu, 28 Apr 2016 14:51:58 +0200 (CEST)

Ok, gmail m'a renvoyé le message plus ou moins gentiement en m'expliquant qu'il n'en voulait pas, et mon serveur essaie de le transférer au spammeur original qui nous a fermé sa porte. Le mail reste donc en queue jusqu'à son expiration... Voilà pourquoi il traîne dans la queue des mails et qu'il n'a été envoyé ni par moi ni pour moi...

Conclusion

J'espère que cette courte plongée dans les en-têtes d'un mail vous a permis de vous familiariser avec le "vrai" contenu d'un mail.

Ici je ne peux plus faire grand chose avec ce mail à part le supprimer de la queue mail, le spammeur ayant certainement déjà été bloqué par son hébergeur ou changé d'adresse IP pour continuer à envoyer sans se faire prendre par les blacklists.

À bientôt pour un nouvel article.

Set-up SQL quarantine with Amavisd-new and ISPConfig

Mathieu — Sun, 06 Dec 2015 16:56:00 +0100

It's documented, but it took me two days to do it correctly, so here is how to reconfigure an ISPConfig installation of Amavis to store quarantined mail in SQL database, in order to install a quarantine viewer like Mailzu.

1. Prerequisites

A working Postfix+Amavis stack with ISPConfig
A working SQL (PostgreSQL, MySQL...) database
Optional : a working mail server with PHP (for Mailzu)

Jump to Installation if you know what you are doing.

2. Off-subject generic explanations

2.1 Amavis and ISPConfig policies

ISPConfig policies, Tag-Levels

In a default ISPConfig installation per-user ISPConfig policies are loaded. The configuration file for Amavis, written by ISPConfig contains :

@lookup_sql_dsn =
   ( ['DBI:mysql:database=dbispconfig;host=127.0.0.1;port=3306', 'ispconfig', 'xxxx'] );
$sql_select_policy =
   'SELECT *,spamfilter_users.id'.
   ' FROM spamfilter_users LEFT JOIN spamfilter_policy ON spamfilter_users.policy_id=spamfilter_policy.id'.
   ' WHERE spamfilter_users.email IN (%k) ORDER BY spamfilter_users.priority DESC';
$sql_select_white_black_list = 'SELECT wb FROM spamfilter_wblist'.
    ' WHERE (spamfilter_wblist.rid=?) AND (spamfilter_wblist.email IN (%k))' .
    ' ORDER BY spamfilter_wblist.priority DESC';

It means that whatever you would set as $sa_spam_subject_tag, $sa_tag_level_deflt, $sa_tag2_level_deflt, $sa_kill_level_deflt, $sa_dsn_cutoff_level, it will be overridden by per-user policies.

The ISPConfig policies can be changed in tab Email => Spamfilter => Policy in ISPConfig panel. If you struggle wondering why your message keeps getting smashed at level 4.5, look at the sa_tag_level in policies. We will have to change values in that policies, to make the SQL quarantine working.

2.2 Lookup DSN and Storage DSN

DSN (Data Source Name) are the connection strings with host, username, and password, used to connect to databases.

Amavis can set two DSN : one for Policies lookup (used to retrieve ISPConfig policies from Panel), and one for storage of mail meta informations and quarantine. We will use the Storage DSN to set up a secondary database for quarantine storage, to not mess with existing ISPConfig database.

2.3 Levels and cutoffs

Amavis uses Spamassassin to score the mail, in order to decide what to do with it. The category of test (spam test, antivirus, etc) and the score along with levels determines the actions Amavis will trigger, and the final destiny where the mail belongs.

Spamassassin levels are :

tag_level : a message above that score will be tagged with X-Spam-Status, X-Spam-Score and X-Spam-Level headers.
tag2_level : a message above that score will be marked as X-Spam-Status: Yes and the subject is changed if sa_spam_modifies_subj is set to true.
kill_level : a message above that score is taken to the final_spam_destiny, and quarantined, it will not be delivered unless D_PASS is set to final_spam_destiny.
dsn_cutoff_level : a message above that level will never trigger a bounce or a reject, whatever spam_destiny is.
quarantine_cutoff_level : a message above that level will not be quarantined.

2.4 Final destinations

Once the message is categorized by Amavis tests (through SpamAssassin, ClamAV, etc), Amavis decides if it should be delivered to user mailbox or not, and if a bounce will be issued.

This is the purpose of $final_virus_destiny, $final_spam_destiny, $final_banned_destiny, $final_bad_header_destiny.

They can take the following values :

D_PASS : mail will be delivered to inbox.
D_BOUNCE : mail will not be delivered, and a delivery status notification will be returned by Postifx to sender (except if the score exceeds the dsn_cutoff level)
D_REJECT : Postfix will answer REJECT to the distant mail server, and the distant mail server may produce a delivery status notification to the user
D_DISCARD : forgive and forget : the mail will not be delivered and the sender is not informed. The mail may be quarantined if the quarantine_cutoff level is not exceeded.

3. Installation

For Amavis : Nothing ! Amavis comes out-of-the-box with SQL storage.
For Mailzu : see Mailzu section.

4. Configuration

4.1 Database

Create an user and a database for quarantine storage :

# mysql -u root -p
mysql> CREATE DATABASE amavis_storage;
mysql> CREATE USER 'amavis_storage'@'localhost' IDENTIFIED BY 'xxxx';
mysql> GRANT ALL PRIVILEGES ON amavis_storage.* TO 'amavis_storage'@'localhost';
mysql> FLUSH PRIVILEGES;

Load the initial schema from Amavis docs (usually located in /usr/share/doc/amavisd-new/ ).

Delete unnecessary tables, as we will be using this database only for mail storage and not for lookups :

# mysql -u amavis_storage -p amavis_storage
mysql> DROP TABLE users;
mysql> DROP TABLE mailaddr;
mysql> DROP TABLE policy;
mysql> DROP TABLE wblist;

Nota Bene : while executing DROP TABLE users, don't be silly, and do not remove mysql users database.

4.2 Amavis

Update your Amavis configuration /etc/amavis/conf.d/50_user :

@storage_sql_dsn = ( ['DBI:mysql:database=amavis_storage;host=127.0.0.1;port=3306', 'amavis_storage', 'xxxx'] );  # none, same, or separate database

# Quarantine SPAM into SQL server.
$spam_quarantine_to = 'spam-quarantine';
$spam_quarantine_method = 'sql:';

# Quarantine VIRUS into SQL server.
$virus_quarantine_to = 'virus-quarantine';
$virus_quarantine_method = 'sql:';

# Quarantine BANNED message into SQL server.
$banned_quarantine_to = 'banned-quarantine';
$banned_files_quarantine_method = 'sql:';

# Quarantine Bad Header message into SQL server.
$bad_header_quarantine_method = 'sql:';
$bad_header_quarantine_to = 'badheader-quarantine';

# Do not store non-quarantined messages info
# You can set it to 1 (the default) to test if Amavis is filling correctly the tables maddr, msgs, and msgcrpt
$sql_store_info_for_all_msgs = 0;

#
# SQL Select statements
#

$sql_select_policy =
   'SELECT *,spamfilter_users.id'.
   ' FROM spamfilter_users LEFT JOIN spamfilter_policy ON spamfilter_users.policy_id=spamfilter_policy.id'.
   ' WHERE spamfilter_users.email IN (%k) ORDER BY spamfilter_users.priority DESC';

$sql_select_white_black_list = 'SELECT wb FROM spamfilter_wblist'.
    ' WHERE (spamfilter_wblist.rid=?) AND (spamfilter_wblist.email IN (%k))' .
    ' ORDER BY spamfilter_wblist.priority DESC';

#
# Quarantine settings
#

$final_virus_destiny = D_BOUNCE;
$final_spam_destiny = D_DISCARD;
$final_banned_destiny = D_BOUNCE;
$final_bad_header_destiny = D_PASS;

# Default settings, we st this very high to not filter aut emails accidently
$sa_spam_subject_tag = '[SPAM] ';
$sa_tag_level_deflt  = 20.0;  # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 60.0; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 60.0; # triggers spam evasive actions
$sa_dsn_cutoff_level = 100;   # spam level beyond which a DSN is not sent
#$sa_debug = 1;

#
# Disable spam and virus notifications for the admin user.
# Can be overridden by the policies in mysql
#

$virus_admin = undef;
$spam_admin = undef;

#
# Enable Logging
#

$DO_SYSLOG = 1;
$LOGFILE = "/var/log/amavis.log";  # (defaults to empty, no log)

# Set the log_level to 5 for debugging
$log_level = 0;                # (defaults to 0)

4.3 ISPConfig policies

Remember that ISPconfig policies are overriding a lot of our configuration in 50_user. In order to majke the quarantine work, you have to reconfigure all the available policies in ISPConfig Panel.

Look at your policies list, you have to change the quarantine settings for every policies :

ISPConfig Mail Spamfilter Policy

When editing a policy, on the Quarantine tab, set the destinations :

ISPConfig Mail Spamfilter Policy Quarantine destinations

If you do not fill something in these fields, Amavis will not store quarantined mails in SQL database, and will just discard it !

These fields correspond to the virus_quarantine_to, spam_quarantine_to, banned_quarantine_to, bad_header_quarantine_to variables in Amavis configuration, and an empty value is overriding those we set in Amavis configuration.

4.4 Test

Send some spam to your server, check if the tables are populated :

mysql> SELECT * FROM maddr;

Check if meta informations are populated :

mysql> SELECT * FROM msgs;
mysql> SELECT * FROM msgrcpt;

And if quarantine is filling :

mysql> SELECT * FROM quarantine;

5. Cleanup !

You should not "setup and forget" your quarantine SQL storage. Messages has to be deleted periodically, otherwise your database will grow forever. Look at the documentaion in /usr/share/docs/amavisd-new to make a cronjob like this :

#!/bin/bash

SQL_HOST="localhost";
SQL_LOGIN="amavis_storage"
SQL_PASSWORD="xxxx"
SQL_DB="amavis_storage"

mysql --user="$SQL_LOGIN" --password="$SQL_PASSWORD" --host="$SQL_HOST" $SQL_DB -e " \
  DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 30*24*3600; \
  DELETE FROM msgrcpt WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id); \
  DELETE FROM quarantine WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id); \
  DELETE FROM maddr WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id) AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id); \
"

6. Mailzu

I have to admit, Mailzu seems a bit obsolete as I had to patch to make it working with Amavis 3.3 tables. But it still works pretty well for a simple task like reading and releasing quarantine mails.

6.1 Installation

Download the source files at http://sourceforge.net/projects/mailzu/.

62. Patch

The existing Mailzu source code is quite old, and the schema of Amavis SQL tables changed. Download and apply this patch in to make Mailzu work.

6.3 Configuration

I suppose that you know how to spawn PHP with CGI to serve the Mailzu files.

Configure your database login and password in config/config.php :

$conf['db']['dbType'] = 'mysql';
$conf['db']['dbUser'] = 'amavis_storage';
$conf['db']['dbPass'] = 'xxxx';
$conf['db']['dbName'] = 'amavis_storage';
$conf['db']['hostSpec'] = 'localhost:3306';

I am using IMAP login to authenticate in Mailzu. Unfortunately, I had to turn off SSL authentication, as it wasn't working. Here is my configuration :

$conf['auth']['serverType'] = 'imap';
$conf['auth']['imap_hosts'] = array( 'localhost:143' );
$conf['auth']['imap_type'] = 'imaptls';
$conf['auth']['imap_domain_name'] = 'example.com';

Don't forget to set yourself "super" :

$conf['auth']['s_admins'] = array ('me@example.com');

And to set your web uri :

$conf['app']['weburi'] = 'https://example.com/mailzu';

6.4 Configure in-app release

Mailzu can also release quarantined mail. I did not implement this function, but you have to set up the amavisd-release internface on an inet socket on port 9998, instead of the existing unix socket located at /var/lib/amavis/amavisd.sock . Read more.

References

Pourquoi mon commentaire n'est pas publié ?

Mathieu — Fri, 18 Jul 2014 22:56:00 +0200

Pour mieux lutter contre le SPAM, j'ai modifié il y a quelques semaines le captcha à remplir pour publier un commentaire ou un rétrolien (commentaire sous forme d'article de blog). C'est à présent une question à laquelle il faut répondre courgette (sans guillemets ni majuscules).

Pour moi, cela a pour conséquence une réduction drastique du nombre de SPAM qui passaient à côté du précédent système (composé principalement d'un filtre bayésien). Mais pour vous, cela signifie que beaucoup plus de commentaires légitimes peuvent se retrouver bloqués, puisque les filtres sont mis en série. Je réalise des tests pour essayer de réduire le nombre de filtres, peut être ne garder que le principal, qui filtre la très grande majorité des commentaires.

Mais le but de cet article ce n'est pas de faire de l'autopromotion sur mon système antispam génial. Je me suis rendu compte, ayant à présent plus de temps pour trier les "bon" commentaires, que beaucoup de gens se trompent sur l'objectif du commentaire.

Si je laisse des commentaires ouvert, ce n'est pas pour que le commentaire que vous écrivez dirige les visiteurs vers votre site ou en fasse une quelconque promotion. Je sélectionne moi-même les blogs ou personnes que je trouve intéressant(e)s, et je les publie dans la barre à droite du blog. Tout commentaire n'apportant rien de concret au fond de l'article et contenant un lien vers un site externe, je ne peux pas le publier, car c'est pour moi une forme de publicité trop facile (vous remarquerez que la publicité est absente de ce blog depuis plusieurs années).

Votre avis et vos encouragements, même s'ils me font plaisir, je ne peux pas les publier s'ils contreviennent à la règle. Ne m'en voulez pas, je sais que vous avez pris du temps pour l'écrire et ça me va droit au cœur, mais comme je me refuse à modifier les commentaires, je ne peux pas non plus retirer le lien présent pour le publier quand même. Publiez-le sans lien, et il est possible que votre commentaire passe.

Il y a bien entendu une exception. Si vous publiez un commentaire contenant un lien, et que ce lien est le point de départ d'un enrichissement de l'article à partir du commentaire qui en est fait, alors il y a de très fortes chances qu'il soit publié. Mais beaucoup de gens confondent cet espace d'expression avec un tableau blanc sur lequel on peut faire sa propre promotion.

Je dois avouer qu'il y a quelques mois, j'ai pensé fermer les commentaires. Mais comme beaucoup de gens n'ont pas d'autres moyens de réagir à mes articles (je ne suis pas présent les réseaux sociaux publics), j'ai laissé les commentaires ouverts et installé le nouveau captcha. Il faut plutôt voir ça comme une chance, les commentaires publiés sont les plus intéressants, il n'y a aucun contenu parasite sur ce blog.

Une dernière chose à savoir : les commentaires sont modéré à priori. Votre commentaire, s'il passe les filtres, arrive dans une liste où je choisis lesquels je publie, ce n'est pas automatique. Soyez donc patient si vous avez publié un commentaire intéressant, ou alors utilisez le formulaire de contact du blog pour m'en informer.

Nobody loves me. But spammers does !

Mathieu — Tue, 25 Jun 2013 22:54:00 +0200

Billet d’humour et d’humeur, ne pas comprendre prouvera que vous êtes normal.

Definitely consider that that you said. Your favourite reason seemed to be on the net the easiest factor to keep in mind of. I say to you, I certainly get annoyed even as people consider worries that they just do not realize about. You managed to hit the nail upon the highest as neatly as outlined out the entire thing without having side effect , other folks could take a signal. Will likely be back to get more. Thank you

Thanks , I’ve just been searching for info approximately this topic for a long time and yours is the greatest I’ve came upon so far. However, what in regards to the conclusion? Are you sure in regards to the source?|What i do not realize is if truth be told how you are no longer really a lot more neatly-liked than you might be now. You’re so intelligent.

You can absolutely visit interest inside function you’re writing. Your segment desires of more ardent freelancers just like you who aren’t frightened to mention the way they feel. At all times focus on a person’s soul.

You could definitely see your skills within the paintings you write. The world hopes for more passionate writers such as you who are not afraid to mention how they believe. All the time go after your heart.

Woah the following blog site is fantastic i favor examining the articles you write. Maintain the great function! You are aware of, many people are seeking circular with this information and facts, you may help them drastically.

Woah this weblog can be magnificent i enjoy learning the articles you write. Be the great! You recognize, plenty of people need about with this information and facts, you may assist them to enormously.

Bon allez, stop, point trop n’en faut, j’en ai des centaines comme ça.