date: 08/21/2008

This HowTo describes the setup of a Gentoo based spam and virus filtering mailgateway. This server is meant to run in front of the servers keeping the mail accounts i.e. Zarafa, Lotus Notes or Microsoft Exchange.
The MTA is postfix which will listen on port 25 for inbound mail. Recieved mails are forwarded to amavisd-new on port 10024. Amavisd-new filters the mail for virii and spam before passing it back to Postfix on port 10025 for final delivery.
In this Setup ClamAV is used for virusscans, if a virus is found the mail is discarded or optional quarantined, if no virus is found the mail first runs through dspam and afterwards through Spamassassin for antispam. dspam gets better detection rates than SpamAssassin's Bayes filter, it writes it's headers which will then be scored by SpamAssassin together with some other tests.

It should be similar for other distributions, except packages and config file paths, in Debian amavisd-new config is splitted for example.

Installing the programs needed

We will start with installing the most important programs.

# emerge postfix amavisd-new spamassassin clamav dspam

Configuring Postfix

First we have to tell postfix which content_filter to use with it's smtp process. Also we ensure that it will only listen for local connections on port 10025. To accomplish this we have to add the following at the end of /etc/postfix/master.cf

# nano -w /etc/postfix/master.cf
smtp inet n - n - - smtpd
  -o content_filter=amavisfeed:[127.0.0.1]:10024
  -o receive_override_options=no_address_mappings

amavisfeed unix - - n - 4 smtp
  -o smtp_data_done_timeout=1200
  -o smtp_send_xforward_command=yes
  -o disable_dns_lookups=yes
  -o max_use=20

127.0.0.1:10025 inet n - n - - smtpd
  -o content_filter=
  -o smtpd_delay_reject=no
  -o smtpd_client_restrictions=permit_mynetworks,reject
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o smtpd_data_restrictions=reject_unauth_pipelining
  -o smtpd_end_of_data_restrictions=
  -o smtpd_restriction_classes=
  -o mynetworks=127.0.0.0/8
  -o smtpd_error_sleep_time=0
  -o smtpd_soft_error_limit=1001
  -o smtpd_hard_error_limit=1000
  -o smtpd_client_connection_count_limit=0
  -o smtpd_client_connection_rate_limit=0
  -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
  -o local_header_rewrite_clients=

The file master.cf tells the postfix master program how to run each individual postfix process. More info with Postfix manual master(8).

If your first attempts to send mail result in messages bouncing, you've likely made a configuration error somewhere. Try temporarily enabling soft_bounce while you work out your configuration issues. This prevents postfix from bouncing mails on delivery errors by treating them as temporary errors. It keeps mails in the mail queue until soft_bounce is disabled or removed.

# postconf -e "soft_bounce = yes"
# /etc/init.d/postfix reload

Once you've finished creating a working configuration, be sure to disable or remove soft_bounce and reload postfix.

Configuring Amavisd-new

Amavisd-new is used to handle all the filtering and allows you to easily glue together severel different technologies. Upon reception of a mail message it will extract the mail, filter it through some custom filters, handle white and black listing, filter the mail through various virus scanners and finally it will filter the mail using dspam and SpamAssassin.

Amavisd-new itself has a number of extra features:

• it identifies dangerous file attachments and has policies to handle them per-user,
• per-domain and system-wide policies for:
  • whitelists
  • blacklists
  • spam score thresholds
  • virus and spam policies

Apart from postfix and freshclam we will run all applications as the user amavis.

Edit the following lines in /etc/amavisd.conf for your needs.

# nano -w /etc/amavisd.conf
$mydomain = 'domain.tld';
$myhostname = 'mx.domain.tld';
$forward_method = 'smtp:[127.0.0.1]:10025'; # where to forward checked mail
$notify_method = $forward_method;
$max_servers = 4; # number of pre-forked children
$max_requests = 20;
$inet_socket_port = [10024];
$inet_socket_bind = '127.0.0.1';
@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
%final_destiny_by_ccat = (
  CC_VIRUS, D_DISCARD,
  CC_BANNED, D_PASS,
  CC_UNCHECKED, D_PASS,
  CC_SPAM, D_DISCARD,
  CC_BADH, D_PASS,
  CC_OVERSIZED, D_PASS,
  CC_CLEAN, D_PASS,
  CC_CATCHALL, D_PASS,
);
$virus_admin = "virusalert\@$mydomain";
$virus_quarantine_method = 'local:virus/%m';
$bad_header_quarantine_method = 'local:badh/%m';
$banned_files_quarantine_method = 'local:banned/%m';
$spam_quarantine_method = 'local:spam/%m';
@bypass_spam_checks_maps = ( [qw( spam@mx.domain.tld nospam@mx.domain.tld noscan.doamin.tld )] );
@whitelist_sender_maps = ( read_hash("$MYHOME/whitelist") );
@blacklist_sender_maps = ( read_hash("$MYHOME/blacklist") );
$dspam = 'dspam';
$sa_tag_level_deflt = -5.17;
$sa_tag2_level_deflt = 5.81;
$sa_tag3_level_deflt = 9.81;
$sa_kill_level_deflt = 12.53;
@spam_subject_tag2_maps = ('[possible-spam: _SCORE_ (_REQD_)] ');
@spam_subject_tag3_maps = ('[spam: _SCORE_ (_REQD_)] ');
$sa_spam_modifies_subj = 1;
@virus_name_to_spam_score_maps = ();

Configuring ClamAV

As virus scanner we use ClamAV as it has a fine detection rate comparable with commercial offerings, it is very fast and it is Open Source Software. We love log files, so make clamd log using syslog and turn on verbose logging. Also do not run clamd as root. Now edit /etc/clamd.conf

# nano -w /etc/clamd.conf
LogSyslog
LogVerbose
LogFacility LOG_MAIL
LocalSocket /var/run/clamav/clamd.sock

ClamAV comes with the freshclam deamon dedicated to periodical checks of virus signature updates. We will make freshclam update virus signatures every 4 hours and send email notifications about update status.

# nano -w /etc/freshclam.conf
LogSyslog
LogVerbose
DatabaseOwner clamav
Checks 6
DatabaseMirror db.XY.clamav.net
OnUpdateExecute echo "ClamAV Database updated successfully at `date +%T` on `date +%D`." | mail -s "Freshclam success" root
OnErrorExecute echo "ClamAV Database failed at `date +%T` on `date +%D`. See logfiles for details." | mail -s "Freshclam Error " root
OnOutdatedExecute echo "ClamAV Version outdated. Please update immediately." | mail -s "Freshclam failed" root

Start clamd with freshclam using the init scripts by modifying /etc/conf.d/clamd.

# nano -w /etc/conf.d/clamav
START_CLAMD=yes
START_FRESHCLAM=yes
CLAMD_NICELEVEL=3
FRESHCLAM_NICELEVEL=19

At last modify amavisd.conf with the new location of the socket.

# nano -w /etc/amavisd.conf
['ClamAV-clamd',
  \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
  qr/\bOK$/, qr/\bFOUND$/,
  qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],

Configuring Spamassassin

Amavis is using the Spamassassin Perl libraries directly so there is no need to start the service. Also this creates some confusion about the configuration as some Spamassassin settings are configured in /etc/mail/spamassassin/local.cf and overridden by options in /etc/amavisd.conf.

# nano -w /etc/spamassassin/local.cf
use_bayes 1
bayes_path /var/amavis/.spamassassin/bayes
skip_rbl_checks 0
ok_languages de en
ok_locales en
bayes_auto_learn 1
use_pyzor 1
use_razor2 1
dns_available yes
bayes_ignore_header X-Bogosity
bayes_ignore_header X-Spam-Flag
bayes_ignore_header X-Spam-Status

Alternatively it is possible to store the bayesian filter data in a SQL database. For activating this feature it is necessary to configure spamassassin to use a different bayes storage module and to configure the database which should be used. This can be done in local.cf

The database schema for storage of the bayesian filter data contains several different tables. To install the tables using the included example, use the following command:

mysql -p databasename < bayes_mysql.sql
Enter password:

Once you have created the database and added the tables, just add the required lines to your global configuration file (local.cf)

# nano -w /etc/spamassassin/local.cf
bayes_store_module Mail::SpamAssassin::BayesStore::SQL
bayes_sql_dsn DBI:mysql:spamassassin:localhost:3306
bayes_sql_username username
bayes_sql_password password
bayes_sql_override_username amavis

Configuring dspam

dspam's classification isn't used by amavisd-new directly, but before the mail is running through spamassassin dspam writes it's Headers. To make spamassassin use dspam's Headers this perl module by Eric Lubow is needed for spamassassin and has to be copied to /usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin/Plugin/. The configuration file can be copied to /etc/mail/spamassassin/, within this directory all *.cf files are used for spamassassin.

White/Blacklisting

Once mail starts passing through this mail gateway you will probably discover that the above setup is not perfect. Maybe some of your customers like to receive mails that others wouldn't. You can whitelist/blacklist envelope senders quite easily. Uncomment/add the following line in amavisd.conf.

@whitelist_sender_maps = ( read_hash("$MYHOME/whitelist") );

Optional but not really necessary, since blocking unwanted domains is better done at the MTA level, you can add blacklisting the same way

@blacklist_sender_maps = ( read_hash("$MYHOME/blacklist") );

You can add the following to amavisd.conf to deliver mails to postmaster and abuse mailboxes above quarantine level. Otherwise add them to bypass_spam_checks_maps as shown below with the training addresses.

@spam_lovers_maps = ( ["postmaster\@$mydomain", "abuse\@$mydomain"] );

For bypassing the training addresses from spamanalysis when retraining false negatives and false positives add the following.

@bypass_spam_checks_maps = ( [qw( spam@mx.domain.tld notspam@mx.domain.tld )] );

Ruleset

One part of SpamAssassins classification of emails is based on static rules. Here is a simple script which can be added to /etc/cron.weekly for example, to keep the ruleset up to date.

Updating ruleset with sa-update

#!/bin/sh
VERSION=`grep -m 1 -E 'VERSION = ' /usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin.pm | cut -f2 -d\"`
rm /var/lib/spamassassin/current
ln -s /var/lib/spamassassin/${VERSION} /var/lib/spamassassin/current
sa-update --channel updates.spamassassin.org &&
/etc/init.d/amavisd restart &>/dev/null

Adding more rules

If you want to use more rules provided by the SARE Ninjas at the SpamAssassin Rules Emporium or by OpenProtect you can easily add and update them using the sa-update mechanism included in Spamassassin.

A brief guide to using SARE rulesets with sa-update can be found here {.external}.

Testing the setup

Now before you start freshclam you can manually verify that it works.

# freshclam
ClamAV update process started at Sat Aug 23 01:02:53 2008
main.cvd is up to date (version: 47, sigs: 312304, f-level: 31, builder: sven)
Downloading daily-8076.cdiff [100%]
daily.cld updated (version: 8076, sigs: 89342, f-level: 33, builder: arnaud)
Database updated (401646 signatures) from database.clamav.net (IP: 62.201.161.84)
Clamd successfully notified about the update.

Now you have updated virus definitions and you know that freshclam.conf is working properly start clamd and amavis with the following commands

# /etc/init.d/clamd start
# /etc/init.d/amavisd start
# /etc/init.d/postfix reload

If everything went well postfix should now be listening for mails on port 25 and for reinjected mails on port 10025. amavisd-new should be listening on port 10024 for mails to be scanned. To verify this check your log file.

# tail /var/log/mail.log

Now if no strange messages appear in the log file it is time for a new test. Use telnet or netcat to manually connect to amavisd on port 10024 and postfix on port 10025 and/or use netstat to check processes for listening ports

# nc localhost 10024
220 [127.0.0.1] ESMTP amavisd-new service ready

# nc localhost 10025
220 domain.tld ESMTP Postfix

You can also send mails through telnet for debugging purposes

# telnet localhost 25
220 domain.tld ESMTP Postfix
EHLO localhost
250-mail.domain.tld
250-PIPELINING
250-SIZE 81920000
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM:<user@domain.tld>
250 2.1.0 Ok
RCPT TO:<user@domain.tld>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
.
250 2.0.0 Ok: queued as E52071B5DF
QUIT
221 2.0.0 Bye
Connection closed by foreign host.

Add amavisd and clamd to the default runlevel.

# rc-update add clamd default
# rc-update add amavisd default

Greylisting Policy Server

When a request for delivery of a mail is received by Postfix via SMTP, the triplet CLIENT_IP / SENDER / RECIPIENT is built. If it is the first time that this triplet is seen, or if the triplet was first seen, less than 5 minutes ago, then the mail gets rejected with a temporary error. Hopefully spammers or viruses will not try again later, as it is however required per RFC. To implement greylisting

# emerge postgrey

In this example we're using unix sockets, so edit the following in /etc/conf.d/postgrey. Adjust the other settings like delay and text for your needs.

# nano /etc/conf.d/postgrey
POSTGREY_TYPE="unix"

...and start the service

# /etc/init.d/postgrey start

To make postfix use postgrey for greylisting features add the following to the end of the recipient_restrictions in postfix's main.cf

# nano /etc/postfix/main.cf
smtpd_recipient_restrictions =
  check_policy_service unix:private/postgrey

Throttling

Policy Daemon

Policyd is an anti-spam plugin for Postfix that does Greylisting, Sender-(envelope, SASL or host/ip)-based throttling (on messages and/or volume per defined time unit), Recipient rate limiting, Spamtrap monitoring / blacklisting, HELO auto blacklisting and HELO randomization preventation. To add policyd for it's throttling features

# echo "mail-filter/policyd ~x86" >>/etc/portage/package.keywords
# emerge mysql policyd

Set up the database

# cp /usr/share/doc/policyd-1.82/DATABASE.mysql.bz2
# bunzip2 DATABASE.mysql.bz2
# mysql -u root -p < DATABASE.mysql

Create a database user

# mysql -u root -p
GRANT ALL ON policyd.* TO 'policyd'@'localhost' IDENTIFIED BY 'password';

To make postfix use policyd add the following to the recipient_restrictions in postfix's main.cf

# nano /etc/postfix/main.cf
smtpd_recipient_restrictions =
  check_policy_service inet:127.0.0.1:10031

Edit the database owner and password in /etc/policyd.conf as setup before, then edit the other settings for your needs, if not using postgrey a greylisting feature can be enabled in policyd also.

Handling false positives and negatives

Training the filter

For feeding the Bayes database with false positives and false negatives, we'll use suitable email addresses, they will be forwarded to different scripts, one for SpamAssassin's Bayes DB and the other one for dspam.

The first script is sa-wrapper.pl by Alexandre Jousset {.external}.
The second script is dspam-retrain.pl by Neale Pickett {.external}.
Just copy these script to your /usr/local/bin or another path of your liking.

Create the training addresses spam@mail.domain.tld for training false negatives
and
nospam@mail.domain.tld for training false positives

We'll add the necessary recipient addresses for the spam/ham training in /etc/mail/aliases

spam: spam@spam.spam,spam@dspam-retrain.spam
notspam: ham@ham.ham,ham@dspam-retrain.ham

After editing the alias table do not forget to postmap it or run newaliases

Next postfix's transport table is edited and the following is added.
/etc/postfix/transport

spam.spam sa-spam:
ham.ham sa-ham:
dspam-retrain.spam dspam-retrain:spam
dspam-retrain.ham dspam-retrain:innocent

The necessary processes have to be configured in /etc/postfix/master.cf.

sa-spam unix - n n - 1 pipe
  user=amavis:amavis argv=/usr/local/bin/sa-wrapper.pl spam ${sender}

sa-ham unix - n n - 1 pipe
  user=amavis:amavis argv=/usr/local/bin/sa-wrapper.pl ham ${sender}

dspam-retrain unix - n n - 1 pipe
  flags=Ru user=amavis argv=/usr/local/bin/dspam-retrain.pl $nexthop $sender $recipient

Finally we will create a restriction class in postfix's main.cf and define it, which will later be used in check_recipient_access. This will only allow authenticated clients to send to our training addresses. The restriction class can also be configured with permit_mynetworks for example or an other of postfix's restriction classes.

smtpd_restriction_classes = authenticated_only
  authenticated_only = permit_sasl_authenticated, reject

Then add the recipient check to the restrictions

smtpd_recipient_restrictions =
  check_recipient_access hash:/etc/postfix/protected_dest

protected_dest contains the following

spam@domain.tld authenticated_only
notspam@domain.tld authenticated_only

Troubleshooting

Amavisd-new

To troubleshoot Amavisd-new start out by stopping it with /etc/init.d/amavisd stop and then start it manually in the foreground with amavisd debug and watch it for anomalies in the output.

Spamassassin

To troubleshoot Spamassassin you can filter an email through it with spamassassin -D < mail. To ensure that the headers are intact you can move it from another machine with IMAP. sa-learn and sa-update can be called with the -D option too.