Bits and thoughts

#!/bin/bash is not rude


Filtering SPAM on server with postfix, dovecot, and sieve

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

Identify spams is one step to fight against them. The next step is to be able to move them around independently of the device you are using to view your inbox.

Filtering spams by configuring the email client is sometimes possible (and sometimes not) but it requires to implement filtering rules on every client. Be it evolution on your computer or some other email client on your smartphone.

The solution is to filter them on the server when they are locally dispatched to recipients by the local delivery agent. (LDA)

In my basic setup of my email server I had postfix that was used as a mail transfer agent (MTA) and LDA. It happens that sieve is a mail filtering solution that comes bundled with dovecot. The setup is then made to :

  • use dovecot as a local delivery agent
  • configure sieve to move spam emails to 'Junk' folder

Of course the pre-required step is that spamassassin has already been set up to identify what emails are spams and what emails are not.

Base configuration

  • Debian Wheezy : 7.4
  • postfix : 2.9.6
  • dovecot with sieve embedded : 2.1.7

Use dovecot as LDA

Modify /etc/postfix/ file to change mailbox_command to dovecot deliver


Configure dovecot to enable sieve

The first step is to configure where sieve should be reading the rules configuration files. This is done by modifying /etc/dovecot/conf.d/90-sieve.conf and adapting sieve_default location. The result is that the behavior will be the same for all users for which a local mail delivery is made. If your MTA is configured to redirect emails to external mailboxes then the spams emails won't be moved to a junk folder.

sieve_default = /etc/dovecot/default.sieve

Enable dovecot user to read the file :

chgrp dovecot /etc/dovecot/conf.d/90-sieve.conf

Next step is to enable sieve plugin for LDA (local delivery agent) in /etc/dovecot/conf.d/15-lda.conf declare sieve as a plugin :

mail_plugins = sieve

Now how can sieve move spams to 'Junk' folders ? This is done by configuring the /etc/dovecot/default.sieve file with this content :

require ["fileinto"];
# Move spam to Junk
folderif header :contains "X-spam-flag" ["YES"] {
  fileinto "Spam";

This file must be binary compiled by sieve compiler

cd /etc/dovecot
sievec default.sieve

A new file default.svbin is created and it must be readable by dovecot user

chgrp dovecot /etc/dovecot/default.svbin

There are some permission problems with /var/mail/<<user>> INBOX with dovecot on Debian. The email can be delivered directly to ~/Maildir with a configuration of /etc/dovecot/10-mail.conf. 

#mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_location = maildir:~/Maildir

You can now restart dovecot and test a specially crafted spam email ;)

service dovecot restart
mail -s "Product for you" an-account-existing@your-server.tld

The following content was flagged as spam by spamassassin :

Online Drugstore can have your order of discounted Viagra shipped to you for only 5 minutes of your time!!!
No Prior Prescriptions Needed  
-Licensed U.S. Physicians are ready to fill your order  
-Guaranteed Lowest Prices Available  
-Discreet Mailing directly to your home or office

Just visit and enjoy the good life today!!!

Have a watch on your logs

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

Want to know what happened on your server ? Logwatch is for you.It sums up and aggregates some information from your log files and you can have it sent by email to you.

Context and installation

The server I'm running is a Debian Wheezy.
apt-get install logwatch
I wanted it to send an email to a specific local user and I wanted the email to contains as much detail as possible. I wrote to etc/logwatch/conf/override.conf
# cat >/etc/logwatch/conf/override.conf <<EOF
logwatch: MailTo =
logwatch: Output = mail
logwatch: Detail = high

The result is an email sent with every important details.

Filtering spam emails with Spamassassin

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

Last time I explained how I setup my own email server . One of the possible improvement was to be able to filter spam on the server-side rather than relying on the client-side configuration. I configured this using spamassassin.The technical background is Debian Wheezy (testing version as it is not yet stable as for now).You will see in another article that we can go further by adding some filtering rules on the server as well ...But for now let's see how to set this up.

Installing spamassassin

Where talking about Debian here ... :
apt-get install spamassassin
One configuration step is to enable spamassassin in its configuration file /etc/default/spamassassin
# sed -i "s/ENABLED=0/ENABLED=1/g" /etc/default/spamassassin
And then the spamd service that must be launched with :
# service spamassassin start
You can check that the spamd daemon is listening to inputs on loopback address :
# netstat -ntpl | grep spamdtcp  
0  0*       LISTEN      20892/
The version installed is :
# spamassassin --version
SpamAssassin version 3.3.2  running on Perl version 5.14.2

Filtering SMTP content through spamassassin

I'll have to configure a service for smtp in /etc/postfix/ by adding a -o option :
smtp inet  n       -       -       -       -            smtpd
  -o content_filter=spamassassin
submission inet n       -       -       -       -       smtpd
   -o smtpd_tls_security_level=encrypt
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o content_filter=spamassassin
smtps     inet  n       -       -       -       -       smtpd
   -o smtpd_tls_wrappermode=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o content_filter=spamassassin
And configure what spamassassin stands for in /etc/postfix/ by adding it at the end of the file :
spamassassin unix -     n       n       -       -       pipe  user=debian-spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail \
-oi -f ${sender} ${recipient}
NB : I edited the line to have it fit in the <pre> section but I guess that it is better if the line starting with "user=" and ending with "${recipient}" is a one-liner.debian-spamd is the user created by the apt-get install.
# getent passwd | grep debian-spamd


You may find some interesting logs in /var/log/mail.log
Nov 17 19:30:44  postfix/pipe[32112]: 596F361DA4: to=<>, relay=spamassassin, delay=1.1, delays=0.76/0.02/0/0.29, dsn=2.0.0, status=sent (delivered via spamassassin service)
Nov 17 19:35:14  postfix/pipe[5098]: C8A6761D6F: to=<>, relay=spamassassin, delay=519, delays=518/0.01/0/0.63, dsn=2.0.0, status=sent (delivered via spamassassin service)

Updating filters

Once in a while, or through a cron entry, you can update filters with this command  (man pages are well written) :
sa-update && service spamassassin reload

Learning behaviour

Look at man pages for sa-learn to improve spamassassin bayesian filters efficiency.

Self-hosting emails

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

Of the few services that where left in the hands of Google some were finally migrated to my own server :  
  • E mails
  • Calendars
  • Contacts
I had in mind to host them for autonomy and privacy reasons (and also because I find it fun to tinker with this ...).

One of my goal was to be able to access emails, contacts and calendar through my home computer (running Debian) and my smartphone (running Android).

The background OS is Debian Wheezy (testing .. not yet a 7.0 stable) as of November 2012.

Hosting emails

Somehow it's a tricky job to implement your own email server !! There are many steps, many tools to integrate and many configurations to tweak. The functionalities that I wanted to provide are :
  • Store my emails on my own server
  • Being able to securely read them from my home computer and my smart-phone
  • Avoid being a spam relay to the nasty bots while alowing to post emails without using my ISP gateway
  • Forward emails to my familly when they want to receive their emails with my domain name :
Of course I wanted to be able to migrate all my emails from Google to this new infrastructure. Over the course of the years I had stored something like 1.7 Gb of email

In a more technical fashion I took these decisions :

Store my emails on my own server

Store my emails in an easily readable format without a database back-end. Maildir format is good for this. I can even read my emails with the simple mail command through an SSH connection ... postfix is my tool of the trade.
  • Incoming messages are delivered to /home/<<user>>/Maildir. in a plain raw format. Each <<user>> has its own file
  • Creating email folders to organize them is made by using linux user's home directory : /home/<<user>>/Maildir/<<subdirectories>>

Being able to securely read them from my home computer and my smart-phone

Reading from different devices requires that the messages are downloaded on said devices but must still be available on the server. Two protocols are available IMAP and POP3. I chose IMAP (mainly because it first transfers email header so that you can sort them out before actually downloading the whole message when you want to read it)  and implemented it using dovecot.
  • Access to the mail boxes are secured with the unix user account's credentials
  • The authentication is made under TLS so that the credentials are encrypted before being sent over the network. I had to uncomment disable_plaintext_auth = yes to force TLS authentication in /etc/dovecot.conf.d/10-auth.conf
#### Authentication processes
### Disable LOGIN command and all other plaintext authentications unless
# SSL/TLS is used (LOGINDISABLED capability). Note that if the remote IP
# matches the local IP (ie. you're connecting from the same computer), the
# connection is considered secure and plaintext authentication is allowed.
disable_plaintext_auth = yes
I also added login authentication mode to /etc/dovecot.conf.d/10-auth.conf :
# Space separated list of wanted authentication mechanisms:
#   plain login digest-md5 cram-md5 ntlm rpa apop anonymous gssapi
#   otp skey gss-spnego# NOTE: See also disable_plaintext_auth
setting.auth_mechanisms = plain login

Avoid being a spam relay to the nasty bots while allowing to post emails without using my ISP gateway

This relies on :
  • Allowing any SMTP incoming emails that must be delivered to my domain. Delivering to is ok. Delivering to is forbidden.
  • Allowing any SMTP incoming emails to be forwarded to another domain ONLY IF the sender is authenticated.
  • Denying access when the message is received from an unqualified domain name server, unknown domain name server or of the sender's email address is not resolvable to an existing one. Examples :
    • if the server is instead of agoodserver.fromdomain.xx it is rejected.
    • if the domain agoodserver.fromdomain.xx does not exists it is rejected
    • if the sender's email is unknown it's rejected
    I have changed this rule a bit for I could not receive or forward messages coming from addresses like noreply@xxxx.xx. It was the case for my ISP invoices notification that where rejected.
  • Denying access to well known spammers through RBL lists
Authentication is made by SMTP protocol and it relies on dovecot :in /etc/postfix/
smtp      inet  n       -       -       -       -       smtpd
#smtp inet n - - - 1 postscreen
#smtpd pass - - - - - smtpd
#dnsblog unix - - - - 0 dnsblog
#tlsproxy unix - - - - 0 tlsproxy
submission inet n - - - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
smtps inet n - - - - smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject

in /etc/postfix/

In bold are the modifications I made after realizing that some legitimate emails where rejected.
#smtpd_sender_restrictions = reject_unknown_sender_domain, 
# reject_unverified_sender
smtpd_recipient_restrictions = reject_invalid_hostname,
smtpd_helo_restrictions = reject_invalid_helo_hostname,
smtpd_client_restrictions = reject_rbl_client,
Delegating authentication to dovecot is made by configuring some other entries in /etc/postfix/
#Activate SASLsmtpd_sasl_auth_enable = yes
#Use Dovecotsmtpd_sasl_type = dovecotsmtpd_sasl_path = private/auth
# Add in header SASL authentication information.
smtpd_sasl_authenticated_header = yes
smtpd_sasl_path = private/auth means that postfix is going to rely on informations provided by dovecot in a file located in /var/spool/postfix/private/auth and configured in /etc/dovecot/conf.d/10-master.conf
service auth {  
# auth_socket_path points to this userdb socket by default. It's typically 
# used by dovecot-lda, doveadm, possibly imap process, etc. Users that have 
# full permissions to this socket are able to get a list of all usernames and 
# get the results of everyone's userdb lookups. 

# The default 0666 mode allows anyone to connect to the socket, but the 
# userdb lookups will succeed only if the userdb returns an "uid" field that 
# matches the caller process's UID. Also if caller's uid or gid matches the 
# socket's uid or gid the lookup succeeds. Anything else causes a failure. 

# To give the caller full permissions to lookup all users, set the mode to 
# something else than 0666 and Dovecot lets the kernel enforce the 
# permissions (e.g. 0777 allows everyone full permissions). 
unix_listener auth-userdb {
    mode = 0666
    user = postfix
    group = postfix
# Postfix smtp-auth 
# 3 following lines uncommented on 11/11/2012 
unix_listener /var/spool/postfix/private/auth
    mode = 0666

Forward emails to my familly when they want to receive their emails with my domain name :

This is configured through /etc/aliases file. If I want my sister's email to be delivered to her Gmail account : goes to then I configure this entry :
This goes live only when using this command line
postalias /etc/aliases

Results and improvements

As for now I have configured Evolution email client and my Android phone with the default Android "E-mail" application.What I'd like to improve is filtering of incoming messages. There are still some spam coming in. I'd like to get rid of them on the server side to avoid to configure my Evolution client and my Android client with the same filters (though I haven't found yet how to writer filter rules on the default Android app ...)

Next step is also to move my calendars and contacts out of Google grasp ... Ok It's already done in fact ... but I'll explain it in an other blog entry ...