emails

Adapt bayesian filters for spam on server

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - 29 march 2016 -

When the detection of spam is activated through spamassassin on the server it would be gold to have the spamassassin daemon to learn what is spam and what is not.

One way to do that is to write a simple script that could be run manually or through a crontab entry :

#!/usr/bin/env bash
basedir=Maildir
ls -d $basedir/.* |  while read dir
do
        processedPart=$(basename "$dir")
        if [ "$processedPart" != "." -a "$processedPart" != ".." ]
        then
                echo -n Processing $processedPart ...
                if [ "$processedPart" == ".Spam" -o "$processedPart" == ".Pourriels" ]
                then
                        echo as SPAM
                        sa-learn --spam "$basedir/$processedPart/{cur,new}"
                else
                        echo as ham
                        sa-learn --ham "$basedir/$processedPart/{cur,new}"
                fi
        fi
done

Of course ".Spam" and ".Pourriels" ara my own choices for IMAP folders to contain spam. Update this script with your own preferences.
I have placed this script in a "spam" file in ~/bin to run it manually.

Filtering SPAM on server with postfix, dovecot, and sieve

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - 26 march 2014 -

Identify spams is one step to fight against them. The next step is to be able to move them around independently of the device you are using to view your inbox.

Filtering spams by configuring the email client is sometimes possible (and sometimes not) but it requires to implement filtering rules on every client. Be it evolution on your computer or some other email client on your smartphone.

The solution is to filter them on the server when they are locally dispatched to recipients by the local delivery agent. (LDA)

In my basic setup of my email server I had postfix that was used as a mail transfer agent (MTA) and LDA. It happens that sieve is a mail filtering solution that comes bundled with dovecot. The setup is then made to :

use dovecot as a local delivery agent
configure sieve to move spam emails to 'Junk' folder

Of course the pre-required step is that spamassassin has already been set up to identify what emails are spams and what emails are not.

Base configuration

Debian Wheezy : 7.4
postfix : 2.9.6
dovecot with sieve embedded : 2.1.7

Use dovecot as LDA

Modify /etc/postfix/main.cf file to change mailbox_command to dovecot deliver

mailbox_command=/usr/lib/dovecot/deliver

Configure dovecot to enable sieve

The first step is to configure where sieve should be reading the rules configuration files. This is done by modifying /etc/dovecot/conf.d/90-sieve.conf and adapting sieve_default location. The result is that the behavior will be the same for all users for which a local mail delivery is made. If your MTA is configured to redirect emails to external mailboxes then the spams emails won't be moved to a junk folder.

sieve_default = /etc/dovecot/default.sieve

Enable dovecot user to read the file :

chgrp dovecot /etc/dovecot/conf.d/90-sieve.conf

Next step is to enable sieve plugin for LDA (local delivery agent) in /etc/dovecot/conf.d/15-lda.conf declare sieve as a plugin :

mail_plugins = sieve

Now how can sieve move spams to 'Junk' folders ? This is done by configuring the /etc/dovecot/default.sieve file with this content :

require ["fileinto"];
# Move spam to Junk 
folderif header :contains "X-spam-flag" ["YES"] {
  fileinto "Spam";
  stop;
}

This file must be binary compiled by sieve compiler

cd /etc/dovecot
sievec default.sieve

A new file default.svbin is created and it must be readable by dovecot user

chgrp dovecot /etc/dovecot/default.svbin

There are some permission problems with /var/mail/<<user>> INBOX with dovecot on Debian. The email can be delivered directly to ~/Maildir with a configuration of /etc/dovecot/10-mail.conf.

#mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_location = maildir:~/Maildir

You can now restart dovecot and test a specially crafted spam email ;)

service dovecot restart
mail -s "Product for you" an-account-existing@your-server.tld

The following content was flagged as spam by spamassassin :

Online Drugstore can have your order of discounted Viagra shipped to you for only 5 minutes of your time!!! 
http://www.justgottago.com/od/azzbc/ 
No Prior Prescriptions Needed   
-Licensed U.S. Physicians are ready to fill your order   
-Guaranteed Lowest Prices Available   
-Discreet Mailing directly to your home or office 

Just visit http://www.justgottago.com/od/azzbc/ and enjoy the good life today!!!

Filtering spam emails with Spamassassin

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - 17 november 2012 -

Last time I explained how I setup my own email server . One of the possible improvement was to be able to filter spam on the server-side rather than relying on the client-side configuration. I configured this using spamassassin.The technical background is Debian Wheezy (testing version as it is not yet stable as for now).You will see in another article that we can go further by adding some filtering rules on the server as well ...But for now let's see how to set this up.

Installing spamassassin

Where talking about Debian here ... :

apt-get install spamassassin

One configuration step is to enable spamassassin in its configuration file /etc/default/spamassassin

# sed -i "s/ENABLED=0/ENABLED=1/g" /etc/default/spamassassin

And then the spamd service that must be launched with :

# service spamassassin start

You can check that the spamd daemon is listening to inputs on loopback address :

# netstat -ntpl | grep spamdtcp  
0  0 127.0.0.1:783  0.0.0.0:*       LISTEN      20892/spamd.pid

The version installed is :

# spamassassin --version
SpamAssassin version 3.3.2  running on Perl version 5.14.2

Filtering SMTP content through spamassassin

I'll have to configure a service for smtp in /etc/postfix/master.cf by adding a -o option :

smtp inet  n       -       -       -       -            smtpd
  -o content_filter=spamassassin
submission inet n       -       -       -       -       smtpd
   -o smtpd_tls_security_level=encrypt
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o content_filter=spamassassin
smtps     inet  n       -       -       -       -       smtpd
   -o smtpd_tls_wrappermode=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o content_filter=spamassassin

And configure what spamassassin stands for in /etc/postfix/master.cf by adding it at the end of the file :

##  SPAMASSASSIN
spamassassin unix -     n       n       -       -       pipe  user=debian-spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail \
                                                                                                  -oi -f ${sender} ${recipient}

NB : I edited the line to have it fit in the <pre> section but I guess that it is better if the line starting with "user=" and ending with "${recipient}" is a one-liner.debian-spamd is the user created by the apt-get install.

# getent passwd | grep debian-spamd
debian-spamd:x:112:116::/var/lib/spamassassin:/bin/sh

Result

You may find some interesting logs in /var/log/mail.log

Nov 17 19:30:44  postfix/pipe[32112]: 596F361DA4: to=<xxx@lebegue.org>, relay=spamassassin, delay=1.1, delays=0.76/0.02/0/0.29, dsn=2.0.0, status=sent (delivered via spamassassin service)
Nov 17 19:35:14  postfix/pipe[5098]: C8A6761D6F: to=<xxx@lebegue.org>, relay=spamassassin, delay=519, delays=518/0.01/0/0.63, dsn=2.0.0, status=sent (delivered via spamassassin service)

Updating filters

Once in a while, or through a cron entry, you can update filters with this command (man pages are well written) :

sa-update && service spamassassin reload

Learning behaviour

Look at man pages for sa-learn to improve spamassassin bayesian filters efficiency.

Self-hosting emails

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - 16 november 2012 -

Of the few services that where left in the hands of Google some were finally migrated to my own server :

E mails
Calendars
Contacts

I had in mind to host them for autonomy and privacy reasons (and also because I find it fun to tinker with this ...).

One of my goal was to be able to access emails, contacts and calendar through my home computer (running Debian) and my smartphone (running Android).

The background OS is Debian Wheezy (testing .. not yet a 7.0 stable) as of November 2012.

Hosting emails

Somehow it's a tricky job to implement your own email server !! There are many steps, many tools to integrate and many configurations to tweak. The functionalities that I wanted to provide are :

Store my emails on my own server
Being able to securely read them from my home computer and my smart-phone
Avoid being a spam relay to the nasty bots while alowing to post emails without using my ISP gateway
Forward emails to my familly when they want to receive their emails with my domain name : myrelatives@lebegue.org

Of course I wanted to be able to migrate all my emails from Google to this new infrastructure. Over the course of the years I had stored something like 1.7 Gb of email

In a more technical fashion I took these decisions :

Store my emails on my own server

Store my emails in an easily readable format without a database back-end. Maildir format is good for this. I can even read my emails with the simple mail command through an SSH connection ... postfix is my tool of the trade.

Incoming messages are delivered to /home/<<user>>/Maildir. in a plain raw format. Each <<user>> has its own file
Creating email folders to organize them is made by using linux user's home directory : /home/<<user>>/Maildir/<<subdirectories>>

Being able to securely read them from my home computer and my smart-phone

Reading from different devices requires that the messages are downloaded on said devices but must still be available on the server. Two protocols are available IMAP and POP3. I chose IMAP (mainly because it first transfers email header so that you can sort them out before actually downloading the whole message when you want to read it) and implemented it using dovecot.

Access to the mail boxes are secured with the unix user account's credentials
The authentication is made under TLS so that the credentials are encrypted before being sent over the network. I had to uncomment disable_plaintext_auth = yes to force TLS authentication in /etc/dovecot.conf.d/10-auth.conf

#### Authentication processes
### Disable LOGIN command and all other plaintext authentications unless
# SSL/TLS is used (LOGINDISABLED capability). Note that if the remote IP
# matches the local IP (ie. you're connecting from the same computer), the
# connection is considered secure and plaintext authentication is allowed.
disable_plaintext_auth = yes

I also added login authentication mode to /etc/dovecot.conf.d/10-auth.conf :

# Space separated list of wanted authentication mechanisms:
#   plain login digest-md5 cram-md5 ntlm rpa apop anonymous gssapi 
#   otp skey gss-spnego# NOTE: See also disable_plaintext_auth 
setting.auth_mechanisms = plain login

Avoid being a spam relay to the nasty bots while allowing to post emails without using my ISP gateway

This relies on :

Allowing any SMTP incoming emails that must be delivered to my domain. Delivering to something@lebegue.org is ok. Delivering to something@xxxx.xxx is forbidden.
Allowing any SMTP incoming emails to be forwarded to another domain ONLY IF the sender is authenticated.
Denying access when the message is received from an unqualified domain name server, unknown domain name server or of the sender's email address is not resolvable to an existing one. Examples :
- if the server is 84.56.12.41 instead of agoodserver.fromdomain.xx it is rejected.
- if the domain agoodserver.fromdomain.xx does not exists it is rejected
- if the sender's email is unknown it's rejected
I have changed this rule a bit for I could not receive or forward messages coming from addresses like noreply@xxxx.xx. It was the case for my ISP invoices notification that where rejected.
Denying access to well known spammers through RBL lists

Authentication is made by SMTP protocol and it relies on dovecot :in /etc/postfix/master.cf

smtp      inet  n       -       -       -       -       smtpd
#smtp      inet  n       -       -       -       1       postscreen
#smtpd     pass  -       -       -       -       -       smtpd
#dnsblog   unix  -       -       -       -       0       dnsblog
#tlsproxy  unix  -       -       -       -       0       tlsproxy
submission inet n       -       -       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       -       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

in /etc/postfix/main.cf

In bold are the modifications I made after realizing that some legitimate emails where rejected.

#smtpd_sender_restrictions = reject_unknown_sender_domain, 
# reject_unverified_sender
smtpd_recipient_restrictions = reject_invalid_hostname,
        reject_unknown_recipient_domain,
        reject_rbl_client sbl.spamhaus.org,
        permit_sasl_authenticated,
        permit_mynetworks,
        reject_unauth_destination,
        permit
smtpd_helo_restrictions = reject_invalid_helo_hostname,
        reject_non_fqdn_helo_hostname        
#reject_unknown_helo_hostname
smtpd_client_restrictions = reject_rbl_client dnsbl.sorbs.net,
         permit_sasl_authenticated

Delegating authentication to dovecot is made by configuring some other entries in /etc/postfix/main.cf

#Activate SASLsmtpd_sasl_auth_enable = yes
#Use Dovecotsmtpd_sasl_type = dovecotsmtpd_sasl_path = private/auth
# Add in header SASL authentication information.
smtpd_sasl_authenticated_header = yes

smtpd_sasl_path = private/auth means that postfix is going to rely on informations provided by dovecot in a file located in /var/spool/postfix/private/auth and configured in /etc/dovecot/conf.d/10-master.conf

service auth {  
# auth_socket_path points to this userdb socket by default. It's typically  
# used by dovecot-lda, doveadm, possibly imap process, etc. Users that have  
# full permissions to this socket are able to get a list of all usernames and  
# get the results of everyone's userdb lookups.  
#  
# The default 0666 mode allows anyone to connect to the socket, but the  
# userdb lookups will succeed only if the userdb returns an "uid" field that  
# matches the caller process's UID. Also if caller's uid or gid matches the  
# socket's uid or gid the lookup succeeds. Anything else causes a failure.  
#  
# To give the caller full permissions to lookup all users, set the mode to  
# something else than 0666 and Dovecot lets the kernel enforce the  
# permissions (e.g. 0777 allows everyone full permissions).  
unix_listener auth-userdb {
    mode = 0666
    user = postfix
    group = postfix
  }  
# Postfix smtp-auth  
# 3 following lines uncommented on 11/11/2012  
unix_listener /var/spool/postfix/private/auth 
{
    mode = 0666
  }

Forward emails to my familly when they want to receive their emails with my domain name : myrelatives@lebegue.org

This is configured through /etc/aliases file. If I want my sister's email to be delivered to her Gmail account : mysister@lebegue.org goes to sistergmailaccount@gmail.com then I configure this entry :

mysister: sistergmailaccount@gmail.com

This goes live only when using this command line

postalias /etc/aliases

Results and improvements

As for now I have configured Evolution email client and my Android phone with the default Android "E-mail" application.What I'd like to improve is filtering of incoming messages. There are still some spam coming in. I'd like to get rid of them on the server side to avoid to configure my Evolution client and my Android client with the same filters (though I haven't found yet how to writer filter rules on the default Android app ...)

Next step is also to move my calendars and contacts out of Google grasp ... Ok It's already done in fact ... but I'll explain it in an other blog entry ...

Rss feed of the tag

Bits and thoughts

#!/bin/bash is not rude