Bits and thoughts

#!/bin/bash is not rude

bits

Changing GNUSocial log files

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

GNUSocial uses syslog to create logs during its execution. As there are no specific indications these logs all go to /var/log/syslog.

Mixing sub-systems

This is also where some other sub systems are writing their own logs.
awk '{print $5}' /var/log/syslog \
| sed -e 's/[0-9].//g' \
| tr -d '[]' \
| sort -u
gives this result :
kernel:
named:
postfix/anvil:
postfix/cleanup:
postfix/local:
postfix/master:
postfix/pickup:
postfix/qmgr:
postfix/smtp:
postfix/smtpd:
rsyslogd:statusnet:
/USR/SBIN/CRON:

Configure GNUSocial to use its own log file

I wanted to have GNUSocial logs separated from the other subsystems. I followed the instructions of the documentation and I modified config.php to add two lines (I did that when it was still status.net)
#logs$config['site']['logdebug'] = false;
$config['site']['logfile'] = '/var/log/statusnet/statusnet.log';
At the first attempt to write a log it failed ... nothing was logged  in /var/log/statusnet/statusnet.log . Looks like a user permissions problem.... (www-data is the group of www-data user who runs Apache)
chown -R root:www-data /var/log/statusnet
chmod g+w /var/log/statusnet/statusnet.log

Limit the growth

Now that GNUSocial logs should be on their own, how can I limit their disk usage over time ? On a linux system the answer is by logrotating them. So I wrote a logrotated configuration file for these logs in /etc/logrotate.d/statusnet
/var/log/statusnet/*.log
{
  rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        create 660 root www-data
}
This a read like this :
        
  • /var/log/statusnet/*.log : consider every file named *.log in /var/log/statusnet directory
  • rotate 7,  daily,  missingok,  notifempty,  delaycompress,  compress : every day create a new log file and keep the seven latest log files. If nothing was written in the log file than do noting. You can also ignore the log rotation if there is no file at all. And, by the way, compress it so that it takes even less room.
  •     
  • create 660 root www-data : every new log file should be created so that users belonging to www-data group and root itself can read and write the file

Strange behaviour

Using the aforementioned configuration of config.php I expect not to see any more debug logs and I expected to see every log in /var/log/statusnet/statusnet.log ... Well that's not what is happening : 
  • /var/log/sysout still contains GNUSocial logs but much less than before. I can read daemon messages and every messages stating that a PHP call is "Including config file: /var/www/statusnet/config.php"
  • LOG_DEBUG messages are in /var/log/statusnet/statusnet.log
Discussing with @jpope I configured LogFilter plugin
addPlugin('LogFilter', array( 'priority' => array(LOG_ERR => true,    LOG_INFO => true,    LOG_DEBUG => false),    'regex' => array('/About to push/i' => false,      '/Including config file/i' => false,      '/Successfully handled item/i' => false)      ));
But these line are still printed out. A quick search indicates that ./lib/statusnet.php is the culprit
# find . -name "*.php" -exec grep -H "Including config file" '{}' \;./config.php:      '/Including config file/i' => false,./lib/statusnet.php:    common_log(LOG_INFO, "Including config file: " . $_config_file);
My next step is to change this message priority to LOG_DEBUG by editing ./lib/statusnet.php ... to no avail for the message is still present ...

Ocam's razor is still sharp :
replace
common_log(LOG_INFO, "Including config file: " . $_config_file);
by
//common_log(LOG_INFO, "Including config file: " . $_config_file);
in ./lib/statusnet.php

Et voilà ...

Self-Hosting on Raspberry Pi

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

I recently bought a RaspberryPi v2.

My goal is to selfhost some of my services :
  • email
  • GNUSocial instance
  • XMPP server
  • This blog !!

I had to migrate all these services from my previously OVH hosted site. I won't bother you much with the details but it took me 2 days and a half to migrate it all.

I started with my email server because I like to start with difficult things. And then my GNUSocial instance. The XMPP Server was a piece of cake (If you retrieve what was parameterized in /var/lib/prosody too ...).

As for now it runs smoothly and very very silently.

The largest memory hog is mysql of course !

Fail2ban Analysis

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -


Having restarted my fail2ban daemon on 2014-12-17 I was curious about the time distribution of bans and their original location.I crafted a small bash one-liner that gives me raw data : 
for ipadd in $(zgrep Ban fail2ban.log.* | awk '{print $NF}' | sort -u); \
do \
zgrep $ipadd fail2ban.log.* | awk -F':' '{print $2 $NF}' ; \
done | awk '{print $1" " $NF}' | \
grep -v '2014-12-16\|2014-12-15\|2014-12-14\|2014-12-13\|2014-12-12\|2014-12-11\|2014-12-10\|2014-12-09\|2014-12-08\|2014-12-07' \
| sort | while read data; \
do \
ipaddr2=$(echo $data | awk '{print $2}'); \
country=$(whois $ipaddr2 | egrep -i "^country:"| awk '{print $NF}' | sort -u); \
echo $data $country; \
done
It represents a total of 270 bans in almost three weeks.The ip adresses for which there is no country code is because the whois returned a Korean UTF-8 content that I couldn't parse. But they are all Koreans (I checked them manually).

IP address bans by day China comes first with 97 bans. It is followed by the United states of america (50 bans) and then Germany (30 bans).

Debian Jessie Multimedia

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

Debian Jessie (Debian 8) is, by default, provided with many multimedia capabilities.

Playing any audio files comes natively with the platform. Watching a video recorded from your phone is easy once the video has been downloaded to the computer where Jessie is installed.

Some DVDs are directly playable and some of them require libdvcss which is not provided on debian repositories because of legal stuff but it can be downloaded as a debian package from videolan.

Flash video playing is not pre-installed. There is a way to install Flash but more and more websites use HTML5 video tag to provide video content. This makes Flash quit useless. But anyway it can still be installed if you add "non-free" repository to your package repository configuration. There are plenty of sites to show you how to do that ...

There are some websites that propose videos on .mp4 format. This working very well with a default install in Iceweasel. The solution is simply to install ffmpeg plugin for gstreamer :
sudo apt install gstreamer0.10-ffmpeg
I don't have a blue-ray drive so I can't tell how well blue-ray is working on Debian Jessie ...

Perhaps there are some other tricks for an advanced usage of multimedia that I'm not aware of !

Calendar and contacts server

Written by ⓘⓓⓔⓝⓣⓛⓤⓓ - -

This post dating from 2012 is still relevant in September 2014 !I'm still on my trajectory to lower my Google dependency . As for now I have already hosted my own emails and I have tried to reduce as much as possible any incoming spam . Calendars and Contacts are the next step.

Standards

Calendars and Contacts are respectively based upon two standards   
Both are derived from WebDAV protocol. There are many products that implement these standards. I chose Baïkal for some reasons :   
  • It looked easy to setup
  • It brought what I needed and not the full range of groupware solution
  • It added a lightweight administration layer to manage users and their Calendars or Address Books.

Baïkal setup

Setting up the web application

I downloaded Baikal and un-tarred it in its own directory : /var/www/baikal. and I made sure that it was owned by default Apache user www-data
cd /tmp
wget http://baikal.codr.fr/get/baikal-regular-0.2.3.tgz
cd /var/www
tar -xzvf /tmp/baikal-regular-0.2.3.tgz
mv baikal-regular/ baikal
chown -R www-data: baikal/
I edited .htaccess file that was alread present in /var/www/baikal/html to add "Allow from all" directive because my Apache configuration is locked by default on every directory.I  created an Apache virtual host to reach Baïkal pointing to /var/www/baikal/html accessible with https through /etc/apache2/sites-available/baikal.lebegue.org-ssl
<VirtualHost *:443>
        ServerAlias baikal.lebegue.org
        DocumentRoot /var/www/baikal/html
        <Directory "/var/www/baikal/html">
                Options None
                Options +FollowSymlinks
                AllowOverride All
        </Directory>
        SSLEngine On
        SSLCertificateFile    /etc/ssl/certs/certificate.pem
        SSLCertificateKeyFile /etc/ssl/private/priv_key.pem
 </VirtualHost>
Any http access will be redirected to https through /etc/apache2/sites-available/baikal.lebegue.org
<VirtualHost *:80>
        ServerAlias baikal.lebegue.org
        <IfModule mod_rewrite.c>
        RewriteEngine On
        RewriteCond %{HTTPS} off
        RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
        </IfModule>
</VirtualHost>
And then I had to enable these configurations in Apache :
a2ensite baikal.lebegue.org baikal.lebegue.org-ssl
It never hurts to check if Apache is happy with this configuration :
apache2ctl configtest
And then an Apache reload
service apache2 reload

Setting up MySql database

By default Baikal uses a SQLite database but it can run on MySql ... and I happen to have a MySql dabase running ...
# mysql -p
mysql> create database baikal;
mysql> grant all on baikal.* to 'baikaluser'@'localhost' identified by 'password';
mysql> \q#
mysql -u baikaluser --database=baikal -p
mysql> @/var/www/baikal/Core/Resources/Db/MySQL/db.sql
mysql> show tables;
+------------------+
| Tables_in_baikal |
+------------------+
| addressbooks     |
| calendarobjects  |
| calendars        |
| cards            |
| groupmembers     |
| locks            |
| principals       |
| users            |
+------------------+
I'm good to go ...

Web application setup

Open a browser and navigate to https://yourserver.xxx/admin/ . The first step is to set up an admin user account ...

And then you need to setup the database connectivity :

Users setup

The final step is to declare users through the administration interface :

Client configuration

Evolution

My desktop client is Evolution here is how I set it up, Calendar:



and Contacts :

Android 4.x

On Android I used one application for caldav and carddav :
        
  • DAVDroid   Open source calendar and contact synchronisation adapter available on independently managed application repository : F-droid
Independently signed certificates such as the one I use (certified by CACert.org) are not recognized by Android. In order for DAVDroid to be able to connect to my https Baïkal server I had to import CACert.org root certificate into the phone :
        
  • On the phone download the CACert.org root certificate
  •     
  • It should propose you to switch the settings of the phone in order to add the certificate to the user's approved container.
  •     
  • At this step you MUST have set up an unlock method for the phone (draw a schema, type a password or a PIN) ... this will be enforced by Android itself
The final setup consists in indicating the servers, the user and its password as in Evolution, using 'Accounts' in Android settings.

Conclusions

Migrating my Google Contacts to Baikal is as simple as a copy and paste in Evolution