Addressbook Synchronisation

Background
I would like to synchronise my address books at my computers, email clients and phone.

My ideal address book would work off-line, automatically synchronise when on-line, and not corrupt data (even not by guessing. If I enter a marriage date in one app, I don't want it to show up as a birthday in another app, messing up the next sync, thank-you-very-much). It would be a relation database, and acknowledge that a person can have multiple email addresses, an organisation can have (non-personal) email addresses, that multiple persons live at the same address, that people have relations (friends may be married and have children), and that there are more dates to remember than birthdays (try forgetting your marriage day if you don't what I mean.). Finally, it is important to group address and persons (I like to separate friends, work, family and organisations for starters.)

In the past I used an LDAP server, but that lacked synchronisation, had a clumsy interface, and the data structure is really hierarchical in nature, rather then relational.

I have used Plaxo to synchronise address books. While it featured quite a few applications (Apple Address book, Thunderbird mail addresses), I have not been entirely satisfied. I experience corruption of data, especially in dates, duplication of data, and was hampered by lack of some minor features.

CardDav
I have recently become aware of the upcoming CardDav standard, based on WebDAV and vCard. At the time of writing, CardDav is in active development at the IETF vcarddav working group. Especially the use of WebDAV seems a very good step forward. I'm not very fond of the vCard standard (the data structure is too linear instead of relational. For example if I want to add the birthday of a child of one of my friends, I have to make a new vCard for the child and retype the address.), but given the wide-spread support for vCards, I consider it the most practical solution.

Apple is an early adopter of CardDav (Apple' Address Book and iPhone's address book both support it since Mac OS 10.6 and iPhone OS 4 respectively), and since I have mostly Apple devices, this was another reason to use CardDav

Calender Server Installation
Of the available CardDav servers, only Apple's Calender Server seems to be an active project at the time of this writing.

Install Prerequisites
I'm attempting to install the carddav server at a G5 computer running Linux (Debian Lenny distro). While Lenny contains a package calendarserver, this has some quirks (it requires a custom version of Python-twisted, spiked my CPU usage to 100%), and above all, has not support for carddav yet.

carddav support was only implemented in calenderdav 3.0, which was not released at the time of writing, so I checked out the code from source:

svn co http://svn.macosforge.org/repository/calendarserver/CalendarServer/trunk calendarserver

(In my case, I checked out revision r6233).

The rest of the installation is based on the quick start guide in the calendarserver documentation.

Calendarserver has quite a few dependencies. The run script tries to download and compile them for you, which is extremely annoying (it simply should have stated which software was missing with a hint how to install it, so the user could decide how to install it). Instead, I tried to fulfill the dependencies with packages in Debian (stable if the version is recent enough, otherwise using testing). If that failed, I installed them by hand.

Packages in Debian Lenny:

aptitude install memcached postgresql python-dateutil python-kerberos python-vobject python-zopeinterface

By mixing stable and testing, the following packages can be installed using Debian Squeeze: pyOpenSSL 0.10, PyGreSQL-4.0:

aptitude -t testing install python-openssl python-pygresql

Finally, select26, xattr 0.5 and Twisted 10 were installed by hand. select26 provides some Python 2.6 API calls for Python 2.5 and has no Debian package; xattr 0.5 (not to be confused with the python-pyxattr package!) is not available in stable or testing, and Twisted 10 has so many dependencies that I rather install it by hand.

sudo aptitude install python-dev wget http://pypi.python.org/packages/source/s/select26/select26-0.1a3.tar.gz tar -xzf select26-0.1a3.tar.gz cd select26-0.1a3 python setup.py build sudo python setup.py install --prefix /usr/local

svn co svn://svn.twistedmatrix.com/svn/Twisted/tags/releases/twisted-10.0.0 cd twisted-10.0.0 python setup.py build sudo python setup.py install --prefix /usr/local sudo twisted --help

svn co http://svn.red-bean.com/bob/xattr/releases/xattr-0.5 cd xattr-0.5 python setup.py build sudo python setup.py install --prefix /usr/local

The run script attempts to install more stuff, which is extremely annoying (it simply should have stated which software was missing with a hint how to install it, so the user could decide how to install it).

I made the following changes in support/build.sh:
 * Change "type postgres" to "type psql", as the name of the postgresql tool on Linux is "psql", not "postgres".

Run the Server In-Place
Calendarserver comes with a run script:

./run

which downloads all requirements, and then runs:

./bin/carddavd -X -L -f ./conf/carddavd-dev.plist -P caldav -t Combined

Rather than using the run script, you should use the carddavd script to start the server. Observe that you can both use the carddavd and caldavd script: both can be configured to run as either CalDAV or CardDAV server in the configuration file.

The carddavd script set the PYTHONPATH and runs:

python /usr/local/bin/twistd -n carddav -f ./conf/carddavd-dev.plist -o ProcessType=Combined -o ErrorLogEnabled=False

Unfortunately, despite the ability to set the PYTHONPATH, this failed because some modules could not be found (e.g. Python quits with and "ImportError"). There are three possible causes for this:
 * Not all prerequisites are installed
 * Twisted is looking for modules and plugins in the wrong place
 * The conf file is not parsed correctly, causing modules to be loaded not available on all platforms (notably opendirectory only exists on Mac OS X).

The next subsections describe how to deal with twisted and the conf file, respectively.

Twisted Install Location
As described above, the server is run using the following command:

export PYTHONPATH=/home/user/calendarserver/usr/lib/python2.5/site-packages /usr/local/bin/twistd -n carddav -f ./conf/carddavd-dev.plist -o ProcessType=Combined -o ErrorLogEnabled=False

(where PYTHONPATH points to the location where you downloaded and compiled calendarserver.)

Observe that this does not contain the full path of the carddav executable. What happens is that twistd (the Twisted daemon) determines the location of the carddav executable by searching its plugin directories. The locations is looks for are determined by the locations of the Python site-packages (as defined by $PYTHONPATH and sys.path in Python itself. Typical locations include:

/usr/lib/python2.5/site-packages/twisted/plugins/ /usr/lib64/python2.5/site-packages/twisted/plugins/ /usr/local/lib/python2.5/site-packages/twisted/plugins/ /home/user/calendarserver/usr/lib/python2.5/site-packages/twisted/plugins/ /opt/calendarserver/usr/lib/python2.5/site-packages/twisted/plugins/

Resetting Twisted Cache
To complicate things, Twisted caches these locations. If you never used a twisted server before, you may get the following error:

exceptions.IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.5/site-packages/twisted/plugins/dropin.cache.new'

In this case, twisted wants to update the cache file, but is not allowed to so so. You can fix this by running any twisted command as root. Printing help works just as well as starting the server:

sudo twistd --help

Resetting Plugin Paths
Twisted looks for both .pyc and .py files. If you move the plugins, your .pyc files can be outdated, and should be remove (they are automatically regenerated from the .py files).

For example, I moved the caldav plugin with (both caldav.py and caldav.pyc) from /home/user/calendarserver/usr/lib/python2.5/site-packages/twisted/plugins/ to /usr/local/lib/python2.5/site-packages/twisted/plugins/</tt>. The following trace still referes to files in the former directory, even though I erased those files!

... File "/usr/local/lib/python2.5/site-packages/twisted/application/app.py", line 679, in subCommands self.loadedPlugins[plug.tapname] = plug File "/home/user/calendarserver/usr/lib/python2.5/site-packages/twisted/plugins/caldav.py", line 29, in getProperty File "/usr/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 351, in namedObject module = namedModule('.'.join(classSplit[:-1])) File "/usr/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 339, in namedModule topLevel = __import__(name) ...

The solution is to simply remove the pyc files:

sudo find /usr/lib /usr/local/lib -name "*.pyc" -exec rm {} \;

If all else fails, the following scripts lists all plugins and resets the case in the process:

from twisted.plugin import IPlugin, getPlugins pluginlist = list(getPlugins(IPlugin)) for p in pluginlist: print p
 * 1) !/usr/bin/python

Put All Plugins in the Same Folder
You are recommended to put all plugins in the same folder, instead of scattering it over multiple folders, such as:

/usr/lib/python2.5/site-packages/twisted/plugins/ /usr/lib64/python2.5/site-packages/twisted/plugins/ /usr/local/lib/python2.5/site-packages/twisted/plugins/ /home/user/calendarserver/usr/lib/python2.5/site-packages/twisted/plugins/ /opt/calendarserver/usr/lib/python2.5/site-packages/twisted/plugins/

If it is scattered over multiple folders, the modules may not be found, even if you included all site-packages paths in your $PYTHONPATH</tt>.

Consider the following ImportError:

File "/opt/calendarserver/usr/lib/python2.5/site-packages/twext/web2/iweb.py", line 152, in   from twisted.web.iweb import IRequest as IOldRequest ImportError: No module named iweb

In this case, the twext.web.iweb modules is trying to include the twisted.web.iweb module, but twisted.web.iweb can not be found. Looking in the folders, I could easily locate it in /usr/local/lib/python2.5/site-packages/twisted/web/iweb.py</tt>. However, since the /usr/lib/python2.5/site-packages/twisted/</tt> already existed, Python was no longer looking in the /usr/local/lib/</tt> folder. Unfortunately, since the twisted</tt> folder /usr/lib</tt> folder contained no iweb.py, Python gave the error.

This could have been avoided by putting all plugins and site-packages in the same directory, or by making symlinks if you prefer that.

Installing the Server
The following command is supposed to install calendarserver in the right place:

./run -I /opt/calendarserver/

However, all it did was:
 * create a folder /opt/calendarserver/bin with a copy of all tools from /usr/bin and /usr/local/bin (huh?)
 * abort with an error rsync: change_dir "/usr/caldavd" failed: No such file or directory (2)</tt>

In short, buggy beyond repair.

After some tips on the mailing list, it was advised to install calenderserver using ./run -i</tt> instead of ./run -I</tt>, and secondly that it is no longer necessary to use ./run -a</tt>, as carddav can be configured in the config files.

I installed calenderserver in /opt/calenderserver</tt> (not trusting it in /urs/local</tt>:

./run -i /opt/calendarserver

I can now start calenderserver with running twisted with a proper PYTHONPATH</tt> set:

export PYTHONPATH=/usr/lib/python2.5/site-packages:/opt/calendarserver/usr/lib/python2.5/site-packages:/opt/calendarserver/usr/lib/python2.5/site-packages/twext/web2 /usr/local/bin/twistd -l - -n caldav -f /opt/calendarserver/etc/caldavd/carddavd.plist