I have been using Firefox Sync self-hosting for a while now. I think I started in 2010, the project was still called Weave at this time. Since Firefox 29, Mozilla added a new protocol, and is still offering the possibility to selfhost it. I’ve been postponing upgrading for a while since the old protocol was still working, while still making a few unsuccessful attempts to upgrade. The sync stopped working a while ago, not sure if it’s Firefox related or maybe a problem on my server. Anyway, I finally managed to selfhost the new sync server after much head scratching, hours lost on google search, coffee, and headaches.

This small guide is an attempt to explain how, and also a reminder for me next time I want to do it 😄. It will cover selfhosting Firefox Sync with the following requirements :

• Hosting on ONE server for all components (Firefox Sync + Firefox Accounts)
• Debian Wheezy.
• Apache (reverse proxy).
• SSL. I am using CACert to sign my certificates, but selfsigned should work as well.
• MySQL.
• Linux and Android sync.

This guide assumes you are familiar with Linux system administration in general, that you more or less understand what you are doing and know how to do use a search engine in case anything goes wrong 😄.

I actually did all my tests on Debian Jessie (current testing) then put in production on Wheezy (current stable). It’s a bit different :

• Apache version : 2.2 vs 2.4, but in our case it’s mostly the authorizations.
• Node.js & NPM availability. It’s only packaged for Jessie.

There is a Node.js package in Wheezy-backports, but it does not include NPM. However, there is a repository with Node.js & NPM from NodeSource. Follow instructions here to install the repo : https://github.com/nodesource/distributions.

General requirements

The new sync has two parts :

• Firefox Sync server.
• Firefox Accounts server. The accounts server is divided in three parts : auth server, database server, and content server.

You can find enough documentation on the Sync server to get it running by using Mozilla official Firefox Accounts server. However, there is almost no documentation on running the Firefox Accounts server, which is why I’m writing this.

First thing first, set up a special user to run the services :

adduser ffsync

Every command after that should be run with this user. Unless root is required of course (apt-get, Apache configuration, …).

Install required packages (run as root) :

apt-get install python-dev python-virtualenv libapache2-mod-wsgi git nodejs libgmp-dev

Configure WSGI properly. Maybe this is needed, maybe not, but it cannot hurt to do it by the book (as root) :

mkdir /usr/local/pythonenv
cd /usr/local/pythonenv
virtualenv --no-site-packages BASELINE

Create a conf file for apache with the following (eg: /etc/apache2/conf.d/modwsgi.conf) :

<IfModule mod_wsgi.c>
WSGIPythonHome /usr/local/pythonenv/BASELINE
</IfModule>

Sync server setup

Install the syncserver :

cd ~
git clone https://github.com/mozilla-services/syncserver.git
cd syncserver
make build

Edit synserver.ini, and specify :

• The public URL. This is the URL as seen by Firefox.
• The database in which to store server data. In the example here, it’s a MySQL database running on the local server, user ffsync, password mypass, database ffdb.
• The secret key. Use the method in the comments to generate it.
• Leave force_wsgi_environ to false. You will (I hope) not need it with the configuration I’m describing here.
• Uncomment the browserid part, and set audiences to the same public URL as above. No trailing slash this time !

[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 5000

[app:main]
use = egg:syncserver

[syncserver]
# This must be edited to point to the public URL of your server,
# i.e. the URL as seen by Firefox.
public_url = https://ffsync.mydomain.net/

# This defines the database in which to store all server data.
sqluri = pymysql://ffsync:mypass@localhost/ffdb

# This is a secret key used for signing authentication tokens.
# It should be long and randomly-generated.
# The following command will give a suitable value on *nix systems:
#
#    head -c 20 /dev/urandom | sha1sum
#
# If not specified then the server will generate a temporary one at startup.
secret = TEHSECRETKEYZOMG

# Set this to "false" to disable new-user signups on the server.
# Only request by existing accounts will be honoured.
# allow_new_users = false

# Set this to "true" to work around a mismatch between public_url and
# the application URL as seen by python, which can happen in certain reverse-
# proxy hosting setups.  It will overwrite the WSGI environ dict with the
# details from public_url.  This could have security implications if e.g.
# you tell the app that it's on HTTPS but it's really on HTTP, so it should
# only be used as a last resort and after careful checking of server config.
force_wsgi_environ = false

# Uncomment and edit the following to use a local BrowserID verifier
# rather than posting assertions to the mozilla-hosted verifier.
# Audiences should be set to your public_url without a trailing slash.
[browserid]
backend = tokenserver.verifiers.LocalVerifier
audiences = https://ffsync.mydomain.net

Make sure the server will load the right configuration file by modifying syncserver.wsgi, putting the full path of the file in ini_file variable :

# setting up logging
#ini_file = os.path.join(_CURDIR, 'syncserver.ini')
ini_file = '/home/ffsync/syncserver/syncserver.ini'
try:
    fileConfig(ini_file)
except NoSectionError:
    pass

We now need to configure Apache. I’m not going to do this in details, there are plenty of available documentation for that. You need the following for one of the virtual hosts :

<VirtualHost *:443>
ServerName ffsync.mydomain.net
DocumentRoot /home/ffsync/syncserver
WSGIProcessGroup ffsync

WSGIDaemonProcess ffsync user=ffsync group=ffsync processes=2 threads=25 python-path=/home/ffsync/syncserver/local/lib/python2.7/site-packages
WSGIPassAuthorization On
WSGIScriptAlias / /home/ffsync/syncserver/syncserver.wsgi

SSLEngine on
SSLStrictSNIVHostCheck on
SSLCertificateFile      /etc/apache2/ssl/ffsync.mydomain.net.crt
SSLCertificateKeyFile   /etc/apache2/ssl/ffsync.mydomain.net.key
</VirtualHost>

You will also need to edit the following file : /home/ffsync/syncserver/local/lib/python2.7/site-packages/requests/cacert.pem and add the CACert root certificate. Or your own root certificate.

If you want to run both services on the same servers like me, you will most probably have issues with SNI. By default it does not work with Python 2.7 requests module. There is a workaround though. You can find it here : https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/. It’s quite simple.

First, install yet another package, needed to build PyOpenSSL :

apt-get install libffi-dev

Then install the needed modules in the syncserver :

cd ~/syncserver
./local/bin/pip install PyOpenSSL
./local/bin/pip install urllib3
./local/bin/pip install ndg-httpsclient
./local/bin/pip install pyasn1

Do not forget to restart Apache after any change to syncserver settings.

Firefox Accounts server setup

Note that you will have 3 URLs to remember, and I will refer to them in the following examples.

• Auth server URL : https://auth.mydomain.net
• Content server URL : https://content.mydomain.net
• Auth DB URL : http://127.0.0.1:8000

I’m using a DB on the local server, so I left it as 127.0.0.1 since it’s only accessed from the Auth server.

Auth server setup

git clone https://github.com/mozilla/fxa-auth-server.git
cd fxa-auth-server
npm install

Edit the configuration file config/dev.json, and do something like the following. All parameters and their default are explained in config/config.js. You need to put the publicUrl, or you will probably have auth failures with « Bad mac » errors in the logs.

{
  "publicUrl": "https://auth.mydomain.net"
  "contentServer": {
    "url": "https://content.mydomain.net"
  },
  "smtp": {
    "host": "smtp.mydomain.net",
    "port": 25,
    "secure": false,
    "sender": "root@mydomain.net",
    "resendBlackoutPeriod": 0
  },
  "verifierVersion": 0,
  "customsUrl": "none",
  "lockoutEnabled": true
}

Edit the starting script scripts/start-local.sh and comment the line starting an auth db server in memory, since we are using a MySQL auth db we do not want this.

#!/usr/bin/env bash
node ./scripts/gen_keys.js
node ./test/mail_helper.js &
MH=$!
#node ./node_modules/fxa-auth-db-mem/bin/server.js &
DB=$!

node ./bin/key_server.js | node ./bin/notifier.js >/dev/null

kill $MH
kill $DB

The way to run the auth server will be the following. The NODE_ENV environnment variable is overriden by the npm start (it’s set as « dev »), that’s why what you set does not matter, and you need to edit config/dev.json.

cd ~/fxa-auth-server
npm start

Create an Apache configuration file like this :

<VirtualHost *:443>
ServerName auth.mydomain.net
DocumentRoot /home/ffsync/fxa-auth-server

SSLEngine on
SSLStrictSNIVHostCheck on
SSLCertificateFile      /etc/apache2/ssl/auth.mydomain.net.crt
SSLCertificateKeyFile   /etc/apache2/ssl/auth.mydomain.net.key

ProxyPreserveHost On
ProxyRequests off
ProxyPass / http://127.0.0.1:9000/
ProxyPassReverse / http://127.0.0.1:9000/

</VirtualHost>

Database backend setup

Two backends are available currently : memory and MySQL. This guide will use the MySQL one. A generic backend template is available there : https://github.com/mozilla/fxa-auth-db-server.

git clone https://github.com/mozilla/fxa-auth-db-mysql.git
cd fxa-auth-db-mysql
npm install

Configure the database backend. Create config/prod.json :

{
	"hostname": "127.0.0.1",
	"port": 8000,
		"master": {
			"user": "fxa",
			"password": "mypass",
			"database": "fxa"
	},
	"slave": {
		"user": "fxa",
		"password": "mypass",
		"database": "fxa"
	}
}

Then create the database and patch it. This is done when you run the backend, but checking if it works now is better :

export NODE_ENV=prod
node bin/db_patcher.js

You will be able to run the database backend like this :

export NODE_ENV=prod
npm start

Content server setup

I had problems to install it, sometimes npm failed. After a few tries, it finally succeeded.

git clone https://github.com/mozilla/fxa-content-server.git
cd fxa-content-server
npm install

Copy server/config/local.json-dist to server/config/local.json and edit it. Pay special attention to public_url and fxaccount_url, and the secret key.

{
  "public_url": "https://content.mydomain.net",
  "fxaccount_url" : "https://auth.mydomain.net",
  "api_proxy": {
    "enabled": true
  },
  "oauth_client_id": "98e6508e88680e1a",
  "oauth_url": "http://127.0.0.1:9010",
  "profile_url": "http://127.0.0.1:1111",
  "profile_images_url": "http://127.0.0.1:1112",
  "client_sessions": {
    "cookie_name": "session",
    "secret": "SECRETKEYZOMG",
    "duration": 86400000
  },
  "env": "development",
  "use_https": false,
  "static_max_age" : 0,
  "i18n": {
    "supportedLanguages": ["af", "an", "ar", "as", "ast", "az", "be", "bg", "bn-BD", "bn-IN", "br", "bs", "ca", "cs", "cy", "da", "de", "dsb", "el", "en", "en-GB", "en-ZA", "eo", "es", "es-AR", "es-CL", "es-MX", "et", "eu", "fa", "ff", "fi", "fr", "fy", "fy-NL", "ga", "ga-IE", "gd", "gl", "gu", "gu-IN", "he", "hi-IN", "hr", "hsb", "ht", "hu", "hy-AM", "id", "is", "it", "it-CH", "ja", "kk", "km", "kn", "ko", "ku", "lij", "lt", "lv", "mai", "mk", "ml", "mr", "ms", "nb-NO", "ne-NP", "nl", "nn-NO", "or", "pa", "pa-IN", "pl", "pt", "pt-BR", "pt-PT", "rm", "ro", "ru", "si", "sk", "sl", "son", "sq", "sr", "sr-LATN", "sv", "sv-SE", "ta", "te", "th", "tr", "uk", "ur", "vi", "xh", "zh-CN", "zh-TW", "zu"]
  },
  "route_log_format": "dev_fxa",
  "logging": {
    "fmt": "pretty",
    "level": "debug"
  },
  "static_directory": "app",
  "metrics": {
    "sample_rate": 0
  },
  "iframe_allowed_origins": ["http://127.0.0.1:8080"]
}

You can run the content server with the following command :

npm start

Create an Apache configuration file like this :

<VirtualHost *:443>
ServerName content.mydomain.net
DocumentRoot /home/ffsync/fxa-content-server

SSLEngine on
SSLStrictSNIVHostCheck on
SSLCertificateFile      /etc/apache2/ssl/content.mydomain.net.crt
SSLCertificateKeyFile   /etc/apache2/ssl/content.mydomain.net.key

ProxyPreserveHost On
ProxyRequests off
ProxyPass / http://127.0.0.1:3030/
ProxyPassReverse / http://127.0.0.1:3030/

</VirtualHost>

Running everything

Run the database backend :

cd ~/fxa-auth-db-mysql
export NODE_ENV=prod
npm start

Run the auth server :

cd ~/fxa-auth-server
export NODE_ENV=prod
npm start

Run the content server :

cd ~/fxa-content-server
npm start

You do not have to run the sync server explicitely, it will be run when requested by Apache.

Tips

I struggled over complicated stuff, but also over simple and stupid issues. Those are a few tips that cover most of the issues I got.

Firewall

If you have a firewall enabled, make sure you can reach out for http, https and git. This is needed for installation, the « npm install » command will try to get git repositories using both http and git.

Also, make sure your syncserver can connect to your Firefox accounts server. For some reason, this was not allowed on my server, and it was not easy to find out why it was not working, until I added logs in one of the python source files as someone suggested in a bug report on github and noticed there was a timeout. Then I took a look at my iptables logs… 😄.

Mail

The confirmation mail may end up in your spam folder. My spamassassin is putting it there 😄. There is a discussion about this issue but it has not been solved yet.

Apache

Make sure the following modules are enabled : wsgi, proxy, proxy_http.

It works fine with the regular MPM prefork which I tested on Jessie. However I’m using MPM ITK on my production server, and I had to compile mod_wsgi to enable daemon mode. I followed this guide : https://klette.us/apache2-mpm-itk-and-libapache2-mod-wsgi.

Node.js

Pay attention to .js vs .json extensions.

While I did not get any issue in npm install on Jessie, I got a few when trying to install fxa-content-server on Wheezy. It’s probably a bug in npm according to comments on the interwebz. Just keep trying if it fails.

Restart

I moved the virtual machine hosting the server to another host recently. Trying to start the database backend resulted in a « Not Found » error. Somehow, the patch level had been removed from the metadata table. You can find it in the schema folder in case this also happens to you.

Android setup

There is information on the interwebz about Android configuration. Unfortunately, I was not able to test it, since the Android sync client does not support SNI - it’s actually not really part of Firefox, it’s a separate app. There is a bug opened about it for a while, which has not been fixed yet.