This document will review the current status about build process, packaging, unit and integration testing of all projects that are part of the tor project, and the tools that are currently used for those tasks. It will try to list the main tasks that should be planned in the near and distant future.

It should answer the following questions :

  • What is the list of official tor projects, and their status regarding build (including deterministic build), testing and packaging automation ?

  • What is the list of tools that we currently use and what are their limitations ? Which ones do we want to keep ? Which one do we want to replace, and what are the available alternatives ? Which new tools do we want to use, for which tasks ?

  • What do we want to test ? What type of regressions do we want to avoid by having tests ? What are the main tests that we want to have ?

  • What automation features do we want soon, and what do we want in long term ?

If you want to send me a patch, the sources for this document are available in a git repository.

Topics definitions

This section will explain the different topics of this document.

Continuous integration

The continuous integration tool is the tool that will monitor a git repository, and automatically start a new job when there are new commits, and publish the result of the job.

Common jobs that are started when a new commit is made are :

  • build the software on different platforms, to detect build errors

  • run a test suite to check that the software is behaving correctly

  • publish the build

Common continuous integration tools are :

  • Jenkins (previously known as Hudson)

  • travis-ci

  • buildbot

Reproducible Builds

Reproducible (or Deterministic) Builds means that building some software twice on two different hosts will produce the exact same result.

This blog post provide some explainations about why it is important : https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise

This blog post provide technical details about how it is currently achieved : https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details

Packaging

Distributions packaging

It’s usually better to package the softwares directly into the distribution repositories, so that users don’t have to add an additional packages repository.

External repositories

Most distributions have constraining policies regarding package maintainance in stable distributions. Having an external packages repository allows use to maintain packages using our own policy.

Some tor packages are available on http://deb.torproject.org/

Tarballs

Sometimes we don’t provide packages, but instead provide a tarball that users will manually extract in their home directory.

It is less convenient than packages, but has some other advantages :

  • it works on all distributions

  • it does not require root access to install

Continuous packaging

Continuous packaging means building new packages as soon as new commits are made, and uploading the packages to a repository.

This make it easier for people who want to run the latest developement version to do it.

If enough disk space is available, history of a few previous builds can be kept, to make git bisect faster when looking for the commit which introduced a bug.

Testing

There can be different levels of testing:

Unit Testing

Unit Tests are low level tests that are checking the functionning of some internal functions used in the program. The tools or framwork used depend on the language of the program.

Functional and acceptance tests

More high level tests, to check that the resulting program is producing the expected results. This is usually done using a test framework that give you a format to write the tests and some tools to run them and report the results. See section about test framworks for more details.

Deployement

Some projects are hosted services, and we need to configure and deploy them when a new release is made. We can use a configuration management tool to automate that part.

Common configuration management tools are :

  • cfengine

  • puppet

  • ansible

  • saltstack

Priorities

.

Projects

In this section we will look at each project that is part of the tor project, and look the build, packaging and testing of this project to see the current status and list some proposals.

tor

Continuous Build

The master, maint-0.2.3, maint-0.2.4 branches are built automatically by Jenkins :

  • on different versions of Debian distributions (squeeze, wheezy, jessie, sid, lucid, precise, quantal, raring, saucy)

  • on Debian sid with clang

  • on Windows

Packaging

Tor is included in the Tor Browser Bundle.

Packages for the stable version of Tor are available in many Linux distributions.

Some packages are also available on http://deb.torproject.org/ :

  • nightly builds of Debian packages based on the tor master, maint-0.2.3, maint-0.2.4 branches are built automatically by Jenkins. Those packages are created using scripts in the jenkins/tools.git repository.

  • RPM packages for Fedora 18, 19 and Red Hat EL 6, for the maint-0.2.4 and master branches. The packages are built using the scripts from the user/hiviah/rpm-build-scripts.git repository. The build is manually started, there is no nightly builds yet.

Testing

Tor includes 189 unit tests in C, testing individual functions used in Tor. A few python scripts are also used to test some parts of Tor. Those tests are run automatically by Jenkins after successful builds.

There is no integration tests that are run automatically yet. There is the beginnings of a set of scripts to run integration tests using chutney. The script src/test/test-network.sh can be used to bootstrap a Tor network.

Example Bugs

TODO: find example of bugs that we would like to detect with automated tests.

Proposals

  • Implement a few integration tests, and make them run by Jenkins

  • Add RPM continuous packaging

tor browser

Tor browser is the web browser that is included as part of the Tor Bundle Browser. It is based on Firefox ESR release, with some patches to improve privacy and make the build reproducible.

The sources are stored in the tor-browser.git repository. Because the tor browser is a series of patches on the Firefox browser, there is no master branch. When a new Firefox release is available, the new Firefox commits are imported in the tor-browser.git repository and tagged as FIREFOX_$VERSION, and the tor browser patches are rebased on those commits in a new branch called tor-browser-$VERSION.

Continuous Build

Tor Browser is not yet built automatically when there are new commits.

Packaging

There is no packages available for Tor Browser yet. The Tor Browser is currently only available as part of the Tor Browser Bundle.

The Tails project is making some Iceweasel packages which include the Tor Browser patches to include in their distribution. They take the latest Debian Iceweasel package and add the Tor Browser patches on top of it, using the process described on this page.

Debian and most distributions have a policy about security updates that require updates to include only minimal changes to fix the security problem. Because of this policy, having an official Tor Browser package in those distributions would be a lot of work. However, having packages on deb.torproject.org would be easier to do.

Testing

Testing is currently done manually by volunteers on the tor-qa mailing list.

There is no automated testing yet. There are plans to add automated testing using Mozilla Try servers soon.

Test Categories

We need some tests to check the following :

Usability tests

We want to check that all websites that work in vanilla Firefox still work in Tor Browser. To check this, we can run the tests from Mozilla. Mozilla will provide us access to their Try Server so we can run their tests.

Fingerprinting tests

Tor Browser includes some patches to make all Tor Browsers look the same. New firefox releases could introduce some changes that make the browser stand out from different versions or different setup, so we need some tests to check that the fingerprint did not change from previous release.

Privacy tests

Tor Browser includes some patches to make sure that the browser does not leak any information about the user, or its network.

Reproducible build test

Tor Browser includes some patches to make the build reproducible. We could have a test that check the reproducibility of the build by building the browser twice.

Test Framework

Depending on what needs to be tested, a different test framework can be used :

Mozilla Unit Tests

Firefox includes several unit tests. We can for instance add xpcshel based unit tests, written in Javascript. Those tests will be run when tor browser is pushed to Mozilla Try.

General test framework

In some cases, tests may be difficult to implement using the Mozilla test frameworks. For instance some tests will require a running tor daemon, or access to an external service. For those tests we will use a more general test framework. See test framework section for details.

Interesting tickets

#10195 : Tor Browser crashing in 64-bit Linux builds

A compiler bug caused the tor browser to crash.

#8143 : Investigate failure of test_async_notification.js and fix it

A mozilla unit test that was failing. We should be able to notice unit test failures when we are pushing to Mozilla’s Try servers.

#10285 : Write test pages for certain FF24 features

This ticket includes some test pages that can maybe be used for automated tests.

#3994 : Get TorBrowser in Debian

This ticket discuss some of the issues of getting a Tor Browser package into Debian.

Proposals

Packaging

Tor browser is currently only available as part of the Tor Browser Bundle. We can produce some packages for Tor Browser, as a first step to do continuous packaging. Those packages can also be useful to Tails developers.

Continuous packaging

If packages are available for Tor Browser, we can also produce packages automatically when there are new commits on the browser. See continuous packaging proposal for details about how to do that. Packages available for nightly builds allows some people to use those builds and hopefully notice and report problems such as #10195 before the release.

Fingerprinting tests

We can implement some tests that extract a fingerprint of a Tor Browser, and compare it with previous version of the Tor Browser. See fingerprint helper for details.

Identifier tests

We can implement some tests to check that cookies and other possible identifiers are saved or not saved depending on the configuration. See identifier helper for details.

Network leak tests

We can implement some tests to check that tor browser does not do any direct connection when configured to use tor. To do that we can make a script that visit various different web sites, using different protocols and different features, while using a tool to monitor network activity. See network helper for details about this tool.

tor launcher

Tor Launcher is a xul browser extension that is responsible for starting the tor daemon in the Tor Browser Bundle.

Packaging

Tor Launcher is packaged as an xpi file (Mozilla extension file) which is included in the Tor Browser Bundle.

The xpi file (a zip file) created by the Makefile is not currently deterministic, so it has to be extracted and recreated in a deterministic way during the bundle build.

Testing

Tor Launcher does not have tests yet.

Using Tor Launcher require the other components of the bundle, so it can be difficult to test itself. It can be tested as part of Tor Browser Bundle.

Proposals

minor change: make the xpi file created by the Makefile deterministic, so it doesn’t have to be recreated during TBB build.

Torbutton

Torbutton is a firefox extension that is included with Tor browser as part of the bundle.

Packaging

Torbutton is packaged as an xpi file (Mozilla extension file) which is included in the Tor Browser Bundle.

The xpi file (a zip file) created by the the script ./makexpi.sh is not currently deterministic, so it has to be extracted and recreated in a deterministic way during the bundle build.

Testing

Torbutton does not have tests yet.

Some tests to check the main functionnalities of Torbutton could be added. A possible tool to implement those tests can be MozMill or one of the other Mozilla frameworks.

Proposals

Add some tests to check the main functionnalities of Torbutton. Those tests should be run as part of bundle testing.

tor browser bundle

Tor Browser Bundle is a bundle containing different tor components bundled in a tarball :

  • the tor browser

  • the tor daemon

  • tor launcher

  • tor button

  • https everywhere

  • noscript

It is built for Windows, Linux (x86 and x86_64), Windows.

The different branches

TBB 2.x.x

The scripts to build the 2.x.x versions of the bundle are in the torbrowser.git repository.

The Windows and Mac OS X versions were built natively on Windows and Mac OS X.

With the release of TBB 3.5, the TBB 2.x.x branch is now deprecated and won’t be used for new releases.

TBB 3.x.x

The main differences between TBB 2.x.x and TBB 3.x.x branches are :

  • bundle builds are now reproducible

  • the Windows and Mac OS X versions are cross compiled

  • Vidalia is no longer used to start the tor daemon and browser. It is replaced by tor launcher. Vidalia is still available as an addon that can be downloaded separately.

The scripts to build the bundle are now located in the builders/tor-browser-bundle.git repository.

Details about the build process can be found in the gitian/README.build file.

Bug Examples

#10444: TBB’s Info.plist needs to be updated each release

The file Info.plist included in the mac builds include some version numbers that should be updated for each new release. We should automate the update of this file this so that it’s not forgotten.

Proposals

  • Replace gitian with an other tool with more features, so that we don’t need to use shell wrapper scripts around it. This should also make the build process more simple and more modular. This tool is described in the build tool proposal.

  • The Bundle can currently only be built from Ubuntu. It would be nice to be able to launch the build from other Linux distributions, so that we don’t have to run Ubuntu on the server that will do automated builds. This is reported in ticket #10125.

  • After improving the build process, run automated builds from Jenkins when there are new commits in any of the components included in the bundle. A more modular build process allows us to rebuild only the parts that changed since the last build.

  • Implement some automated tests and run them automatically after new builds.

tor pluggable transports

.

tor pluggable transports bundle

The tor pluggable transports bundle is a bundle similar to the Tor Browser Bundle, but which includes some pluggable transports:

  • pyptlib (python pluggable transport library)

  • obfsproxy

  • flashproxy

Build Process

The repository pluggable-transports/bundle.git contains the Makefile that is used to build the 2.x.x series of the bundle. This Makefile downloads the classical Tor Browser Bundle, extract it, add the pluggable transports and a modified torrc configuration file, then create and sign new tarballs.

With the 3.5 bundle release, the build process has been changed to use gitian. A new pluggable-transports bundle has also been created using gitian, using descriptors currently in the repository user/dcf/tor-browser-bundle.git. This work is described in ticket #9444.

Testing

The pluggable transports bundle does not currently have automated tests.

Proposals

  • A proposal for a new build process is described here.

  • Automate build of PTTBB with Jenkins when there are new commits in any of its components, and keep an archive of recent builds

  • Implement some automated tests for the pluggable transports, and run them on new commits

TorBirdy

TorBirdy is an extension that brings safe Tor support to Thunderbird.

Continuous Build

TorBirdy is not currently built by Jenkins. However, its current build process is to zip the sources as an xpi file, so it has no reason to fail.

Packaging

TorBirdy is distributed as an xpi file (a zip file containing the sources of the plugin), to be installed with thunderbird.

There is no distribution package available yet. Ticket #6018 is about creating a Debian package to add it to torproject.org repository, and ticket #8030 is about adding it to Debian..

Testing

TorBirdy does not seem to have tests yet. It might be possible to implement some tests using xpcshell.

Proposals

torsocks

.

tordnsel

.

vidalia

.

tor cloud

.

orbot

.

arm

.

stem

.

pytorctl

pytorctl is mostly unmaintained according to its README file. It can be replaced by Stem.

chutney

.

atlas

.

compass

.

check

.

bridgedb

Continuous Build

BridgeDB is not currently built and tested by Jenkins.

However, Isis Lovecruft has a personnal development fork on github that is built and tested by travis-ci.org: https://travis-ci.org/isislovecruft/bridgedb/ The configuration file used by travis-ci is: https://gitweb.torproject.org/user/isis/bridgedb.git/blob/refs/heads/develop:/.travis.yml

Packaging

BridgeDB does not have packages. It is currently deployed using a Python virtualenv.

Testing

Some unit tests are implemented in lib/bridgedb/Tests.py and can be run with the command python setup.py test.

Proposals

  • Add BridgeDB build and test to Jenkins (#10417). We should run the tests with multiple versions of python, and try to have coverage reports similar to coveralls.io, using the Jenkins Cobertura Plugin.

ooni

.

pyonionoo

.

gettor

.

metrics db

.

doctor

.

tor-ramdisk

.

tails

Tails is an independant project. However it includes a lot of tor software so it is interesting to look at what they are already doing, to avoid duplicating work that they already did.

Testing

Tails have some automated tests to check different features. Some of them are related to Tor.

The tests can be found in their git repository in the diretory features.

The tests are written using the Cucumber tests framework. The GUI tests are made using Sikuli.

An interesting test is the torified_browsing.feature test checking that when visiting http://check.torproject.org/ using the tor browser, all traffic flows through tor.

Tools

Continuous integration

This section will discuss the use of Jenkins, possible improvements or alternatives.

Jenkins

Jenkins (previously known as Hudson from which it was forked) is a continuous integration tool written in Java.

There is already an instance of Jenkins running on https://jenkins.torproject.org/

Jenkins Current Setup

Jenkins is configured using Jenkins Job Builder. This is a tool that allows configuring Jenkins jobs using YAML files instead of the web interface. The Jenkins jobs configuration is stored in the jenkins/jobs.git repository.

The jenkins/tools.git repository contains scripts and tools to run the jobs.

Pros

  • It’s already setup

  • Lots of plugins are available to display test results.

  • It does what we need: trigger an action when there are some new commits, and publish the results.

Cons

  • Jenkins might be less flexible than other tools like BuildBot. However it’s not clear that we need that flexibility.

Travis CI

Travis CI is an hosted continuous integration service. An open source version that you can install on your own server is also available. However the README file discourage doing that:

Travis CI is not currently a good fit for closed in-house installations.
It's made up of multiple applications which evolve rapidly. We are
working on providing a way to install a closed in-house installation,
but we have no ETA for this at the moment.
Pros

  • Free for open source projects

  • Easy to use and good documentation

  • Out of the box support for various languages and databases

Cons

  • Requires the git repository to be hosted on github

  • You can’t easily host your own instance

  • It concentrates on the most common use cases. We don’t control the server where it is hosted, so we are limited in what we can do.

Integration tests framework

The test framework is the tool that is responsible for :

  • defining the format in which to express the tests

  • running the tests

  • reporting the results

TODO: implement some basic tests using both framework for comparison

Cucumber

Cucumber is a test framework written in Ruby. The tests are written using the Gherkin language. The differents steps used in the tests are mapped to actions in Ruby.

A Jenkins plugin is available to display the tests results.

Cucumber is the test framework that is used by the Tails project. Their tests are available in their git repository in the features directory.

Robot Framework

Robot Framework is an other test framework, written in Python. The tests are written in plain text using some keyworks. The keywords are implemented in Python or Java. The RemoteLibrary can be used to implement keywords in a different language.

Robot Framework also supports the Gherkin syntax.

A Jenkins plugin is available to display the tests results.

Packaging tools

This section will discuss the use of some tools for packaging.

Reproducible build tools

This section will discuss the use of gitian to produce reproducible builds.

Static analysis tools

This section will discuss the use of static analysys tools.

Proposal details

The proposals are now described in a separate document.

Hardware resources

This section will list the servers which are already available, the servers which will become available soon, and the new servers we should consider getting.

Summary

This section will give a summary of the current status on tor automation, and plans for the future.