Where to add a new test
Depending on what you want to test, it can be added in different places.
There are two categories of tests that we are running:
-
The tor browser unit tests. Those tests are included in the tor browser sources tree, in tor-browser.git, and can only be run from the sources tree. We are currently running all the xpcshell and mozmill tests that we find in the sources tree.
-
The tor browser bundle tests. Those tests are run on a complete bundle. The tests in the browser can be implemented using marionette. We also have other types of tests checking that pluggable transports are working with and without using an http proxy, or checking that binary files have been compiled with hardenning options.
If the goal is to check that the behavior of the browser is correct after some changes from one of our patches, then it is probably a good idea to have the test as a patch on tor-browser.git. The Mozilla wiki has some documentation about adding mochitest and XPCShell tests.
If the goal is to check the behavior of something that could be changed by an extension included in the bundle, some build options or some settings, then it should be included in the bundle tests. How to add this kind of test is described below.
Adding a browser bundle test
The list of tests to be run by the test suite is defined in file TBBTestSuite/TestSuite/BrowserBundleTests.pm in the @tests variable.
Each test is defined with the following properties:
- name
-
The name of the test. For marionette, tor_bootstrap tests this name is used to find the test filename.
- type
-
The type of tests. This can be marionette, tor_bootstrap, command, or something else if you want to write the test in perl. In the later case, you need to define it in the %test_types variable.
- descr
-
A short description.
- fail_type
-
The default is error. If you set this to fatal, when this test fails the following tests are not run. If you set this to warning, failure of this test does not fail the report.
- pre
-
A reference to a perl subroutine that will be run before the test.
- post
-
A reference to a perl subroutine that will be run after the test.
- enable
-
A reference to a perl subroutine that should return false when the test should be skipped.
- run_once
-
If set to a true value, when running the testsuite on a series of bundles, this test will only be run on the first bundle.
Adding a marionette test
To add a new test based on marionette, you will add something like this in TBBTestSuite/TestSuite/BrowserBundleTests.pm:
{
name => 'example-test',
type => 'marionette',
descr => 'Example marionette test',
},The marionette test files are stored in the directory marionette/tor_browser_tests/. The name of the file is test_$name.py where $name is the name of the test, unless marionette_test is defined, in which case it is used instead of the name.
It is possible to share a single marionette file for multiple tests. The marionette test can access values from the test definition to do something different in each test. It is also possible to access the test suite options. In this example, we are accessing the "timeout" value from the test definition, and the "test_data_url" from the the testsuite options:
import testsuite
class Test(MarionetteTestCase):
def setUp(self):
MarionetteTestCase.setUp(self)
ts = testsuite.TestSuite()
timeout = ts.t['test']['timeout']
test_data_url = ts.t['options']['test_data_url']A common option is test_data_url which is an URL where test data have been deployed. If you want to add more data, it should be added to the test-data directory at the root of the repository.
If the test data does not need to be accessed remotely, the test_data_dir option can be used instead.
Adding a simple test page
If what you want to test is the behavior of some specific page and you don’t need any of the marionette specific features, you can reuse the page marionette test.
To do that, put your page in the directory test-data. The name of the test will be the name of the page without the .html extension.
Your page will do something in javascript, and save its result in an element whose id is test_result. If the test is successful, the result should be "OK", and anything else will be considered as an error message and a failed test.
Adding a successful test result can be done like this:
var result = document.createElement('div');
result.innerHTML = 'OK';
result.setAttribute("id", "test_result");
document.getElementsByTagName("body")[0].appendChild(result);The test definition will look like this:
{
name => 'some_name',
type => 'marionette',
descr => 'Some description',
marionette_test => 'page',
remote => 1,
timeout => 50000,
},If remote is set to 1, then the page will be accessed using the URL defined in the test_data_url option, otherwise it will be accessed locally with the test_data_dir option.
The timeout value is the time in milliseconds after which the test should be considered as failed if the test_result element is still not present.
Adding a command test
To run a command on some files, and check the output. You will add something like this in TBBTestSuite/TestSuite/BrowserBundleTests.pm:
{
name => 'readelf_RELRO',
type => 'command',
descr => 'Check if binaries are RELocation Read-Only',
files => [ 'Browser/firefox', 'Tor/tor', ],
command => [ 'readelf', '-l' ],
check_output => sub { $_[0] =~ m/GNU_RELRO/ },
},The command option is the command that should be run. The files option is the list of files on which it should be run. If files is a reference to a subroutine, then it is expected to return an array reference containing the files list. The check_output is optional and is a reference to a subroutine that checks the output of the command. The skip_files option can contain a list of files to skip. The test will fail if the command returns non-zero, or the check_output sub returns false, on any of the files.
Testing the new test
When you are creating a new test, you probably don’t want to run all the tests to try your new test. In this case, the --enable-tests option is useful to run only the test that you added:
$ ./tbb-testsuite --enable-tests my_new_test /path/to/tor-browser-linux64-3.6.6_ko.tar.xzIf your test requires a running Tor daemon, you may need to enable the tor_bootstrap test too:
$ ./tbb-testsuite --enable-tests tor_bootstrap,my_new_test /path/to/tor-browser-linux64-3.6.6_ko.tar.xzAlternatively you may use an already running Tor daemon with the --tor-socks-port option:
$ ./tbb-testsuite --enable-tests my_new_test --tor-socks-port=9150 /path/to/tor-browser-linux64-3.6.6_ko.tar.xz