Scroll to navigation

ooniprobe(1) ooniprobe ooniprobe(1)

NAME

ooniprobe - a network censorship measurement tool.

SYNOPSIS

ooniprobe [-hnspgN] [--version] [--spew] [-o reportfile] [-i testdeck] [-c collector] [-b bouncer] [-l logfile] [-O pcapfile] [-f configfile] [-d datadir] [-a annotations] test_name

DESCRIPTION

ooniprobe is tool for performing internet censorship measurements. Our goal is to achieve a common data format and set of methodologies for conducting censorship related research.

OPTIONS

-h or --help
Display this help and exit.
-n or --no-collector
Disable reporting the net test results to an oonib collector.
-N or --no-yamloo
Disable writing report file to disk.
-g or --no-geoip
Disable performing geoip lookup.
-s or --list
List all of the available net tests.
-p or --printdeck
Print to standard output the specified command line options as an ooniprobe test deck.
-o or --reportfile
Specify the path to report file to write.
-i or --testdeck
Specify as input a test deck: a yaml file containing the tests to run and their arguments.
-c or --collector
Specify the address of the collector of net test results. It is advisable to always specify and bouncer and let it return a collector for the test or test deck you are running.
-b or --bouncer
Address of the bouncer that will inform the probe of which collector to use and the addresses of test helpers. default: httpo://nkvphnp3p6agi5qq.onion
-l or --logfile
Path to the log file to write
-O or --pcapfile
Prefix to the pcap file name.
-f or --configfile
Specify a path to the ooniprobe configuration file.
-d or --datadir
Specify a path to the ooniprobe data directory
-a or --annotations
Annotate the report with a key:value[, key:value] format.
--spew
Print an insanely verbose log of everything that happens. Useful when debugging freezes or locks in complex code.
--version
Display the ooniprobe version and exit.

OONIPROBE

Is the tool that volunteers and researches interested in contributing data to the project should be running.
ooniprobe allows the user to select what test should be run and what backend should be used for storing the test report and/or assisting them in the running of the test.
ooniprobe tests are divided into two categories: Traffic Manipulation and Content Blocking.
Traffic Manipulation tests aim to detect the presence of some sort of tampering with the internet traffic between the probe and a remote test helper backend. As such they usually require the selection of a oonib backend component for running the test.
Content Blocking are aimed at enumerating the kind of content that is blocked from the probes network point of view. As such they usually require to have specified an input list for running the test.

Threat Model

Our adversary is capable of doing country wide network surveillance and manipulation of network traffic.
The goals of our adversary are:
Restrict access to certain content, while not degrading overall quality of the network
Monitor the network in a way that they are able to identify misuse of it in real time



More specific to the running of network filtering detection tests:
1.
Detect actors performing censorship detection tests
2.
Fool people running such tests into believing that the network is unrestricted

Note that while 2) => 1) it is not true that 1) => 2) as the identification of such actors does not necessarily have to happen in real time. While our intention is to minimize the risk of users running OONI probe to be identified, this comes with a tradeoff in accuracy. It is therefore necessary in certain tests to trade-off fingerprintability in favour of tests accuracy.
This is why we divide tests based on what risk the user running it can face, allowing the user to freely choose what threat model they wish to adhere to.

Installation

Read this before running ooniprobe!
Running ooniprobe is a potentially risky activity. This greatly depends on the jurisdiction in which you are in and which test you are running. It is technically possible for a person observing your internet connection to be aware of the fact that you are running ooniprobe. This means that if running network measurement tests is something considered to be illegal in your country then you could be spotted.
Furthermore, ooniprobe takes no precautions to protect the install target machine from forensics analysis. If the fact that you have installed or used ooni probe is a liability for you, please be aware of this risk.

Debian based systems

sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list'
gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4
gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add -
sudo apt-get update && sudo apt-get install ooniprobe

Linux

We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as well as versions of Ubuntu such as natty and later releases. Running ooniprobe without installing it is supported with the following commands:
git clone https://git.torproject.org/ooni-probe.git
cd ooni-probe
./setup-dependencies.sh
python setup.py install

Setting up development environment

On debian based systems this can be done with:
Vsudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper
mkvirtualenv ooniprobe
python setup.py install
pip install -r requirements-dev.txt

Other platforms (with Vagrant)

Install Vagrant and Install Virtualbox
On OSX:
If you don't have it install homebrew
brew install git
On debian/ubuntu:
sudo apt-get install git
1.
Open a Terminal and run:

git clone https://git.torproject.org/ooni-probe.git
cd ooni-probe/
vagrant up
2.
Login to the box with:

vagrant ssh
ooniprobe will be installed in /ooni.
3.
You can run tests with:

ooniprobe blocking/http_requests -f /ooni/inputs/input-pack/alexa-top-1k.txt

Using ooniprobe

Net test is a set of measurements to assess what kind of internet censorship is occurring.
Decks are collections of ooniprobe nettests with some associated inputs.
Collector is a service used to report the results of measurements.
Test helper is a service used by a probe for successfully performing its measurements.
Bouncer is a service used to discover the addresses of test helpers and collectors.

Configuring ooniprobe

You may edit the configuration for ooniprobe by editing the configuration file found inside of ~/.ooni/ooniprobe.conf.
By default ooniprobe will not include personal identifying information in the test result, nor create a pcap file. This behavior can be personalized.

Running decks

You will find all the installed decks inside of /usr/share/ooni/decks.
You may then run a deck by using the command line option -i:
As root:
ooniprobe -i /usr/share/ooni/decks/mlab.deck
Or as a user:
ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck
Or:
As root:
ooniprobe -i /usr/share/ooni/decks/complete.deck
Or as a user:
ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck
The above tests will require around 20-30 minutes to complete depending on your network speed.
If you would prefer to run some faster tests you should run: As root:
ooniprobe -i /usr/share/ooni/decks/fast.deck
Or as a user:
ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck

Running net tests

You may list all the installed stable net tests with:
ooniprobe -s
You may then run a nettest by specifying its name for example:
ooniprobe manipulation/http_header_field_manipulation
It is also possible to specify inputs to tests as URLs:
ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1
You can find the result of the test in your current working directory.
By default the report result will be collected by the default ooni collector and the addresses of test helpers will be obtained from the default bouncer.
You may also specify your own collector or bouncer with the options -c and -b.

(Optional) Install obfsproxy

Install the latest version of obfsproxy for your platform.
Download Obfsproxy

Bridges and obfsproxy bridges

ooniprobe submits reports to oonib report collectors through Tor to a hidden service endpoint. By default, ooniprobe uses the installed system Tor, but can also be configured to launch Tor (see the advanced.start_tor option in ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path to a file that should contain a set of "bridge" lines (of the same format as used in torrc, and as returned by https://bridges.torproject.org). If obfsproxy bridges are to be used, the path to the obfsproxy binary must be configured. See option advanced.obfsproxy_binary, in ooniprobe.conf.

Setting capabilities on your virtualenv python binary

If your distribution supports capabilities you can avoid needing to run OONI as root:
setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python

Core ooniprobe Tests

The source for Content blocking tests and Traffic Manipulation tests can be found in the nettests/blocking and nettests/manipulation directories respectively.

Content Blocking Tests

DNSConsistency
HTTP Requests
TCP Connect



Traffic Manipulation Tests

HTTP Invalid Request Line:
DNS Spoof
HTTP Header Field Manipulation
Traceroute
HTTP Host



Other tests

We also have some other tests that are currently not fully supported or still being experimented with.
You can find these in:
ooni/nettests/experimental



Tests that don't do a measurement but are useful for scanning can be found in:
ooni/nettests/scanning



Tests that involve running third party tools may be found in:
ooni/nettests/third_party



Reports

The reports collected by ooniprobe are stored on https://ooni.torproject.org/reports/0.1/ CC /
Where CC is the two letter country code as specified by ISO 31666-2.
For example the reports for Italy ( CC is it) of the may be found in:
https://ooni.torproject.org/reports/0.1/IT/
This directory shall contain the various reports for the test using the following convention:
testName - dateInISO8601Format - probeASNumber .yamloo
The date is expressed using ISO 8601 including seconds and with no : to delimit hours, minutes, days.
Like so:
YEAR - MONTH - DAY T HOURS MINUTES SECONDS Z
Look here for the up to date list of ISO 8601 country codes
The time is always expressed in UTC.
If a collision is detected then an int (starting with 1) will get appended to the test.
For example if two report that are created on the first of January 2012 at Noon (UTC time) sharp from MIT (AS3) will be stored here:
Note: it is highly unlikely that reports get created with the same exact timestamp from the same exact ASN. If this does happen it could be index of some malicious report poisoning attack in progress.

Report format version changelog

In here shall go details about the major changes to the reporting format.

version 0.1

Initial format version.

Writing OONI tests

The OONI testing API is heavily influenced and partially based on the python unittest module and twisted.trial.

Test Cases

The atom of OONI Testing is called a Test Case. A test case class may contain multiple Test Methods.
class ooni.nettest.NetTestCase
This is the base of the OONI nettest universe. When you write a nettest you will subclass this object.
inputs: can be set to a static set of inputs. All the tests (the methods starting with the "test" prefix) will be run once per input. At every run the _input_ attribute of the TestCase instance will be set to the value of the current iteration over inputs. Any python iterable object can be set to inputs.
inputFile: attribute should be set to an array containing the command line argument that should be used as the input file. Such array looks like this:
["commandlinearg", "c", "default value" "The description"]


The second value of such arrray is the shorthand for the command line arg. The user will then be able to specify inputs to the test via:
ooniprobe mytest.py --commandlinearg path/to/file.txt


or
ooniprobe mytest.py -c path/to/file.txt


inputProcessor: should be set to a function that takes as argument a filename and it will return the input to be passed to the test instance.
name: should be set to the name of the test.
author: should contain the name and contact details for the test author. The format for such string is as follows:
The Name <email@example.com>


version: is the version string of the test.
requiresRoot: set to True if the test must be run as root.
usageOptions: a subclass of twisted.python.usage.Options for processing of command line arguments
localOptions: contains the parsed command line arguments.

Quirks: Every class that is prefixed with test must return a twisted.internet.defer.Deferred.

If the test you plan to write is not listed on the Tor OONI trac page, you should add it to the list and then add a description about it following the Test Template
Tests are driven by inputs. For every input a new test instance is created, internally the _setUp method is called that is defined inside of test templates, then the setUp method that is overwritable by users.
Gotchas: never call reactor.start of reactor.stop inside of your test method and all will be good.

Inputs

Inputs are what is given as input to every iteration of the Test Case. Iflyou have 100 inputs, then every test case will be run 100 times.
To configure a static set of inputs you should define the ooni.nettest.NetTestCase attribute inputs. The test will be run len(inputs) times. Any iterable object is a valid inputs attribute.
If you would like to have inputs be determined from a user specified input file, then you must set the inputFile attribute. This is an array that specifies what command line option may be used to control this value.
By default the inputProcessor is set to read the file line by line and strip newline characters. To change this behavior you must set the inputProcessor attribute to a function that takes as argument a file descriptor and yield the next item. The default inputProcessor looks like this:
def lineByLine(filename):
    fp = open(filename)
    for x in fp.xreadlines():
        yield x.strip()
    fp.close()


Setup and command line passing

Tests may define the setUp method that will be called every time the Test Case object is instantiated, in here you may place some common logic to all your Test Methods that should be run before any testing occurs.
Command line arguments can be parsed thanks to the twisted twisted.python.usage.UsageOptions class.
You will have to subclass this and define the NetTestCase attribute usageOptions to point to a subclass of this.
class UsageOptions(usage.Options):
  optParameters = [['backend', 'b', 'http://127.0.0.1:57001',
                      'URL of the test backend to use']
                  ]
class MyTestCase(nettest.NetTestCase): usageOptions = UsageOptions
inputFile = ['file', 'f', None, "Some foo file"] requiredOptions = ['backend']
def test_my_test(self): self.localOptions['backend']


You will then be able to access the parsed command line arguments via the class attribute localOptions.
The requiredOptions attributes specifies an array of parameters that are required for the test to run properly.
inputFile is a special class attribute that will be used for processing of the inputFile. The filename that is read here will be given to the ooni.nettest.NetTestCase.inputProcessor method that will yield, by default, one line of the file at a time.

Test Methods

These shall be defined inside of your ooni.nettest.NetTestCase subclass. These will be class methods.
All class methods that are prefixed with test_ shall be run. Functions that are relevant to your test should be all lowercase separated by underscore.
To add data to the test report you may write directly to the report object like so:
def test_my_function():
    result = do_something()
    self.report['something'] = result


OONI will then handle the writing of the data to the final test report.
To access the current input you can use the input attribute, for example:
def test_with_input():
    do_something_with_input(self.input)


This will at each iteration over the list of inputs do something with the input.

Test Templates

Test templates assist you in writing tests. They already contain all the common functionality that is useful to running a test of that type. They also take care of writing the data they collect that is relevant to the test run to the report file.
Currently implemented test templates are ooni.templates.scapyt for tests based on Scapy, ooni.templates.tcpt for tests based on TCP, ooni.templates.httpt for tests based on HTTP, and ooni.templates.dnst for tests based on DNS.

Scapy based tests

Scapy based tests will be a subclass of ooni.templates.scapyt.BaseScapyTest.
It provides a wrapper around the scapy send and receive function that will write the sent and received packets to the report with sanitization of the src and destination IP addresses.
It has the same syntax as the Scapy sr function, except that it will return a deferred.
To implement a simple ICMP ping based on this function you can do like so (Taken from nettest.examples.example_scapyt.ExampleICMPPingScapy)
from twisted.python import usage
from scapy.all import IP, ICMP
from ooni.templates import scapyt
class UsageOptions(usage.Options): optParameters = [['target', 't', '127.0.0.1', "Specify the target to ping"]]
class ExampleICMPPingScapy(scapyt.BaseScapyTest): name = "Example ICMP Ping Test"
usageOptions = UsageOptions
def test_icmp_ping(self): def finished(packets): print packets answered, unanswered = packets for snd, rcv in answered: rcv.show()
packets = IP(dst=self.localOptions['target'])/ICMP() d = self.sr(packets) d.addCallback(finished) return d


The arguments taken by self.sr() are exactly the same as the scapy send and receive function, the only difference is that instead of using the regular scapy super socket it uses our twisted driven wrapper around it.
Alternatively this test can also be written using the twisted.defer.inlineCallbacks() decorator, that makes it look more similar to regular sequential code.
from twisted.python import usage
from twisted.internet import defer
from scapy.all import IP, ICMP
from ooni.templates import scapyt
class UsageOptions(usage.Options): optParameters = [['target', 't', '127.0.0.1', "Specify the target to ping"]]
class ExampleICMPPingScapyYield(scapyt.BaseScapyTest): name = "Example ICMP Ping Test"
usageOptions = UsageOptions
@defer.inlineCallbacks def test_icmp_ping(self): packets = IP(dst=self.localOptions['target'])/ICMP() answered, unanswered = yield self.sr(packets) for snd, rcv in answered: rcv.show()


Report Format

###########################################
# OONI Probe Report for Example ICMP Ping Test test
# Thu Nov 22 18:20:43 2012
###########################################
---
{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha,
  start_time: 1353601243.0, test_name: Example ICMP Ping Test, test_version: 0.1}
...
---
input: null
report:
  answer_flags: [ipsrc]
  answered_packets:
  - - raw_packet: !!binary |
        RQAAHAEdAAAuAbjKCAgICH8AAAEAAAAAAAAAAA==
      summary: IP / ICMP 8.8.8.8 > 127.0.0.1 echo-reply 0
  sent_packets:
  - - raw_packet: !!binary |
        RQAAHAABAABAAevPfwAAAQgICAgIAPf/AAAAAA==
      summary: IP / ICMP 127.0.0.1 > 8.8.8.8 echo-request 0
test_name: test_icmp_ping
test_started: 1353604843.553605
...


TCP based tests

TCP based tests will subclass ooni.templates.tcpt.TCPTest.
This test template facilitates the sending of TCP payloads to the wire and recording the response.
from twisted.internet.error import ConnectionRefusedError
from ooni.utils import log
from ooni.templates import tcpt
class ExampleTCPT(tcpt.TCPTest): def test_hello_world(self): def got_response(response): print "Got this data %s" % response
def connection_failed(failure): failure.trap(ConnectionRefusedError) print "Connection Refused"
self.address = "127.0.0.1" self.port = 57002 payload = "Hello World!\n\r" d = self.sendPayload(payload) d.addErrback(connection_failed) d.addCallback(got_response) return d


The possible failures for a TCP connection are:
twisted.internet.error.NoRouteError that corresponds to errno.ENETUNREACH
twisted.internet.error.ConnectionRefusedError that corresponds to errno.ECONNREFUSED
twisted.internet.error.TCPTimedOutError that corresponds to errno.ETIMEDOUT

Report format

The basic report of a TCP test looks like the following (this is an report generated by running the above example against a TCP echo server).
###########################################
# OONI Probe Report for Base TCP Test test
# Thu Nov 22 18:18:28 2012
###########################################
---
{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha,
  start_time: 1353601108.0, test_name: Base TCP Test, test_version: '0.1'}
...
---
input: null
report:
  errors: []
  received: ["Hello World!\n\r"]
  sent: ["Hello World!\n\r"]
test_name: test_hello_world
test_started: 1353604708.705081
...


TODO finish this with more details

HTTP based tests

HTTP based tests will be a subclass of ooni.templates.httpt.HTTPTest.
It provides methods ooni.templates.httpt.HTTPTest.processResponseBody() and ooni.templates.httpt.HTTPTest.processResponseHeaders() for interacting with the response body and headers respectively.
For example, to implement a HTTP test that returns the sha256 hash of the response body (based on nettests.examples.example_httpt):
from ooni.utils import log
from ooni.templates import httpt
from hashlib import sha256
class SHA256HTTPBodyTest(httpt.HTTPTest): name = "ChecksumHTTPBodyTest" author = "Aaron Gibson" version = 0.1
inputFile = ['url file', 'f', None, 'List of URLS to perform GET requests to'] requiredOptions = ['url file']
def test_http(self): if self.input: url = self.input return self.doRequest(url) else: raise Exception("No input specified")
def processResponseBody(self, body): body_sha256sum = sha256(body).hexdigest() self.report['checksum'] = body_sha256sum


Report format

###########################################
# OONI Probe Report for ChecksumHTTPBodyTest test
# Thu Dec  6 17:31:57 2012
###########################################
---
options:
  collector: null
  help: 0
  logfile: null
  pcapfile: null
  reportfile: null
  resume: 0
  subargs: [-f, hosts]
  test: nettests/examples/example_http_checksum.py
probe_asn: null
probe_cc: null
probe_ip: 127.0.0.1
software_name: ooniprobe
software_version: 0.0.7.1-alpha
start_time: 1354786317.0
test_name: ChecksumHTTPBodyTest
test_version: 0.1
...
---
input: http://www.google.com
report:
  agent: agent
  checksum: d630fa2efd547d3656e349e96ff7af5496889dad959e8e29212af1ff843e7aa1
  requests:
  - request:
      body: null
      headers:
      - - User-Agent
        - - [Opera/9.00 (Windows NT 5.1; U; en), 'Opera 9.0, Windows XP']
      method: GET
      url: http://www.google.com
    response:
      body: '<!doctype html><html ... snip ...  </html>'
      code: 200
      headers:
      - - X-XSS-Protection
        - [1; mode=block]
      - - Set-Cookie
        - ['PREF=ID=fada4216eb3684f9:FF=0:TM=1354800717:LM=1354800717:S=IT-2GCkNAocyXlVa;
            expires=Sat, 06-Dec-2014 13:31:57 GMT; path=/; domain=.google.com', 'NID=66=KWaLbNQumuGuYf0HrWlGm54u9l-DKJwhFCMQXfhQPZM-qniRhmF6QRGXUKXb_8CIUuCOHnyoC5oAX5jWNrsfk-LLJLW530UiMp6hemTtDMh_e6GSiEB4GR3yOP_E0TCN;
            expires=Fri, 07-Jun-2013 13:31:57 GMT; path=/; domain=.google.com; HttpOnly']
      - - Expires
        - ['-1']
      - - Server
        - [gws]
      - - Connection
        - [close]
      - - Cache-Control
        - ['private, max-age=0']
      - - Date
        - ['Thu, 06 Dec 2012 13:31:57 GMT']
      - - P3P
        - ['CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
            for more info."']
      - - Content-Type
        - [text/html; charset=UTF-8]
      - - X-Frame-Options
        - [SAMEORIGIN]
  socksproxy: null
test_name: test_http
test_runtime: 0.08298492431640625
test_started: 1354800717.478403
...


DNS based tests

DNS based tests will be a subclass of ooni.templates.dnst.DNSTest.
It provides methods ooni.templates.dnst.DNSTest.performPTRLookup() and ooni.templates.dnst.DNSTest.performALookup()
For example (taken from nettests.examples.example_dnst):
from ooni.templates.dnst import DNSTest
class ExampleDNSTest(DNSTest): def test_a_lookup(self): def gotResult(result): # Result is an array containing all the A record lookup results print result
d = self.performALookup('torproject.org', ('8.8.8.8', 53)) d.addCallback(gotResult) return d


Report format

###########################################
# OONI Probe Report for Base DNS Test test
# Thu Dec  6 17:42:51 2012
###########################################
---
options:
  collector: null
  help: 0
  logfile: null
  pcapfile: null
  reportfile: null
  resume: 0
  subargs: []
  test: nettests/examples/example_dnst.py
probe_asn: null
probe_cc: null
probe_ip: 127.0.0.1
software_name: ooniprobe
software_version: 0.0.7.1-alpha
start_time: 1354786971.0
test_name: Base DNS Test
test_version: 0.1
...
---
input: null
report:
  queries:
  - addrs: [82.195.75.101, 86.59.30.40, 38.229.72.14, 38.229.72.16]
    answers:
    - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=82.195.75.101
        ttl=782>]
    - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=86.59.30.40
        ttl=782>]
    - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.14
        ttl=782>]
    - [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.16
        ttl=782>]
    query: '[Query(''torproject.org'', 1, 1)]'
    query_type: A
    resolver: [8.8.8.8, 53]
test_name: test_a_lookup
test_runtime: 0.028924942016601562
test_started: 1354801371.980114
...


For a more complex example, see: nettests.blocking.dnsconsistency

GLOSSARY

Here we will summarize some of the jargon that is unique to OONI.
Test Case: a set of measurements performed on a to be tested network that are logically grouped together
Report: is the output of a test run containing all the information that is require for a researcher to assess what is the output of a test.
Yamlooni: The format we use for Reports, that is based on YAML.
Input: What is given as input to a TestCase to perform a measurement.

AUTHOR

The Tor Project

COPYRIGHT

2016, The Tor Project
April 26, 2016 1.4.0