nullhypothesis / exitmap Goto Github PK

In addition to one-off scans, it would be great to have support for continuous scanning. exitmap should then download the hourly consensus and have a smart way to scan new relays and re-scan existing relays.

After iterating over all exit relays, scan statistics should be shown.

Exitmap seems to crash whenever "too much" is done over a circuit. It's not clear what "too much" means, exactly. It has happened with a module that fetched several X.509 certificates, and apparently with another module that resolved a large set of domains over a simple exit relay.

First, the problem must be narrowed down. It's not clear if the issue is in exitmap, Stem, or tor. The nature of the problem is also not clear yet. Does it happen when too much data is sent, or if a given timeout is reached?

Sometimes, exitmap won't delete all temporary pymp directories in /tmp/. This might only happen in case of a non-clean shutdown. Either way, it should be fixed.

There are a various amount of pep8 violations that could be fixed. Some of these include removing ununsed imports, etc.

When trying to get exitmap to run and executing 'checktest' I get the following error:

./exitmap checktest
2016-03-01 11:47:06,584 [INFO]: Attempting to invoke Tor process in directory "/tmp/exitmap_tor_datadir-jurre".  This might take a while.
2016-03-01 11:47:06,585 [INFO]: No first hop given.  Using randomly determined first hops for circuits.
2016-03-01 11:47:06,853 [INFO]: Tor Bootstrapped 0%: Starting
2016-03-01 11:47:07,855 [INFO]: Tor Bootstrapped 5%: Connecting to directory server
2016-03-01 11:47:07,942 [INFO]: Tor Bootstrapped 10%: Finishing handshake with directory server
2016-03-01 11:47:08,261 [INFO]: Tor Bootstrapped 15%: Establishing an encrypted directory connection
2016-03-01 11:47:08,352 [INFO]: Tor Bootstrapped 20%: Asking for networkstatus consensus
2016-03-01 11:47:08,453 [INFO]: Tor Bootstrapped 25%: Loading networkstatus consensus
2016-03-01 11:47:16,315 [INFO]: Tor Bootstrapped 40%: Loading authority key certs
2016-03-01 11:47:17,106 [INFO]: Tor Bootstrapped 45%: Asking for relay descriptors
2016-03-01 11:47:17,712 [INFO]: Tor Bootstrapped 50%: Loading relay descriptors
2016-03-01 11:47:54,291 [INFO]: Tor Bootstrapped 55%: Loading relay descriptors
2016-03-01 11:47:55,107 [INFO]: Tor Bootstrapped 61%: Loading relay descriptors
2016-03-01 11:47:55,654 [INFO]: Tor Bootstrapped 69%: Loading relay descriptors
2016-03-01 11:47:55,911 [INFO]: Tor Bootstrapped 75%: Loading relay descriptors
2016-03-01 11:47:56,159 [INFO]: Tor Bootstrapped 80%: Connecting to the Tor network
2016-03-01 11:47:56,159 [INFO]: Successfully started Tor process (PID=4656).
2016-03-01 11:47:56,272 [INFO]: Running module 'checktest'.
Traceback (most recent call last):
  File "./exitmap", line 32, in <module>
    sys.exit(main())
  File "/home/jurre/dev/exitmap/src/exitmap.py", line 260, in main
    run_module(module_name, args, controller, socks_port, stats)
  File "/home/jurre/dev/exitmap/src/exitmap.py", line 341, in run_module
    exit_relays = select_exits(args, module)
  File "/home/jurre/dev/exitmap/src/exitmap.py", line 302, in select_exits
    hosts=hosts)
  File "/home/jurre/dev/exitmap/src/relayselector.py", line 109, in get_exits
    for desc in stem.descriptor.parse_file(cached_descriptors_path):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/__init__.py", line 161, in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/__init__.py", line 226, in _parse_file_for_path
    for desc in parse_file(desc_file, *args, **kwargs):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/__init__.py", line 211, in parse_file
    for desc in file_parser(descriptor_file):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 157, in _parse_file
    yield RelayDescriptor(descriptor_text, validate, annotations, **kwargs)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 665, in __init__
    super(RelayDescriptor, self).__init__(raw_contents, validate, annotations)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 302, in __init__
    self._parse(entries, validate)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 819, in _parse
    ServerDescriptor._parse(self, entries, validate)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 494, in _parse
    raise ValueError('Extra-info digests should consist of forty hex digits: %s' % value)
ValueError: Extra-info digests should consist of forty hex digits: 17FCF4A1A749301A0B4710323ABE1C123DC30824 p3Hrg+Cn86Vc13zsyFYDBtETPZJC/fKphySA3R5WI0A

Relays can lie about their exit policy, e.g., they claim that they exit to port 80 and 443, but won't actually do it when they receive data from clients. We should have an exitmap module that verifies that a relay follows its own policy.

This could be done by attempting to establish TCP connections to a dedicated scanning host that has all of its port open. That would make it possible for relays to game the system, however, by whitelisting this dedicated scanning host and still refusing to exit to other IP addresses.

If a relay only allows exiting to port 80, and thus doesn't get an exit flag, exitmap won't scan it, and it won't get checked for .exe patching when i'm scanning with https://github.com/leviathansecurity/exitmap/blob/master/src/modules/patchingCheck.py

Hi,

2016-01-09 14:38:44,700 [ERROR]: Found false negative for https://atlas.torproject.org/#details/816CBF7FCF565F87195C6618FB2FAF8AE71B99F4. Desc addr is 120.56.163.114 and check addr is 120.56.163.114.

Ran a checktest and got a hit but the des addr and check addr were the same.

I think there is an issue.

Currently digging further.

We should investigate to which degree we can use Tor Browser programmatically instead of just mimicing its request headers. That would help us detect attacks that target Tor Browser.

While testing the dnspoison module, dnspython was a requirement, which should be included in the requirements.txt file for exitmap.

At the moment, it's not possible to run several exitmap instances at the same time. It shouldn't be too hard to add that functionality. It might be sufficient to create separate Tor data directories for every exitmap instance.

In the meanwhile, it would be good if exitmap would refuse to start if another instance is already running.

The GeoIP code is slow and comes with licensing weirdness. Instead, Maxmind's API can be used or Onionoo can be queried with the country option: https://onionoo.torproject.org/protocol.html

When the exit relay returns IPv6 addresses for a tested domain the dnspoison module claims to have found a DNS poisoning exit, but this is a false positive since the module does not support IPv6 and interprets the returned data as IPv4 address.

This is causing almost daily false positives.

It should be possible to pass parameters to modules. For example, the DNS module could then be invoked with custom domains and their according IP addresses:

exitmap dns -d www.example.com 1.2.3.4

When starting exitmap for the first time, its data directory is empty. As a result, it can take quite a while to bootstrap, and we might hit the timeout. We should increase the timeout and add log messages to inform the user about what's going on. Maybe there's something else we can do?

So I was making a module that uses the openssl command as it was a lot more complicated to do it in just python. I was just using subprocess and run_python_over_tor until I realized it wasn't actually running the command over tor so I went to use run_cmd_over_tor.
It hasn't been used in any of the other modules or anything I could find on google so I spent some time reading it and here is a simple module for testing I came up with:

import logging
log = logging.getLogger(__name__)

def print_callback(input,proc_kill):
    log.info('Command Output: ' + input)
def probe(exit_desc, run_python_over_tor, run_cmd_over_tor, **kwargs):
    run_cmd_over_tor.execute(['curl','-s','https://wtfismyip.com/text'],output_callback=print_callback)

The only output I could get with it (even when the command was changed to something like pwd):

2016-08-28 00:37:10,903 modules.test [INFO] Command Output: [Aug 28 00:37:10] DEBUG torsocks[18470]: Logging subsytem initialized. Level 5, file (null), time 1 (in init_logging() at torsocks.c:286)
I would just like to be able to get all the output after the command completes.
Any help would be appreciated :)

We might also want to be able to submit reports to OONI's collector.

Test coverage is almost zero at this point, which means that there's quite a bit of code that has never been executed.

This ticket is to Improve transparent HTTP proxy detection with SSL test on port 80 as described on https://trac.torproject.org/projects/tor/ticket/2697 .

phw: naif: so far, there's only a module that looks for the "Via" http header, so there's room for improvement.

We should log the exit relays for which we were unable to create circuits during a scan. Perhaps this information should even be logged to disk, so it's easier to later find relays for which circuits failed repeatedly.

Right now it's not possible to run exitmap in Whonix,

2017-03-13 12:43:09,944 exitmap [INFO] Attempting to invoke Tor process in directory "/tmp/exitmap_tor_datadir-user".  This might take a while.
2017-03-13 12:43:09,944 exitmap [INFO] No first hop given.  Using randomly determined first hops for circuits.
2017-03-13 12:43:09,982 exitmap [ERROR] Couldn't launch Tor: Process terminated: Timed out.  Maybe try again?

At this point, we use 45678 as SOCKS port and 45679 as control port. If there's already a local application bound to either one of these ports, invoking the Tor process will fail. To fix this, the SOCKS port and control port should be determined dynamically.

This ticket is to detect Tor Exit doing sniffing by passively detecting HTML link/url/js/css parsers & viewer by unique DNS.

The idea is to generate HTML content over HTTP that contain in several tags (link, url, js, css, image inclusion, etc) URIs that point to a specific DNS entry, done for that specific Tor exit, that if loaded will enable to detect passively a sniffer.

The idea is that a sniffer, after it store the data, will have some piece of software processing it automatically and/or at a later stage a viewer component of a sniffer software that will enable the "sniffer operators" looking at the "sniffed data" to see an offline copy of the traffic.

If the software processing the sniffed data with HTML content or the viewer will try to load any external resources, not present in the traffic sniffed, it will trigger a DNS request.

A sub-domain with a wildcard such as *.testme.domain.org could be configured so that all requests will go that specific DNS where a DNS server with logging (or some custom Twisted DNS server code) is enabled.

When scanning Tor exit with fingerprint XXXX1234, all the URIs included in the HTML content being transmitted could be XXXX1234.testme.domain.org .

Whenever a dns resolution query would come to XXXX1234.testme.domain.org is surely because some piece of software sniffed traffic on Tor Exit XXXX1234 and then processed it.

The DNS hostname used, could also include a "timestamp" to know when it has been sniffed, and by looking and timestamp when the DNS query has been received it's possible to know when the HTML content has been processed and/or viewed.

HTML content could be transported also over other protocols, such as POP3/IMAP with RFC822/MIME format or FTP by downloading and/or uploading an HTML file somewhere.

When using a Stem version that is too old, we run into the error:
ValueError: Extra-info digests should consist of forty hex digits: ...
Exitmap should check Stem's version number and warn when it's too old, instead of throwing an obscure exception.

Sometimes, the following warning message shows up:
2015-08-13 10:57:43,703 [WARNING]: Failed to attach stream because: Unknown circuit "870"

It is not clear why the circuit was unknown to Stem. Was it torn down by a relay in the circuit?

Sometimes, we want to issue HTTP requests. Passive adversaries should have a hard time distinguishing our HTTP requests (fairly obvious user agent) from TorBrowser requests. Instead of dealing with this issue in every module independently, there should be an API for that. Adding the necessary HTTP headers using urllib2's add_header() methods might be good enough.

It would be cool if there was an API to allow this functionality. If it is there, my apologies, wasn't apparent.