Giter VIP home page Giter VIP logo

nullhypothesis / exitmap Goto Github PK

View Code? Open in Web Editor NEW
458.0 41.0 107.0 1.55 MB

A fast and modular scanner for Tor exit relays. The canonical repository (including issue tracker) is at https://gitlab.torproject.org/tpo/network-health/exitmap

Home Page: https://www.cs.kau.se/philwint/spoiled_onions/

License: GNU General Public License v3.0

Python 100.00%
tor python scanner security tor-network

exitmap's Introduction

exitmap logo

This repository is unmaintained. Refer to The Tor Project's fork.

Overview

Exitmap is a fast and modular Python-based scanner for Tor exit relays. Exitmap modules implement tasks that are run over (a subset of) all exit relays. If you have a background in functional programming, think of exitmap as a map() interface for Tor exit relays: Modules can perform any TCP-based networking task like fetching a web page, uploading a file, connecting to an SSH server, or joining an IRC channel.

In practice, exitmap is useful to monitor the reliability and trustworthiness of exit relays. The Tor Project uses exitmap to check for false negatives on the Tor Project's check service and to find malicious exit relays. It is easy to develop new modules for exitmap; just have a look at the file HACKING in the doc/ directory or check out one of the existing modules.

Exitmap uses Stem to create circuits to all given exit relays. Each time tor notifies exitmap of an established circuit, a module is invoked for the newly established circuit. Modules can be pure Python scripts or executables. For executables, torsocks is necessary.

Finally, note that exitmap is a network measurement tool and of little use to ordinary Tor users. The Tor Project is already running the tool regularly. More exitmap scans just cause unnecessary network load. The only reason exitmap is publicly available is because its source code and design might be of interest to some.

Installation

Exitmap uses the library Stem to communicate with Tor. There are plenty of ways to install Stem. The easiest might be to use pip in combination with the provided requirements.txt file:

$ pip install -r requirements.txt

Running exitmap

The only argument exitmap requires is the name of a module. For example, you can run exitmap with the checktest module by running:

$ ./bin/exitmap checktest

The command line output will then show you how Tor bootstraps, the output of the checktest module, and a scan summary. If you don't need three hops and prefer to use two hops with a static first hop, run:

$ ./bin/exitmap --first-hop 1234567890ABCDEF1234567890ABCDEF12345678 checktest

To run the same test over German exit relays only, execute:

$ ./bin/exitmap --country DE --first-hop 1234567890ABCDEF1234567890ABCDEF12345678 checktest

If you want to pause for five seconds in between circuit creations to reduce the load on the Tor network and the scanning destination, run:

$ ./bin/exitmap --build-delay 5 checktest

Note that 1234567890ABCDEF1234567890ABCDEF12345678 is a pseudo fingerprint that you should replace with an exit relay that you control.

To learn more about all of exitmap's options, run:

$ ./bin/exitmap --help

Exitmap comes with batteries included, providing the following modules:

  • testfds: Tests if an exit relay is able to fetch the content of a simple web page. If an exit relay is unable to do that, it might not have enough file descriptors available.
  • checktest: Attempts to find false negatives in the Tor Project's check service.
  • dnspoison: Attempts to resolve several domains and compares the received DNS A records to the expected records.
  • dnssec: Detects exit relays whose resolver does not validate DNSSEC.
  • patchingCheck: Checks for file tampering.
  • cloudflared: Checks if a web site returns a CloudFlare CAPTCHA.
  • rtt: Measure round-trip times through an exit to various destinations.

Configuration

By default, exitmap tries to read the file .exitmaprc in your home directory. The file accepts all command line options, but you have to replace minuses with underscores. Here is an example:

[Defaults]
first_hop = 1234567890ABCDEF1234567890ABCDEF12345678
verbosity = debug
build_delay = 1
analysis_dir = /path/to/exitmap_scans

Alternatives

Don't like exitmap? Then have a look at tortunnel, SoaT, torscanner, DetecTor, or SelekTOR.

Tests

Before submitting pull requests, please make sure that all unit tests pass by running:

$ pip install -r requirements-dev.txt
$ py.test --cov-report term-missing --cov-config .coveragerc --cov=src test

Feedback

Contact: Philipp Winter [email protected]
OpenPGP fingerprint: B369 E7A2 18FE CEAD EB96 8C73 CF70 89E3 D7FD C0D0

exitmap's People

Contributors

apexhat avatar arlolra avatar atagar avatar donnchac avatar gunesacar avatar intchloe avatar kushaldas avatar loganaden avatar manuteleco avatar nogoegst avatar nullhypothesis avatar oorestisime avatar qbi avatar radman404 avatar s4chin avatar zackw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

exitmap's Issues

Exitmap fails when circuits are "overused"

Exitmap seems to crash whenever "too much" is done over a circuit. It's not clear what "too much" means, exactly. It has happened with a module that fetched several X.509 certificates, and apparently with another module that resolved a large set of domains over a simple exit relay.

First, the problem must be narrowed down. It's not clear if the issue is in exitmap, Stem, or tor. The nature of the problem is also not clear yet. Does it happen when too much data is sent, or if a given timeout is reached?

Bind Tor to dynamically determined ports

At this point, we use 45678 as SOCKS port and 45679 as control port. If there's already a local application bound to either one of these ports, invoking the Tor process will fail. To fix this, the SOCKS port and control port should be determined dynamically.

ValueError: Extra-info digests should consist of forty hex digits

When trying to get exitmap to run and executing 'checktest' I get the following error:

./exitmap checktest
2016-03-01 11:47:06,584 [INFO]: Attempting to invoke Tor process in directory "/tmp/exitmap_tor_datadir-jurre".  This might take a while.
2016-03-01 11:47:06,585 [INFO]: No first hop given.  Using randomly determined first hops for circuits.
2016-03-01 11:47:06,853 [INFO]: Tor Bootstrapped 0%: Starting
2016-03-01 11:47:07,855 [INFO]: Tor Bootstrapped 5%: Connecting to directory server
2016-03-01 11:47:07,942 [INFO]: Tor Bootstrapped 10%: Finishing handshake with directory server
2016-03-01 11:47:08,261 [INFO]: Tor Bootstrapped 15%: Establishing an encrypted directory connection
2016-03-01 11:47:08,352 [INFO]: Tor Bootstrapped 20%: Asking for networkstatus consensus
2016-03-01 11:47:08,453 [INFO]: Tor Bootstrapped 25%: Loading networkstatus consensus
2016-03-01 11:47:16,315 [INFO]: Tor Bootstrapped 40%: Loading authority key certs
2016-03-01 11:47:17,106 [INFO]: Tor Bootstrapped 45%: Asking for relay descriptors
2016-03-01 11:47:17,712 [INFO]: Tor Bootstrapped 50%: Loading relay descriptors
2016-03-01 11:47:54,291 [INFO]: Tor Bootstrapped 55%: Loading relay descriptors
2016-03-01 11:47:55,107 [INFO]: Tor Bootstrapped 61%: Loading relay descriptors
2016-03-01 11:47:55,654 [INFO]: Tor Bootstrapped 69%: Loading relay descriptors
2016-03-01 11:47:55,911 [INFO]: Tor Bootstrapped 75%: Loading relay descriptors
2016-03-01 11:47:56,159 [INFO]: Tor Bootstrapped 80%: Connecting to the Tor network
2016-03-01 11:47:56,159 [INFO]: Successfully started Tor process (PID=4656).
2016-03-01 11:47:56,272 [INFO]: Running module 'checktest'.
Traceback (most recent call last):
  File "./exitmap", line 32, in <module>
    sys.exit(main())
  File "/home/jurre/dev/exitmap/src/exitmap.py", line 260, in main
    run_module(module_name, args, controller, socks_port, stats)
  File "/home/jurre/dev/exitmap/src/exitmap.py", line 341, in run_module
    exit_relays = select_exits(args, module)
  File "/home/jurre/dev/exitmap/src/exitmap.py", line 302, in select_exits
    hosts=hosts)
  File "/home/jurre/dev/exitmap/src/relayselector.py", line 109, in get_exits
    for desc in stem.descriptor.parse_file(cached_descriptors_path):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/__init__.py", line 161, in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/__init__.py", line 226, in _parse_file_for_path
    for desc in parse_file(desc_file, *args, **kwargs):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/__init__.py", line 211, in parse_file
    for desc in file_parser(descriptor_file):
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 157, in _parse_file
    yield RelayDescriptor(descriptor_text, validate, annotations, **kwargs)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 665, in __init__
    super(RelayDescriptor, self).__init__(raw_contents, validate, annotations)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 302, in __init__
    self._parse(entries, validate)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 819, in _parse
    ServerDescriptor._parse(self, entries, validate)
  File "/usr/lib/python2.7/dist-packages/stem/descriptor/server_descriptor.py", line 494, in _parse
    raise ValueError('Extra-info digests should consist of forty hex digits: %s' % value)
ValueError: Extra-info digests should consist of forty hex digits: 17FCF4A1A749301A0B4710323ABE1C123DC30824 p3Hrg+Cn86Vc13zsyFYDBtETPZJC/fKphySA3R5WI0A

Inform users that it can take a while to bootstrap

When starting exitmap for the first time, its data directory is empty. As a result, it can take quite a while to bootstrap, and we might hit the timeout. We should increase the timeout and add log messages to inform the user about what's going on. Maybe there's something else we can do?

Making it compatible with Whonix

Right now it's not possible to run exitmap in Whonix,

2017-03-13 12:43:09,944 exitmap [INFO] Attempting to invoke Tor process in directory "/tmp/exitmap_tor_datadir-user".  This might take a while.
2017-03-13 12:43:09,944 exitmap [INFO] No first hop given.  Using randomly determined first hops for circuits.
2017-03-13 12:43:09,982 exitmap [ERROR] Couldn't launch Tor: Process terminated: Timed out.  Maybe try again?

Add support for continuous scanning

In addition to one-off scans, it would be great to have support for continuous scanning. exitmap should then download the hourly consensus and have a smart way to scan new relays and re-scan existing relays.

dnspoison module: IPv6 addresses cause false positives

When the exit relay returns IPv6 addresses for a tested domain the dnspoison module claims to have found a DNS poisoning exit, but this is a false positive since the module does not support IPv6 and interprets the returned data as IPv4 address.

This is causing almost daily false positives.

Failed to attach stream because of unknown circuit.

Sometimes, the following warning message shows up:
2015-08-13 10:57:43,703 [WARNING]: Failed to attach stream because: Unknown circuit "870"

It is not clear why the circuit was unknown to Stem. Was it torn down by a relay in the circuit?

Pep8 Violations

There are a various amount of pep8 violations that could be fixed. Some of these include removing ununsed imports, etc.

Detect Tor Exit doing sniffing by passively detecting unique DNS query (via HTML & PCAP parsing/viewing)

This ticket is to detect Tor Exit doing sniffing by passively detecting HTML link/url/js/css parsers & viewer by unique DNS.

The idea is to generate HTML content over HTTP that contain in several tags (link, url, js, css, image inclusion, etc) URIs that point to a specific DNS entry, done for that specific Tor exit, that if loaded will enable to detect passively a sniffer.

The idea is that a sniffer, after it store the data, will have some piece of software processing it automatically and/or at a later stage a viewer component of a sniffer software that will enable the "sniffer operators" looking at the "sniffed data" to see an offline copy of the traffic.

If the software processing the sniffed data with HTML content or the viewer will try to load any external resources, not present in the traffic sniffed, it will trigger a DNS request.

A sub-domain with a wildcard such as *.testme.domain.org could be configured so that all requests will go that specific DNS where a DNS server with logging (or some custom Twisted DNS server code) is enabled.

When scanning Tor exit with fingerprint XXXX1234, all the URIs included in the HTML content being transmitted could be XXXX1234.testme.domain.org .

Whenever a dns resolution query would come to XXXX1234.testme.domain.org is surely because some piece of software sniffed traffic on Tor Exit XXXX1234 and then processed it.

The DNS hostname used, could also include a "timestamp" to know when it has been sniffed, and by looking and timestamp when the DNS query has been received it's possible to know when the HTML content has been processed and/or viewed.

HTML content could be transported also over other protocols, such as POP3/IMAP with RFC822/MIME format or FTP by downloading and/or uploading an HTML file somewhere.

Provide API to mimic TorBrowser HTTP requests

Sometimes, we want to issue HTTP requests. Passive adversaries should have a hard time distinguishing our HTTP requests (fairly obvious user agent) from TorBrowser requests. Instead of dealing with this issue in every module independently, there should be an API for that. Adding the necessary HTTP headers using urllib2's add_header() methods might be good enough.

run_cmd_over_tor lacking examples

So I was making a module that uses the openssl command as it was a lot more complicated to do it in just python. I was just using subprocess and run_python_over_tor until I realized it wasn't actually running the command over tor so I went to use run_cmd_over_tor.
It hasn't been used in any of the other modules or anything I could find on google so I spent some time reading it and here is a simple module for testing I came up with:

import logging
log = logging.getLogger(__name__)

def print_callback(input,proc_kill):
    log.info('Command Output: ' + input)
def probe(exit_desc, run_python_over_tor, run_cmd_over_tor, **kwargs):
    run_cmd_over_tor.execute(['curl','-s','https://wtfismyip.com/text'],output_callback=print_callback)

The only output I could get with it (even when the command was changed to something like pwd):

2016-08-28 00:37:10,903 modules.test [INFO] Command Output: [Aug 28 00:37:10] DEBUG torsocks[18470]: Logging subsytem initialized. Level 5, file (null), time 1 (in init_logging() at torsocks.c:286)
I would just like to be able to get all the output after the command completes.
Any help would be appreciated :)

Log which circuits failed during scan.

We should log the exit relays for which we were unable to create circuits during a scan. Perhaps this information should even be logged to disk, so it's easier to later find relays for which circuits failed repeatedly.

Add module to verify a relay's exit policy

Relays can lie about their exit policy, e.g., they claim that they exit to port 80 and 443, but won't actually do it when they receive data from clients. We should have an exitmap module that verifies that a relay follows its own policy.

This could be done by attempting to establish TCP connections to a dedicated scanning host that has all of its port open. That would make it possible for relays to game the system, however, by whitelisting this dedicated scanning host and still refusing to exit to other IP addresses.

Handle multiple parallel instances better

At the moment, it's not possible to run several exitmap instances at the same time. It shouldn't be too hard to add that functionality. It might be sufficient to create separate Tor data directories for every exitmap instance.

In the meanwhile, it would be good if exitmap would refuse to start if another instance is already running.

Warn user when Stem version is too old

When using a Stem version that is too old, we run into the error:
ValueError: Extra-info digests should consist of forty hex digits: ...
Exitmap should check Stem's version number and warn when it's too old, instead of throwing an obscure exception.

Use Tor Browser programmatically for scanning

We should investigate to which degree we can use Tor Browser programmatically instead of just mimicing its request headers. That would help us detect attacks that target Tor Browser.

Make modules parameterised

It should be possible to pass parameters to modules. For example, the DNS module could then be invoked with custom domains and their according IP addresses:

exitmap dns -d www.example.com 1.2.3.4

Increase test coverage

Test coverage is almost zero at this point, which means that there's quite a bit of code that has never been executed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.