tribler / py-ipv8 Goto Github PK

View Code? Open in Web Editor NEW

222.0 25.0 48.0 4.14 MB

Python implementation of Tribler's IPv8 p2p-networking layer

License: GNU Lesser General Public License v3.0

Python 99.96% R 0.04%

python overlay p2p networking framework identity anonymity

py-ipv8's Introduction

FAQ:

Q: Is this the new official Layer 3 solution for the Internet?
A: No, the naming is a 10-years-old mockery of the deployment failure of IPv6 (which we sincerely hope will be deployed properly at some point in time).

Unit tests:

Mutation Tests:

Read the Docs:

What is IPv8 ?

IPv8 aims to provide authenticated communication with privacy. The design principle is to enable communication between public key pairs: IP addresses and physical network attachment points are abstracted away. This Python 3 package is an amalgamation of peer-to-peer communication functionality from Dispersy and Tribler, developed over the last 19 years by students and employees of the Delft University of Technology. The IPv8 library allows you to easily create network overlays on which to build your own applications.

IPv8 Objectives

Authentication. We offer mutual authentication using strong cryptography. During an IPv8 communication session, both parties can be sure of the other party’s identity. IPv8 users are identified by their public key. The initial key exchange is designed so that secrets are never transmitted across the Internet, not even in encrypted form. We use a standard challenge/response protocol with protection against spoofing, man-in-the-middle, and replay attacks.
Privacy. IPv8 is specifically designed for strong privacy protection and end-to-end encryption with perfect forward secrecy. We enhanced the industry standard onion routing protocol, Tor, for usage in a trustless environment (e.g. no trusted central directory servers).
No infrastructure dependency. Everybody is equal in the world of IPv8. No central web server, discovery server, or support foundation is needed.
Universal connectivity. IPv8 can establish direct communication in difficult network situations. This includes connecting people behind a NAT or firewall. IPv8 includes a single simple and effective NAT traversal technique: UDP hole-punching. This is essential when offering privacy without infrastructure and consumer-grade donated resources.
Trust. You can enhance your security if you tell IPv8 which people you know and trust. It tries to build a web-of-trust automatically.

Dependencies

The dependencies for IPv8 are collected in the requirements.txt file and can be installed using pip:

python3 -m pip install --upgrade -r requirements.txt

On Windows or MacOS you will need to install Libsodium separately, as explained here.

Tests

Running tests can be done by running:

python3 run_all_tests.py

Running the test suite on Python 3.7 requires the installation of asynctest (python3 -m pip install asynctest).

Running code coverage requires the coverage package (python3 -m pip install coverage). A coverage report can be generated by running:

python3 create_test_coverage_report.py

Getting started

You can start creating your first network overlay by following the overlay creation tutorial.

We provide additional documentation on configuration, key generation and message serialization formats on our ReadTheDocs page.

py-ipv8's People

Contributors

Stargazers

Watchers

py-ipv8's Issues

AttributeError: 'NoneType' object has no attribute 'execute'

IPv8 is having problem on shutting down. See the logs:

ERROR   1524667809.84              rest_manager:53   (RESTRequest)  [Failure instance: Traceback: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'execute'
/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py:571:dataReceived
/usr/lib/python2.7/dist-packages/twisted/web/http.py:1657:lineReceived
/usr/lib/python2.7/dist-packages/twisted/web/http.py:1736:allContentReceived
/usr/lib/python2.7/dist-packages/twisted/web/http.py:762:requestReceived
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/web/server.py:183:process
/usr/lib/python2.7/dist-packages/twisted/web/server.py:234:render
/usr/lib/python2.7/dist-packages/twisted/web/resource.py:250:render
/home/sandip/tudelft/tribler/Tribler/Core/Modules/restapi/trustchain_endpoint.py:87:render_GET
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py:14:helper
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py:26:blockingCallFromThread
/home/sandip/tudelft/tribler/Tribler/community/triblerchain/community.py:44:get_statistics
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py:80:get_latest
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py:49:_get
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py:40:wrapper
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py:302:execute
]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 183, in process
    self.render(resrc)
  File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 234, in render
    body = resrc.render(self)
  File "/usr/lib/python2.7/dist-packages/twisted/web/resource.py", line 250, in render
    return m(request)
  File "/home/sandip/tudelft/tribler/Tribler/Core/Modules/restapi/trustchain_endpoint.py", line 87, in render_GET
    return json.dumps({'statistics': triblerchain_community.get_statistics()})
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py", line 14, in helper
    return blockingCallFromThread(reactor, func, *args, **kargs)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py", line 26, in blockingCallFromThread
    return f(*args, **kwargs)
  File "/home/sandip/tudelft/tribler/Tribler/community/triblerchain/community.py", line 44, in get_statistics
    latest_block = self.persistence.get_latest(public_key)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 80, in get_latest
    u"WHERE public_key = ?)", (buffer(public_key), buffer(public_key)))
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 49, in _get
    db_result = list(self.execute(self.get_sql_header() + query, params, fetch_all=False))
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 40, in wrapper
    return f(self, *args, **kwargs)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 302, in execute
    result = self._cursor.execute(statement, bindings)
AttributeError: 'NoneType' object has no attribute 'execute'

Similar: Exception while handling packet #103

Add Attestation challenge batching

Right now all the Attestation challenges are being sent at once. This is bad for network health.

Also, this allows us to implement honesty batches.

TrustChain tests unstable

For some reason the Mac tests for TrustChain are unstable. This may have to do with a different commit protocol on Mac.

UPDATE: There seems to be a race condition in the test, also reproduced on Ubuntu 16.10.
Ran the following to produce:

#!/bin/bash
export PYTHONPATH=.
while $(nosetests test/attestation/trustchain/)
do
    echo ""
done

======================================================================
FAIL: Check if missing blocks are retrieved through a crawl request.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/utils.py", line 201, in runWithWarningsSuppressed
    reraise(exc_info[1], exc_info[2])
  File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/utils.py", line 197, in runWithWarningsSuppressed
    result = f(*a, **kw)
  File "/home/quinten/tribler/py-ipv8/test/util.py", line 125, in errback
    failure.raiseException()
  File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/quinten/tribler/py-ipv8/test/attestation/trustchain/test_community.py", line 229, in test_retrieve_missing_block
    self.assertEqual(self.nodes[node_nr].overlay.persistence.get(his_pubkey, 1).link_sequence_number, 1)
  File "/home/quinten/.local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 432, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
FailTest: 2 != 1
-------------------- >> begin captured logging << --------------------
twisted: INFO: --> test.attestation.trustchain.test_community.TestTrustChainCommunity.test_retrieve_missing_block <--
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 1024) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a56aa579d3ca2b931de20b71a50882643378309b95095633667cccee32bb1ed0af52876c04964339034d9f610d1a3994b4b01153e5cec670efb8049cff999fe3d
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 1024) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a67148d49f0e1d601147f6c9f1b5bc4c9d85095039b554d06b1d14a99fd0bd7213a6137048a89b167e45b00ca830a9c2077f99f21b1e6ed929cc3d19802e7d6c7
TrustChainCommunity: INFO: Signed block to 02e7d6c7 (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Signed block to 02e7d6c7 (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x7f91bfb33aa0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x7f91bfb33aa0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Request block could not be validated sufficiently, crawling requester. (<function no_info at 0x7f91bfb33aa0>, [])
RequestCache: DEBUG: add <CrawlRequestCache crawl-81589634>
TrustChainCommunity: INFO: Requesting crawl of node f999fe3d:1 with id 81589634
TrustChainCommunity: INFO: Received crawl request from node 02e7d6c7 for sequence number 1
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Sent 2 blocks
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x7f91bfb338c0>, [], (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x7f91bfb338c0>, [], (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Signed block to f999fe3d (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<156.112.38.66:56845, DTjdP/pkFJzvti2JZ7sXR8bVXWU=> (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainCommunity: DEBUG: Received already known block (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Signed block to f999fe3d (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<156.112.38.66:56845, DTjdP/pkFJzvti2JZ7sXR8bVXWU=> (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainCommunity: DEBUG: Received already known block (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------

Offer IPv8 as a service

Right now ipv8.py is and example file to invoke the modular framework. ipv8.py could be offered as a full-fledged modular service instead.

Strange message from Tribler

In Tribler someone is sending the following message:

STRUCT            DATA                       DESCRIPTION
====================================================================
DISPERSY_VERSION  00                         0
COMMUNITY_VERSION 02                         2
COMMUNITY_ID      7e 31 36 85 | c1 91 2a 14
                  12 79 f8 24 | 8f c8 db 58
                  99 c5 df 5a                DiscoveryCommunity
MESSAGE_ID        f6                         IntroductionRequest
GLOBAL_TIME       00 00 00 00 | 00 00 00 e5  229
DESTINATION       59 3d 0b 80 | 53 c7        (89.61.11.128, 21447)
LAN               ed c3 04 f2 | 20 37        (237.195.4.242, 8247)
WAN               97 ff ed 0c | 74 99        (151.255.237.12, 29849)
FLAGS             67                         (connection_type_0=0,
                                              connection_type_1=1,
                                              dflag0=1,
                                              dflag1=0,
                                              dflag2=0,
                                              tunnel=1,
                                              sync=1,
                                              advice=1)
IDENTIFIER        32 f3                      13043
TIME_LOW          91 5e 7f f5 | ce 13 c0 a8  10474950477011468456
TIME_HIGH         01 71 1e 4f | bc cc c3 82  103897594225017730
MODULO            1e 4f                      7759
OFFSET            01 84                      388
FUNCTIONS         9c                         10011100
BLOOMFILTER_SIZE  <N/A>                      MISSING!
BLOOMFILTER       <N/A>                      MISSING!

The contents seem like garbage, but this message has been correctly signed though.

Error when running IPV8

When running IPV8, I encountered a stack trace:

CRITICAL:root:Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/home/crawler/py-ipv8/ipv8/messaging/interfaces/endpoint.py", line 44, in _deliver_later
    listener.on_packet(packet)
  File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 237, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 58, in on_similarity_response
    auth, dist, payload = self._ez_unpack_auth(SimilarityResponsePayload, data)
  File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 43, in _ez_unpack_auth
    dist, payload, unknown_data = self.serializer.unpack_to_serializables(format, remainder[23:])
  File "/home/crawler/py-ipv8/ipv8/messaging/serialization.py", line 266, in unpack_to_serializables
    out.append(serializable.from_unpack_list(*unpack_list))
  File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery_payload.py", line 83, in from_unpack_list
    [(tb_overlap[i:i+20], unpack(">I", tb_overlap[i+20:i+24])) for i in range(0, len(tb_overlap), 24)]]
struct.error: unpack requires a string argument of length 4

Mac TrustChain tests failing

======================================================================
 test/attestation/trustchain/test_community.py:TestTrustChainCommunity
======================================================================
Test broadcasting a half block ... ok
Test broadcasting a half block pair ... ok
Check if a block can be crawled. ... ok
Check if the default crawl strategy produces blocks. ... ok
Check if a block can be crawled by negative range. ... ok
Check if blocks don't magically appear. ... ok
Check if a both halves of a fully signed block link to each other. ... ok
Check if blocks created in parallel will properly be stored in the database. ... ok
Check if missing blocks are retrieved through a crawl request. ... FAIL

======================================================================
FAIL: Check if missing blocks are retrieved through a crawl request.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/Library/Python/2.7/site-packages/twisted/internet/utils.py", line 201, in runWithWarningsSuppressed
    reraise(exc_info[1], exc_info[2])
  File "/Library/Python/2.7/site-packages/twisted/internet/utils.py", line 197, in runWithWarningsSuppressed
    result = f(*a, **kw)
  File "/Users/tribler/Documents/workspace/workspace/ipv8/test_ipv8_mac/test/util.py", line 125, in errback
    failure.raiseException()
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/Users/tribler/Documents/workspace/workspace/ipv8/test_ipv8_mac/test/attestation/trustchain/test_community.py", line 226, in test_retrieve_missing_block
    self.assertIsNotNone(self.nodes[node_nr].overlay.persistence.get(his_pubkey, 1))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 962, in assertIsNotNone
    self.fail(self._formatMessage(msg, standardMsg))
  File "/Library/Python/2.7/site-packages/twisted/trial/_synctest.py", line 368, in fail
    raise self.failureException(msg)
FailTest: unexpectedly None
-------------------- >> begin captured logging << --------------------
twisted: INFO: --> test_ipv8_mac.test.attestation.trustchain.test_community.TestTrustChainCommunity.test_retrieve_missing_block <--
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 4096) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a9774c700dd2bab1179c90a9ad454145177d3953a70832e8be75c36e4cdde846f0779fa63fd480cc169ec022825082068aeb8c0f48374ce6aef1a987d22325233
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 4096) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a30ae524d08c89b13a3fbb4bc93b880d2427014588a9b2011b0ef602cff8009434790236e887cd80ff5cc93c08ae1df095325421efff830c5626aab171f71fe53
TrustChainCommunity: INFO: Signed block to 1f71fe53 (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (215.136.3.36:40395) (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
RequestCache: DEBUG: add <HalfBlockSignCache sign-18142001>
TrustChainCommunity: INFO: Signed block to 1f71fe53 (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (215.136.3.36:40395) (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
RequestCache: DEBUG: add <HalfBlockSignCache sign-18142002>
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x10d417f50>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x10d417f50>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Request block could not be validated sufficiently, crawling requester. (<function no_info at 0x10d417f50>, [])
RequestCache: DEBUG: add <CrawlRequestCache crawl-95915699>
TrustChainCommunity: INFO: Requesting crawl of node 22325233:1 with id 95915699
TrustChainCommunity: INFO: Received crawl request from node 1f71fe53 for sequence number 1
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<215.136.3.36:40395, 34hUFsyOvKICXK8xv7aDqYDkPXE=> (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<215.136.3.36:40395, 34hUFsyOvKICXK8xv7aDqYDkPXE=> (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Sent 2 blocks
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x10d417d70>, [], (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x10d417d70>, [], (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Signed block to 22325233 (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (1.47.221.125:16905) (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Signed block to 22325233 (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (1.47.221.125:16905) (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainCommunity: DEBUG: Received already known block (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
TrustChainCommunity: DEBUG: Received already known block (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
--------------------- >> end captured logging << ---------------------

Add message handler chains

Right now programmers have to manually start waiting for new RequestCaches when handling messages. This leads to a lot of duplicate code. Instead, handler chains could be specified, with the message handlers returning the RequestCache __init__ arguments for the next cache.

For example, the declaration:

IntroductionRequest -> IntroductionResponse;

Could fire the IntroductionRequest_handler(message, request_cache=None) when an IntroductionRequest message is received and then construct a new RequestCache waiting for the IntroductionResponse to be received.

Add support for smaller primes in attestation library

Right now the smallest primes we can generate are 512 bit. This makes sense for RSA, but this leads to1024 bit EC keys, which is ridiculous. We can use the gensafeprime OpenSSL bindings for generating primes smaller than 512 bit.

import gensafeprime 
gensafeprime.generate(bitspace)

We will need to update the Jenkins Android builder and the requirements.txt.

TrustChain cache timeout traceback

There is a bug on TrustChain cache timeout.

INFO    1522056800.81          caches:66    Timeout for crawl with id 46814472
2018-03-26T11:33:20+0200 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 396, in startReactor
	    self.config, oldstdout, oldstderr, self.profiler, reactor)
	  File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
	    reactor.run()
	  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1243, in run
	    self.mainLoop()
	  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1252, in mainLoop
	    self.runUntilCurrent()
	--- <exception caught here> ---
	  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 878, in runUntilCurrent
	    call.func(*call.args, **call.kw)
	  File "/home/jenkins/tribler/Tribler/pyipv8/ipv8/requestcache.py", line 146, in _on_timeout
	    cache.on_timeout()
	  File "/home/jenkins/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/caches.py", line 67, in on_timeout
	    self.community.request_cache.pop(u"crawl", self.number)
	  File "/home/jenkins/tribler/Tribler/pyipv8/ipv8/requestcache.py", line 132, in pop
	    cache = self._identifiers.pop(identifier)
	exceptions.KeyError: u'crawl:46814472'

Explore: Linux kernel and hardware support for IPv8

Possible research directions: towards a fast clean-slate network stack or support multi-path routing out of the box. problem of protocol ossification. Prior work: how to receive a million packets per second. This is opposed to moving the IPv8 research to the database side of the stack. There is also the DHT, connect to anybody connect() primitive with network mobility.

Linux kernel support for UDP tunneling from 2015. Open Hardware support, like White Rabbit Network Interface Card. Scientific article a multi-port 10GbE PCIe NIC featuring UDP offload and GPUDirect capabilities. Other work Realtime Ethernet Device Driver.

Congestion control layer in/on-top of IPv8

Transport any binary blob:

Create human readable error message for Serializer

Right now error messages appear as follows:

error: unpack_from requires a buffer of at least 4 bytes

Combined with a stack trace, this is debuggable. However, a more human readable format can and should be made.

Update gitignore

Add the sqlite files and key files to the gitignore.

Create Android port

IPv8 should function as an Android app.

All of the dependencies have been compiled before for the Tribler project, these should/could be reused for the py-to-app compiler.

Refactor TunnelCommunity logic into Cell class

Right now we do a lot of operations on a cell. It would make sense to group this logic into a class of the same name.

TunnelCommunity send_destroy for None

Traceback (most recent call last):
  File "/home/quinten/Documents/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 296, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/home/quinten/Documents/tribler/Tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 797, in on_cell
    self.send_destroy(circuit.sock_addr, circuit_id, 0)
AttributeError: 'NoneType' object has no attribute 'sock_addr'

AttestationCommunity database error

There is an error when loading keys for the AttestationCommunity:

04-26 09:57:49.876 4441-4461/? I/IPV8Service: DEBUG:Launching AttestationCommunity with prefix 0002b42c93d167a0fc4a0843f917d4bf1e9ebb340ec4.
04-26 09:57:49.877 4441-4461/? I/IPV8Service: DEBUG:loading database [sqlite/attestations.db]
04-26 09:57:49.877 4441-4461/? I/IPV8Service: DEBUG:open database [sqlite/attestations.db]
04-26 09:57:49.907 4441-4461/? I/IPV8Service: DEBUG:PRAGMA page_size = 8192 (no change) [sqlite/attestations.db]
04-26 09:57:49.907 4441-4461/? I/IPV8Service: DEBUG:PRAGMA journal_mode = WAL (no change) [sqlite/attestations.db]
04-26 09:57:49.907 4441-4461/? I/IPV8Service: DEBUG:PRAGMA synchronous = NORMAL (previously: 2) [sqlite/attestations.db]
04-26 09:57:49.932 4441-4461/? I/IPV8Service: Unhandled Error
04-26 09:57:49.932 4441-4461/? I/IPV8Service: Traceback (most recent call last):
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/base.py", line 417, in fireEvent
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     DeferredList(beforeResults).addCallback(self._continueFiring)
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/defer.py", line 317, in addCallback
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     callbackKeywords=kw)
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/defer.py", line 306, in addCallbacks
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     self._runCallbacks()
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     current.result = callback(current.result, *args, **kw)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: --- <exception caught here> ---
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/base.py", line 430, in _continueFiring
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     callable(*args, **kwargs)
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/plugins/ipv8_plugin.py", line 48, in start_ipv8
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     self.ipv8 = IPv8(config)
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8_service.py", line 89, in __init__
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     overlay_instance = overlay_class(my_peer, self.endpoint, self.network, **overlay['initialize'])
04-26 09:57:49.933 4441-4461/? I/IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/attestation/wallet/community.py", line 56, in __init__
04-26 09:57:49.933 4441-4461/? I/IPV8Service:     self.attestation_keys[hash] = BonehPrivateKey.unserialize(key)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: exceptions.TypeError: writable buffers are not hashable

Crash after a while when running IPv8 in Tribler

I have the TunnelCommunity and DiscoveryCommunity enabled.

Unhandled error in Deferred:

ERROR   1518620389.98  tribler_window:84    Traceback (most recent call last):
  File "/Users/martijndevos/Documents/tribler/TriblerGUI/event_request_manager.py", line 123, in on_read_data
    raise RuntimeError(json_dict["event"]["text"])
RuntimeError: [Failure instance: Traceback: <type 'exceptions.RuntimeError'>: dictionary changed size during iteration
/Library/Python/2.7/site-packages/twisted/internet/base.py:1199:run
/Library/Python/2.7/site-packages/twisted/internet/base.py:1208:mainLoop
/Library/Python/2.7/site-packages/twisted/internet/base.py:828:runUntilCurrent
/Library/Python/2.7/site-packages/twisted/internet/task.py:239:__call__
--- <exception caught here> ---
/Library/Python/2.7/site-packages/twisted/internet/defer.py:150:maybeDeferred
/Users/martijndevos/Documents/tribler/Tribler/pyipv8/ipv8_service.py:112:on_tick
/Users/martijndevos/Documents/tribler/Tribler/pyipv8/ipv8/peerdiscovery/discovery.py:104:take_step
/Users/martijndevos/Documents/tribler/Tribler/pyipv8/ipv8/peerdiscovery/network.py:188:get_introductions_from
]

Add random challenges to Attestation verification

Right now we are blindly trusting the Attestation owner to respond truthfully to challenges. Random challenges should also be sent to verify key ownership and honesty.

Name attestation_request_callback structure in AttestationCommunity

Regarding https://github.com/Tribler/py-ipv8/blob/master/ipv8/attestation/wallet/community.py#L51, see https://github.com/Tribler/py-ipv8/pull/119/files#r184669956 :

Two comments: first, I think the naming of the variables should be more clear and precise. Second, I see two callbacks here but I have no idea what they are doing exactly. Could you use a named tuple or even a dictionary instead to store these callbacks by a human-readable key?

Fix crawler branch

A quick 4 minute crawl gives over 300 peers. This means the statistics.tribler.org information is incorrect and should be fixed.

Make Network handle snapshotting

For quick startup, the Network class should be able to snapshot the known peers (for writing to file) and load snapshots.

This should contain verified peers with at least:

Full public key
IP address
Port

Perhaps, they should just be added to the walkable addresses instead without the public key?

Add BPASE state of the art

Some recent advancements in key fragmentation (pyUmbral) and proofing (Bulletproofs) can be integrated into IPv8's ID framework for large performance gain.

Add request caches to Attestation community

Use request caches to prevent memory leaks in the Attestation community when the network is unreliable or when the peer is attacked.

Build triggers being dropped

Since 02/28/2018 the response from Jenkins is a HTTP 403:

Headers

Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 415
Content-Type: text/html;charset=iso-8859-1
Date: Wed, 28 Feb 2018 14:30:34 GMT
Server: Jetty(9.4.z-SNAPSHOT)
X-Content-Type-Options: nosniff

Body

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 403 No valid crumb was included in the request</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /job/ipv8/job/test_ipv8_linux/build. Reason:
<pre>    No valid crumb was included in the request</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>

</body>
</html>

distributed bootstrap service

https://njal.la/list/?search=ip-v8

Accepted Payments
We accept payments via Bitcoin, Litecoin, Monero, DASH, Bitcoin Cash and PayPal.

Who's behind Njalla?
We're a team of committed internet activists and we're also involved in other privacy projects such as the IPredator VPN service. Some of us have also been involved in projects like The Pirate Bay and Piratbyrån to mention a few things. Or to keep it simple; we've got a long history and serious understanding of the issues of anonymity on the internets.

Add a window size to RandomWalk

Right now the RandomWalk just keeps firing off requests. There should be a window of outstanding requests instead.

Infinite loop when receiving invalid block during crawl request when wanting to sign a block

I found an infinite loop when a node A wants to create a new block with a counterparty B. Suppose that the latest block of the chain of A is invalid (possibly due to a software bug or he could have inserted this himself). Now, when A creates a new block with node B and B does not have the latest blocks in the chain of A, B will issue a crawl request. Now, A will send the invalid block to B, which B will not insert in his database. While this is expected, the problem here is that B will now call the process_half_block method again and the process continues.

I've written the following test to confirm this:

    @twisted_wrapper
    def test_invalid_block(self):
        yield self.introduce_nodes()
        invalid_block = TestBlock(key=self.nodes[0].overlay.my_peer.key)
        invalid_block.signature = 'a' * 64
        self.nodes[0].overlay.persistence.add_block(invalid_block)

        his_pubkey = self.nodes[0].network.verified_peers[0].public_key.key_to_bin()
        yield self.nodes[0].overlay.sign_block(self.nodes[0].network.verified_peers[0], public_key=his_pubkey,
                                               transaction={})

        yield self.deliver_messages()

@qstokkink any comments on this?

Exception while handling packet

ERROR   1523354803.66       community:299   Exception occurred while handling packet!
Traceback (most recent call last):
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 296, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 33, in wrapper
    return f(self, *args, **kwargs)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 168, in received_half_block
    self.process_half_block(block, peer)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 33, in wrapper
    return f(self, *args, **kwargs)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 227, in process_half_block
    validation = self.validate_persist_block(blk)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 211, in validate_persist_block
    validation = block.validate(self.persistence)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/block.py", line 156, in validate
    blk = database.get(self.public_key, self.sequence_number)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 63, in get
    return self._get(u"WHERE public_key = ? AND sequence_number = ?", (buffer(public_key), sequence_number))
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 49, in _get
    db_result = list(self.execute(self.get_sql_header() + query, params, fetch_all=False))
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 40, in wrapper
    return f(self, *args, **kwargs)
  File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 302, in execute
    result = self._cursor.execute(statement, bindings)
AttributeError: 'NoneType' object has no attribute 'execute'

Interface specification between Self-Sovereign Identity and passport-level authentication

M.V.P. IDEMIA Interface

This document will entail the interface specifications for the onboarding procedure between the TU Delft IPv8 Blockchain solution (IPv8) and the IDEMIA biometry platform, from IPv8’s point of view. The Public Key based claim is the preferred solution from an ideological “the user is in control” perspective.

Context

This work defines the communication standard between 2 stand-alone apps. We provide an open standard for multiple biometric vendors, this is the key factor for large-scale adoption. As such and for security, the applications should be isolated as much as possible. Therefore any authentication app can talk to the IPv8 app. Alternative identity management apps such as Soverin or uPort can also use this neutral interface definition. This specification is specifically crafted to protect against man-in-the-middle attacks on the phone of a user. To deal with the isolation we need secure bi-directional authentication between apps. A simple REST api is used to communicate. For external communication QR-codes must be used.

There are two keypairs:

An IDEMIA keypair, assigned by IDEMIA
An IPv8 keypair, generated by the user through IPv8

Onboarding claim: Public Key based

The onboarding claim has the following format in IPv8:

Claim property	Type	Content
Name	String	“IDEMIA_ONBOARD_PK”
Format	String	“IDEMIA_CURVE”
Link	String	<IDEMIA Public Key>

Use-case

At the municipality:

IDEMIA: During the onboarding process in the municipality, the user onboards using the IDEMIA onboarding process (as specified in previous documents).
IDEMIA: The IDEMIA public key is sent to IPv8.
IPv8: IPv8 packs the public key into the claim format, as shown above.
IPv8: The municipality signs the claim format using the IPv8 key.
IPv8: The user signs the claim format signed by the municipality using the IPv8 key and publishes this.

At the verifier (store/airport/business/etc.):

IPv8: The user presents the claim to the verifier.
IPv8: The verifier issues a challenge (random data) to the user.
IDEMIA: Through the IDEMIA app, the user signs the challenge with his IDEMIA private key
IPv8: The verifier confirms that the signature belongs to the public key, as advertised through the claim format.

Required interface IDEMIA -> IPv8

Interface	Type	Details
sign(String: challenge)	String	Sign data using the Private Key for this device

Required interface IPv8 -> IDEMIA

Interface	Type	Details
createPKClaim(String: publicKey)	None	Create the IDEMIA_ONBOAD_PK claim for a certain IDEMIA public key

Onboarding claim: IDEMIA identifier based

The onboarding claim has the following format in IPv8:

Claim property	Type	Content
Name	String	“IDEMIA_ONBOARD_ID”
Format	String	“IDEMIA_ID”
Link	String	<IDEMIA identifier>

Use-case

At the municipality:

IDEMIA: During the onboarding process in the municipality, the user onboards using the IDEMIA onboarding process (as specified in previous documents).
IDEMIA: The IDEMIA identifier is sent to IPv8.
IPv8: IPv8 packs the identifier into the claim format, as shown above.
IPv8: The municipality signs the claim format using the IPv8 key.
IPv8: The user signs the claim format signed by the municipality using the IPv8 key and publishes this.

At the verifier (store/airport/business/etc.):

IPv8: The user presents the claim to the verifier.
IDEMIA: The verifier contacts IDEMIA with the identifier and verifies the user based on the reported attributes.

Required interface IPv8 -> IDEMIA

Interface	Type	Details
createIDClaim(String: identifier)	None	Create the IDEMIA_ONBOAD_ID claim for a certain IDEMIA id

Improve performance of Trustchain by first checking whether the block is in the database already

We can improve the performance of Trustchain by first checking whether the block is in the database in the validate_persist_block method (see https://github.com/Tribler/py-ipv8/blob/master/ipv8/attestation/trustchain/community.py#L205). This is because it is more efficient to do a database lookup than the block validation (which includes checking the signature(s) which is a relatively CPU-intensive task). If the block is already in the database, we forget about the validation.

In addition, it seems we are doing validation twice when the process_half_block method is invoked: first in validate_persist_block and next in process_half_block. This should be improved.

I will try to add this to #77.

Problem with pysqlite2 import

IPv8 fails to run on MacOS with following stacktrace:

  File "ipv8.py", line 5, in <module>
    from twisted.plugins.ipv8_plugin import Options, service_maker
  File "/Users/sandippandey/tudelft/personal/py-ipv8/twisted/plugins/ipv8_plugin.py", line 18, in <module>
    from ipv8.attestation.identity.community import IdentityCommunity
  File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/attestation/identity/community.py", line 6, in <module>
    from ..trustchain.community import TrustChainCommunity
  File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/attestation/trustchain/community.py", line 11, in <module>
    from .database import TrustChainDB
  File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/attestation/trustchain/database.py", line 6, in <module>
    from ...database import Database
  File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/database.py", line 18, in <module>
    import pysqlite2.dbapi2 as sqlite3
  File "/Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/dbapi2.py", line 28, in <module>
    from pysqlite2._sqlite import *
ImportError: dlopen(/Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/_sqlite.so, 2): Symbol not found: _sqlite3_enable_load_extension
  Referenced from: /Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/_sqlite.so
  Expected in: flat namespace
 in /Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/_sqlite.so

The problem is with pysqlite2 import
import pysqlite2.dbapi2 as sqlite3

For my case, using sqlite3 resolves the issue
from sqlite3 import dbapi2 as sqlite3

Wrong error when endpoint is closed

Traceback (most recent call last):
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 286, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 31, in on_introduction_response
    self.endpoint.send(source_address, packet)
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/messaging/interfaces/udp/endpoint.py", line 75, in send
    self.assert_open()
  File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/messaging/interfaces/udp/endpoint.py", line 52, in assert_open
    raise EndpointClosedException()
TypeError: __init__() takes exactly 2 arguments (1 given)

Instead of a EndpointClosedException, a TypeError is raised.

Add Twistd plugin

Add a way to use IPv8 as a twistd plugin.

Windows tests failing completely

Traceback (most recent call last):
File "c:\python27\lib\site.py", line 62, in
import os   
File "c:\python27\lib\os.py", line 400, in
import UserDict   
File "c:\python27\lib\UserDict.py", line 116, in
import _abcoll   
File "c:\python27\lib\_abcoll.py", line 11, in
from abc import ABCMeta, abstractmethod
File "C:\Users\heteoma\Documents\tribler\abc.py", line 15, in
import wx   
ImportError: No module named wx

This is a problem with the build machine import path.

Refactor Peer lifetime to Discovery mechanism

Like in Dispersy, the Peer objects contain their own lifespan: this should be refactored to a DiscoveryStrategy instead (https://github.com/qstokkink/py-ipv8/blob/master/ipv8/peerdiscovery/churn.py in particular).

Hardcode DNS names of Dispersy trackers

For faster resolving of peers, the DNS addresses of the Dispersy trackers should be added to the DiscoveryCommunity.

Refactor REST and test folders

The current folder structure does not allow parent modules to import files from these locations. Simply placing the REST and test folders within ipv8 will mitigate this.

Un-deprecate on-missing-id

Especially on phones, on-missing-identity messages are more frequently received from old Dispersy instances.

This possibly causes #7.

AssertionError when running IPV8

When running IPV8, I got the following error:

CRITICAL:root:Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/home/crawler/py-ipv8/ipv8/messaging/interfaces/endpoint.py", line 44, in _deliver_later
    listener.on_packet(packet)
  File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 237, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 58, in on_similarity_response
    auth, dist, payload = self._ez_unpack_auth(SimilarityResponsePayload, data)
  File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 47, in _ez_unpack_auth
    unknown_data.encode('HEX'))
exceptions.AssertionError: Incoming packet SimilarityResponsePayload (00027e313685c1912a141279f8248fc8db5899c5df5a020080307e301006072a8648ce3d020106052b81040024036a000400be6c4d4db7412784e0a72d6dad5c5b31d749f54f14fb7e6b748de5cf12f98b3377bc4a839bd73adb4fbf6e12df4dc84ae4613400d3d482bb422372ab8e3b63e5da0fd55aa9404e9aa9c4fa05d94fa2e6e2d1da75540d6645b38f502c9e9ccdb58e083472c3d81d0000000000000002663300017e313685c1912a141279f8248fc8db5899c5df5ac8e701e581942330a0d347eedcdb5048d5d1993e00000001fb23d2f92f3d2256f9af7bd216320eab1c7168bd00000001918552282c7ad9ce5442955540ef406562e72fca000000010a4aca64c779298ed5e26e4e71a859c90191888d00000001d62590448b6c013ebe1a30be18eb5837c06853f8000000012ef3a1ba8f3515ea13881360db451c7e78d8a689000000010e826d8689a9f6d7e20dd16ecfab651d56325149000000014283cc07c7e8262d1541e452b7e13f131caa065b000000012af6ed98b4f5b9446280c2a213fd38922caca05a00000001f694aa4578151549f206969504bae7f54494e6fc00000001c1ce314c76d2a55ccd4fc64bfeb144c777723f1a00000001d0fe3254a87e5f07926bfae86fa8b5207e8c6249000000011761b004430b51fa2508219cbcd67f5bca62f91500000001617975ffab60ce91a34e009f78eab62b03fad0af00000001fab3bc767235090226159744bdd7e43f060f14b800000001cf5fcb34d7b19cb57f4ca5410f35dbb55e2178b1000000014a356204ab68793194a09d5cd6c3d13f468ed1af0000000109e159243c76d7b868635a7b9d70f904ad38ebf900000001d7d6e11e03da72eb6efe5f40498917c755a6927100000001c5205c5e69cd33bd56549060e52a485a7a01647b0000000166ee5b7cb7686352b75f8e273dc1fb50f14e1c8300000001dd60f8406bf991211949d22d0becd006a968d6d50000000174abfb28a2de6f0827a91459b6d84a15a62e837c00000001a7632f3058f61996902c54ef1f03202c59f452af000000015fba30c1475c67aa5a02ac29098a9e3de1c60ef500000001001997f27903eb87cb78f3c23aa6684a5b73aeb04724cc8091f69f46a4de47e06eaffafa1cdad378cc345dca617e6e7fc86ea0210008b27c92be9764f9aef53950e23c6e064a04007b5ec5eba83aeb600eb1b419e091ad03227aca67c585b7699c848fff9183b924) has extra data: (fb23d2f92f3d2256f9af7bd216320eab1c7168bd00000001918552282c7ad9ce5442955540ef406562e72fca000000010a4aca64c779298ed5e26e4e71a859c90191888d00000001d62590448b6c013ebe1a30be18eb5837c06853f8000000012ef3a1ba8f3515ea13881360db451c7e78d8a689000000010e826d8689a9f6d7e20dd16ecfab651d56325149000000014283cc07c7e8262d1541e452b7e13f131caa065b000000012af6ed98b4f5b9446280c2a213fd38922caca05a00000001f694aa4578151549f206969504bae7f54494e6fc00000001c1ce314c76d2a55ccd4fc64bfeb144c777723f1a00000001d0fe3254a87e5f07926bfae86fa8b5207e8c6249000000011761b004430b51fa2508219cbcd67f5bca62f91500000001617975ffab60ce91a34e009f78eab62b03fad0af00000001fab3bc767235090226159744bdd7e43f060f14b800000001cf5fcb34d7b19cb57f4ca5410f35dbb55e2178b1000000014a356204ab68793194a09d5cd6c3d13f468ed1af0000000109e159243c76d7b868635a7b9d70f904ad38ebf900000001d7d6e11e03da72eb6efe5f40498917c755a6927100000001c5205c5e69cd33bd56549060e52a485a7a01647b0000000166ee5b7cb7686352b75f8e273dc1fb50f14e1c8300000001dd60f8406bf991211949d22d0becd006a968d6d50000000174abfb28a2de6f0827a91459b6d84a15a62e837c00000001a7632f3058f61996902c54ef1f03202c59f452af000000015fba30c1475c67aa5a02ac29098a9e3de1c60ef500000001),

List index out of range exception from anonymization community

Leecher errors, when trying to download through exit node (1 hop). Download does not work.

ERROR:TriblerTunnelCommunity:Exception occurred while handling packet!
Traceback (most recent call last):
  File "/mnt/leecher/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 283, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/mnt/leecher/tribler/Tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 767, in on_cell
    message_type = [k for k, v in message_to_payload.iteritems() if v[0] == payload.message_type][0]
IndexError: list index out of range

Tunnel helper (exit node) errors:

WARNING 1524131154.26       community:976   Got exception InvalidTag() when trying to decrypt relay message: "\x00\x00\x00\x00\x00\x00\x00\x020000000000000000\x02\x00\x0btest1.local\x1b9\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x17'\x10\x19\x80\x00\x00\x00\x00H\xa9\x7f\x1e" received for circuit_id: 461418816, is_data: 1, 
WARNING 1524131155.27       community:794   Got exception InvalidTag() when trying to decrypt relay message: '\x00\x00\x00\x00\x00\x00\x00\x020000000000000000P\x13g\x106\xb4\x9f-[\x98\xf2\xd0\x0b\x06\rI\xe1\xaa\x89\xad\xb1\xb5' received for circuit_id: 1606279686, is_data: 0, 
WARNING 1524131163.26       community:794   Got exception InvalidTag() when trying to decrypt relay message: '\x00\x00\x00\x00\x00\x00\x00\x020000000000000000\xb3\xda' received for circuit_id: 2585321991, is_data: 0,

The only file that was modified for running the local testnet is pyipv8/ipv8/deprecated/community.py:

_DEFAULT_ADDRESSES = [
    ("10.0.3.4", 10034)
]


_DNS_ADDRESSES = [
]

Issue when running IPv8 on Android

When running the library on Android, I continuously get the following stack trace:

12-07 10:21:54.514 11966 11989 I IPV8Service: DEBUG:Exception occurred while handling packet!
12-07 10:21:54.514 11966 11989 I IPV8Service: Traceback (most recent call last):
12-07 10:21:54.514 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/anonymization/community.py", line 169, in on_packet
12-07 10:21:54.514 11966 11989 I IPV8Service:     self.decode_map_private[data[22]](source_address, data, circuit_id)
12-07 10:21:54.514 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/anonymization/community.py", line 798, in on_created
12-07 10:21:54.514 11966 11989 I IPV8Service:     dist, payload = self._ez_unpack_noauth(CreatedPayload, data)
12-07 10:21:54.514 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/deprecated/community.py", line 65, in _ez_unpack_noauth
12-07 10:21:54.514 11966 11989 I IPV8Service:     dist, payload, unknown_data = self.serializer.unpack_to_serializables(format, data[23:])
12-07 10:21:54.514 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 265, in unpack_to_serializables
12-07 10:21:54.515 11966 11989 I IPV8Service:     serializable.optional_format_list, offset)
12-07 10:21:54.515 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 218, in unpack_multiple
12-07 10:21:54.515 11966 11989 I IPV8Service:     out.append(self.unpack(format, data, current_offset))
12-07 10:21:54.515 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 193, in unpack
12-07 10:21:54.515 11966 11989 I IPV8Service:     return self._packers[format].unpack_from(data, offset)
12-07 10:21:54.515 11966 11989 I IPV8Service:   File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 101, in unpack_from
12-07 10:21:54.515 11966 11989 I IPV8Service:     out = super(DefaultStruct, self).unpack_from(buffer, offset)
12-07 10:21:54.515 11966 11989 I IPV8Service: error: unpack_from requires a buffer of at least 4 bytes

Add Android baking code to the Tribler user

Right now the code for baking the IPv8 Android service is tied to my username (https://github.com/qstokkink/ipv8-android-service). We should move this repository to the Tribler user/team.

REST API wrong value

https://github.com/qstokkink/py-ipv8/blob/master/REST/attestation_endpoint.py#L87

Gives a string representation of the blocks. This should be the name fields in the block transactions.

[crawler branch] statistics.tribler.org unknown peers

http://statistics.tribler.org/ has a lot of peers with no community affinity, ~~this is probably related to the malformed incoming SimilarityResponses. This is most likely a bug in how SimilarityResponses are read.~~ This is not due to malformed SimilarityResponses as previously mentioned (the invalid message of #3 is readable in the current version of IPv8).

We need to send more SimilarityRequests to find out which community everyone belongs to. We should pobably also exclude NaCL peers too.

Create an IPv8 overlay creation tutorial

Current tutorial W.I.P.:
https://github.com/qstokkink/py-ipv8/blob/ipv8tutorial/doc/overlay_tutorial.md

Feedback is appreciated.

TODO:

More explanations

Add CI for Pull Requests

Add the unit tests as a step in the GitHub PR checks.

Another error when running IPV8

CRITICAL:root:Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/home/crawler/py-ipv8/ipv8/messaging/interfaces/endpoint.py", line 44, in _deliver_later
    listener.on_packet(packet)
  File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 237, in on_packet
    self.decode_map[data[22]](source_address, data)
  File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 48, in on_introduction_response
    packet = self.create_similarity_request()
  File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 81, in create_similarity_request
    return self._ez_pack(self._prefix, 1, [auth, dist, payload])
  File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 25, in _ez_pack
    packet += self.serializer.pack_multiple(format_list)
  File "/home/crawler/py-ipv8/ipv8/messaging/serialization.py", line 182, in pack_multiple
    out += self.pack(*packable)
  File "/home/crawler/py-ipv8/ipv8/messaging/serialization.py", line 170, in pack
    return self._packers[format].pack(*data)
struct.error: 'H' format requires 0 <= number <= 65535

Issue when running Tribler

When running Tribler using twistd, I suddenly got the following stack trace:

Mar 14 14:18:35 tribler-runner1 twistd[1936]:         Traceback (most recent call last):
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1199, in run
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             self.mainLoop()
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1208, in mainLoop
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             self.runUntilCurrent()
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 828, in runUntilCurrent
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             call.func(*call.args, **call.kw)
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/usr/lib/python2.7/dist-packages/twisted/internet/task.py", line 239, in __call__
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             d = defer.maybeDeferred(self.f, *self.a, **self.kw)
Mar 14 14:18:35 tribler-runner1 twistd[1936]:         --- <exception caught here> ---
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             result = f(*args, **kw)
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/opt/tribler/Tribler/pyipv8/ipv8_service.py", line 108, in on_tick
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             strategy.take_step(service)
Mar 14 14:18:35 tribler-runner1 twistd[1936]:           File "/opt/tribler/Tribler/pyipv8/ipv8/peerdiscovery/discovery.py", line 107, in take_step
Mar 14 14:18:35 tribler-runner1 twistd[1936]:             last_verified = self.under_construction[root][-1]
Mar 14 14:18:35 tribler-runner1 twistd[1936]:         exceptions.KeyError: <Tribler.pyipv8.ipv8.peer.Peer object at 0x7f53a4233e50>