tribler / py-ipv8 Goto Github PK
View Code? Open in Web Editor NEWPython implementation of Tribler's IPv8 p2p-networking layer
License: GNU Lesser General Public License v3.0
Python implementation of Tribler's IPv8 p2p-networking layer
License: GNU Lesser General Public License v3.0
Possible research directions: towards a fast clean-slate network stack or support multi-path routing out of the box. problem of protocol ossification. Prior work: how to receive a million packets per second. This is opposed to moving the IPv8 research to the database side of the stack. There is also the DHT, connect to anybody connect() primitive with network mobility.
Linux kernel support for UDP tunneling from 2015. Open Hardware support, like White Rabbit Network Interface Card. Scientific article a multi-port 10GbE PCIe NIC featuring UDP offload and GPUDirect capabilities. Other work Realtime Ethernet Device Driver.
Right now the code for baking the IPv8 Android service is tied to my username (https://github.com/qstokkink/ipv8-android-service). We should move this repository to the Tribler user/team.
This document will entail the interface specifications for the onboarding procedure between the TU Delft IPv8 Blockchain solution (IPv8) and the IDEMIA biometry platform, from IPv8’s point of view. The Public Key based claim is the preferred solution from an ideological “the user is in control” perspective.
This work defines the communication standard between 2 stand-alone apps. We provide an open standard for multiple biometric vendors, this is the key factor for large-scale adoption. As such and for security, the applications should be isolated as much as possible. Therefore any authentication app can talk to the IPv8 app. Alternative identity management apps such as Soverin or uPort can also use this neutral interface definition. This specification is specifically crafted to protect against man-in-the-middle attacks on the phone of a user. To deal with the isolation we need secure bi-directional authentication between apps. A simple REST api is used to communicate. For external communication QR-codes must be used.
There are two keypairs:
The onboarding claim has the following format in IPv8:
Claim property | Type | Content |
---|---|---|
Name | String | “IDEMIA_ONBOARD_PK” |
Format | String | “IDEMIA_CURVE” |
Link | String | <IDEMIA Public Key> |
Use-case
At the municipality:
At the verifier (store/airport/business/etc.):
Required interface IDEMIA -> IPv8
Interface | Type | Details |
---|---|---|
sign(String: challenge) | String | Sign data using the Private Key for this device |
Required interface IPv8 -> IDEMIA
Interface | Type | Details |
---|---|---|
createPKClaim(String: publicKey) | None | Create the IDEMIA_ONBOAD_PK claim for a certain IDEMIA public key |
The onboarding claim has the following format in IPv8:
Claim property | Type | Content |
---|---|---|
Name | String | “IDEMIA_ONBOARD_ID” |
Format | String | “IDEMIA_ID” |
Link | String | <IDEMIA identifier> |
Use-case
At the municipality:
At the verifier (store/airport/business/etc.):
Required interface IPv8 -> IDEMIA
Interface | Type | Details |
---|---|---|
createIDClaim(String: identifier) | None | Create the IDEMIA_ONBOAD_ID claim for a certain IDEMIA id |
ERROR 1523354803.66 community:299 Exception occurred while handling packet!
Traceback (most recent call last):
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 296, in on_packet
self.decode_map[data[22]](source_address, data)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 33, in wrapper
return f(self, *args, **kwargs)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 168, in received_half_block
self.process_half_block(block, peer)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 33, in wrapper
return f(self, *args, **kwargs)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 227, in process_half_block
validation = self.validate_persist_block(blk)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/community.py", line 211, in validate_persist_block
validation = block.validate(self.persistence)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/block.py", line 156, in validate
blk = database.get(self.public_key, self.sequence_number)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 63, in get
return self._get(u"WHERE public_key = ? AND sequence_number = ?", (buffer(public_key), sequence_number))
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 49, in _get
db_result = list(self.execute(self.get_sql_header() + query, params, fetch_all=False))
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 40, in wrapper
return f(self, *args, **kwargs)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 302, in execute
result = self._cursor.execute(statement, bindings)
AttributeError: 'NoneType' object has no attribute 'execute'
Right now we are blindly trusting the Attestation owner to respond truthfully to challenges. Random challenges should also be sent to verify key ownership and honesty.
There is a bug on TrustChain cache timeout.
INFO 1522056800.81 caches:66 Timeout for crawl with id 46814472
2018-03-26T11:33:20+0200 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 396, in startReactor
self.config, oldstdout, oldstderr, self.profiler, reactor)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
reactor.run()
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1243, in run
self.mainLoop()
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1252, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 878, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/jenkins/tribler/Tribler/pyipv8/ipv8/requestcache.py", line 146, in _on_timeout
cache.on_timeout()
File "/home/jenkins/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/caches.py", line 67, in on_timeout
self.community.request_cache.pop(u"crawl", self.number)
File "/home/jenkins/tribler/Tribler/pyipv8/ipv8/requestcache.py", line 132, in pop
cache = self._identifiers.pop(identifier)
exceptions.KeyError: u'crawl:46814472'
The current folder structure does not allow parent modules to import files from these locations. Simply placing the REST
and test
folders within ipv8
will mitigate this.
Current tutorial W.I.P.:
https://github.com/qstokkink/py-ipv8/blob/ipv8tutorial/doc/overlay_tutorial.md
Feedback is appreciated.
TODO:
Add a way to use IPv8 as a twistd plugin.
For faster resolving of peers, the DNS addresses of the Dispersy trackers should be added to the DiscoveryCommunity.
======================================================================
test/attestation/trustchain/test_community.py:TestTrustChainCommunity
======================================================================
Test broadcasting a half block ... ok
Test broadcasting a half block pair ... ok
Check if a block can be crawled. ... ok
Check if the default crawl strategy produces blocks. ... ok
Check if a block can be crawled by negative range. ... ok
Check if blocks don't magically appear. ... ok
Check if a both halves of a fully signed block link to each other. ... ok
Check if blocks created in parallel will properly be stored in the database. ... ok
Check if missing blocks are retrieved through a crawl request. ... FAIL
======================================================================
FAIL: Check if missing blocks are retrieved through a crawl request.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/Library/Python/2.7/site-packages/twisted/internet/utils.py", line 201, in runWithWarningsSuppressed
reraise(exc_info[1], exc_info[2])
File "/Library/Python/2.7/site-packages/twisted/internet/utils.py", line 197, in runWithWarningsSuppressed
result = f(*a, **kw)
File "/Users/tribler/Documents/workspace/workspace/ipv8/test_ipv8_mac/test/util.py", line 125, in errback
failure.raiseException()
File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = g.send(result)
File "/Users/tribler/Documents/workspace/workspace/ipv8/test_ipv8_mac/test/attestation/trustchain/test_community.py", line 226, in test_retrieve_missing_block
self.assertIsNotNone(self.nodes[node_nr].overlay.persistence.get(his_pubkey, 1))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 962, in assertIsNotNone
self.fail(self._formatMessage(msg, standardMsg))
File "/Library/Python/2.7/site-packages/twisted/trial/_synctest.py", line 368, in fail
raise self.failureException(msg)
FailTest: unexpectedly None
-------------------- >> begin captured logging << --------------------
twisted: INFO: --> test_ipv8_mac.test.attestation.trustchain.test_community.TestTrustChainCommunity.test_retrieve_missing_block <--
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 4096) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a9774c700dd2bab1179c90a9ad454145177d3953a70832e8be75c36e4cdde846f0779fa63fd480cc169ec022825082068aeb8c0f48374ce6aef1a987d22325233
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 4096) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a30ae524d08c89b13a3fbb4bc93b880d2427014588a9b2011b0ef602cff8009434790236e887cd80ff5cc93c08ae1df095325421efff830c5626aab171f71fe53
TrustChainCommunity: INFO: Signed block to 1f71fe53 (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (215.136.3.36:40395) (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
RequestCache: DEBUG: add <HalfBlockSignCache sign-18142001>
TrustChainCommunity: INFO: Signed block to 1f71fe53 (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (215.136.3.36:40395) (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
RequestCache: DEBUG: add <HalfBlockSignCache sign-18142002>
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x10d417f50>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x10d417f50>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Request block could not be validated sufficiently, crawling requester. (<function no_info at 0x10d417f50>, [])
RequestCache: DEBUG: add <CrawlRequestCache crawl-95915699>
TrustChainCommunity: INFO: Requesting crawl of node 22325233:1 with id 95915699
TrustChainCommunity: INFO: Received crawl request from node 1f71fe53 for sequence number 1
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<215.136.3.36:40395, 34hUFsyOvKICXK8xv7aDqYDkPXE=> (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<215.136.3.36:40395, 34hUFsyOvKICXK8xv7aDqYDkPXE=> (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Sent 2 blocks
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x10d417d70>, [], (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x10d417d70>, [], (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block 912e3938 from ...22325233:1 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Signed block to 22325233 (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (1.47.221.125:16905) (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: INFO: Signed block to 22325233 (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {}) validation result (<function partial_next at 0x10d417e60>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to (1.47.221.125:16905) (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 8dc82eb3 from ...22325233:2 links ...1f71fe53:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainCommunity: DEBUG: Received already known block (Block d4f8dfe3 from ...1f71fe53:1 links ...22325233:1 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x10d417e60>, [], (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
TrustChainCommunity: DEBUG: Received already known block (Block ab49fa5a from ...1f71fe53:2 links ...22325233:2 for {})
--------------------- >> end captured logging << ---------------------
In Tribler someone is sending the following message:
STRUCT DATA DESCRIPTION
====================================================================
DISPERSY_VERSION 00 0
COMMUNITY_VERSION 02 2
COMMUNITY_ID 7e 31 36 85 | c1 91 2a 14
12 79 f8 24 | 8f c8 db 58
99 c5 df 5a DiscoveryCommunity
MESSAGE_ID f6 IntroductionRequest
GLOBAL_TIME 00 00 00 00 | 00 00 00 e5 229
DESTINATION 59 3d 0b 80 | 53 c7 (89.61.11.128, 21447)
LAN ed c3 04 f2 | 20 37 (237.195.4.242, 8247)
WAN 97 ff ed 0c | 74 99 (151.255.237.12, 29849)
FLAGS 67 (connection_type_0=0,
connection_type_1=1,
dflag0=1,
dflag1=0,
dflag2=0,
tunnel=1,
sync=1,
advice=1)
IDENTIFIER 32 f3 13043
TIME_LOW 91 5e 7f f5 | ce 13 c0 a8 10474950477011468456
TIME_HIGH 01 71 1e 4f | bc cc c3 82 103897594225017730
MODULO 1e 4f 7759
OFFSET 01 84 388
FUNCTIONS 9c 10011100
BLOOMFILTER_SIZE <N/A> MISSING!
BLOOMFILTER <N/A> MISSING!
The contents seem like garbage, but this message has been correctly signed though.
When running Tribler using twistd, I suddenly got the following stack trace:
Mar 14 14:18:35 tribler-runner1 twistd[1936]: Traceback (most recent call last):
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1199, in run
Mar 14 14:18:35 tribler-runner1 twistd[1936]: self.mainLoop()
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1208, in mainLoop
Mar 14 14:18:35 tribler-runner1 twistd[1936]: self.runUntilCurrent()
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 828, in runUntilCurrent
Mar 14 14:18:35 tribler-runner1 twistd[1936]: call.func(*call.args, **call.kw)
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/usr/lib/python2.7/dist-packages/twisted/internet/task.py", line 239, in __call__
Mar 14 14:18:35 tribler-runner1 twistd[1936]: d = defer.maybeDeferred(self.f, *self.a, **self.kw)
Mar 14 14:18:35 tribler-runner1 twistd[1936]: --- <exception caught here> ---
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
Mar 14 14:18:35 tribler-runner1 twistd[1936]: result = f(*args, **kw)
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/opt/tribler/Tribler/pyipv8/ipv8_service.py", line 108, in on_tick
Mar 14 14:18:35 tribler-runner1 twistd[1936]: strategy.take_step(service)
Mar 14 14:18:35 tribler-runner1 twistd[1936]: File "/opt/tribler/Tribler/pyipv8/ipv8/peerdiscovery/discovery.py", line 107, in take_step
Mar 14 14:18:35 tribler-runner1 twistd[1936]: last_verified = self.under_construction[root][-1]
Mar 14 14:18:35 tribler-runner1 twistd[1936]: exceptions.KeyError: <Tribler.pyipv8.ipv8.peer.Peer object at 0x7f53a4233e50>
Especially on phones, on-missing-identity
messages are more frequently received from old Dispersy instances.
This possibly causes #7.
Like in Dispersy, the Peer
objects contain their own lifespan: this should be refactored to a DiscoveryStrategy
instead (https://github.com/qstokkink/py-ipv8/blob/master/ipv8/peerdiscovery/churn.py in particular).
Regarding https://github.com/Tribler/py-ipv8/blob/master/ipv8/attestation/wallet/community.py#L51, see https://github.com/Tribler/py-ipv8/pull/119/files#r184669956 :
Two comments: first, I think the naming of the variables should be more clear and precise. Second, I see two callbacks here but I have no idea what they are doing exactly. Could you use a named tuple or even a dictionary instead to store these callbacks by a human-readable key?
Right now programmers have to manually start waiting for new RequestCache
s when handling messages. This leads to a lot of duplicate code. Instead, handler chains could be specified, with the message handlers returning the RequestCache __init__
arguments for the next cache.
For example, the declaration:
IntroductionRequest -> IntroductionResponse;
Could fire the IntroductionRequest_handler(message, request_cache=None)
when an IntroductionRequest
message is received and then construct a new RequestCache
waiting for the IntroductionResponse
to be received.
Some recent advancements in key fragmentation (pyUmbral) and proofing (Bulletproofs) can be integrated into IPv8's ID framework for large performance gain.
When running IPV8, I encountered a stack trace:
CRITICAL:root:Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
task()
File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_team.py", line 190, in doWork
task()
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/home/crawler/py-ipv8/ipv8/messaging/interfaces/endpoint.py", line 44, in _deliver_later
listener.on_packet(packet)
File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 237, in on_packet
self.decode_map[data[22]](source_address, data)
File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 58, in on_similarity_response
auth, dist, payload = self._ez_unpack_auth(SimilarityResponsePayload, data)
File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 43, in _ez_unpack_auth
dist, payload, unknown_data = self.serializer.unpack_to_serializables(format, remainder[23:])
File "/home/crawler/py-ipv8/ipv8/messaging/serialization.py", line 266, in unpack_to_serializables
out.append(serializable.from_unpack_list(*unpack_list))
File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery_payload.py", line 83, in from_unpack_list
[(tb_overlap[i:i+20], unpack(">I", tb_overlap[i+20:i+24])) for i in range(0, len(tb_overlap), 24)]]
struct.error: unpack requires a string argument of length 4
Right now the smallest primes we can generate are 512 bit. This makes sense for RSA, but this leads to1024 bit EC keys, which is ridiculous. We can use the gensafeprime
OpenSSL bindings for generating primes smaller than 512 bit.
import gensafeprime
gensafeprime.generate(bitspace)
We will need to update the Jenkins Android builder and the requirements.txt
.
IPv8 fails to run on MacOS with following stacktrace:
File "ipv8.py", line 5, in <module>
from twisted.plugins.ipv8_plugin import Options, service_maker
File "/Users/sandippandey/tudelft/personal/py-ipv8/twisted/plugins/ipv8_plugin.py", line 18, in <module>
from ipv8.attestation.identity.community import IdentityCommunity
File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/attestation/identity/community.py", line 6, in <module>
from ..trustchain.community import TrustChainCommunity
File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/attestation/trustchain/community.py", line 11, in <module>
from .database import TrustChainDB
File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/attestation/trustchain/database.py", line 6, in <module>
from ...database import Database
File "/Users/sandippandey/tudelft/personal/py-ipv8/ipv8/database.py", line 18, in <module>
import pysqlite2.dbapi2 as sqlite3
File "/Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/dbapi2.py", line 28, in <module>
from pysqlite2._sqlite import *
ImportError: dlopen(/Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/_sqlite.so, 2): Symbol not found: _sqlite3_enable_load_extension
Referenced from: /Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/_sqlite.so
Expected in: flat namespace
in /Users/sandippandey/tudelft/viro/lib/python2.7/site-packages/pysqlite2/_sqlite.so
The problem is with pysqlite2 import
import pysqlite2.dbapi2 as sqlite3
For my case, using sqlite3 resolves the issue
from sqlite3 import dbapi2 as sqlite3
Transport any binary blob:
Add the unit tests as a step in the GitHub PR checks.
http://statistics.tribler.org/ has a lot of peers with no community affinity, this is probably related to the malformed incoming SimilarityResponses. This is most likely a bug in how SimilarityResponses are read. This is not due to malformed SimilarityResponses as previously mentioned (the invalid message of #3 is readable in the current version of IPv8).
We need to send more SimilarityRequests to find out which community everyone belongs to. We should pobably also exclude NaCL peers too.
Right now we do a lot of operations on a cell. It would make sense to group this logic into a class of the same name.
Right now error messages appear as follows:
error: unpack_from requires a buffer of at least 4 bytes
Combined with a stack trace, this is debuggable. However, a more human readable format can and should be made.
A quick 4 minute crawl gives over 300 peers. This means the statistics.tribler.org
information is incorrect and should be fixed.
There is an error when loading keys for the AttestationCommunity:
04-26 09:57:49.876 4441-4461/? I/IPV8Service: DEBUG:Launching AttestationCommunity with prefix 0002b42c93d167a0fc4a0843f917d4bf1e9ebb340ec4.
04-26 09:57:49.877 4441-4461/? I/IPV8Service: DEBUG:loading database [sqlite/attestations.db]
04-26 09:57:49.877 4441-4461/? I/IPV8Service: DEBUG:open database [sqlite/attestations.db]
04-26 09:57:49.907 4441-4461/? I/IPV8Service: DEBUG:PRAGMA page_size = 8192 (no change) [sqlite/attestations.db]
04-26 09:57:49.907 4441-4461/? I/IPV8Service: DEBUG:PRAGMA journal_mode = WAL (no change) [sqlite/attestations.db]
04-26 09:57:49.907 4441-4461/? I/IPV8Service: DEBUG:PRAGMA synchronous = NORMAL (previously: 2) [sqlite/attestations.db]
04-26 09:57:49.932 4441-4461/? I/IPV8Service: Unhandled Error
04-26 09:57:49.932 4441-4461/? I/IPV8Service: Traceback (most recent call last):
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/base.py", line 417, in fireEvent
04-26 09:57:49.933 4441-4461/? I/IPV8Service: DeferredList(beforeResults).addCallback(self._continueFiring)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/defer.py", line 317, in addCallback
04-26 09:57:49.933 4441-4461/? I/IPV8Service: callbackKeywords=kw)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/defer.py", line 306, in addCallbacks
04-26 09:57:49.933 4441-4461/? I/IPV8Service: self._runCallbacks()
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
04-26 09:57:49.933 4441-4461/? I/IPV8Service: current.result = callback(current.result, *args, **kw)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: --- <exception caught here> ---
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/internet/base.py", line 430, in _continueFiring
04-26 09:57:49.933 4441-4461/? I/IPV8Service: callable(*args, **kwargs)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/twisted/plugins/ipv8_plugin.py", line 48, in start_ipv8
04-26 09:57:49.933 4441-4461/? I/IPV8Service: self.ipv8 = IPv8(config)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8_service.py", line 89, in __init__
04-26 09:57:49.933 4441-4461/? I/IPV8Service: overlay_instance = overlay_class(my_peer, self.endpoint, self.network, **overlay['initialize'])
04-26 09:57:49.933 4441-4461/? I/IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/attestation/wallet/community.py", line 56, in __init__
04-26 09:57:49.933 4441-4461/? I/IPV8Service: self.attestation_keys[hash] = BonehPrivateKey.unserialize(key)
04-26 09:57:49.933 4441-4461/? I/IPV8Service: exceptions.TypeError: writable buffers are not hashable
Right now all the Attestation challenges are being sent at once. This is bad for network health.
Also, this allows us to implement honesty batches.
Leecher errors, when trying to download through exit node (1 hop). Download does not work.
ERROR:TriblerTunnelCommunity:Exception occurred while handling packet!
Traceback (most recent call last):
File "/mnt/leecher/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 283, in on_packet
self.decode_map[data[22]](source_address, data)
File "/mnt/leecher/tribler/Tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 767, in on_cell
message_type = [k for k, v in message_to_payload.iteritems() if v[0] == payload.message_type][0]
IndexError: list index out of range
Tunnel helper (exit node) errors:
WARNING 1524131154.26 community:976 Got exception InvalidTag() when trying to decrypt relay message: "\x00\x00\x00\x00\x00\x00\x00\x020000000000000000\x02\x00\x0btest1.local\x1b9\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x17'\x10\x19\x80\x00\x00\x00\x00H\xa9\x7f\x1e" received for circuit_id: 461418816, is_data: 1,
WARNING 1524131155.27 community:794 Got exception InvalidTag() when trying to decrypt relay message: '\x00\x00\x00\x00\x00\x00\x00\x020000000000000000P\x13g\x106\xb4\x9f-[\x98\xf2\xd0\x0b\x06\rI\xe1\xaa\x89\xad\xb1\xb5' received for circuit_id: 1606279686, is_data: 0,
WARNING 1524131163.26 community:794 Got exception InvalidTag() when trying to decrypt relay message: '\x00\x00\x00\x00\x00\x00\x00\x020000000000000000\xb3\xda' received for circuit_id: 2585321991, is_data: 0,
The only file that was modified for running the local testnet is pyipv8/ipv8/deprecated/community.py
:
_DEFAULT_ADDRESSES = [
("10.0.3.4", 10034)
]
_DNS_ADDRESSES = [
]
When running IPV8, I got the following error:
CRITICAL:root:Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
task()
File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_team.py", line 190, in doWork
task()
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/home/crawler/py-ipv8/ipv8/messaging/interfaces/endpoint.py", line 44, in _deliver_later
listener.on_packet(packet)
File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 237, in on_packet
self.decode_map[data[22]](source_address, data)
File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 58, in on_similarity_response
auth, dist, payload = self._ez_unpack_auth(SimilarityResponsePayload, data)
File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 47, in _ez_unpack_auth
unknown_data.encode('HEX'))
exceptions.AssertionError: Incoming packet SimilarityResponsePayload (00027e313685c1912a141279f8248fc8db5899c5df5a020080307e301006072a8648ce3d020106052b81040024036a000400be6c4d4db7412784e0a72d6dad5c5b31d749f54f14fb7e6b748de5cf12f98b3377bc4a839bd73adb4fbf6e12df4dc84ae4613400d3d482bb422372ab8e3b63e5da0fd55aa9404e9aa9c4fa05d94fa2e6e2d1da75540d6645b38f502c9e9ccdb58e083472c3d81d0000000000000002663300017e313685c1912a141279f8248fc8db5899c5df5ac8e701e581942330a0d347eedcdb5048d5d1993e00000001fb23d2f92f3d2256f9af7bd216320eab1c7168bd00000001918552282c7ad9ce5442955540ef406562e72fca000000010a4aca64c779298ed5e26e4e71a859c90191888d00000001d62590448b6c013ebe1a30be18eb5837c06853f8000000012ef3a1ba8f3515ea13881360db451c7e78d8a689000000010e826d8689a9f6d7e20dd16ecfab651d56325149000000014283cc07c7e8262d1541e452b7e13f131caa065b000000012af6ed98b4f5b9446280c2a213fd38922caca05a00000001f694aa4578151549f206969504bae7f54494e6fc00000001c1ce314c76d2a55ccd4fc64bfeb144c777723f1a00000001d0fe3254a87e5f07926bfae86fa8b5207e8c6249000000011761b004430b51fa2508219cbcd67f5bca62f91500000001617975ffab60ce91a34e009f78eab62b03fad0af00000001fab3bc767235090226159744bdd7e43f060f14b800000001cf5fcb34d7b19cb57f4ca5410f35dbb55e2178b1000000014a356204ab68793194a09d5cd6c3d13f468ed1af0000000109e159243c76d7b868635a7b9d70f904ad38ebf900000001d7d6e11e03da72eb6efe5f40498917c755a6927100000001c5205c5e69cd33bd56549060e52a485a7a01647b0000000166ee5b7cb7686352b75f8e273dc1fb50f14e1c8300000001dd60f8406bf991211949d22d0becd006a968d6d50000000174abfb28a2de6f0827a91459b6d84a15a62e837c00000001a7632f3058f61996902c54ef1f03202c59f452af000000015fba30c1475c67aa5a02ac29098a9e3de1c60ef500000001001997f27903eb87cb78f3c23aa6684a5b73aeb04724cc8091f69f46a4de47e06eaffafa1cdad378cc345dca617e6e7fc86ea0210008b27c92be9764f9aef53950e23c6e064a04007b5ec5eba83aeb600eb1b419e091ad03227aca67c585b7699c848fff9183b924) has extra data: (fb23d2f92f3d2256f9af7bd216320eab1c7168bd00000001918552282c7ad9ce5442955540ef406562e72fca000000010a4aca64c779298ed5e26e4e71a859c90191888d00000001d62590448b6c013ebe1a30be18eb5837c06853f8000000012ef3a1ba8f3515ea13881360db451c7e78d8a689000000010e826d8689a9f6d7e20dd16ecfab651d56325149000000014283cc07c7e8262d1541e452b7e13f131caa065b000000012af6ed98b4f5b9446280c2a213fd38922caca05a00000001f694aa4578151549f206969504bae7f54494e6fc00000001c1ce314c76d2a55ccd4fc64bfeb144c777723f1a00000001d0fe3254a87e5f07926bfae86fa8b5207e8c6249000000011761b004430b51fa2508219cbcd67f5bca62f91500000001617975ffab60ce91a34e009f78eab62b03fad0af00000001fab3bc767235090226159744bdd7e43f060f14b800000001cf5fcb34d7b19cb57f4ca5410f35dbb55e2178b1000000014a356204ab68793194a09d5cd6c3d13f468ed1af0000000109e159243c76d7b868635a7b9d70f904ad38ebf900000001d7d6e11e03da72eb6efe5f40498917c755a6927100000001c5205c5e69cd33bd56549060e52a485a7a01647b0000000166ee5b7cb7686352b75f8e273dc1fb50f14e1c8300000001dd60f8406bf991211949d22d0becd006a968d6d50000000174abfb28a2de6f0827a91459b6d84a15a62e837c00000001a7632f3058f61996902c54ef1f03202c59f452af000000015fba30c1475c67aa5a02ac29098a9e3de1c60ef500000001),
We can improve the performance of Trustchain by first checking whether the block is in the database in the validate_persist_block
method (see https://github.com/Tribler/py-ipv8/blob/master/ipv8/attestation/trustchain/community.py#L205). This is because it is more efficient to do a database lookup than the block validation (which includes checking the signature(s) which is a relatively CPU-intensive task). If the block is already in the database, we forget about the validation.
In addition, it seems we are doing validation twice when the process_half_block
method is invoked: first in validate_persist_block
and next in process_half_block
. This should be improved.
I will try to add this to #77.
Since 02/28/2018 the response from Jenkins is a HTTP 403:
Headers
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 415
Content-Type: text/html;charset=iso-8859-1
Date: Wed, 28 Feb 2018 14:30:34 GMT
Server: Jetty(9.4.z-SNAPSHOT)
X-Content-Type-Options: nosniff
Body
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 403 No valid crumb was included in the request</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /job/ipv8/job/test_ipv8_linux/build. Reason:
<pre> No valid crumb was included in the request</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>
</body>
</html>
Traceback (most recent call last):
File "c:\python27\lib\site.py", line 62, in
import os
File "c:\python27\lib\os.py", line 400, in
import UserDict
File "c:\python27\lib\UserDict.py", line 116, in
import _abcoll
File "c:\python27\lib\_abcoll.py", line 11, in
from abc import ABCMeta, abstractmethod
File "C:\Users\heteoma\Documents\tribler\abc.py", line 15, in
import wx
ImportError: No module named wx
This is a problem with the build machine import path.
IPv8 is having problem on shutting down. See the logs:
ERROR 1524667809.84 rest_manager:53 (RESTRequest) [Failure instance: Traceback: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'execute'
/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py:571:dataReceived
/usr/lib/python2.7/dist-packages/twisted/web/http.py:1657:lineReceived
/usr/lib/python2.7/dist-packages/twisted/web/http.py:1736:allContentReceived
/usr/lib/python2.7/dist-packages/twisted/web/http.py:762:requestReceived
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/web/server.py:183:process
/usr/lib/python2.7/dist-packages/twisted/web/server.py:234:render
/usr/lib/python2.7/dist-packages/twisted/web/resource.py:250:render
/home/sandip/tudelft/tribler/Tribler/Core/Modules/restapi/trustchain_endpoint.py:87:render_GET
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py:14:helper
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py:26:blockingCallFromThread
/home/sandip/tudelft/tribler/Tribler/community/triblerchain/community.py:44:get_statistics
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py:80:get_latest
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py:49:_get
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py:40:wrapper
/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py:302:execute
]
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 183, in process
self.render(resrc)
File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 234, in render
body = resrc.render(self)
File "/usr/lib/python2.7/dist-packages/twisted/web/resource.py", line 250, in render
return m(request)
File "/home/sandip/tudelft/tribler/Tribler/Core/Modules/restapi/trustchain_endpoint.py", line 87, in render_GET
return json.dumps({'statistics': triblerchain_community.get_statistics()})
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py", line 14, in helper
return blockingCallFromThread(reactor, func, *args, **kargs)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/util.py", line 26, in blockingCallFromThread
return f(*args, **kwargs)
File "/home/sandip/tudelft/tribler/Tribler/community/triblerchain/community.py", line 44, in get_statistics
latest_block = self.persistence.get_latest(public_key)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 80, in get_latest
u"WHERE public_key = ?)", (buffer(public_key), buffer(public_key)))
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/attestation/trustchain/database.py", line 49, in _get
db_result = list(self.execute(self.get_sql_header() + query, params, fetch_all=False))
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 40, in wrapper
return f(self, *args, **kwargs)
File "/home/sandip/tudelft/tribler/Tribler/pyipv8/ipv8/database.py", line 302, in execute
result = self._cursor.execute(statement, bindings)
AttributeError: 'NoneType' object has no attribute 'execute'
Similar: Exception while handling packet #103
When running the library on Android, I continuously get the following stack trace:
12-07 10:21:54.514 11966 11989 I IPV8Service: DEBUG:Exception occurred while handling packet!
12-07 10:21:54.514 11966 11989 I IPV8Service: Traceback (most recent call last):
12-07 10:21:54.514 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/anonymization/community.py", line 169, in on_packet
12-07 10:21:54.514 11966 11989 I IPV8Service: self.decode_map_private[data[22]](source_address, data, circuit_id)
12-07 10:21:54.514 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/anonymization/community.py", line 798, in on_created
12-07 10:21:54.514 11966 11989 I IPV8Service: dist, payload = self._ez_unpack_noauth(CreatedPayload, data)
12-07 10:21:54.514 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/deprecated/community.py", line 65, in _ez_unpack_noauth
12-07 10:21:54.514 11966 11989 I IPV8Service: dist, payload, unknown_data = self.serializer.unpack_to_serializables(format, data[23:])
12-07 10:21:54.514 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 265, in unpack_to_serializables
12-07 10:21:54.515 11966 11989 I IPV8Service: serializable.optional_format_list, offset)
12-07 10:21:54.515 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 218, in unpack_multiple
12-07 10:21:54.515 11966 11989 I IPV8Service: out.append(self.unpack(format, data, current_offset))
12-07 10:21:54.515 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 193, in unpack
12-07 10:21:54.515 11966 11989 I IPV8Service: return self._packers[format].unpack_from(data, offset)
12-07 10:21:54.515 11966 11989 I IPV8Service: File "/data/user/0/org.internetofmoney.android/files/lib/python2.7/site-packages/ipv8/messaging/serialization.py", line 101, in unpack_from
12-07 10:21:54.515 11966 11989 I IPV8Service: out = super(DefaultStruct, self).unpack_from(buffer, offset)
12-07 10:21:54.515 11966 11989 I IPV8Service: error: unpack_from requires a buffer of at least 4 bytes
For quick startup, the Network
class should be able to snapshot the known peers (for writing to file) and load snapshots.
This should contain verified peers with at least:
Perhaps, they should just be added to the walkable addresses instead without the public key?
For some reason the Mac tests for TrustChain are unstable. This may have to do with a different commit
protocol on Mac.
UPDATE: There seems to be a race condition in the test, also reproduced on Ubuntu 16.10.
Ran the following to produce:
#!/bin/bash
export PYTHONPATH=.
while $(nosetests test/attestation/trustchain/)
do
echo ""
done
======================================================================
FAIL: Check if missing blocks are retrieved through a crawl request.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/utils.py", line 201, in runWithWarningsSuppressed
reraise(exc_info[1], exc_info[2])
File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/utils.py", line 197, in runWithWarningsSuppressed
result = f(*a, **kw)
File "/home/quinten/tribler/py-ipv8/test/util.py", line 125, in errback
failure.raiseException()
File "/home/quinten/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/home/quinten/tribler/py-ipv8/test/attestation/trustchain/test_community.py", line 229, in test_retrieve_missing_block
self.assertEqual(self.nodes[node_nr].overlay.persistence.get(his_pubkey, 1).link_sequence_number, 1)
File "/home/quinten/.local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 432, in assertEqual
super(_Assertions, self).assertEqual(first, second, msg)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
raise self.failureException(msg)
FailTest: 2 != 1
-------------------- >> begin captured logging << --------------------
twisted: INFO: --> test.attestation.trustchain.test_community.TestTrustChainCommunity.test_retrieve_missing_block <--
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 1024) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a56aa579d3ca2b931de20b71a50882643378309b95095633667cccee32bb1ed0af52876c04964339034d9f610d1a3994b4b01153e5cec670efb8049cff999fe3d
TrustChainDB: DEBUG: loading database [:memory:]
TrustChainDB: DEBUG: TrustChain database path: :memory:
TrustChainDB: DEBUG: open database [:memory:]
TrustChainDB: DEBUG: PRAGMA page_size = 8192 (previously: 1024) [:memory:]
TrustChainDB: DEBUG: PRAGMA journal_mode = MEMORY (no change) [:memory:]
TrustChainDB: DEBUG: PRAGMA synchronous = NORMAL (previously: 2) [:memory:]
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: The trustchain community started with Public Key: 4c69624e61434c504b3a67148d49f0e1d601147f6c9f1b5bc4c9d85095039b554d06b1d14a99fd0bd7213a6137048a89b167e45b00ca830a9c2077f99f21b1e6ed929cc3d19802e7d6c7
TrustChainCommunity: INFO: Signed block to 02e7d6c7 (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Signed block to 02e7d6c7 (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x7f91bfb33aa0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function no_info at 0x7f91bfb33aa0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Request block could not be validated sufficiently, crawling requester. (<function no_info at 0x7f91bfb33aa0>, [])
RequestCache: DEBUG: add <CrawlRequestCache crawl-81589634>
TrustChainCommunity: INFO: Requesting crawl of node f999fe3d:1 with id 81589634
TrustChainCommunity: INFO: Received crawl request from node 02e7d6c7 for sequence number 1
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Sending block for crawl request to Peer<1.28.122.61:37216, O4TnSGduJ5QEF4dL/m8XPKSl9j8=> (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Sent 2 blocks
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x7f91bfb338c0>, [], (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function valid at 0x7f91bfb338c0>, [], (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Signed block to f999fe3d (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<156.112.38.66:56845, DTjdP/pkFJzvti2JZ7sXR8bVXWU=> (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainCommunity: DEBUG: Received already known block (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Received request block addressed to us (Block c7580377 from ...f999fe3d:1 links ...02e7d6c7:0 for {})
TrustChainCommunity: INFO: Signed block to f999fe3d (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {}) validation result (<function partial_next at 0x7f91bfb339b0>, [])
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Sending block to Peer<156.112.38.66:56845, DTjdP/pkFJzvti2JZ7sXR8bVXWU=> (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Received already known block (Block f490ad82 from ...f999fe3d:2 links ...02e7d6c7:0 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainCommunity: DEBUG: Received already known block (Block c4639c16 from ...02e7d6c7:1 links ...f999fe3d:2 for {})
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
TrustChainDB: DEBUG: commit [:memory:]
TrustChainCommunity: DEBUG: Block validation result <function partial_next at 0x7f91bfb339b0>, [], (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
TrustChainCommunity: DEBUG: Received already known block (Block 69209d79 from ...02e7d6c7:2 links ...f999fe3d:1 for {})
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
CRITICAL:root:Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
task()
File "/usr/local/lib/python2.7/dist-packages/twisted/_threads/_team.py", line 190, in doWork
task()
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/home/crawler/py-ipv8/ipv8/messaging/interfaces/endpoint.py", line 44, in _deliver_later
listener.on_packet(packet)
File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 237, in on_packet
self.decode_map[data[22]](source_address, data)
File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 48, in on_introduction_response
packet = self.create_similarity_request()
File "/home/crawler/py-ipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 81, in create_similarity_request
return self._ez_pack(self._prefix, 1, [auth, dist, payload])
File "/home/crawler/py-ipv8/ipv8/deprecated/community.py", line 25, in _ez_pack
packet += self.serializer.pack_multiple(format_list)
File "/home/crawler/py-ipv8/ipv8/messaging/serialization.py", line 182, in pack_multiple
out += self.pack(*packable)
File "/home/crawler/py-ipv8/ipv8/messaging/serialization.py", line 170, in pack
return self._packers[format].pack(*data)
struct.error: 'H' format requires 0 <= number <= 65535
I found an infinite loop when a node A wants to create a new block with a counterparty B. Suppose that the latest block of the chain of A is invalid (possibly due to a software bug or he could have inserted this himself). Now, when A creates a new block with node B and B does not have the latest blocks in the chain of A, B will issue a crawl request. Now, A will send the invalid block to B, which B will not insert in his database. While this is expected, the problem here is that B will now call the process_half_block
method again and the process continues.
I've written the following test to confirm this:
@twisted_wrapper
def test_invalid_block(self):
yield self.introduce_nodes()
invalid_block = TestBlock(key=self.nodes[0].overlay.my_peer.key)
invalid_block.signature = 'a' * 64
self.nodes[0].overlay.persistence.add_block(invalid_block)
his_pubkey = self.nodes[0].network.verified_peers[0].public_key.key_to_bin()
yield self.nodes[0].overlay.sign_block(self.nodes[0].network.verified_peers[0], public_key=his_pubkey,
transaction={})
yield self.deliver_messages()
@qstokkink any comments on this?
Traceback (most recent call last):
File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 286, in on_packet
self.decode_map[data[22]](source_address, data)
File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/peerdiscovery/deprecated/discovery.py", line 31, in on_introduction_response
self.endpoint.send(source_address, packet)
File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/messaging/interfaces/udp/endpoint.py", line 75, in send
self.assert_open()
File "/home/jenkins/workspace/GH_Tribler_PR_tests_linux/tribler/Tribler/pyipv8/ipv8/messaging/interfaces/udp/endpoint.py", line 52, in assert_open
raise EndpointClosedException()
TypeError: __init__() takes exactly 2 arguments (1 given)
Instead of a EndpointClosedException
, a TypeError
is raised.
Right now the RandomWalk
just keeps firing off requests. There should be a window of outstanding requests instead.
https://github.com/qstokkink/py-ipv8/blob/master/REST/attestation_endpoint.py#L87
Gives a string representation of the blocks. This should be the name
fields in the block transactions.
I have the TunnelCommunity
and DiscoveryCommunity
enabled.
Unhandled error in Deferred:
ERROR 1518620389.98 tribler_window:84 Traceback (most recent call last):
File "/Users/martijndevos/Documents/tribler/TriblerGUI/event_request_manager.py", line 123, in on_read_data
raise RuntimeError(json_dict["event"]["text"])
RuntimeError: [Failure instance: Traceback: <type 'exceptions.RuntimeError'>: dictionary changed size during iteration
/Library/Python/2.7/site-packages/twisted/internet/base.py:1199:run
/Library/Python/2.7/site-packages/twisted/internet/base.py:1208:mainLoop
/Library/Python/2.7/site-packages/twisted/internet/base.py:828:runUntilCurrent
/Library/Python/2.7/site-packages/twisted/internet/task.py:239:__call__
--- <exception caught here> ---
/Library/Python/2.7/site-packages/twisted/internet/defer.py:150:maybeDeferred
/Users/martijndevos/Documents/tribler/Tribler/pyipv8/ipv8_service.py:112:on_tick
/Users/martijndevos/Documents/tribler/Tribler/pyipv8/ipv8/peerdiscovery/discovery.py:104:take_step
/Users/martijndevos/Documents/tribler/Tribler/pyipv8/ipv8/peerdiscovery/network.py:188:get_introductions_from
]
https://njal.la/list/?search=ip-v8
Accepted Payments
We accept payments via Bitcoin, Litecoin, Monero, DASH, Bitcoin Cash and PayPal.
Who's behind Njalla?
We're a team of committed internet activists and we're also involved in other privacy projects such as the IPredator VPN service. Some of us have also been involved in projects like The Pirate Bay and Piratbyrån to mention a few things. Or to keep it simple; we've got a long history and serious understanding of the issues of anonymity on the internets.
Add the sqlite files and key files to the gitignore.
Traceback (most recent call last):
File "/home/quinten/Documents/tribler/Tribler/pyipv8/ipv8/deprecated/community.py", line 296, in on_packet
self.decode_map[data[22]](source_address, data)
File "/home/quinten/Documents/tribler/Tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 797, in on_cell
self.send_destroy(circuit.sock_addr, circuit_id, 0)
AttributeError: 'NoneType' object has no attribute 'sock_addr'
Right now ipv8.py
is and example file to invoke the modular framework. ipv8.py
could be offered as a full-fledged modular service instead.
IPv8 should function as an Android app.
All of the dependencies have been compiled before for the Tribler project, these should/could be reused for the py-to-app compiler.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.