1. Does it have the no-copy bug like go-ipfs, where the pinned file will be duplicated in the disk?
   • https://github.com/ipfs/go-ipfs/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+nocopy
2. Does Py-IPFS has hashes algorithms for SHA256, BLAKE2b, BLAKE2s and Skein?
   • For reference SHA256 and BLAKE2 are in the standard library, and Skein is in https://pythonhosted.org/pyskein/

Following help/issue should be done to stimulate the development.

1. Clear Timeline setup for the project
2. Development setup
3. chat channel instructions
4. Weekly/monthly online meet.
5. Special instructions for first-time contributors to this repo.

Feel free to add others

I don't know how/if it can be useful, but on python-dev there is a PEP proposal to establish a uniform protocol to treat filesystem's path:

1. https://mail.python.org/pipermail/python-dev/2016-May/144506.html
2. https://mail.python.org/pipermail/python-dev/2016-May/144569.html

Compatibility Target 0.4 go-ipfs

Start with implementing libp2p: https://github.com/ipfs/specs/blob/master/protocol/network/6-interfaces.md#63-swarm

• https://github.com/dabeaz/curio
• Transports (Session Layer Protocols)
• Protocols (DHTs/Bitswap)
• Multistream/Interchange

👶 ←#2

@whyrusleeping

• Don't support Python 2 at all
• Support to 2.7

@BrendanBenshoof

• Don't support Python 2 at all
• Support to 2.7

@sirMackk

• Don't support Python 2 at all
• Support to 2.7

@SilverWingedSeraph

• Don't support Python 2 at all
• Support to 2.7

@mvanveen

• Don't support Python 2 at all
• Support to 2.7

@bmcorser

• Don't support Python 2 at all
• Support to 2.7

@JulienPalard

• Don't support Python 2 at all
• Support to 2.7

Not sure if there are any other stakeholders ...

Since #1 is clogged with all the many comments I open a new issue here. Feel free to continue the discussion below and I'll keep the following updated as things develop. Also feel free to create separate issues / repos to coordinate and I'll add the relevant links below.

Next steps (networking, stalled – please see the “storage” section below):

• We could use some documentation for the py-libp2p library
  • See issue libp2p/py-libp2p#35 for status
  • Current work: libp2p/py-libp2p#330
• Documentation for the Bitswap protocol would be very useful as well – and not just for py-ipfs
  • See issue ipfs/js-ipfs-bitswap#21 for status
  • Currently available information:
    • Protocol: https://github.com/ipfs/specs/blob/master/BITSWAP.md
    • API: https://ipfs.github.io/js-ipfs-bitswap/#bitswapget
• Finally, some documentation on the currently used DHT would be nice
  • See https://github.com/ipfs/camp/blob/master/DEEP_DIVES/02-scaling-up-the-dht.md for some relevant links and current developments
• Implement the Bitswap protocol in Python 3
  • Current py-ipfs-bitswap library: https://github.com/AliabbasMerchant/py-ipfs-bitswap
  • Current status/discussion: AliabbasMerchant/py-bitswap#1
  • → Having this would allow us to implement the ipfs block * API for fetching blocks of nodes we are connected to – fetching blocks of non-connected nodes needs the DHT.
  • To interact with go-IPFS you can start it with ipfs daemon --disable-transport-encryption, but note that you will not be able to connect to any regular peers until one of the transport encryption methods is implemented
• (Low priority) Improve the multistream-select code of py-libp2p to support actually dialing other nodes (MOSTLY FIXED UPSTREAM – ls is still missing and an mss-nc implementation could still be useful)
  • Write a mss-nc like utility on top of this code to demonstrate that you are able to connect to go-ipfs nodes and negotiate
  • Here's some very simple sample code demonstrating the main mode of MSS:

    import socket
    s = socket.socket(socket.AF_INET)
    s.connect(("127.0.0.1", 4001))  # The connect will already exist
    s.sendall(b'\x13/multistream/1.0.0\n') # Send your supported version of MSS
    s.recv(1024) # → b'\x13/multistream/1.0.0\n' – Receive supported version of MSS by other party & validate!
    s.sendall(b"\x0d/secio/1.0.0\n") # Request the protocol you'd like to upgrade too
    s.recv(1024) # → b'\0x0d/secio/1.0.0\n…' – Confirmation that protocol is available + Protocol data OR
                 # → b'\0x03na\n' – Protocol was Not Available

  • The binary values at the start are varints and you need to read them byte-by-byte until you're done decoding them, then read the remainder of each message lines based on the received length value
    To do this you'll need to create an async stream based version of https://github.com/fmoo/python-varint/blob/master/varint.py
  • Additionally there is also a special ls mode in which MSS will return a list of supported protocols, see https://github.com/multiformats/multistream-select/blob/master/README.md for the complete spec
  • Subtask: Figure out how to actually dial a node using py-libp2p and document this using an example program.
• Implement SecIO libp2p transport security (easier than TLS, but will be phased out eventually)
  • You'll need to coordinate with the py-libp2p guys on transport security modules are added exactly
  • Some background with crypto is highly recommended!
• Implement DHT peer lookups in libp2p
  • Currently available documentation: https://github.com/libp2p/specs/tree/8b89dc2521b48bf6edab7c93e8129156a7f5f02c/kad-dht
  • Go Implementation: https://github.com/libp2p/go-libp2p-kad-dht
  • JavaScript Implementation: https://github.com/libp2p/js-libp2p-kad-dht
  • For the current status see: libp2p/py-libp2p#150 (also see the referenced PRs on that issue)
  • Interesting PRs: #129, #153, #157
  • Contacts: @alexh @zaibon @zixuanzh
• (Stretch goal) Factor out https://github.com/zixuanzh/py-libp2p/tree/master/protocol_muxer into a separate py-multistream-select library and update py-libp2p to use it (Easy!, stalled – needs your help!)
  • Current py-multistream-select library: https://github.com/dheatovwil/py-multistream-select
  • Current status: libp2p/py-libp2p#101
• (Stretch goal) Implement TLSv1.3 libp2p transport security
  • You'll need to coordinate with the py-libp2p guys on transport security modules are added exactly
  • Some background with crypto/X.509/TLS is highly recommended!

Next steps (storage, simpler):

1. Port https://github.com/ipfs/py-datastore to Python 3
   (Suggestion: Use Python's lib2to3 and just drop Python 2 entirely.)
   • Current port: https://github.com/dheatovwil/datastore
2. Convert datastore to use async/await using some library (maybe https://pypi.org/project/aiofiles/ ?) for file access The trio framework is used for async I/O now
3. Implement a https://github.com/ipfs/go-ds-flatfs compatible backend for the above library
4. Write a minimal py-ipfs “implementation” that can fetch blocks from the local $IPFS_PATH directory and expose them with an API similar to what https://github.com/ipfs/py-ipfs-http-client currently offers (goal here is to eventually have a drop-in replacement)
   • In progress by @Alexander255 (no public code yet, most work happens in py-datastore)
5. Implement a simple Python HTTP server that emulates the block/{get,put,rm,stat} API that serves blocks from the local $IPFS_PATH directory
   • Recommendation: Use the trio-quart ASGI web microframework for this. (Whatever you choose it will have to be compatible with trio as that is the AIO framework used in the stack.)
6. (Stretch goal) Implement a badgerds compatible backend for py-datastore
   • There is an issue requesting Python bindings for the Go library, but no work has been done yet:
     dgraph-io/badger#984
7. Beyond: Start integrating IPLD to expose the UnixFS files stored in those raw blocks…

I thought of getting multihash module ready as a part of building supporting modules and have been basing it on JS implementation. I have been coming across several doubts regarding my implementation.

Can someone check if this is going correctly or can it be written better?

Address: multihash

I am interested in contributing to Python implementation of IPFS but I don't know where to begin. I initially had an idea of doing this via basing it on JS variant but every library I look at -- libp2p or peer-id or ipld they all have internal dependencies to each other and other libraries like multihashing or libp2p.

So I am right now stuck on where to begin contribution. I understood libp2p is important and so I peeked at js variant. It's dependent on so many libraries which are in-turn dependent on other libraries.

What to do? Can anyone please help? AS a beginner to this, I'd like some guidance please!

Before moving on to the rest of libp2p and py-ipfs, we should look at the required generic utilities and determine which existing implementation we want to use, or if there isn't one, who wants to work on it. Here is what we need (see #21 (comment) for more info):

• PeerID
  • js impl: https://github.com/diasdavid/js-peer-id
• PeerInfo
  • js impl: https://github.com/diasdavid/js-peer-info
• multihash
  • specs: https://github.com/jbenet/multihash
  • go impl: https://github.com/jbenet/go-multihash
  • js impl: https://github.com/jbenet/js-multihash
  • py impl: https://github.com/ivilata/pymultihash
• multihashing (optional, pymultihash delegates this mostly to hashlib)
  • specs: ???
  • js impl: https://github.com/jbenet/js-multihashing
• multiaddr
  • specs: https://github.com/jbenet/multiaddr
  • go impl: https://github.com/jbenet/go-multiaddr
  • js impl: https://github.com/jbenet/js-multiaddr
  • py impl: https://github.com/sbuss/py-multiaddr
• multistream
  • specs: https://github.com/jbenet/multistream
  • js impl: https://github.com/diasdavid/js-multistream
• multicodec
  • specs: https://github.com/jbenet/multicodec
  • go impl: https://github.com/jbenet/go-multicodec
  • py impl: https://github.com/multiformats/py-multicodec
• ipld
  • specs: ???
  • discussion: ipld/go-ipld-deprecated#8
  • go impl: https://github.com/ipfs/go-ipld
  • py impl: https://github.com/ipld/py-ipld-dag

I know we already have several multihash implementations, and I hacked together a multiaddr implementation that could use some love. If anyone has already worked on any of these or is interested on working on any of them, please comment here. Also if anyone has anymore links to references or specs for the generic modules, post them here.

Would the maintainers be averse to doing this the (easier) way of building on C (I'm thinking kenCode's implementation) or CGo extensions?

The main con would probably be the loss of (some, perhaps not all) of the feedback on the spec, and the need to publish wheels. Pros would be implementation time, execution time (by a huge margin initially) and faster updates (in the case of C/Go internals).

@jbenet calls this out in #1. Judging by #8 people seem to agree with the principle that developing against a testable spec is a priority.

I'm curious how we're going to go about accomplishing and structuring this goal. I realize the actual written spec is still in progress (I myself am still coming up to speed on the protocol specifics).

I'd like to use this issue to center discussion on planning for how we're going to tackle this issue.

I'm cross-posting this from the Orbit github since Morphis is written completely in Python and is really similar to IPFS so it can help speed along the development of py-ipfs.

The main developer was shot and killed 2 weeks ago and it would mean a lot for his work to live on. https://morph.is is a distributed datastore like IPFS and there was a ui being developed for a distributed irc\slack which you can get if you pull the source and git checkout f-dds (the idea was to have it be completely functional without any JavaScript for those who prefer the added security of not running any JavaScript, with optional settings and tools for those who don't mind JavaScript)

the distributed irc was almost done heres a mockup of what it was going to look like https://ipfs.pics/Qmab8KzU4jCEYwwHQaDRNBQGu2eRSf9Qvkhfjn7Q63qFTH
if you run a morphis node with the f-dds branch you can see and use most of it in action. There were some tweaks to the ui we had talked about but never got around to finish.

Heres a backup of the code including all branches (which is also currently still available on the official morph.is git repo until the servers go offline)

https://u.teknik.io/XWzPR.zip

Despite this project not having any code written yet, we should support Python back to at least 2.7 to help with uptake of users who can't (or don't want to) work in a Python 3 environment.

Python 2.7 smoothes over many issues with __future__, but some guidance for developers would be useful.

Resources:

• wiki 3 stdlib in 2, general info
• six compatibility lib
• a guide print, exceptions, imports, L, unicode
• tox testing
• setuptools packaging provision

I'm the developer of Argos Translate, an open source neural machine translation library, and currently have models available for IPFS download. I'd like to add support for automatically downloading from IPFS at some point (currently you need to manually download and install the models). What's the recommended timeline/strategy for doing this? My current plan is to wait for this project to be production ready at which point adding IPFS support would be pretty straightforward. Is there any sort of timeline estimate for when this will be? Is there a better strategy like trying to use the Go implementation?

Thanks

I've been steadily diving head first into IPFS docs and am still getting up to speed. Along the way, I noticed that there is a pre-existing weekly sprint process defined for a lot of the specific current initiatives (go, node, testing + ci, protocol + specs, etc.).

Would this be something that this team would be interested in adopting?

I think one of the benefits is it could solidify strong initial momentum and team to get things going, but it might not fit into everyone's schedule and might not be everyone's favored process. Since this is an ipfs project, I also think it might be good to stick to their existing defined processes & best practices.

Hey everyone. I was just starting to look around because I want to contribute to this project. I read that an implementation of py-libp2p should be one of the first step but currently the link found in the roadmap is broken.
As any work been done yet and if so where is it?

Should we include .idea/ for PyCharm? I use PyCharm extensively on my desktop dev machine, and I think it would make the process of onboarding new developers easier if we had a .idea/ project directory already set up with execution settings for tests, etc. already set up. Otherwise, we will end up with a lot of duplication of work.

I'd like to have a consensus on how to implement the HTTP(S) API. I would suggest Flask but that is only because of my inexperience with other options (Bottle, Pyramid, etc).

Once we have a decision on this, a skeleton API could be written with the actual implementation filled in as the functionality materializes. It will mostly be boilerplate and there is no reason for implementers of actually hard things to have to write that code as well.

i want to join to dev python-ipfs. i saw this project starting. i am a python developer. i work for eos blockchain. i am interested in blockchain and ipfs. i used python,golang,php. ipfs is cool. so i want to join it and dev a new internet.
so how can i join python-ipfs. first maybe i can pr, but there are maybe other developer build it. so i create this issue. i hope to join them,but i don’t how to do that.
and sorry to create this issue just to do for that. but i’m really interested in ipfs.

if you have free time, please give a replay. and if i can’t join it,please give me some advice to tell me how to contribute.

my email: [email protected]

thanks,
goooo

I suggest that we separate the ipfs package into packages for each IPFS subsystem:
* ipfs-api - Used to connect to Py-IPFS by applications. This is where most users should touch. From https://github.com/ipfs/python-ipfs-api
* ipfs-naming or ipfs-ipns - Provides access to the IPNS pulbic key infrastructure and naming system. Handles HRNs and mutable pointers
* ipfs-merkledag - Provides access to Merkledag for resolving paths
* ipfs-block - Provides access to the block exchange system for transferring data.
* ipfs-routing - Provides access to the routing system, probably mostly DHT and mdns
* ipfs-network - Provides access to the lowest level IPFS network transport functions, like NAT traversal/hole punching, encryption/signing, and multi/broadcasting

Client applications would initialize the stack from the bottom up: choose a transport and encryption with ipfs-network, connect to the DHT with ipfs-routing, then use ipfs-naming (ipfs-ipns) to find the location of desired content and ipfs-merkledag and ipfs-block to recall that data.
On the other hand, if an instance of IPFS is already running on the system (which is preferred), they could simply use ipfs-api to interact with it.

I want to try and plan a developer hangout to start planning out the project and perhaps prepare for the first sprint.

We are a global "team" so scheduling is a mess.
I can commit to being available for any time chosen on this sat. or sun. with advanced notice.
If you are interested please post your availability for this weekend (make sure to mark timezone info or use utc)

There are 4 open PRs from a year ago, and it would be nice if we could delegate someone to do the reviews

Lots of work is being done, towards making py-ipfs a complete project.
However, since the dependencies and all need to be worked up from scratch, not much work is being done directly in this repo. Various contributors are working on various other repos.
Maybe, we can put a link to those repos in the README.md file, so that new contributors know where to look for issues and where to contribute.
1 more suggestion: We can separate those repos as:

1. Core jobs (The protocols ...)
2. Dependencies (libp2p ...)

Traceback (most recent call last):
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/http_httpx.py", line 173, in _request
closables, session = self._access_session()
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/http_common.py", line 417, in _access_session
session = self._make_session()
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/http_httpx.py", line 119, in _make_session
connection_pool = httpcore.SyncConnectionPool(
AttributeError: module 'httpcore' has no attribute 'SyncConnectionPool'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wongasley/Desktop/RBS/Post.py", line 3, in
client = ipfshttpclient.connect()
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/client/init.py", line 119, in connect
assert_version(client.apply_workarounds()["Version"])
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/client/init.py", line 228, in apply_workarounds
version_info = self.version()
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/client/base.py", line 245, in wrapper2
result = func(*args, **kwargs)
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/client/miscellaneous.py", line 204, in version
return self._client.request('/version', decoder='json', **kwargs)
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/http_common.py", line 613, in request
closables, res = self._request(
File "/home/wongasley/.local/lib/python3.8/site-packages/ipfshttpclient/http_httpx.py", line 202, in _request
for closable in closables:
UnboundLocalError: local variable 'closables' referenced before assignment

With the following code
api = ipfsapi.connect('127.0.0.1', 5001)
api.get(<hash>)
The file is downloaded and is renamed as hash. I'm not certain if this is done on purpose or an issue that needs to be resolved

As there is no official Protocol Buffers working support yet for Python3, it would be great if server and client could support https://capnproto.org/

See ipfs/ipfs#21

I dont have the bandwidth to lead this effort, so ideally we could find a maintainer that can take a lead on. Ideally someone who understands ipfs at both a technical, ideological, and product levels.

Doing this well will in short term will need:

• stable specs https://github.com/ipfs/specs/ (which we're headed toward)
• passing a common test suite (which we'll extract from go-ipfs)
• coordination with others in ipfs world in general (come hang out in #ipfs)

we're super happy to support this effort however we can. (biggest need is docs i think). i'd encourage you to play with existing go-ipfs, and come hang out in #ipfs.

Just noticed there are (at least) two multihash implementations out there. Oddly, it appears as if they were created a few weeks apart:

• https://github.com/tehmaze/python-multihash
• https://github.com/bmcorser/py-multihash

@tehmaze's implementation is mentioned in the main ipfs documentation and actually does the hashing for the user, my implementation offers an interface and __eq__. Can they be merged under a common repo?

cc: @BrendanBenshoof @JulienPalard @bmcorser @SilverWingedSeraph @mvanveen

Our kick-off meeting went pretty well (#20) and I think we now have a pretty good idea about how to start this project. I think the plan is to build a python port of libp2p first and then build the IPFS protocol on top of that. I was thinking we should probably schedule another meeting to discuss libp2p in more detail and perhaps delegate some tasks. Thoughts?

Is this still under active development? What are the current items that are being implemented? Would love to work on some of the implementation (would need a LOT of help, of course).

Is this project still in good maintenence or is it unmaintained? Please let me know I am very much interested in this.

• setup.py
• tox, pytest
• Sphnix
• gh-pages

I suggest that, since we are starting with both a spec and and a reference implementation, we go the Test Driven Development route.

This raises the question, what test framework do we use? I'm most comfortable with unittest, but am entirely open to learning pytest or Nose.

Additionally, I was considering an essentially 3-branch structure: release, master, and unimplemented, where unimplemented holds tests for new features that have yet to be implemented. Upon those tests passing, master would receive both the tests and the code to pass them. That way, we can enforce an always-passing rule on master more easily.

Hey everyone, py-ipfs seems a bit dormant for now. that's fine.

But things many people have requested is:

• py-ipld to match https://github.com/ipfs/js-ipld
• completeness of api ipfs-shipyard/py-ipfs-http-client#42

In particular, it would be very helpful to have py-ipld.