I just updated ipfs-pack to use go-ipfs 0.4.14. This should fix some of the memory issues people were running into when adding really large datasets.

We should push out a new version with these changes, and maybe fix a the other issue @Stebalien pointed out.

In order to implement pack make, which generates a PackManifest file, we need to know how PackManifest files should be formatted.

Some references:

• bagit payload manifests -- extremely similar to what we need
• data packages format -- this json format is designed to capture metadata about a dataset. Not likely to be included in PackManifest, but could be included in the pack's contents... (if you have ideas around this, please create issues in this repo to discuss)

From the spec

> cat PackManifest
QmVP2aaAWFe21QjUujMw5hwYRKD1eGx3yYWEBbMtuxpqXs <fmtstr> moreData/0
QmV7eDE2WXuwQnvccsoXSzK5CQGXdFfay1LSadZCwyfbDV <fmtstr> moreData/1
QmaMY7h9pmTcA5w9S2dsQT5eGLEQ1CwYQ32HwMTXAev5gQ <fmtstr> moreData/2
QmQjYU5PscpCHadDbL1fDvTK4P9eXirSwD8hzJbAyrd5mf <fmtstr> moreData/3
QmRErwActoLmffucXq7HPtefBC19MjWUcj1DdBoaAnMm6p <fmtstr> moreData/4
QmeWvL929Tdhzw27CS5ZVHD73NQ9TT1xvLvCaXCgi7a9YB <fmtstr> moreData/5
QmXbzZeh44jJEUueWjFxEiLcfAfzoaKYEy1fMHygkSD3hm <fmtstr> moreData/6
QmYL17nYZrZsAhJut5v7ooD9hmz2rBotC1tqC9ZPxzCfer <fmtstr> moreData/7
QmPKkidoUYX12PyCuKzehQuhEJofUJ9PPaX2Gc2iYd4GRs <fmtstr> moreData/8
QmQAubXA3Gji5v5oaJhMbvmbGbiuwDf1u9sYsN125mcqrn <fmtstr> moreData/9
QmYbYduoHMZAUMB5mjHoJHgJ9WndrdWkTCzuQ6yHkbgqkU <fmtstr> someJSON.json
QmeWiZD5cdyiJoS3b7h87Cs9G21uQ1sLmeKrunTae9h5qG <fmtstr> someXML.xml
QmVizQ5fUceForgWogbb2m2v5RRrE8xEm8uSkbkyNB4Rdm <fmtstr> moreData
QmZ7iEGqahTHdUWGGZMUxYRXPwSM3UjBouneLcCmj9e6q6 <fmtstr> .

It would also be helpful to have ipfs-pack serve show how far it is with verifying the pack integrity. My 30+ GB pack takes about 7-8 minutes - and unless I am actively monitoring the console there is no way to really know when it starts...

Could there be a way to add something like a webhook in the config, that the daemon pings when it is up - or sends a heartbeat to? Sometimes the daemon will not be accessible via NAT traversal and for some use-cases it would be great to be able to have it report to a monitoring endpoint.

There is a link in the ipfs's documentation to this lib in the section for working with the big files. Nonetheless, the latest commit was made about 2 years ago, which brings suspicion that it is outdated/deprecated/not supported any more, is it?

Right now the manifest contains all paths as

QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH  f0000120001     ./abcd

This is not consistent with spec (which has just regular relative paths) and makes the file slightly more annoying to parse. Manifest still verifies after removing the prefix so maybe we should remove it?

(this may be idiosyncratic to OS X)

File Attributes (either in POSIX or extended attributes) of UNIX File System Objects SHOULD be preserved as long as they allow canonical serialisation (i.e. can be uniquely hashed regardless of env).

References:

1. Nix Archive: preserve executable flag and symlinks (required for software build process); strip permissions, timestamps, and ownerships (since they are environment-dependent) http://nixos.org/~eelco/pubs/phd-thesis.pdf#5.2.1
2. Git: preserve executable flag and symlinks. Several tools exist to extend git to preserve ownership and permission,
3. Torrent: extended file attributes and symlinks still WIP, an issue is raised for potential data loss after deduplication http://www.bittorrent.org/beps/bep_0047.html
4. BagIt: unspecified
5. DataPackage: unspecified (but MAY contain other metadata http://specs.frictionlessdata.io/data-packages/#optional-fields-1)
6. WARC: unspecified (but header MUST contain several fields that may overlap with optional metadata in DataPackage, e.g. https://github.com/internetarchive/warc/blob/8f05a000a23bbd6501217e37cfd862ffdf19da7f/warc/warc.py#L43)

Given
There are files and/or sub-directories that I do not want added to the pack,

Then
I add the files paths, or path matching patterns, to a .ipfsignore file and then build the pack manifest.

Then
The manifest does not include those files

If there was a well written spec, then it would be easier to diagnose problems, write external tests and even autogenerate better documentation.

ipfs/interface-ipfs-core/SPEC does an excellent job of this.

We should add some solid tests here, a lot of edge cases to test.

• Test basic creation, serving, verify, repo regen rm and other commands
• Test different permissions issues (how does pack make fail under bad permissions)
• Test pack creation from different directories (passing 'dir' parameter for different commands)
• Test changing files in the pack before and/or during serve
• Test moving, deleting, changing perms, etc on files before/during serve

cc @chriscool, do you think you would have time to help us out on this?

I built a pack based on my working copy of this git repo. Immediately after making the PackManifest I ran ipfs-pack verify and got a bunch of errors:

Checksum mismatch on ./vendor/src/github.com/ipfs/go-ipfs/bin/gx. (QmcNhnTpKWEeHoBegJjjbDsvtpNxCroWxWFg8yzHVmXmyh)
Checksum mismatch on ./vendor/src/github.com/ipfs/go-ipfs/bin/gx-go. (QmfNsYvjmztqya5RipyDoFQU7Ps4YNhd3BeXk4EzNNraD8)
error: in manifest, missing from pack: ./vendor/src/github.com/multiformats/go-multihash/test/sharness/bin
error: checking file ./vendor/src/golang.org/x/crypto/pbkdf2: open ./vendor/src/golang.org/x/crypto/pbkdf2: too many open files
[... about a couple hundred repetitions of that error with different paths ...]
Pack verify found some corruption.

ipfs-pack PackManifest files have a similar structure to BitTorrent's "manifest-like" .torrent files. Torrent files play a prominent role in the BitTorrent UX. This issue is aimed at comparing & contrasting PackManifests and torrent files so that we can consider any implications for the ipfs-pack UX.

First-Pass at Comparison/Contrast

While IPFS has some distinct advantages over bit torrent, especially for distributed archives, there are strong UX advantages to using manifest files as a convenient, compact, relatable way of identifying and sharing datasets.

Implications

The main UX benefit for manifest files is that it allows UX scenarios where I give you a PackManifest (which is a lot smaller than the pack), or the hash of the PackManifest (which is only a few bytes), and you can use that to acquire the rest of the pack's contents directly from the network.

Some implications of this:

• ipfs-pack repo should include the PackManifest in the object store that it builds.
• Consider: in scenarios where I share a datasets, should I share the hash of the PackManifest or should I share the root hash of the dataset? What are the implications of each approach, since both of them ultimately let me retrieve the same blocks.

Background

The original draft proposal for ipfs-pack lists:

Other tooling examples

• BagIt - https://tools.ietf.org/html/draft-kunze-bagit-06#section-2.1.3
• WARC - https://en.wikipedia.org/wiki/Web_ARChive
• BitTorrent's "manifest-like" .torrent file

The ipfs-pack spec addresses the relationship to BagIt bag files in the background and the ipfs-pack bag method description, but it doesn't address the relationships to torrent files. This issue is aimed at gathering that info so we can update the spec and consider any implications for the ipfs-pack UX.

This seems unmaintained and sadly, we have little bandwidth to maintain it as a core component. I suggest moving to ipfs-shipyard.

Will do in 1 week if no voices against.

Newlines are technically valid in filenames, but since the PackManifest file is newline delimited this poses a slight problem.

Unsure how exactly to progress here. Do we disallow newlines in names? or escape them? Probably escape, but its worth a little bit of discussion.

ipfs-pack serve appears to chose a random port. Is there a way to chose a specific port, or alternatively, which ports should be opened in a firewall for ipfs-pack serve to work?

We should always check type assertions.

Given the considerations mentioned in ipld/legacy-unixfs-v2#15 (comment) (following which I found ipfs-pack) for data that chunked in a content-aware fashion (there are mentioned video streams chunked at keyframes; my applications would rather have raw camera files that contain the literal jpeg inside them, images with XMP that is served as a copy in a sidecar file, or zip files without compression), I think that there are practical cases in which having a fmtstr that only tells the chunker and its parameters may be insufficient to reproduce the data already published on IPFS.

Would it be viable to have the option to explicitly describe the chunking, rather than giving rules the recipient might not be able to follow?

Straw man example:

Qm... <f0000120001> file1.jpg
Qm... <(3200, f0000120001) (4096000, f0000120001), (50000000, f0000120001)> file1.raw
Qm... <f0000120001> .

where file1.jpg is 4096000 bytes long and somewhere in the first part of file1.raw (with 3k of header and 50M of actual raw image at the tail). To reproduce the hash of file1.raw that has been published on IPFS, one needs to concatenate the results of chunking the first 3k, the jpeg data and the rest with the DefaultRaw chunker each rather than running it over the whole file.

I am currently unaware of tools that produce such content-aware chunkings of data (only tried it out manually so far), but the above-linked suggestion indicates that this is more than a pet-peeve of mine.

@whyrusleeping said:

The 'ipfs-pack make' process is computing the total size of the files being added in the background. this will take some time, so until it finishes you won't get any sort of a time estimate

I think this is signaled by the ? and the "working" bar in this image:

• I don't think just ? is super clear.
• maybe there's more cut off in the rest of the image
• some products use calc or calculating total or something
• we should avoid giving an ETA in time until the total is calculated. (that could say calc or calculating or estimating too)

It just shows "[command]" which is not helpful:

$ ipfs-pack help help
NAME:
    - Shows a list of commands or help for one command

USAGE:
    [command]

Not sure if this should be here - will move to go-ipfs if its more appropriate...

So, I have a pretty big set of video files that I have packed (over 30 GB) - the ones of interest are 720p (ca 2.5 Mbs) - all of which have been correctly transcoded for streaming using h264 baseline encoding.

I have been testing using the https://ipfs.io/ipfs/Qm... gateway. Most of the videos playback just freaking great, not a framedrop, smoother than vimeo, as a matter of fact. So this report is also a report about the success of the IPFS system and the ease with which ipfs-pack can be used. Fast and easy. 💯

However, and its a big however, I have noticed that some (1 in 10 maybe?) of them stall at a specific point (unique to each file, but generally it is at about 10 sec) - my guess is that it is at a block boundary... This was bugging me. Sometimes I would get them to play after reloading the browser several times, but one of them consistently stalled on several different machines. Less then 100% is not an option, so I needed to get to the bottom of it.

This video was 42 seconds long and was consistently stopping at around 34 seconds - on several different computers. Both playback and direct download would stall at the 7.5 MB mark...

Was the file corrupt / broken?

• rsynced to a local machine from the remote box
• file size checks out (10MB, not 7.5MB)
• plays back appropriately in firefox and vlc.

So it wasn't a corrupt file. Then I remembered that I actually moved this particular file from one folder to another folder, renamed it and rebuilt the pack (PACK v2). Before I did this I shared the link with someone, who tried to watch the film the next day. (Several hours ago, actually.) Even though I sent them a new link to the film, they still tried to watch it with the stale link from PACK v1.

PACK v1
/packroot
  /folder
    wrongName.mp4
  /folder2

PACK v2
/packroot
  /folder
  /folder2
    rightName.mp4

I was no longer serving the "stale" link with ipfs-pack, because I regenerated the PackList and made a new PackRoot hash (PACK v2). But they were watching it, even though I wasn't serving it - I suppose it got cached on the IPFS network, which is technically awesome. The weird thing was that the video referenced by both the old link from PACK v1 and the new link from PACK v2 (that I definitely was serving) were stopping at exactly the same place!

I checked the ipfs-pack HUD, and saw that indeed, ipfs-pack (PACK v2) was throwing an error that ./folder/wrongName.mp4 couldn't be found. So I changed directory to ./packroot and wrote cp ./folder2/rightName.mp4 ./folder1/wrongName.mp4 sigkilled ipfs-pack and started it back up again...

Now the file is accessible from links at either PACK v1 or PACK v2, serve to completion and ipfs-pack doesn't throw any more errors.

Probably not important, just thought it was funny and I should share...

Just a quick question

Is the ipfs-pack bag available in the latest version 0.6.0?

Hitting ^C causes ipfs pack to echo "Shutting down ipfs-pack node..." and then it hangs.

So, I have a fundamental question about maintaining packs. Can you, @whyrusleeping give me some advice?

I am having to manually rebuild my 34GB pack almost every day. This size of a pack is kind of the limit of usefulness, because that means that I have to tear down the daemon while I am rebuilding - which takes about 15 minutes. Our use case, however, foresees Packs growing to 3TB size (the size of a current consumer HDD) - and this is a problem, because packing that many files TAKES A VERY LONG TIME®.

These are the cases that I would like to look into:

1. An unhashed file needs to be created and added to the pack.
2. A hashed file needs to have its metadata changed. (Is this even possible? Can I just switch out the first block...?)
3. A hashed file needs to be in a different directory.
4. A hashed file was transcoded wrong, and needs to be unpinned (EVERYWHERE) and replaced.

In all cases, it is not really acceptable to kill the main ipfs-pack thread. Can I manually change or add a hash to the PackManifest instead of rebuilding the pack - while still serving? I have found two files that I need to replace (my file ingest was borked, need to do it again for the main file and transcode it for the second file). For now rebuilding the pack would work, but like I said above, our data volume is going to explode and we can't be offline for 30 minutes. It would be great to have commands like:

 $ ipfs-pack inject ./folder1/somefile.mp4 
 $ ipfs-pack move ./folder1/somefile.mp4 ./folder2/somefile.mp4
 $ ipfs-pack purge ./folder2/somefile.mp4

Of course, this could all be managed by integrating a watchfolder pattern and updating the PackManifest accordingly. I am not sure how possible that is, but it would really make the daemon superpowered. Like:

ipfs pack serve --watchfs

Of course it is also possible that I am trying to find ways to use ipfs-pack that are inappropriate anti-patterns and for which other tools exist. If that is the case, I would love to hear about a better way to do this...

Should we remove the Files on IPFS Lesson from the Decentralized Web Primer and tell people to use ipfs-pack instead?

The old Files on IPFS lesson revolves around using ipfs add -w to add file content and wrap it in directory info. Should we still include that in the "introductory" material?

If we keep that older lesson, which gets closer to the core of how IPFS works, how should we frame the lesson that covers creating ipfs-packs and serving their contents?

This decision is important because it will influence how people encounter IPFS for the first time and how they think it's supposed to be used.

go get -u github.com/jbenet/go-random-files/random-files

With a pack that contains symlinks, ipfs-pack verify reports the symlinks as corruption.

For example, I cloned a copy of this repo and ran

make build
ipfs-pack make
ipfs-pack verify

Which resulted in this output:

Checksum mismatch on ./vendor/src/github.com/ipfs/go-ipfs/bin/gx. (QmcNhnTpKWEeHoBegJjjbDsvtpNxCroWxWFg8yzHVmXmyh)
Checksum mismatch on ./vendor/src/github.com/ipfs/go-ipfs/bin/gx-go. (QmfNsYvjmztqya5RipyDoFQU7Ps4YNhd3BeXk4EzNNraD8)
error: in manifest, missing from pack: ./vendor/src/github.com/multiformats/go-multihash/test/sharness/bin
Pack verify found some corruption.

Couldn't get this to work...

From the spec, I have seen lots of duplications in functionality with the vanilla ipfs, such as ipfs repo *, ipfs tar *, ...

This results in more documentations and cases to write for each of these new commands, even more documentations to describe how these differ from the default ones.

The only conceivable reason is that restructuring go-ipfs might necessarily result in breaking change, and ipfs-pack was meant to be a fresh, cleaner implementation for managing one repository whereas ipfs itself handles many repositories (~ npm vs npm -g) and doesn't default to filestore. But most api calls people have used are add/get/object, unaffected by these details of archive/exchange formats.

Wouldn't it be simpler to have them all be:

• ipfs add -> ipfs add
• ipfs tar add -> ipfs add --format [bag|warc|car|nar|tar]
• ipfs-pack make -> ipfs add --nocopy --pack . or ipfs add --nocopy --format car
• ipfs-pack verify -> ipfs repo verify --local
• ipfs-pack ls -> ipfs repo ls --local # implies filestore is being used
• ipfs-pack gc -> ipfs repo gc --local
• ipfs-pack serve -> ipfs daemon --local
• ipfs get/ipfs cat detects whether the hashed content is formatted in bag/warc/car/nar/tar. ipfs get has options to write the output to be whether in bag/warc/car/nar/tar

Given I have the root hash of a pack
Then I should be able to view the pack manifest with ipfs cat ROOTHASH/PackManifest

This would allow me to rebuild an exact replica of the pack. It also allows me to quickly and consistently see what's in any dataset that's been published using pack.

✔fil:tutut [ master | ✔ ]> pwd
/Users/fil/Source/ipfs-pack/tutut
✔fil:tutut [ master | ✔ ]> ../ipfs-pack make
wrote PackManifest
✔fil:tutut [ master | ✔ ]> ../ipfs-pack serve
verified pack, starting server...
seek PackManifest: invalid argument
✔fil:tutut [ master | ✔ ]> cat PackManifest
zb2rhXbLoTfgP8xqk912oE8eynnDsMLhmZHxCSiVymfdNE6Hq	f0000120001	./hell1.txt
zb2rhXbLoTfgP8xqk912oE8eynnDsMLhmZHxCSiVymfdNE6Hq	f0000120001	./hellO.txt
QmXKJxD7MPJKtfo9oYo7x6yko5vYtHUwGRjj3R5YSKPvdc	f0000120001	.

(an OSX issue?)
@whyrusleeping