haskell / hackage-security Goto Github PK

• Use library to extract .cabal files (and verify signatures once we introduce author signing; until that time this is just an optimization).
• Avoid creation of the cabal-specific tar index in favour of the index created by hackage-security (though note that the latter does not contain any information about preferred-versions). This should speed up cabal update.
• Once cabal-install only access the index through the security library, rename 00-index.tar and co to 01-index.tar (unless we can about older cabal's access?)

Currently we hardcode the trailer length as 1024; see tarTrailer in cacheRemoteFile.

Not precisely sure how this ought to be done (might also be relevant for #28).

Currently we rebuild it from scratch every time we update the index (even if we do an incremental download). As it turns out, this is quite time consuming. This will require changes to the API exposed by Codec.Archive.Tar.Index.

Currently we hardcode the order in which we try the mirrors as: the mirrors from the cabal config file in order, and then the mirrors from the TUF mirrors list, in order.

Should use a service like http://freegeoip.net/json/ ; the client asks about their own ip address to get a broad region, and then orders mirrors in that region above others.

This is documented in detail in downloadPackage.

I'm not sure what else there is to do, but I think there are a few lose ends to be tidied up. See also #6.

See TODO in FromJSON instance.

This should document things such as

• Bootstrapping
• Key thresholds
• How many keys we have of each type
• Key validity (expiry time)
• Implications for key compromise
• Who holds keys, and how should they store them.

Point people to

• The snapshots (see #14)
• Github repos for cabal-install and hackage-security (we won't make a tarball available for the alpha release)
• Discuss how to set up own server (using hackage-server or using a file server)

Mention what is still likely to change (have a read over https://github.com/well-typed/hackage-security/milestones/Beta ; in particular, #12).

We should do some sanity check on the root.json and mirrors.json files when we load them into the server.

If a cache happens to have, say, an older snapshot than it has a timestamp, a client downloading from this cache will see a security validation error. This is not a problem per se because we will anyway just try again (and on the retry we will provide Cache-Control: max-age=0); however, we can be a little bit smarter here and catch this error before the security checks see it, possibly by taking advantage of Last-Modified.

Since the local repo may actually live on a SSH mounted file system:

• Make sure to deal with I/O errors correctly (existing code should already deal with this correctly, and try again n minutes later) -- Left the existing logic in place.
• Make sure to use atomic file operations
• Think about what the repo looks like when one file is changed but updating the next file fails.
• For the RepoLocal case, do NOT read the index directly from the local repo (because that would be too slow -- because reading on every poll). Instead keep it in a cache (like we do for remote repos).
• Still need to write the index to the repo though (but only on changes).
• Once we are using hackage-security to download stuff from source repo, downloads should be small so can set polling frequency quite high (say, every 5 minutes). -- The polling frequency is not hardcoded into the mirror client, but the max polling frequency is, and was set to 30min. Reduced to 5min.
• Make sure mirroring tool can deal with index containing TUF metadata (currently it tries to mirror each package twice)

This should document:

• Repository layout -- See RepoLayout and co
• Order in which files are to be changed and updated
• Which proxy hints should be provided (esp for 'dumb' file mirrors)
• Specification of the index tarball (for instance, see commercialhaskell/all-cabal-files#8).

Currently we use the layout on Hackage as is, but this isn't quite ideal because it means the layout on Hackage is different from the layout inside the index. See https://github.com/well-typed/hackage-security#footnote-paths.

We need to estimate the size of the timestamp and the size of the root information (in case we need to download it after a validation error).

We should add (the equivalent of?) Cache-Control: no-transform, but should also provide additional caching hints. Currently we don't provide any.

(and update the pattern stuff to deal with Path rather than FilePath)

• Avoid storing copies of all packages in the mirror-cache (this would require a lot of disk space)
• No need to create the index/ directory with all .cabal files
• When mirroring from a particular server, we probably don't want to look at other mirrors (in particular, ourselves!).

See withResponse. Unfortunately however this will probably require upstream changes in the HTTP library (see haskell/HTTP#86).

See https://mail.haskell.org/pipermail/beginners/2008-December/000560.html . This is used in hackage-secure-local to find out all packages in a file system.

• Decide on SHA package -- Done, using cryptohash
• Update hackage-security to use new package
• In hackage-security we avoid loading files into memory -- separate ticket #134
• Update hackage-server to use the new package -- separate ticket #135
• In hackage-server we make sure these SHAs are fully evaluated -- separate ticket #136

(the TODOs are listed in Remote.HTTP). Alternatively, perhaps we could do some of these checks once and for all in the Remote repository implementation.

If we don't, people would also have to install and download a custom patched version of tar.

• which will have the right keys hardcoded in for testing purposes. If the binary is called cabal-secure-beta rather than cabal people can easily switch back and forth.
• test with central server once security stuff is live.
• Test it builds against packages releasd on Hackage
• Mention cabal install in blogpost
• Test with old-style Hackage URI

Having two means we can test mirroring.
Might even want to introduce some intentional errors.

The blog post describes what error we want to introduce; once these repos are set up, fix the blog post to point to the right locations (and remove the TODOs).

For instance, but not limited to, in-code TODOs

• formatRecoverableException ("Can we do better than show for custom exceptions?" -- Yes, we can, but only for base 4.8 and up. See #82).
• unexpectedUncompressedTar in checkForUpdates -- This case is eliminated a the type level instead. Yay! :)
• describeFile for Targets -- describeFile is gone as part of dd2bbca
• wrapCustomEx in hackage-security-HTTP -- We now wrap all exceptions in SomeRemoteError, and then deal with them generically.
• wrapCustomEx in hackage-security-http-client (and the TODO in get) -- As above, opened separate ticket (#85) for the TODO in get.

Also,

• If we only display the final verification exception (the one that is thrown up), it is important that this isn't just "verification loop".

This is the flip-side of #33.

Currently all commands take a path to keys/ directory that contains all keys (public and private) for all roles. Ultimately of course these keys will all live in different physical locations, and key signing will have to be done in a more distributed manner.

And then support both the current Hackage layout as well as the ideal Hackage layout. Test with the current hackage layout (we've so far been testing with the ideal layout instead).

I.e., in a keys section of, say, the root metadata, if we have an entry (keyId, key), we should verify that keyId is in fact the key ID of key key.

See FromJSON instance in Key.Env.

We currently do an incremental update whenever we already have a local copy and the server provides it; however, we might want to do additional checks (@dcoutts has a sketch at https://gist.github.com/dcoutts/7bbad299d87d23a806a7).

Currently we search the entire index for packages that we need to migrate, but we can take advantage of the fact that @dcoutts also introduced a separate change into to schema, which we can detect using single call. If we do all migrations at the same time, we don't need to do any searching.

Currently we use the system temp directory everywhere, which may not be what the user wants and moreover might cause trouble wherever we call renameFile. This affects the core library and hackage-security-local, as well as potentially any of the hackage-security-* HTTP libraries.