google / trillian Goto Github PK

They sometimes fail with the server complaining that a port is already in use.

Pretty sure that both of these aren't implemented atm as that work predated the split.

In the CT example frontend.

Consider removing TrillianStatusCode so that clients have a single place to check for errors.

Updates made to CompactMerkleTree in d1ee609 (PR #180) to return a copy of the internal Node state, have not properly covered the edge case where the tree is perfectly balanced (i.e. has 2^n leaf nodes).

In that situation the set of nodes should be empty: the size & root hash alone describe the tree, as evidenced by the fact that when merkle.NewCompactMerkleTreeWithState() is called it will make 0 calls to the backing store via its getNodeFunc to retrieve hashes.

Write a log integration test for consistency proofs between various tree sizes.

Public clients only need an array of neighbor hashes to verify proofs.
Including extra data invites API missuse.

It can be done via existing API but needs client to issue two RPCs. Might as well provide this in the backend. It will be useful for debugging.

Fetching correct node set and stuff. Note: for this milestone initially only at tree sizes where we have an STH because the intermediate recalculations are complex. Needs to support by hash and by index, which are fairly similar operations.

Don't do this until all the handler PRs are merged to avoid rework.

Should trigger termination of the sequencer goroutine(s) and block until it's done.

Including at least:

• operation counts
• status breakdown by operation
• sequencer failures

Return a generic storage object rather than new objects for each tree.

• Eliminates the need for caching within the storage factory.
• Eliminates the need to create a fake tree in order to test DB connectivity.
• Simplifies testing.

Partial work started here

The KeyManager interface currently supports returning error for several of the Get* functions.
This dramatically increases code complexity for calling functions. All these error params can be eliminated if the New* functions require a valid key to be loaded before returning a KeyManager object. Is there a strong reason to support starting Trillian without key material?

type KeyManager interface {
	Signer() (crypto.Signer, error)
	SignatureAlgorithm() spb.DigitallySigned_SignatureAlgorithm
	HashAlgorithm() crypto.Hash
	GetPublicKey() (crypto.PublicKey, error)
	GetRawPublicKey() ([]byte, error)
}

Proposed interface:

New(...) (KeyManager, error)
type KeyManager interface {
	Signer() crypto.Signer
	SignatureAlgorithm() spb.DigitallySigned_SignatureAlgorithm
	HashAlgorithm() crypto.Hash
	GetPublicKey() crypto.PublicKey
	GetRawPublicKey() []byte
}

Existing tests are at a small tree size (up to 7 leaves). This is not enough to be sure about the code. Needs more extensive tests.

In the example CT frontend including all the RFC 6962 gubbins.

Make it possible to prevent very recent leaves from being eligible for sequencing. For more details see the existing CT repository.

Hasher needs to be an interface to support alternative hashing implementations.
Here's a proposal that supports both logs and maps.
Individual implementations are not required to incorporate all input fields into their hash.

// TreeHasher provides hash functions for tree implementations.
type TreeHasher interface {
	HashLeaf(treeID, index []byte, depth int, dataHash []byte) Hash
	HashEmpty(treeID, index []byte, depth int) Hash
	HashInterior(left, right Hash) Hash
}

Steps:

• Move HashKey to Personality
• Create Hasher interface
• Create test implementations
• Migrate to new function interfaces.

Write a log integration test for inclusion proofs at various tree sizes.

Similar to backend tests. Demonstrate that the log operates correctly, signs and returns the correct objects etc. and generally behaves itself.

If I add --alsologtostderr to the invocation of ./trillian_map_server in integration/map_integration_test.sh, I see lots of warnings:

W0131 09:37:43.527794   11656 subtree_cache.go:185] Unexpectedly reading from within GetNodeHash()

The comment there says "This should never happen - we should've already read all the data we need above, in Preload()"...

We need some kind of administrative API that supports creating and deleting LogIDs.
There's ongoing work on an admin API. Not sure how to connect this bug to that.

Currently, the storage.QueueLeaves API effectively enforces an all-or-nothing failure mode for the addition of each leaf in the array, which is not ideal. E.g. this would cause a batch of additions to fail if the log didn't accept dupes but one was presented.

When adding a batch of leaves some leaves could fail, e.g. due to dupe keys, and the log shouldn't ditch the rest of the batch due to an unrelated dupe entry. Rather, it should submit as many as it can, and report those which failed to the caller.

Currently we can only obtain proofs where the tree size is at an internal STH but it should be possible to return proofs at arbitrary sizes as in existing implementations.

This requires dynamic rehashing of some proof nodes and is complex to implement. Research indicates it's feasible to unroll the rehashing chain so storage can fetch all the involved nodes and the rehashing can be done by post-processing.

Changes involved (plus tests of course):

• NodeReader interface must be able to optionally request tree revisions for sizes >= specified size
• Merkle path code must handle rehashing and annotate node fetches that are part of a rehash
• Proof fetching + rehashing should be broken out of the server to allow separate testing and reuse
• After the storage fetch the rehashing operations must be carried out using the annotations
• API should include a flag to allow rehashing so we can test it without impacting current users

Tests that prove end to end that it's doing the correct crypto and other stuff related to adding and querying entries, signing / sequencing, proof serving.

The Map API is intended to expose a key / value interface, yet the "value" aka. MapLeaf currently contains the key aka Index. Index is also contained in IndexValue, IndexInclusionProof and several other messages, producing confusion about where and when to set Index.

Proposal:
Use proto3 map in Set / Get if we can find a scalar value for index. Perhaps a hex string?
Or standadize on the IndexValue message.

Blocking google/keytransparency#486

Current tests verify inclusion proofs by rebuilding a parallel tree and checking that the inclusion proofs are the same. We should migrate these tests to use a Log Verifier that computes the root hash from the neighbor nodes.

Create a pure go implementation that can verify all the responses that are returned from the bits of Trillian that implement an append only log.

Components:

• SetLeaves
  • SetLeaf
  • GetLeaf
    • Hashing Interface #331
    • GetInclusionProofByHash #320
      • InclusionProofVerifier #334
    • GetSignedLogRoot
      • VerifySignature #351
      • ConsistencyProofVerifier #334

Supports google/keytransparency#384

• Rework schema + backend to add support for application tree heads.
• Update CT example to create / serve application STHs.
• Clean up anything in schema that this obsoletes (public / private STH?)

The current Map Hasher interface contains a HashKey function to turn a string into a sha256 index in the map. This index, however, should be computed by the personality, not the map. Key Transparency, for instance, computes the index as the output of a privately keyed signature function.

If this sounds good, I'll convert the Map interfaces to accept a index []byte rather than key []byte or HashedKey []byte, and remove the HashKey function from the MapHasher.

Fetching correct node sets and doing computations etc. Again only at tree sizes where we have an STH.

https://travis-ci.org/google/trillian/jobs/182679607#L1061

Running test(s)
# github.com/google/certificate-transparency/go/merkletree
cc1plus: error: unrecognized command line option ‘-std=c++11’
FAIL	github.com/google/trillian/integration [build failed]

This resolves the "Too many open connections at once" issue.
It also allows tests to create test databases directly.

Currently the log backend rejects them. Should probably do the same thing as current log implementations, which I think is returning an empty proof but no error.

The signatures library doesn't have a companion verification function.
Investigate importing the one from Key Transparency

Most of the work is done. The API needs to read the hash from the new leaf_value_hash field. When leaves are queried the backend should return both the merkle_leaf_hash and leaf_value_hash.

It's in the API and could be useful for clients but not implemented in the server.

Including at least:

• operation counts
• HTTP status count by operation
• backend returning invalid data

This should be the last remaining handler for the example CT implementation.

Should use the one with the lowest sequence number for the leaf index.

Examples:

log_integration_test.go:97: Test failed: failed to queue leaves: rpc error: code = 2 desc = Unsequenced: Error 1213: Deadlock found when trying to get lock; try restarting