ctclone Fails with Incomplete CT Log Responses about trillian-examples HOT 5 CLOSED

Hey, @mhutchinson,

Thanks for getting back to that issue and for the feedback. I will give it a try this week, hopefully will be able to contribute.

from trillian-examples.

Thanks for the well written bug report and the attempted fix in #1045. This sounds like it should be an easy fix, but I think that appearance is deceptive. I wasn't aware of any logs trying to coerce alignment as specific indices, and the design of the cloner doesn't lend itself well to readjusting to these hints; the current design does a very greedy strategy in the batch.go Bulk method where it starts goroutine workers with statically computed ranges to fetch.

I think the BatchFetch mechanism will need some way to elegantly report up that it only got partial results. The cloner will then need to cancel other fetches in progress and restart fetching again, using a start index based on the last result fetched from the incomplete range. To give confidence that this works, we'd probably want to add some kind of functional test that emulates a log performing this coercion.

I'm open to receiving PRs for this. If none are forthcoming I'll try to make time for this but it's unlikely to happen in the next couple of weeks, at least.

from trillian-examples.

I agree, got actually caught up in that "deception" and still have not fixed an issue.

Another option could be to run at first just one request, check it for the alignment and spin workers accordingly received result.
In the approach you offer I think it's important to also implement the cleanup of cancelled or finished fetchers, as it might break the following insertion transactions.

What do you think would be the preferred way to approach the issue?

from trillian-examples.

I like your proposal to have special-casing only on startup. Once a full chunk has been fetched, then it can enter full parallel mode as it currently does, and die on incomplete chunks.

The reason I like this is:

• it's simpler than trying to realign mid-run
• if the whole process is run by some orchestrator (e.g. docker) then this can be configured to restart the process on death. The events of dying/restarting will fix any transient alignment problems when the startup special casing reruns

I think this approach is worth investigating. It seems strictly better than the current behaviour. Is this something you'd be able to contribute, @tired-engineer ?

from trillian-examples.

Sent the first implementation for review, a bit later than expected. I think it covers more or less everything we've discussed on this thread.

from trillian-examples.