In #6 we discussed and fixed many initial user feedback issues.
In the course of a layout tool user testing session, it was noted that when looking at stdout/stderr of the used tool (in-toto-mock) it was unclear if the command was executed properly or not.

Here are some general user feedback related items open for discussion and fixing:

• Make better use of verbosity levels
• Reduce the output if -v flag is not specified
• Make clear if a command failed or not
• Colorize output
• Do something about subprocess32 warning (seems to confuse users)

Layout metadata and link metadata is stored in canonical json format. It would be nice to have a way of showing metadata in a logically structured and human friendly way as well.

• order fields logically instead of alphabetically
• strip keys and signatures

in-toto steps are usually executed using the in-toto-run command line utility, whose functionality is implemented in the according library function runlib.in_toto_run.

Since this command requires a signing key and a bunch of command line arguments, it would be helpful to have a stripped down variant of that command that can be used to easily mock in-toto-run's metadata generation aspect, without the need to specify a key and knowing all the command line arguments.

This is particularly interesting for in-toto's layout tool, where we ask un-experienced in-toto users to (mock) run their supply chain, so that we can create a layout using the (mock) link metadata.

Here is an example usage of the envisioned command:

in-toto-mock -n <step name> -- <step command>

The resulting link metadata <step name>.link-mock is not signed and records materials and products at the current working directory where the command is executed.

I think in-toto-run's --verbose and --record-byproducts flags should be on by default.

UPDATE:
As mentioned above in-toto-mock most likely will be used to reverse generate in-toto layouts with an initial set of custom artifact rules. Therefor it might be helpful to store the full path of the current working directory in resulting mock link files.
I think for now we should only store it when running in-toto-mock but make it so that we can easily opt in later to also store it when using in-toto-run.

Meaningful exceptions are preferred over zero vs. non-zero return values in verifylib and runlib (this is the pythonic way).
The command line tools toto-run and toto-verify on the other hand could catch those exceptions, print meaningful output to stderr and return zero or non-zero values.

@SantiagoTorres proposed to use a datetime object for the layout's expires field with a default timeout (e.g. a month). This is a good idea. Also we should check this upon toto-verify.

As pointed out by @awwad in-toto/layout-web-tool#14 (comment) the usage message of in-toto-sign is inaccurate.

The short option form -k KEYS does not actually work with the indicated whitespace.

Also, I think we should change the subparser names. in-toto-sign sign ... sounds stupid.

Steps and Inspection are both items of the software supply supply chain defined in the layout.
Steps are performed by functionaries and correspond to one or more link metadata files, whereas Inspections are performed during toto-verification, which also produces link metadata.
Both item's link metadata is verified against the defined matchrules in a similar fashion. This bears potential for generalization.

I'm inclined to call the generic type Item or LayoutItem. Any suggestions would be appreciated.

Note:
@JustinCappos requested a name change fortoto-run's --name flag which sets the name field of a link metadata file and has to correspond to a Step or Inspection name defined in the Layout. He suggested --step or --step-name. I'd like to keep it abstract for Steps and Inspection, e.g. --item-name.

People will want to use and try toto without downloading a multi-gig virtual machine (with old code).

• Provide README.md with easy installation and instructions for a simple workflow
• Update pip requirements file

Currently we store the list representation of Matchrule objects in the according Step and Inspection fields. This is a workaround that allows us to use attr's convenient asdict function when dumping the layout to a file.

I suggest to implement our own asdict function allowing us to store Matchrules as objects but returning their list representation instead of their dict representation when dumping the container layout.

I think we should allow the rule to be written as strings. A .split() when parsing the rule is not very expensive, whereas a split when writing the rule somehow feels stupid. I for instance script my rules like so:

material_matchrules = ["MATCH PRODUCT yadda yadda".split(), ...]

Currently in-toto raises builtin Python errors as well as custom exceptions defined in securesystemslib.exceptions and in-toto.exceptions.

It might be worth taking some time to grep through those raise statements and re-assess the suitability of each raised error and its message.

See also @SantiagoTorres' suggestion in #123 (comment).

in-toto Layouts and Links are both signable.

Let's add a command line utility to handle in-toto metadata signatures. Here are some functions I consider helpful:

• add signature, while keeping existing signature(s)
• replace existing signature(s) with new signature
• verify signature(s)

Feel free to add your own ideas.

We decided to rename the framework to 'in-toto' (lat. as a whole) to avoid collisions with existing software projects.
We need to refactor the code base accordingly.

According to recent specification changes in-toto sublayout verification expects link metadata files pertaining to a sublayout to reside in a fileystem directory with the following naming scheme:

<SUBLAYOUT NAME>.<SUBLAYOUT OWNER KEYID PREFIX>

SUBLAYOUT NAME: Name of step in superlayout that is carried out as sublayout
SUBLAYOUT OWNER: Functionary authorized to perform the step of superlayout that is carried out as sublayout
KEYID PREFIX: Left-most 8 characters of keyid

See 4.5.2 Namespacing link metadata for more details

The readme explains in-toto's general purpose briefly, then starts explaining what a layout contains, without providing the few sentences that I think would help a reader understand the in-toto workflow.

In particular, I think that the last paragraph of the first section of the readme (sorry, can't link line numbers in .md in github) would be better with a little bit more detail. For example, I might change it to something like this:

In-Toto Workflow

in-toto requires a project layout that specifies the functionaries and the tasks they are supposed to perform. The project designer (((or whatever term is preferred))) creates this project layout as a sequence of individual steps, referred to as links, designating functionaries to be trusted for each step. The functionaries in charge of each link produce link metadata once they've completed the work in their link, and provide this metadata to (etc.....). The chain of link metadata travels with the final product, and can be validated manually by the final client or via automated tooling in, for example, a package manager.

The specification recommends storing information about the environment, in which an in-toto step is executed, to the resulting link metadata.

See section 4.4.1 Environment Information of the specification for more details about what should be stored.

Some preliminary questions are:

• Should storing be parameterizable, e.g. via command line options to in-toto-run/in-toto-record?
• If so, what should be the default?
• What do we want to store under the filesystem key?
• How do we infer which files/directories/filesystem layers are relevant?
• Do we already have use cases for this information?

Note:
Storing the working directory as suggested would eliminate the need for
the MockLink class, a Link subclass with an additional working_directory property
added in #95 to create a Layout based on Link metadata

The current rule type MATCH uses the following language:

MATCH (MATERIAL|PRODUCT) <path pattern> [AS <target path pattern>] FROM <step name>

Currently this reads (my interpretation of the specification):

MATERIALS or PRODUCTS, depending on the rule's second element, of a step with <step name> and matched by <target path pattern> (if specified) or <path pattern> (else), must have a hash that matches the hash of a MATERIAL or PRODUCT, depending on whether the rule is in the material_matchrules or product_matchrules, of a step that matches <path pattern> and has not been matched by a previous rule. The rule also fails if the target path pattern does not find any artifacts.

When I and @SantiagoTorres, who came up with the language and implemented it, tried to apply these rules in real world software supply chains, we both had difficulties to express what we wanted, which IMHO indicates that the language should be clarified and freed from ambiguities.
The following (or at least most of it) should be unmistakingly clear by just reading the rule:

• What is the match source and what is the match target?
  rule in one of step's rule lists VS. ... FROM <step name>

• What is the source artifact type and what is the target artifact type?
  rule in material_matchrules or product_matchrules VS. MATCH (MATERIAL|PRODUCT)...

• Should there be a target artifact for every source artifact? (currently no)
  material_matchrule (source): MATCH PRODUCT * FROM target-step
  source materials : hash(foo) = x, hash(bar) = y
  target products: hash(foo) = x
  Pass?

• Should there be a source artifact for every target artifact? (currently yes)
  material_matchrule (source): MATCH PRODUCT * FROM target-step
  source materials : hash(baz) = x
  target products: hash(baz) = x , hash(zap) = y
  Fail?

• Should the rule fail if the path pattern does not find any artifacts?
  MATCH PRODUCT foo ... should probably fail if there is no product foo?
  MATCH PRODUCT * ... should probably not fail if there is no product?

• Do we solely match by hash in case of wildcard patterns?
  material_matchrule (source): MATCH PRODUCT * FROM target-step
  source materials : hash(new/path/foo) = x, hash(bar) = y
  target products: hash(foo) = x, hash(baz) = y

• Can artifacts be matched more than once by different MATCH rules? Would AND, OR operators to concatenate MATCHRULES help here?

Below is a super explicit language to start the discussion. This might be exaggerated but at the same time still does not express everything. I don't say, maximum explicitness is a must, nonetheless implicit behavior should be defined somewhere, e.g. in the specs.

MATCH [ALL] (MATERIAL|PRODUCT) <source path pattern> FROM <source step name> WITH [ALL] (MATERIAL|PRODUCT) <target path pattern>  FROM <target step name>

Currently we provide a command line flag to toto-run to either don't touch stdout and stderr, which is required if toto-run wraps interactive commands like e.g. vi, or to redirect and store them as byproducts to the link metadata.
Ideally we would like to duplicate the streams to allow both.

• Using a pipe won't work because processes like vi will complain
• Wrapping stdout/sterr in Python won't work because the suprocess will only take the filedescriptor and then uses it natively
• Reading from /dev/(stdout|stderr), /dev/tty is *NIX specific

We have been silently accepting a minor metadata format mismatch between specs and implementation. I think it is time to fix that by choosing one or the other.

Reason for the format as specified: Adopted from TUF
Reason for the format as implemented: is-a relationship seemed (from an OO perspective) more intuitive and easier to handle than has-a realtionship.

Implementation

{
  "_type": "layout" or "link",
  ... # rest of the respective properties,
  "signatures" : [
    { "keyid" : KEYID, "method" : METHOD, "sig" : SIGNATURE },
  , ... ]
}

Specification

{
  "signed" : ROLE, # layout or link metadata object [**]
  "signatures" : [
    { "keyid" : KEYID, "method" : METHOD, "sig" : SIGNATURE },
  , ... ]
}

# [**] Btw. I find the term `ROLE` for the metadata type really confusing. 
# While it makes sense for TUF, # where each role corresponds to 
# an individual metadata format, it does not for in-toto, which does 
# not have the some notion of roles.

Delegation is provided by defining a step in the root layout that will be represented by a toto link metadata file of type layout in verification.
This needs to be enabled in toto-verify/verifylib.

The README should link to the project's website. There is also an option of adding a website to the repo by clicking on "Edit" in the repo's description.

@SantiagoTorres made a valid point in preferring "{}".format over "%s" % () for string formatting.

See more info in the Python docs:

Note: The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and dictionaries correctly). Using the newer str.format() interface helps avoid these errors, and also provides a generally more powerful, flexible and extensible approach to formatting text.

Inspection commands defined in the layout provide the possibility to execute arbitrary commands at verification time.
Inspections, like Steps, must have rules to authorize/enforce materials and products that are recorded before and after an Inspection command is run. Currently a failing rule is the only way to make an Inspection fail the verification.
Additionally, it might be desired to simply fail verification when the inspection command returns a non-zero value.

• verifylib.py :Currently the verifylib is implemented as one long function. We should separate the single verification steps into function units that are called from the commandline tool toto-verify.py.
  Also, functions in verifylib should not contain log/debug/info statements but rather raise meaningful exceptions that might be caught in toto-verify.py
• runlib.py
  Runlib is already split into separate functions but there is one "call-them-all" function in runlib which should be moved to toto-run.py. Same as above applies for log statements here.

The current code base is mostly Python 3 compatible. Some minor changes are required though to use in-toto in both Python 2 and 3.

• Replace print with print() (cf. @SantiagoTorres's and @vladimir-v-diaz's comments
• ...

We need schema validation for Links similar to what we do for Layouts, Steps and Inspections.

PR #85 modified an for loop inside the signature verification routines. This for loop could be further simplified and factored out.

I've noticed that we are doing imports as follows:

from x import y, z, \
           i

While this is a correct way of doing it, I'd suggest we move to the parentheses construction instead:

from x import (y, z,
         i)

The rationale behind is that our code style (which is based on pep8) strongly hints of using parentheses for line continuation^*. PEP8 also makes the same suggestion:

The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses. These should be used in preference to using a backslash for line continuation.

If I recall correctly, tools such as flake or pep8 mark this as a style error....

_{*Should we also consider updating the example in our style guideline?}

We share crypto specific libraries with the update framework and started off toto using TUF's format/schema validation. As we are more on the OO side than TUF, we will want to replace format/schema validation with model validators, e.g. using attr's validator module.

• implement validators (the downside of attr.validate is that it seems to be hooked to the __init__, which is not ideal if we want to set attributes post-instantiation and then validate)
• remove toto/formats.py
• keep using and toto/ssl_crypto/formats.py for crypto specific data structures, and add where appropriate

Some of the unit test code needs to be refactored and cleaned up.

This issue is left up ambiguous intentionally, as it will be discussed in an upcoming hackathon.

We need a better plan for how to integrate link files, the layout, delegated layouts, etc. into software we distribute. Asking the user to copy a bunch of things really doesn't cut it.

Should there be an in-toto-bundle step which automates this (somehow)? Should we do something different?

It can occur that the directory in which toto-run is executed and the base of the directory to record materials and products are not the same.
Consider below directory layout and a toto-run to be executed in b and record materials in a:

├─/path/to/project
          ├── a
          │   ├── bar
          │   └── foo
          └── b
# in /path/to/project/b
toto-run.py --materials ../a

this results in a materials list:

materials = [
   "../a/bar" : <hash>
   "../a/foo" : <hash>
]

Currently matchrules would need to add dots or double-dots in the path pattern to match the recorded artifacts. This can be circumvented by using wildcard patterns, which does not seem ideal.

Two possible solutions:

• Ignore leading dots/double dots at verification time
• Add a base-path command line option to toto-run, record absolute artifact paths, leftstrip base path from absolute artifact path before writing link metadata.

• Support file path wildcards for artifact flags in toto-run and consequently in toto-verify
• Re-think "./" left-stripping of paths when recording artifacts (maybe this could be done in verification instead?)
• revise record_artifacts_as_dict function

Later:

• think of a more sophisticated way of recording artifacts than by passing file paths as command line arguments (this might deserve a separate ticket)

Currently we provide a throw-away create_and_persist_or_load_key function which served its purpose during development.
In future we definitely want something a little more sophisticated, although PKI is not part of the toto specification.

• Add pki helper interface to create and store public/private keypairs (offer standardized format, eg. PEM)
• Change key loading calls throughout the codebase to load keys in various formats or die
• Disable the possibility of adding the private portion of a key to the toto layout

The in_toto.util module serves as convenience interface for securesystemslib.keys. We could probably move the whole module over to securesystemslib and revise the code while we are at it.

For instance:
@SantiagoTorres pointed out that there should be a better way to determine if a key is password encrypted, than by simply loading it without pw first and then with pw if a generic CryptoError is raised. I agree with him.

The docstring of in-toto's Layout as well as the default value suggest that the keys property should be a list. This does not conform with the in-toto specification and the demo implementation.
The correct format is ssl_crypto.formats.KEYDICT_SCHEMA.

Please change docstring default value, and while you are at it, add model validators and unit tests!

It would be nice to have bash completion support for in-toto command line interface. I was trying out the demo application and found it that the bash completion is not present.
Adding bash completion would be a good feature IMO and I'll be happy to take it as a task.
Thoughts @lukpueh @JustinCappos @SantiagoTorres

As of the 0.9 version of the in-toto specification, the Layout metadata should include an optional README field.

We need to update the Layout model to reflect this change.

Currently we use default constructors for all in-toto classes. There is a "factory-like" read method which has been used in the demo so far.

It would be useful to have intelligent constructors or factories for objects that will be created manually by the project owner, i.e. Layout, Step and Inspection, with the following capabilities.

• Take key dictionary or key list and create key dictionary from it (Layout) (wontfix)
• Take expected command in list or string format (Layout)
• Add default expiration if not specified (Layout)
• Take matchrules in list or string format (Step, Inspection) (wontfix, would require escaping)
• Validate passed data

The names of the artifact rule properties of in-toto steps and inspections have been updated in the specification to expected_materials and expected_products whereas in the implementation they are still called material_matchrules and product_matchrules.

These names have to be refactored in the codebase and documentation.

Currently we pass command verification upon strict (must be equal) and relaxed (must have an equal prefix) alignment of recorded command and expected command. The following examples show in what cases the alignment passes or fails.

# Left command is the command recorded by toto-run.py
# Right command is the expected command defined in the layout

# Passes:
["vi", "file1", "file2"] ~ ["vi", "file1", "file2"]

# Passes with warning:
["vi", "file1", "file2"] ~  ["vi"]
["make", "install"] ~  ["make"]

# Fails:
["vi", "file1", "file2"] ~ ["make", "install"]
["/usr/bin/vi", "file1"] ~ []
["vi", "file1", "file2"] ~ ["vi", "file2", "file1"]
["/usr/bin/vi", "file1"] ~ ["vi", "file1"]

As one can see in above examples that semantically equal commands sometimes fail whereas syntactically (relaxed) equal commands can pass although they have very different semantics.

@vladimir-v-diaz brought up the idea of using match patterns. We could use regex or a subset of regex like implemented by fnmatch. This would allow either fail or pass and get rid of the relaxed pass.

In any case command alignment does not provide strong security guarantees, since recorded commands can be easily faked using alias, symlinks or the like.

Currently in-toto provides some utility functions to create (password encrypted) RSA key pairs programatically.

It would be nice to additionally have a command line utility, e.g. in-toto-keygen, for key creation (like a stripped down ssh-keygen).

Keep in mind that we might want to drop the util module at some point and move its (updated) functionality to securesystemslib/keys.

in-toto inspection commands that are executed upon verification need to be on the verifier's PATH. How do we want to integrate custom inspections?

I see two options:

• Integrate them with in-toto (feasible if it's only a few)
• Maintain them as a separate project and list them as dependency for in-toto

As it is now, there are two places that perform signature verification. Following review #77, we decided that both routines could be placed in the same place, and have the second place just verify that the signature matches the keyid prefix in the filename.

Make in-toto use the tip of secure-systems-lab/ssl_crypto and secure-systems-lab/ssl_commons as submodules.

The tip of in-toto currently uses vladimir-v-diaz/ssl_crypto@542caed as crypto submodule. The common package has not been sub-modularized yet at all.

This issue can be broken down into:

• Make sure that secure-systems-lab/ssl_crypto and secure-systems-lab/ssl_commons are up-to-date with what in-toto requires.
• Replace package in-toto ssl_commons with submodule
• Adapt in-toto to work with changes that were made in the submodules

Currently the one-month default expiration date for the layout is created in the layout's read method. This should rather happen in the layout's constructor.

Currently in-toto-verify calls verifylib.in_toto_verify and fails immediately if any of the verification sub-routines fails.

It would be useful to allowin-toto-verify and verifylib to call verification routines individually and independently, e.g. only

• a section of the supply chain
• artifact rules
• signatures
  ...

We need more real world in-toto examples! This time it's Debian.

The idea is to show how in-toto can be used to guarantee the integrity of a software project from upstream development to installation using a Debian package management tool.

Let's use our great in-toto Connman OpenSUSE demo as a basis.

Here are some relevant pointers:

• Connman Homepage and Upstream Repo
• Debian Developers' Corner
• Debian Maintainer Guide
• Debian Connman Package Tracker (with links to package site and git repo)

Additionally, this might be helpful:

• A Debian Developer's thoughts on supply chain security
• A Debian hello world package and a packaging diagram (Note: I am not guaranteeing the freshness or relevance of those links)

Command alignment verification passes if the step command as defined in the layout and the related command as recorded by the link metadata are completely equal. If only the first x parameters are equal (where x is the minimum count of both commands) toto-verify should output a warning.

The layout model must support an additional threshold field which contains an integer indicating how many pieces of link metadata must be provided to verify this step. This field is intended to be used for steps that require a higher degree of trust, so multiple functionaries must perform the operation and report the same results.

• add field to model and model validation
• change naming scheme for link metadata files: [name].[KEYID-PREFIX].link
• add in toto-verify/verifylib

In order to verify a layout file that was signed by multiple keys we simply need allow passing multiple keys to toto-verify