To provide something like

$ gto describe iris --version v1
# Output can include DVC metrics, params, plots, dependencies, etc.
# Output can also include MLEM types, frameworks, methods, etc.
# Future output could include LDB data.

we need to implement modules like mlem.gto and dvc.gto to be later automatically discovered by GTO and used to fetch all details about artifacts.

To implement them, we first need to implement abstract interface they must satisfy.

Users Making input mistakes is common, and thus those errors would be very common. Stack traces in the terminal should either never appear or be VERY rare (something critical happened)
So, suggest to handle "easy" and recurring errors (user input errors fall within that category, but maybe other errors as well) elegantly with printing a clear message but hiding stack trace

Examples:

❯ gto show -f text
Traceback (most recent call last):
  File "/Users/oded/iterative/workspace/gto/venv/bin/gto", line 33, in <module>
    sys.exit(load_entry_point('great-tool-ops==0.0.6', 'console_scripts', 'gto')())
  File "/Users/oded/iterative/workspace/gto/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/oded/iterative/workspace/gto/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/oded/iterative/workspace/gto/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/oded/iterative/workspace/gto/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/oded/iterative/workspace/gto/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/oded/iterative/workspace/gto/venv/lib/python3.8/site-packages/gto/cli.py", line 203, in show
    raise NotImplementedError("Unknown format")
NotImplementedError: Unknown format

❯ gto which nn v1.0.0
No version of 'nn' with label 'v1.0.0' active
❯ gto which rf v1.0.0
No version of 'rf' with label 'v1.0.0' active
❯ gto which bla v1.0.0
Traceback (most recent call last):
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/bin/gto", line 8, in <module>
    sys.exit(cli())
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/cli.py", line 147, in which
    version = gto.api.find_active_label(repo, name, label)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/api.py", line 79, in find_active_label
    return GitRegistry.from_repo(repo).which(name, label, raise_if_not_found=False)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/registry.py", line 177, in which
    return self.state.which(name, label, raise_if_not_found)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/base.py", line 148, in which
    latest_labels = self.find_object(name).latest_labels
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/base.py", line 130, in find_object
    raise ObjectNotFound(name)
gto.exceptions.ObjectNotFound: Requested 'bla' wasn't found in registry

❯ gto demote oded mystaging2
Traceback (most recent call last):
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/bin/gto", line 8, in <module>
    sys.exit(cli())
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/cli.py", line 160, in demote
    gto.api.demote(repo, name, label)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/api.py", line 65, in demote
    return GitRegistry.from_repo(repo).demote(name, label)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/registry.py", line 158, in demote
    raise NoActiveLabel(label=label, name=name)
gto.exceptions.NoActiveLabel: No active label 'mystaging2' was found for 'oded'

❯ gto add oded model models2/oded.txt
Traceback (most recent call last):
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/bin/gto", line 8, in <module>
    sys.exit(cli())
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/cli.py", line 68, in add
    gto.api.add(repo, name, type, path)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/api.py", line 27, in add
    return FileIndexManager(path=repo).add(name, type, path)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/index.py", line 89, in add
    index.add(name, type, path)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/index.py", line 29, in inner
    return func(self, *args, **kwargs)
  File "/Users/oded/.local/share/virtualenvs/gto-example-8a8TDK3V/lib/python3.9/site-packages/gto/index.py", line 62, in add
    raise GTOException(f"Artifact {name} already exists")
gto.exceptions.GTOException: Artifact oded already exists

After running showcase.sh in the example repo and then unregistering a model, all commands seem to throw errors:

$ gto show
                 environment
          latest     staging production
features    None         NaN        NaN
nn        v0.0.1      v0.0.1        NaN
rf        v1.0.1      v1.0.1     v1.0.0
$ gto unregister rf v1.0.0
Unregistered rf version v1.0.0
$ gto show
Traceback (most recent call last):
  File "/Users/dave/miniforge3/bin/gto", line 8, in <module>
    sys.exit(cli())
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/cli.py", line 195, in show
    click.echo(gto.api.show(repo, dataframe=True))
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/api.py", line 108, in show
    for o in reg.state.objects.values()
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/registry.py", line 60, in state
    state = self.env_manager.update_state(state, index)
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/tag.py", line 185, in update_state
    state.objects[tag.name] = index_tag(state.objects[tag.name], tag.tag)
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/tag.py", line 165, in index_tag
    obj.labels.append(label_from_tag(tag, obj))
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/tag.py", line 148, in label_from_tag
    version=obj.find_version(
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/base.py", line 103, in find_version
    raise GTOException(
gto.exceptions.GTOException: 0 versions of object rf found, skipping unregistered

It would be cool if we could recommend registering some artifacts. To do that, we can analyze DVC PL and MLEM models, as well as implement some other heuristics.
Not sure if this should be a part of this product, but if can be helpful both in Studio and in CLI.

As an option, this search run in CLI could produce something like artifact_unregistered.yaml.

E.g. if you broke artifacts.yaml when doing things locally or created and pushed an incorrect git tag (incorrect in terms of Artifact Registry).
Beside creating GH action, pre-commit hook could also be helpful.

There are cases when you would prefer to have a more simple git tag format. Few examples:

1. You have only one env to promote to. Then model-name-promote should suffice.
2. You have only one object type. Then name-promote-production should suffice.
3. If both are true, then name-promote should suffice.

The question is how to

1. Set the settings. How a tool should know what format is? (Probably it should be dumped as a file in a repo).
2. Keep this backward compatible. E.g. if I had only one env, and then I introduce the second one, can I make sense of the whole history?
3. Avoid confusion between tags. E.g. name-register-v1 may conflict with model-name-promote, as they all consist of three parts. Or having register or promote as part of name may also bring confusion if tag format can vary.

Usage: gto add [OPTIONS] NAME TYPE PATH
e.g.

$ gto add rf model models/random-forest.pkl)

IMO should be: gto add [OPTIONS] TYPE NAME PATH
so:

$ gto add model rf models/random-forest.pkl 

Reasoning - more natural structure, loosely goes from general to specific ("model"(=type) is more general than the name of the specific model). Similar to kubectl command structure - kubectl [command] [TYPE] [NAME] [flags] (industry standard)

It is not clear from help what is add vs register
gto help shows:

add          Add an object to the Index
...
register     Register new object version

Can be argued it's not very helpful (vs promote which is helpful)

Some Suggestions (not sure it's accurate)
Add an object to the Index -> Insert the object to the registry Index
Register new object version -> Tag the object with a version (git tags)

Working on it in #1

Suggest to display verbose options in a more friendly way (not single line)

Now:

❯ gto show -v
INDEX='artifacts.yaml' VERSION_BASE='tag' VERSION_CONVENTION='semver' VERSION_REQUIRED_FOR_ENV=True ENV_BASE='tag' ENV_WHITELIST=[] ENV_BRANCH_MAPPING={} LOG_LEVEL='INFO' DEBUG=False CONFIG_FILE='gto.yaml'
                 environment
          latest     staging production mystaging2 mystaging
features    None         NaN        NaN        NaN       NaN
nn        v0.0.1      v0.0.1        NaN        NaN       NaN
rf        v1.0.1      v1.0.1     v1.0.0        NaN       NaN
oded      v0.0.2      v0.0.2        NaN     v0.0.1    v0.0.1

Suggestion

❯ gto show -v
INDEX='artifacts.yaml'
VERSION_BASE='tag'
VERSION_CONVENTION='semver'
VERSION_REQUIRED_FOR_ENV=True
ENV_BASE='tag'
ENV_WHITELIST=[]
ENV_BRANCH_MAPPING={}
LOG_LEVEL='INFO'
DEBUG=False
CONFIG_FILE='gto.yaml'

                 environment
          latest     staging production mystaging2 mystaging
features    None         NaN        NaN        NaN       NaN
nn        v0.0.1      v0.0.1        NaN        NaN       NaN
rf        v1.0.1      v1.0.1     v1.0.0        NaN       NaN
oded      v0.0.2      v0.0.2        NaN     v0.0.1    v0.0.1

Indices are on a separate row from other headers (because they are indixes), looks weird in tabular form imo, e.g.

❯ gto audit all

=== Registration audit trail ===
                         version   author                             commit_hexsha unregistered_date
creation_date       name
2022-03-11 21:00:08 rf    v1.0.1  omesser  8bbfbc0a5a9312c1fbacee5d0f0c18f3b258cf7e              None
2022-03-11 20:59:03 nn    v0.0.1  omesser  59a82808e3bef717ca45cff042f4b61a6b7a3dd4              None
2022-03-11 20:59:02 rf    v1.0.0  omesser  59a82808e3bef717ca45cff042f4b61a6b7a3dd4              None
...
<same for promotion audit trail>

Right now there are many special tag-based functions which should be made more general. They should work not with git.Repo directly, but with it's internal state representation. After generalization they should be moved outside of tag.py, e.g. to base.py.

This will allow reusing them in branch-based and file-based approaches, if that is possible.

This is needed for Studio and possibly for some users.
This may be deducted from gto.api.show (then we need to make sure the logic is correct), but at the same time may be a separate API function.

This is probably already on the to-do list, but when I first installed and ran showcase.sh, it broke everything for me (even gto --help) because it's outdated:

$ gto --help
Traceback (most recent call last):
  File "/Users/dave/miniforge3/bin/gto", line 5, in <module>
    from gto.cli import cli
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/__init__.py", line 1, in <module>
    import gto.log  # noqa
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/log.py", line 6, in <module>
    from gto.config import CONFIG
  File "/Users/dave/miniforge3/lib/python3.9/site-packages/gto/config.py", line 162, in <module>
    CONFIG = RegistryConfig()
  File "pydantic/env_settings.py", line 38, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for RegistryConfig
VERSIONS_CONVENTION
  extra fields not permitted (type=value_error.extra)

Now that there is https://github.com/iterative/gto-example with an updated showcase.sh, I would suggest removing showcase.sh here to avoid confusing people.

And possibly make "demote" and "deprecate" the same cmd?

Right now they work on local repos only, that's a limitation of gitpython.

Prints nothing

❯ gto audit something

It seems appropriate that if only promotion, registration and all are allowed - they should be arg choices and anything else should give an error

Something like gto history $ALIAS should give an answer to:

1. In which commits the model was present.
2. In which commits descriptions/tags were changed.
3. In which commits versions were registered.
4. In which commits envs were promoted.

History can be linear by commits (aligned with commits) or linear by time (aligned with time).

Something like what we can see in Studio UI mockups by Yaroslav should work well.

This can make file-based approach possible. Need to investigate how to do this better.

gto audit:

gto audit registrations should give the same output as gto audit registration
(it doesn't, it's just empty, see #57 )

gto add:

Types are free-text here (no assumptions?) so that's debatable, but I think it's worth singularizing them before adding to artifacts.yaml otherwise we get this unwanted behavior due to common (somewhat natural) cli usage (There's the order issue as well, see #53), where's it's apparently clear both artifacts should be of type model:

> gto add oded model models/mymodel.txt
> gto add oded2 models models/mymodel2.txt
> cat artifacts.yaml
...
oded:
  name: oded
  path: models/mymodel.txt
  type: model
oded2:
  name: oded2
  path: models/mymodel2.txt
  type: models

The goal of GitOps is to use it from CI/CD. How this is supposed to be used from CI/CD?

Scenario: User needs to deploy a production/staging model from CI/CD.

A bit more advanced scenario: User needs to deploy a production model from CI/CD if production tag was changed.

It would be great to see code examples. The goal - make sure GitOps provides a smooth experience in CI/CD stage.

PS: it would be great if the same/similar code can work with branch-based approach.

Current behavior (text format)

❯ gto show
Empty DataFrame
Columns: [(, latest)]
Index: []

Suggestion:

❯ gto show
No Tracked artifacts detected in the current workspace

softprops/action-gh-release@v1 is failing, e.g.
https://github.com/iterative/gto/runs/5464250428?check_suite_focus=true

File is called artifacts.yaml (I believe artifacts is the right one here)
But, print-state lists everything under objects entry which is a bit inconsistent

❯ gto print-state
objects:
...

Asking this from Dave and Oded this week.

Initial idea to introduce gto demote $MODEL $VERSION $ENV was based on the assumption that user may want to undo the action of promotion. E.g. if he mistakenly promoted a experimental version to production.

One corner case I found out for this demote cmd is this.

1. Let's suppose that you have a rf with v1 and v2. Then you promote to production v2, then v1, then again v2.
2. Now you demote what's in production right now. As a result, v1 now in production.
3. Now you demote again. As a result, v2 now in production. But you already tried to undo promotion of v2 to production, so it's confusing.

I see two options here:

1. Keep things as they are ^. Maybe there are some user scenarios in which this makes sense.
2. Make another assumption. If you demote v2 from production, that means that it shouldn't appear there until you promote it again. Then the example above will end up without any version of rf promoted to production.
3. Disallow demote instruction and require to promote things instead.
4. Use option 3 + still allow to have demote shortcut cmd which will actually execute promote $PREV_VERSION_IN_THAT_ENV, but that doesn't look very intuitive in complex scenarios also. As you can't distinguish between promote made on purpose and promote made because user called demote, if you run demote 2+ times, you'll stuck between two last versions in that env forever.

I'm leaning towards option 3 right now because it's simpler. Later we can introduce demote command as in option 1 or 2, however we decide.

WDYT?
CC @dmpetrov @mike0sv @dberenbaum @omesser @shcheklein

💅 - Could be nice to not display NaN in gto show output, replacing them with - or -- should do the trick and make the table look much nicer IMO

There's nothing preventing this but I don't think it makes sense, perhaps we'd want to prevent that? It doesn't make sense for models but also not sure it makes sense for any other trackable artifact and will cause troubles if happening by mistake
e.g.

❯ gto add mymodel model models/m1.txt
❯ gto add mymodel2 models models/m1.txt

and artifacts.yaml will show:

mymodel:
  name: mymodel
  path: models/m1.txt
  type: model
mymodel2:
  name: mymodel2
  path: models/m1.txt
  type: model

To reproduce:

git checkout main
cd models
gto audit all

Throws the following Git error: git.exc.InvalidGitRepositoryError

CC @Suor @jellebouwman

In the last week, we’ve decided that this Gitops prototype will be developed as a separate product for now. The idea is to decouple Index and Registry parts from DVC and MLEM as much as possible. That means this product could be used without DVC or MLEM.

I plan to set up an example repo so you could start playing around with it.

It's still unclear how Studio BE integration with MLEM will change. My assumption is that you may need to first call this tool to get a list of models, and then call MLEM API to get more info about those models. Python objects that represent MLEM models/datasets that MLEM returns shouldn't change.

1. Add more tests for things that can be tested without any repo
2. Set up some tests repo (maybe local will suffice?) and test most part of functionality on it (this will probably require adding teardown-able demote and unregister equivalents)

After running showcase.sh in the example repo and then doing gto demote rf production, the same version still shows in production:

$ gto show
                 environment
          latest     staging production
features    None         NaN        NaN
nn        v0.0.1      v0.0.1        NaN
rf        v1.0.1      v1.0.1     v1.0.0
$ gto which rf production
v1.0.0
$ gto demote rf production
Demoted rf from label production
$ gto show
                 environment
          latest     staging production
features    None         NaN        NaN
nn        v0.0.1      v0.0.1        NaN
rf        v1.0.1      v1.0.1     v1.0.0
$ gto which rf production
v1.0.0

Sorry if I'm misunderstanding the expected behavior.

show vs print-index vs print-state

• What's the difference between showing and printing?
• Unless you're familiar with the tool, It's unclear what to expect from the command names (what does "show" show if not the state? what's the difference between a state and an index?).
  🤔

Suggestion (not happy with it, need to iterate):
gto show -> gto show labels
gto print-index -> gto show index
gto print-state -> gto show journal (?? see #69)
Thoughts @aguschin @dberenbaum ?

Let's suppose a downstream system is trying to fetch specific version of a model. How can we warn the system if that version was deprecated?
Examples of possible requests (not sure the 3rd one is valid):

give me a model of the version X
check if a model of the version X is not deprecated
check if a model of the version X marked as "prod" was not deprecated

For enumerated inputs, show valid options/choices in command help (e.g. for -f/--format)
Today

> gto show --help
...
  -f, --format TEXT  Output format
...

Suggestion:

> gto show --help
...
  -f, --format [TABLE|JSON|YAML]  Output format [default=TABLE]
...

In artifacts.yaml we can potentially list artifacts from external paths:

features:
  path: s3://bucket/path
  type: dataset

Some question that come to my mind:

• Does it mean that path can be optional at all? (because we can't check or use this external path anyhow)
• How does the workflow look like with these artifacts?