Normally one can do

require:
  - a
/X:
  require+:
    - b

In the tmt the test case normally defined require: so it replaces eventual parents require:.

Now, the use case is to be able to force some requirement from the parent to the childs.

+require:
  - a
/X:
  require:
    - b

This +require: would ackt the same as require+: but applied in opposite order. So every child node would append it. It might be reset as usual +require: '' or added, deducted by +require+:, +require-: respectively.
The same forced deduction might work with -require:.

What I want to achieve:
instead of updating all the test with requirement for a library referenced with url/name I would propagate it this way so I can manage it centrally.

I just created an fmf file like this:

---
/tests/...

The problem is that fmf tool couldn't handle the three leading dashes which are a common way to start a yaml document: https://stackoverflow.com/questions/50788277/why-3-dashes-hyphen-in-yaml-file

If there is no distro version specified in the context, comparing against distro specific version gives False:

>>> import fmf
>>> fmf.context.Context(distro='fedora').matches('distro < fedora-33')
False

@lukaszachy, shouldn't this raise a CannotDecide exception instead?

$ tree
.
├── main
│   ├── main.fmf
│   └── test
│       └── main.fmf
└── main.fmf

2 directories, 3 files

$ cat main/main.fmf
something: "This come from main"
/in_main:
  sum: "There should be just 'sum' and 'something'"

$ cat main/test/main.fmf
/inside:
  that: "This is from main.fmf inside, there should be more than this"

$ cat main.fmf
/main/test/inside:
  this: "Should be just in last '/main/test/inside:'"

$ fmf
test/main/in_main
something: This come from main
sum: There should be just 'sum' and 'something'
this: Should be just in last '/main/test/inside:'

test/main/test/inside
something: This come from main
that: This is from main.fmf inside, there should be more than this
this: Should be just in last '/main/test/inside:'

It appears data inherited to particular leaf are being inherited all the way through the tree. 'this' attribute should be only in 'main/test/inside' but it is also in 'main/in_main'.

It would be useful to be able to undefine an inherited attribute. For example having an extra directory with helper files which should not be considered as tests. Perhaps something like this?

test: None

If you specify attributes on deeper layer than actual + 1, this gets interpreted as standalone leaf and does not get merged. To demonstrate this I created 2 entries that should get merged into one, as they appear to be on the same position in the tree. But the result is 2 different leaves, 'test -> test/directory' and 'test -> test -> directory' ('->' symbolizes branch of the tree).

[root@localhost test]# tree
.
├── main.fmf
└── test
    └── directory
        └── main.fmf

2 directories, 2 files

[root@localhost test]# fmf
test/test/directory
test: True
test/test/directory
test: True

[root@localhost test]# cat main.fmf              
/test/directory:
  test: True
[root@localhost test]# cat test/directory/main.fmf 
test: True

According to documentation, test case identifiers should be unique within current fmf structure. However, it is possible to work with a configuration text file, that has duplicated identifiers.

Example:

/python/Sanity/sys:
  component:
    - python3
  environment:
    PYTHON: /usr/libexec/platform-python
    PACKAGES: python3

/python/Sanity/sys:
  component:
    - python

Having that test.fmf, fmf show in the root directory shows

/Sanity/sys/test/CoreOS/python/Sanity/sys
component: python

so the first test was overwritten by the second one with duplicated identifier with no error returned.

As a user, I would like to be aware of the fact that I have duplicated test case identifiers in .fmf file, either with a visible warning or error and abortion of execution.

(Note: this is just a quick feedback from exploration of fmf; IOW I haven't read all docs and might be wrong in what I expect.)

@work:~/tmp/sandbox$ mkdir fmf-foo
@work:~/tmp/sandbox$ cd fmf-foo/
@work:~/tmp/sandbox/fmf-foo$ fmf
fmf-foo
@work:~/tmp/sandbox/fmf-foo$

README.md says this command should "List all metadata stored in the current directory", but obviously there's none, so why does it show anything?

fmf has inconsistency of behavior of --name and --filter options and also function insides:
example --name a --name b means logical OR
although using more --filter statements means logical AND
and also it is very hard to write NOT statement (possible to write via regexp: '^(?!.*inverse_match).*') and logical AND via name filter like --name '^((?!.*value1)|(?!.*value2)).*'

possible solution

would be good to copy name -> data["name"] then I will be able to use filter expressions as for other elements via --filter . But this lead to side effect that user cannot define "name" item. and also not easy to understand if name is just this item name or whole path name like /a/b/c and then how to deal with items.

and also there is another issue is that --filter function does not allow to use some asterisk patters like: tags: Tier* to match everything, what will be important to filter names because it is not easy to filter it via full path.

What do you think, is there some simple solution to do this or we do not plan to change this behaior.
Actual solution is fine, but little bit mixing two concepts together and two types of logical statements.

BTW it is close to relevancy. Relevancy syntax should provide these item types for filters.

Add method to get metadata based on #52 so tools as psss/tmt discover do not need to figure out git clone or use cache & run fmf on their own.

E.g. Checkout remote under XDG_CACHE_HOME and keep it there for further use.

This is probably more of a design issue, so I understand if there is some resistance, but I think I can jusify it.

It seems that fmf does not have concept of whole item tree but rather tends to just discover items in all subdirectories. This is nice and simple, but creates the problem that items don't really have "identity" and meta-data actually depends on where I ask.

For example, I have repository like this:

.
└── demo
    ├── foo
    │   └── main.fmf
    └── main.fmf

In the top main.fmf, I have

project: demo
maintainer: joe

and in the foo/main.fmf

description: The Foo

Now I get different results about foo depending on where I ask:

$ fmf
fmf-tree/demo/foo
description: The Foo
maintainer: joe
project: demo
$ cd demo
$ fmf
demo/foo
description: The Foo
maintainer: joe
project: demo
$ cd foo
$ fmf
foo
description: The Foo

If I have inheritancy, doesn't it imply that I already treat my items as tree? If so, shouldn't they always be resolved as part of the same tree?

Or, from another angle: why can't I "ask" for complete set of meta-data when I'm in the directory of a in leaf item?

Currently when searching for tests fmf returns relative paths:

# fmf /root/tests/ --brief --key test --filter component:libselinux
tests/upstream-tests/selinux/libselinux/setenforce
tests/upstream-tests/selinux/libselinux/getsebool

What I would like to see would be absolute paths:

# fmf /root/tests/ --brief --key test --filter component:libselinux --absolute_path
/root/tests/upstream-tests/selinux/libselinux/setenforce
/root/tests/upstream-tests/selinux/libselinux/getsebool

Sometimes it might be useful to define an attribute just for the current node and disable for it inheritance. It could look like this:

description.:
    This long description is intended just for this node and
    should not be inherited to children.

I suggest to use a dot . as the suffix as it signalizes that something is finishing here. Ideas?

Background

The Beakerlib metdatada file Makefile contains, among other things, fields Requires and Rhts-Requires, which hold test dependencies. This is a yet another important piece of information, which should be stored along with the test itself, however, in its current form (in Beakerlib) it's pretty inflexible, as there is no way to split such dependencies for multiple versions of a distribution, let alone for different distributions.

That said, this real-life example:

	@echo "Requires:        systemd yum-utils network-scripts" >> $(METADATA)

is pretty problematic, as:

• yum-utils has been replaced by dnf-utils on RHEL8, but there is no way to conditionally affect this during dependency resolution
• yum-utils may not be present on other distributions, so using these dependencies directly without any modifications is basically impossible
• etc.

This can be solved (and has been done so in many tests) by either placing both yum-utils and dnf-utils into the test dependencies and hope dnf/yum can handle it (and usually help it by using --skip-broken), or edit the test itself to do something like:

if rlIsRHEL "<=7"; then
    yum -q -y install yum-utils
else
    dnf -q -y install dnf-utils
fi

which is ugly, unportable, and in many cases even unstable.

Proposals

As we're going to discuss this more in depth next week, I'll just dump here some ideas which can be used as a starting point.

1) Re-using relevancy machinery to do dependency resolution as well

To make the dependency resolution more flexible, and in this case more dynamic, we could, theoretically, re-use part of the mechanism from relevancy resolution, as it already provides functionality necessary to resolve dependencies as well.

For example:

example: |
    requires:
       - "distro >= f-22 | distro >= rhel-8: systemd dnf-utils cryptsetup"
       - "distro < f-21 | distro < rhel-8: systemd yum-utils cryptsetup"

The one of the disadvantages of this syntax is increased verbosity, as there is no way (at least known to me), to do something like distro > f-22: dnf-utils; else: yum-utils.

2) Sticking with text-only dependencies

On contrary to the first solution, this one is more passive, as it's the plain key-value mapping with a pre-defined distro format string.

requires: |
    fedora-[12][01]:
        - systemd yum-utils
    fedora-[2-9].*:
        - systemd dnf-utils

However, this is really confusing and error-prone format (and I'm not even sure if it's a valid YAML).

The Tree.node() method fetches fmf metadata into ~/.cache/fmf. This can grow in time. It would be nice to implement an easy way to clean up old/all cached stuff. There should be available both from python API and command line. Brainstorming command line usage:

fmf gc
fmf gc --all
fmf gc --old

I've found that when I use filter on some data, it will leads to another output (test is part of list)

14:58 $ fmf --key test moduleframework/tests/ --format "{}/{}\n" --value "sources[-1]" --value 'data["test"]'
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/modulelint.py:ModuleLintPackagesCheck
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/check_compose.py
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/dockerlint.py:DockerfileLinterInContainer
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/rpmvalidation.py
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/dockerfile_lint.py:DockerInstructionsTests
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/dockerfile_lint.py:DockerLabelsTests
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/helpmd_lint.py:HelpFileSanity

and same command just with with filter

$ fmf --key test moduleframework/tests/ --format "{}/{}\n" --value "sources[-1]" --value 'data["test"]' --filter 'tags:static'
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/[u'dockerfile_lint.py:DockerInstructionsTests']
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/[u'dockerfile_lint.py:DockerLabelsTests']
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/[u'helpmd_lint.py:HelpFileSanity']

Actual format of FMF output lost info about FMF config file location, what will have more usages:

• debugging - able to see from which file data cames from
• test path location - help to find where is the test: key

like: main.fmf

/a/1:
  test: 1
/a/2:
  test: 2

and second.fmf

/b/1:
  test: 1

produces:

$ fmf
tmp/second/b/1
test: 1
tmp/a/2
test: 2
tmp/a/1
test: 1

would be:

$ fmf
fmf_invocation_path: /home/jscotka/tmp
tmp/second/b/1
test: 1
fmf_metadata_file: second.fmf
tmp/a/2
test: 2
fmf_metadata_file: main.fmf
tmp/a/1
test: 1
fmf_metadata_file: main.fmf

just little question is how to deal with merges? does that make sense to have there list of files, instead of one file string, or that there is most relevant (latest) is enough and sufficient?
Honza

for one project I would like to see some semantics to be able to reference nodes via name (this simple reference should be enough)
see example code:

/maintainer_label:
  test: script.py
  tags: ["a", "b"]

/sanity
  /from_tag_not_latest
    test: anotherscript.py

/tests
  /checks:
    /check1@maintainer_label:
      tags+: ["required"]
    /check2@/sanity/from_tag_not_latest:
      tags+: ["required"]
      usable_targets: ["dockerfile"]

what will use special symbol @ to use this item name as sources and data will be copied from this item instead of parent in tree structure.
so that list item /tests/checks/check1 will contain:

  test: script.py
  tags: ["a", "b", "required"]

does that make sense?

Possible issues

• wait to initialize all elements without @ items and then these
• allow add there non trivial references, like a@b@c and allow also references to other refs items
• could leads to circular deps -> should be able to detect cycles
• will be data copied, or "really" references, so that change of one will change also second one

Image a node where I define a reference to another git repo where the branch continues.
I can image a reserved attribute

fmf_subtree:
  url: https://github.com/...
  reference: master

such repo would be pulled and the tree would be simply extended by this part.

It is still questionable how the inheritance should behave at such points where a new fmf tree begins. This might be controlled by another attribute of fmf_subtree, e.g. boolean inheritance.

fmf_subtree:
  url: https://github.com/...
  reference: master
  inheritance: true

When trying to run tmt init as root in a container I get:

# tmt init
Traceback (most recent call last):
File "/usr/bin/tmt", line 11, in <module>
    tmt.cli.main()
File "/usr/lib/python3.7/site-packages/click/core.py", line 763, in __call__
    return self.main(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/core.py", line 716, in main
    rv = self.invoke(ctx)
File "/usr/lib/python3.7/site-packages/click/core.py", line 1136, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.7/site-packages/click/core.py", line 955, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.7/site-packages/click/core.py", line 554, in invoke
    return callback(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python3.7/site-packages/tmt/cli.py", line 747, in init
    echo("Tree '{}' initialized.".format(tree.root))
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 472, in root
    return self.tree.root
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 462, in tree
    self._tree = fmf.Tree(self._path)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 84, in __init__
    self.grow(data)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 291, in grow
    dirpath, dirnames, filenames = next(os.walk(path))
StopIteration

# rpm -q tmt
tmt-0.8-1.fc31.noarch

# rpm -q python3-fmf
python3-fmf-0.10-1.fc31.noarch

please provide tool what will be able to transform various metadata format like.
if you have metadata in one single file, tranform them to files in directory structure, or vice versa, from separate simple files, generate one complex file.

it would also nice to have there some generator what for example generates basic metadata files recursively in directory structure,
could work like find . -name "runtest.sh" -print "some strucutre to this directiry."
it could create basic test metadata templates

This example is very tricky and not sure if this way is good to show, because parental inheritance is easy to understand, but this defines same level in more files, and not easy to understand what happen and output is strange.

It is very hard to understand what happen there and why there are these values and priority of loadind these files is bad.

Example data structure

09:22 $ tree .
.
├── a
│   ├── b
│   │   ├── c
│   │   │   └── main.fmf
│   │   └── main.fmf
│   └── main.fmf
└── main.fmf

3 directories, 4 files

09:23 $ cat main.fmf 
/a/b/c:
   x: 1
   first: True

09:23 $ cat a/main.fmf 
/b/c:
   x: 2
   second: True

09:23 $ cat a/b/main.fmf 
/c:
   x: 3
   third: True

09:23 $ cat a/b/c/main.fmf 
x: 4
fourth: True

Actual output

09:23 $ fmf
xxx/a/b/c
fmf_config: ./a/main.fmf
second: True
x: 2
xxx/a/b/c
fmf_config: ./a/b/c/main.fmf
fourth: True
third: True
x: 4
xxx/a/b/c
first: True
fmf_config: ./main.fmf
x: 1

09:24 $ fmf --whole
xxx
fmf_config: ./main.fmf
xxx/a
fmf_config: ./a/main.fmf
xxx/a/b/c
fmf_config: ./a/main.fmf
second: True
x: 2
xxx/a/b
fmf_config: ./a/b/main.fmf
xxx/a/b/c
fmf_config: ./a/b/c/main.fmf
fourth: True
third: True
x: 4
xxx/a/b/c
first: True
fmf_config: ./main.fmf
x: 1

Expected output

$ fmf
xxx/a/b/c
fmf_config: ./a/b/c/main.fmf
first: True
second: True
third: True
fourth: True
x: 4

https://apps.fedoraproject.org/packages/fmf
https://apps.fedoraproject.org/packages/s/fmf

Both show:

no summary in mdapi

(Note: this is just a quick feedback from exploration of fmf; IOW I haven't read all docs and might be wrong in what I expect.)

I tried to get a trivial test suite (eg. just one test) working with fmf. I'm struggling to find simplest possible working example. My idea is, that if I have test suite like this:

mysut
   '--- mytest1
            '---- test.sh

where test.sh is something my test harness understands, what should I do as very first step to make this test suite "fmf-compliant"?

(Note that at this point, calling fmf from mysut merely lists mysut/mytest1, which is not even correct--this is just an arbitrary folder; I can't see why fmf should care about it. Which is in fact a case of issue #12.)

Examples section on redthedocs.io was not very helpful; they show examples of content but don't provide answer as to what file should I create.

Currently there is no recommendation about how an attribute key should look like. For example if there is a need for key consisting of multiple words what would be the best choice?

Keeping spaces? Which could possibly cause problems if not handled properly?

/smoke:
    summary: Basic smoke test
    tcms id: TC#1234567

Substituting them with dashes?

/smoke:
    summary: Basic smoke test
    tcms-id: TC#1234567

Or underscores?

/smoke:
    summary: Basic smoke test
    tcms_id: TC#1234567

Or give no recommendation at all? Ideas?

As a User I want to be able to filter name by using fmf.filter so I have consistency with filter by attributes.

fmf ls --filter tier:1 but fmf ls --name /test/me.
I want be able to use fmf ls --filter name:/test/me as well

Hello.
tmt raises traceback when CWD lost rights to read/execute.

How to reproduce:

under root user:

cd ~

ls -lad /root/

dr-xr-x---. 4 root root 226 Nov 14 03:26 /root/
[root@koza-6 ~]# su rhack
[rhack@koza-6 root]$ tmt
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 96, in _initialize
while ".fmf" not in next(os.walk(root))[1]:
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/bin/tmt", line 11, in
tmt.cli.main()
File "/usr/lib/python3.7/site-packages/click/core.py", line 763, in call
return self.main(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/core.py", line 716, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.7/site-packages/click/core.py", line 1113, in invoke
return Command.invoke(self, ctx)
File "/usr/lib/python3.7/site-packages/click/core.py", line 955, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.7/site-packages/click/core.py", line 554, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python3.7/site-packages/tmt/cli.py", line 159, in main
tmt.Test.overview(tree)
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 122, in overview
style(str(test), fg='red') for test in tree.tests()]
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 489, in tests
return [Test(test) for test in self.tree.prune(
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 466, in tree
self._tree = fmf.Tree(self._path)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 74, in init
self._initialize(path=data)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 103, in _initialize
raise utils.FileError("Invalid directory path: {0}".format(root))
fmf.utils.FileError: Invalid directory path: /root

Something better than traceback would be nice :). Yeah, It's corner case but.. I happened! :)

I would like to have there some similar syntax what I has in my PoC for metadata downloading some files from URLs or git repositories.

User story

It allows me to have metadata and load them also from network or another location
see example config lines like: https://github.com/fedora-modularity/meta-test-family/blob/master/mtf/metadata/examples/general-component/tests/metadata.yaml#L8

download_urls:
  downloaded_test.py: "https://raw.githubusercontent.com/fedora-modularity/meta-test-family/devel/mtf/metadata/examples/general-component/tests/simple.py"
  somedir/metadata.yaml: "some link to another metadata"
clone_gits:
  downloaded_git: "https://github.com/container-images/memcached.git"
import_tests:
  /some/localfs/location/where/to/search/for/metadata

It will clone or dowload files locally as files or subdirectories

How it should work

• load metadata
• find these items what downloads/refer something
• reload metadata again with these updates
  (do not do it recursively) -> can causes loops, to allow just one round of this reloading

summary

• have stored metadata elegantly in various places, easy to use
  • eg. I want to load metadata for some generic tests / linters and run them together with my tests what are in some github repo
  • or another location on same disc means that I can add there this url param with file:///... for example
  • or via this import_tests as simplier solution how to not copy local files to another location and load these metedatas
• this is important feature for my usecases in MTF
• could be implemented by separated class, module, file, whatwill use metadata and this recursive loading could be lead externally or via some option, cmd line argument. I think it is beneficial to have it directly there and not as part of MTF, what will do this part and interpret this items in metadata files
• It helps to deploy tests for local scheduler and keep tests close to metadata

Hi,

I am not sure if this is a wrong example, but it fails:

http://fmf.readthedocs.io/en/latest/examples.html

fmf      
Traceback (most recent call last):
  File "/usr/bin/fmf", line 35, in <module>
    fmf.cli.main()
  File "/usr/lib/python2.7/site-packages/fmf/cli.py", line 120, in main
    tree = fmf.Tree(path)
  File "/usr/lib/python2.7/site-packages/fmf/base.py", line 57, in __init__
    self.grow(data)
  File "/usr/lib/python2.7/site-packages/fmf/base.py", line 146, in grow
    data = yaml.load(datafile)
  File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 115, in check_token
    while self.need_more_tokens():
  File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 149, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 289, in stale_possible_simple_keys
    "could not found expected ':'", self.get_mark())
yaml.scanner.ScannerError: while scanning a simple key
  in "/home/astepano/shared/osci/standard-test-roles/3.fmf", line 6, column 1
could not found expected ':'
  in "/home/astepano/shared/osci/standard-test-roles/3.fmf", line 7, column 16





➜  standard-test-roles git:(master_copr_build) ✗ cat 3.fmf 
description: Check basic download options
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min

/requirements
    requirement: Various download options working correctly
    priority: low
    /get-file:
        coverage: wget/download
    /output-document:
        coverage: wget/download
    /continue:
    /timestamping:
    /tries:
    /no-clobber:
        coverage: wget/download
    /progress:
    /quota:
    /server-response:
    /bind-address:
    /spider:

setup.py says so (and, doesn't even mention Python 3.x). But, text/data/encoding issues exist when running on Python 2.7:

$ fmf > foo 
Traceback (most recent call last):
  File "/home/cleber/.local/bin/fmf", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/cleber/src/fmf/bin/fmf", line 35, in <module>
    fmf.cli.main()
  File "/home/cleber/src/fmf/fmf/cli.py", line 127, in main
    print(show, end="")
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0160' in position 140: ordinal not in range(128)

$ git rev-parse HEAD
3d59b97b8beaa75035526fdf992530adef26f19a

Tree structures are usually visualized from top to bottom, even wikipedia says that "Root: The top node in a tree". With that in mind name 'climb' is a bit confusing. Function is doing tree traversal.

Proposed name: traverse

Hi,

python2-fmf-0.5-1.fc27.noarch

In [1]: import fmf
 
In [3]: fmf.Tree(".")
---------------------------------------------------------------------------
FileError                                 Traceback (most recent call last)
<ipython-input-3-b75dcdf15015> in <module>()
----> 1 fmf.Tree(".")
 
/usr/lib/python2.7/site-packages/fmf/base.pyc in __init__(self, data, name, parent)
     62             self.name = "/"
     63             if not isinstance(data, dict):
---> 64                 self._initialize(path=data)
     65                 data = self.root
     66         # Handle child node creation
 
/usr/lib/python2.7/site-packages/fmf/base.pyc in _initialize(self, path)
     88                     raise utils.FileError(
     89                         "Unable to find tree root for '{0}'.".format(
---> 90                             os.path.abspath(path)))
     91                 root = os.path.abspath(os.path.join(root, os.pardir))
     92         except StopIteration:
 
FileError: Unable to find tree root for '/root/file/tests'.

FMF is now a part of standard-test-roles.
Some tests do not uses FMF.
If I try to init fmf Tree it fails.
It would be nice to have an Exception of specific type for such case.

Hi,

I am not sure if this is a bug, but cmdline tool doesn't recognise any command:

http://fmf.readthedocs.io/en/latest/overview.html

ls show
ls: cannot access show: No such file or directory

rpm -q fmf
fmf-0.4-1.el7.noarch

The path given on the command line defines the root of the metadata tree which is then used for constructing the object identifiers. We should explicilty document this both in the --help message and man page.

Same holds for require. I have a test which has two tags

tag: [ 'A', 'B']

But if I do fmf show I get

tag: A and B

Since tags are case sensitive the word 'and' is difficult to recognize it is not a tag.
In the case of require one cannot directly use copy&pasted list of packages.

Sometimes it is necessary to investigate deeper structures of the data dictionary in order to perform the desired filtering. In addition to the --filter option we could also provide a --condition which would allow arabitrary python expressions to be evaluated. Here's an example use case:

for node in fmf.Tree(path).climb():
    if node.get(['execute', 'how']) == "dependency":
        print(node.get(['execute', 'components']))

In order to filter only metadata nodes with the dependency implementation of the execute step it's necessary to access second level of the dictionary which is not possible using --filter option. Together with --format this would allow much more flexibility right from the command line.

When exploring directories fmf should probably ignore other metadata trees in the directory tree. Yesterday we've discussed the following real-life scenario which seems to support this approach:

Standard Test Roles support fetching tests from remote repositories. At the same time there can be tests stored directly in dist-git. When these two options are mixed we should still be able to gather reasonable metadata for both.

Here's a simple example directory structure:

.
├── fetched
│   ├── fetched1
│   ├── fetched2
│   ├── fetched3
│   ├── .fmf
│   │   └── version
│   └── main.fmf
├── .fmf
│   └── version
├── main.fmf
├── stored1
├── stored2
└── stored3

Current output in the main tests directory (note the modified node identifiers):

/stored1
/stored2
/stored3
/fetched/fetched1
/fetched/fetched3
/fetched/fetched2

Current output for the fetched test cases in tests/fetched:

/fetched1
/fetched3
/fetched2

Expected output for the main directory:

/stored1
/stored2
/stored3

Guys, what are your thoughts on this?

Updating to FMF version 0.3 breaks the inheritance feature. All of the missing attributes in 0.3 are defined somewhere in the parents and they are not getting passed down to the child.

0.2:

$ fmf --name vdo/vdo_cli/various_commands/stop_start_rebuild_success
tests/vdo/vdo_cli/various_commands/stop_start_rebuild_success
component: vdo and kmod-kvdo
description: Testing 'vdo stop' and 'vdo start' with '--forceRebuild', commands should succeed
requires_install: vdo and kmod-kvdo
requires_setup: vdo/setup/storage/setup_* and vdo/setup/create_vdo
test: stop_start_rebuild.py
tester: me...
tier: 1
time: 1 min
vdo_name: vdo_test

0.3:

$ fmf --name vdo/vdo_cli/various_commands/stop_start_rebuild_success
tests/vdo/vdo_cli/various_commands/stop_start_rebuild_success
description: "Testing 'vdo stop' and 'vdo start' with '--forceRebuild', commands should succeed"
test: 'stop_start_rebuild.py'
tier: 1

in https://github.com/sopos/tests:

$ fmf ls 
/fapolicyd/Sanity/trust-db
/luksmeta/Regression/bz1461448-running-luksmeta-on-a-non-LUKS-device-returns-IO-error
/usbguard/Sanity/bz1772149-try-to-parse-match-all-rule
/usbguard/Sanity/config-sanity
/usbguard/Sanity/selinux
/usbguard/Sanity/service-sanity

$ fmf ls --filter 'tag: Fedora_missing_external_dependencies'
/usbguard/Sanity/selinux

$ fmf ls --filter 'tag: -Fedora_missing_external_dependencies'
/usbguard/Sanity/config-sanity

For the negative tag I would expect everything but /usbguard/Sanity/selinux to be listed.

I hit an easy to reproduce traceback when defining same leaf with different methods (using 2 files from same location). If I define it using main.fmf and 1/main.fmf, there is no issue. But doing the same thing using main.fmf and 1.fmf leads to following traceback:

# tree
.
├── 1.fmf
└── main.fmf

# cat 1.fmf
in_1: True

# cat main.fmf
/1:
  in_main: True

# fmf
Traceback (most recent call last):
  File "/usr/bin/fmf", line 4, in <module>
    __import__('pkg_resources').run_script('fmf==0.4.1', 'fmf')
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 658, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1445, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python2.7/site-packages/fmf-0.4.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in <module>

  File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 120, in main
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 57, in __init__
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 152, in grow
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 107, in child
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 124, in grow
AttributeError: 'dict' object has no attribute 'rstrip'

This is reproducible since the very beginning, so no regression:

# fmf
Traceback (most recent call last):
  File "/usr/bin/fmf", line 4, in <module>
    __import__('pkg_resources').run_script('fmf==0.1.1', 'fmf')
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 658, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1445, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python2.7/site-packages/fmf-0.1.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in <module>

  File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 96, in main
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 50, in __init__
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 115, in grow
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 82, in child
  File "build/bdist.linux-x86_64/egg/fmf/base.py", line 90, in grow
AttributeError: 'dict' object has no attribute 'rstrip'

$ fmf init
Traceback (most recent call last):
File "/usr/bin/fmf", line 4, in
import('pkg_resources').run_script('fmf==0.5.1', 'fmf')
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 658, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 1445, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/fmf-0.5.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in

File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 194, in main
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 73, in init
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 145, in command_init
File "/usr/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/home/jkrysl/tests/python-stqe/tests/.fmf'

Currently each node in the metadata tree structure has a unique identifier called name so it is possible to reference individual objects from other attributes. This can be useful for example for mapping test coverage where tests are referenced from requirements.

This works nicely within a single metadata tree. However, we need to define a clear way how to reference objects which are outside of the current metadata tree. For this we could combine repository url with object name using a defined separator.

Also, in order to prevent too long identifiers and repeating urls it would be good to provide a way how to define a common prefix or remote repository shortcut. Perhaps it could work in a similar way how remotes like origin are defined in git.

HEAD is not advanced after git fetch in the fmf repo cache after tests are modified in test repo.
The update should happen every 20 minutes, but does not.

It would be useful to be able to merge multiple structures into a single metadata tree. An example use case can be merging custom configuration with default settings. As a quick hack we can do something like this:

tree = fmf.Tree("default")
tree.grow("custom")

For such cases it would be probably better to have possibility to initialize empty tree and then grow it multiple times:

tree = fmf.Tree()
tree.grow("default")
tree.grow("custom")

We could also add support for multiple trees directly in the constructor (which seems like a little bit more clean solution):

tree = fmf.Tree(['default', 'custom'])

However there are a few things which need to be solved: There are attributes which kind of expect a single tree only: root, version, original_data. Also, repeated call of grow() results in inherit() being applied multiple times which might cause problems.

Hi,
I think about using fmf command and how it deals with directories and root items.
dir structure like: a/b/c/main.fmf and first main.fmf is in c directory could produce structure like:

c/fist ....
c/second ...

and root item is a/b

and it could be invoked like

fmf --brief .
fmf --brief a
fmf --brief a/b
fmf --brief `pwd`

and everytime it will produce same output what I see very beneficial for referencing, than now it is very dependent on invocation of command.

then I can imagine, that you have tests for beakerlib and you have two components like bash and wget and you checkout these gits locally, call fmf tool and it will produce consistent sructure not dependent on place of invocation.
imagine for example you have ~/git/wget and ~/otherpath/git/bash and invocation like fmf ~ will produce output like

wget/sanity
wget/second
bash/sanity
bash/regression

What do you think? maybe could be beneficial also for merging trees
Regards
Honza

Documentation explicitly says that it is discouraged. This is aimed for automation, so usage should enforce readability as rules will be written once but read multiple times.

Please remove unwanted functionality.

As discussed in #26 it would be useful to have support for subcommands which would give us a nice flexibility for different use cases. Here are some examples:

• fmf init ... to initialize metadata tree
• fmf ls ... list available objects (just identifiers)
• fmf show ... show available metadata (what now fmf does)

Hi,

fmf ls hangs up and runs forever.

(venv)➜  empty git:(master_copr_build) ✗ mkdir empty 
(venv)➜  empty git:(master_copr_build) ✗ cd empty 
(venv)➜  empty git:(master_copr_build) ✗ fmf ls

10734 astepano  20   0  170704  20588   2368 R 50.3  2.0   0:18.81 fmf                                                                                                                                        

As outlined in #43 it would be good to have an easy way to discover all metadata trees present under given directory on the filesystem. This would allow us to easily process them one-by-one. A new subcommand find should be added which will list root directories of all subtrees found. Additional options can be provided later to optionally list the metadata as well. Example output:

> cd /usr/share/doc/fmf-0.5/
> fmf find .

./examples/child
./examples/scatter
./examples/deep
./examples/touch
./examples/wget
./examples/merge