teemtee / fmf Goto Github PK
View Code? Open in Web Editor NEWFlexible Metadata Format
License: GNU General Public License v2.0
Flexible Metadata Format
License: GNU General Public License v2.0
I've found that when I use filter on some data, it will leads to another output (test is part of list
)
14:58 $ fmf --key test moduleframework/tests/ --format "{}/{}\n" --value "sources[-1]" --value 'data["test"]'
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/modulelint.py:ModuleLintPackagesCheck
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/check_compose.py
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/dockerlint.py:DockerfileLinterInContainer
/home/jscotka/git/meta-test-family/moduleframework/tests/generic/main.fmf/rpmvalidation.py
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/dockerfile_lint.py:DockerInstructionsTests
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/dockerfile_lint.py:DockerLabelsTests
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/helpmd_lint.py:HelpFileSanity
and same command just with with filter
$ fmf --key test moduleframework/tests/ --format "{}/{}\n" --value "sources[-1]" --value 'data["test"]' --filter 'tags:static'
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/[u'dockerfile_lint.py:DockerInstructionsTests']
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/[u'dockerfile_lint.py:DockerLabelsTests']
/home/jscotka/git/meta-test-family/moduleframework/tests/static/main.fmf/[u'helpmd_lint.py:HelpFileSanity']
If there is no distro version specified in the context, comparing against distro specific version gives False
:
>>> import fmf
>>> fmf.context.Context(distro='fedora').matches('distro < fedora-33')
False
@lukaszachy, shouldn't this raise a CannotDecide
exception instead?
Same holds for require
. I have a test which has two tags
tag: [ 'A', 'B']
But if I do fmf show
I get
tag: A and B
Since tags are case sensitive the word 'and' is difficult to recognize it is not a tag.
In the case of require one cannot directly use copy&pasted list of packages.
This is probably more of a design issue, so I understand if there is some resistance, but I think I can jusify it.
It seems that fmf does not have concept of whole item tree but rather tends to just discover items in all subdirectories. This is nice and simple, but creates the problem that items don't really have "identity" and meta-data actually depends on where I ask.
For example, I have repository like this:
.
└── demo
├── foo
│ └── main.fmf
└── main.fmf
In the top main.fmf, I have
project: demo
maintainer: joe
and in the foo/main.fmf
description: The Foo
Now I get different results about foo depending on where I ask:
$ fmf
fmf-tree/demo/foo
description: The Foo
maintainer: joe
project: demo
$ cd demo
$ fmf
demo/foo
description: The Foo
maintainer: joe
project: demo
$ cd foo
$ fmf
foo
description: The Foo
If I have inheritancy, doesn't it imply that I already treat my items as tree? If so, shouldn't they always be resolved as part of the same tree?
Or, from another angle: why can't I "ask" for complete set of meta-data when I'm in the directory of a in leaf item?
The path given on the command line defines the root of the metadata tree which is then used for constructing the object identifiers. We should explicilty document this both in the --help message and man page.
Tree structures are usually visualized from top to bottom, even wikipedia says that "Root: The top node in a tree". With that in mind name 'climb' is a bit confusing. Function is doing tree traversal.
Proposed name: traverse
Sometimes it is necessary to investigate deeper structures of the data dictionary in order to perform the desired filtering. In addition to the --filter
option we could also provide a --condition
which would allow arabitrary python expressions to be evaluated. Here's an example use case:
for node in fmf.Tree(path).climb():
if node.get(['execute', 'how']) == "dependency":
print(node.get(['execute', 'components']))
In order to filter only metadata nodes with the dependency
implementation of the execute
step it's necessary to access second level of the dictionary which is not possible using --filter
option. Together with --format
this would allow much more flexibility right from the command line.
Currently each node in the metadata tree structure has a unique identifier called name
so it is possible to reference individual objects from other attributes. This can be useful for example for mapping test coverage where tests are referenced from requirements.
This works nicely within a single metadata tree. However, we need to define a clear way how to reference objects which are outside of the current metadata tree. For this we could combine repository url with object name using a defined separator.
Also, in order to prevent too long identifiers and repeating urls it would be good to provide a way how to define a common prefix or remote repository shortcut. Perhaps it could work in a similar way how remotes like origin are defined in git.
HEAD is not advanced after git fetch in the fmf repo cache after tests are modified in test repo.
The update should happen every 20 minutes, but does not.
Documentation explicitly says that it is discouraged. This is aimed for automation, so usage should enforce readability as rules will be written once but read multiple times.
Please remove unwanted functionality.
Normally one can do
require:
- a
/X:
require+:
- b
In the tmt the test case normally defined require:
so it replaces eventual parents require:
.
Now, the use case is to be able to force some requirement from the parent to the childs.
+require:
- a
/X:
require:
- b
This +require:
would ackt the same as require+:
but applied in opposite order. So every child node would append it. It might be reset as usual +require: ''
or added, deducted by +require+:
, +require-:
respectively.
The same forced deduction might work with -require:
.
What I want to achieve:
instead of updating all the test with requirement for a library referenced with url/name I would propagate it this way so I can manage it centrally.
I just created an fmf file like this:
---
/tests/...
The problem is that fmf tool couldn't handle the three leading dashes which are a common way to start a yaml document: https://stackoverflow.com/questions/50788277/why-3-dashes-hyphen-in-yaml-file
(Note: this is just a quick feedback from exploration of fmf; IOW I haven't read all docs and might be wrong in what I expect.)
@work:~/tmp/sandbox$ mkdir fmf-foo
@work:~/tmp/sandbox$ cd fmf-foo/
@work:~/tmp/sandbox/fmf-foo$ fmf
fmf-foo
@work:~/tmp/sandbox/fmf-foo$
README.md says this command should "List all metadata stored in the current directory", but obviously there's none, so why does it show anything?
As discussed in #26 it would be useful to have support for subcommands which would give us a nice flexibility for different use cases. Here are some examples:
Sometimes it might be useful to define an attribute just for the current node and disable for it inheritance. It could look like this:
description.:
This long description is intended just for this node and
should not be inherited to children.
I suggest to use a dot .
as the suffix as it signalizes that something is finishing here. Ideas?
(Note: this is just a quick feedback from exploration of fmf; IOW I haven't read all docs and might be wrong in what I expect.)
I tried to get a trivial test suite (eg. just one test) working with fmf. I'm struggling to find simplest possible working example. My idea is, that if I have test suite like this:
mysut
'--- mytest1
'---- test.sh
where test.sh is something my test harness understands, what should I do as very first step to make this test suite "fmf-compliant"?
(Note that at this point, calling fmf
from mysut merely lists mysut/mytest1, which is not even correct--this is just an arbitrary folder; I can't see why fmf should care about it. Which is in fact a case of issue #12.)
Examples section on redthedocs.io was not very helpful; they show examples of content but don't provide answer as to what file should I create.
$ fmf show
Traceback (most recent call last):
File "/usr/bin/fmf", line 4, in <module>
__import__('pkg_resources').run_script('fmf==0.5.1', 'fmf')
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 654, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1441, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/fmf-0.5.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in <module>
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 194, in main
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 73, in __init__
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 131, in command_show
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 156, in show
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 80, in __init__
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 248, in grow
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
return loader.get_single_data()
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
node = self.get_single_node()
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/composer.py", line 129, in compose_mapping_node
item_key = self.compose_node(node, None)
File "/home/jkrysl/.local/lib/python2.7/site-packages/yaml/composer.py", line 69, in compose_node
% anchor.encode('utf-8'), event.start_mark)
yaml.composer.ComposerError: found undefined alias 'test'
in "/home/jkrysl/tests/fmf_test/main.fmf", line 1, column 1
$ cat main.fmf
*test: string
$ ls -la
total 16
drwxrwxr-x. 3 jkrysl jkrysl 4096 Oct 5 12:28 .
drwxrwxr-x. 20 jkrysl jkrysl 4096 Oct 5 10:49 ..
drwxrwxr-x. 2 jkrysl jkrysl 4096 Oct 5 10:49 .fmf
-rw-rw-r--. 1 jkrysl jkrysl 14 Oct 5 12:27 main.fmf
$ nano main.fmf
$ cat main.fmf
*test+: string
$ fmf show
ERROR Failed to parse '/home/jkrysl/tests/fmf_test/main.fmf'
while scanning an alias
in "/home/jkrysl/tests/fmf_test/main.fmf", line 1, column 1
expected alphabetic or numeric character, but found '+'
in "/home/jkrysl/tests/fmf_test/main.fmf", line 1, column 6
$ tree
.
├── main
│ ├── main.fmf
│ └── test
│ └── main.fmf
└── main.fmf
2 directories, 3 files
$ cat main/main.fmf
something: "This come from main"
/in_main:
sum: "There should be just 'sum' and 'something'"
$ cat main/test/main.fmf
/inside:
that: "This is from main.fmf inside, there should be more than this"
$ cat main.fmf
/main/test/inside:
this: "Should be just in last '/main/test/inside:'"
$ fmf
test/main/in_main
something: This come from main
sum: There should be just 'sum' and 'something'
this: Should be just in last '/main/test/inside:'
test/main/test/inside
something: This come from main
that: This is from main.fmf inside, there should be more than this
this: Should be just in last '/main/test/inside:'
It appears data inherited to particular leaf are being inherited all the way through the tree. 'this' attribute should be only in 'main/test/inside' but it is also in 'main/in_main'.
This example is very tricky and not sure if this way is good to show, because parental inheritance is easy to understand, but this defines same level in more files, and not easy to understand what happen and output is strange.
It is very hard to understand what happen there and why there are these values and priority of loadind these files is bad.
09:22 $ tree .
.
├── a
│ ├── b
│ │ ├── c
│ │ │ └── main.fmf
│ │ └── main.fmf
│ └── main.fmf
└── main.fmf
3 directories, 4 files
09:23 $ cat main.fmf
/a/b/c:
x: 1
first: True
09:23 $ cat a/main.fmf
/b/c:
x: 2
second: True
09:23 $ cat a/b/main.fmf
/c:
x: 3
third: True
09:23 $ cat a/b/c/main.fmf
x: 4
fourth: True
09:23 $ fmf
xxx/a/b/c
fmf_config: ./a/main.fmf
second: True
x: 2
xxx/a/b/c
fmf_config: ./a/b/c/main.fmf
fourth: True
third: True
x: 4
xxx/a/b/c
first: True
fmf_config: ./main.fmf
x: 1
09:24 $ fmf --whole
xxx
fmf_config: ./main.fmf
xxx/a
fmf_config: ./a/main.fmf
xxx/a/b/c
fmf_config: ./a/main.fmf
second: True
x: 2
xxx/a/b
fmf_config: ./a/b/main.fmf
xxx/a/b/c
fmf_config: ./a/b/c/main.fmf
fourth: True
third: True
x: 4
xxx/a/b/c
first: True
fmf_config: ./main.fmf
x: 1
$ fmf
xxx/a/b/c
fmf_config: ./a/b/c/main.fmf
first: True
second: True
third: True
fourth: True
x: 4
Add method to get metadata based on #52 so tools as psss/tmt discover do not need to figure out git clone or use cache & run fmf on their own.
E.g. Checkout remote under XDG_CACHE_HOME
and keep it there for further use.
Actual format of FMF output lost info about FMF config file location, what will have more usages:
test:
keylike: main.fmf
/a/1:
test: 1
/a/2:
test: 2
and second.fmf
/b/1:
test: 1
produces:
$ fmf
tmp/second/b/1
test: 1
tmp/a/2
test: 2
tmp/a/1
test: 1
would be:
$ fmf
fmf_invocation_path: /home/jscotka/tmp
tmp/second/b/1
test: 1
fmf_metadata_file: second.fmf
tmp/a/2
test: 2
fmf_metadata_file: main.fmf
tmp/a/1
test: 1
fmf_metadata_file: main.fmf
just little question is how to deal with merges? does that make sense to have there list of files, instead of one file string, or that there is most relevant (latest) is enough and sufficient?
Honza
Hi,
I am not sure if this is a bug, but cmdline tool doesn't recognise any command:
http://fmf.readthedocs.io/en/latest/overview.html
ls show
ls: cannot access show: No such file or directory
rpm -q fmf
fmf-0.4-1.el7.noarch
Updating to FMF version 0.3 breaks the inheritance feature. All of the missing attributes in 0.3 are defined somewhere in the parents and they are not getting passed down to the child.
$ fmf --name vdo/vdo_cli/various_commands/stop_start_rebuild_success
tests/vdo/vdo_cli/various_commands/stop_start_rebuild_success
component: vdo and kmod-kvdo
description: Testing 'vdo stop' and 'vdo start' with '--forceRebuild', commands should succeed
requires_install: vdo and kmod-kvdo
requires_setup: vdo/setup/storage/setup_* and vdo/setup/create_vdo
test: stop_start_rebuild.py
tester: me...
tier: 1
time: 1 min
vdo_name: vdo_test
$ fmf --name vdo/vdo_cli/various_commands/stop_start_rebuild_success
tests/vdo/vdo_cli/various_commands/stop_start_rebuild_success
description: "Testing 'vdo stop' and 'vdo start' with '--forceRebuild', commands should succeed"
test: 'stop_start_rebuild.py'
tier: 1
https://apps.fedoraproject.org/packages/fmf
https://apps.fedoraproject.org/packages/s/fmf
Both show:
no summary in mdapi
Currently there is no recommendation about how an attribute key should look like. For example if there is a need for key consisting of multiple words what would be the best choice?
Keeping spaces? Which could possibly cause problems if not handled properly?
/smoke:
summary: Basic smoke test
tcms id: TC#1234567
Substituting them with dashes?
/smoke:
summary: Basic smoke test
tcms-id: TC#1234567
Or underscores?
/smoke:
summary: Basic smoke test
tcms_id: TC#1234567
Or give no recommendation at all? Ideas?
fmf has inconsistency of behavior of --name
and --filter
options and also function insides:
example --name a --name b
means logical OR
although using more --filter
statements means logical AND
and also it is very hard to write NOT statement (possible to write via regexp: '^(?!.*inverse_match).*'
) and logical AND via name filter like --name '^((?!.*value1)|(?!.*value2)).*'
would be good to copy name
-> data["name"]
then I will be able to use filter expressions as for other elements via --filter
. But this lead to side effect that user cannot define "name" item. and also not easy to understand if name
is just this item name or whole path name like /a/b/c
and then how to deal with items.
and also there is another issue is that --filter
function does not allow to use some asterisk patters like: tags: Tier*
to match everything, what will be important to filter names because it is not easy to filter it via full path.
What do you think, is there some simple solution to do this or we do not plan to change this behaior.
Actual solution is fine, but little bit mixing two concepts together and two types of logical statements.
BTW it is close to relevancy. Relevancy syntax should provide these item types for filters.
I would like to have there some similar syntax what I has in my PoC for metadata downloading some files from URLs or git repositories.
It allows me to have metadata and load them also from network or another location
see example config lines like: https://github.com/fedora-modularity/meta-test-family/blob/master/mtf/metadata/examples/general-component/tests/metadata.yaml#L8
download_urls:
downloaded_test.py: "https://raw.githubusercontent.com/fedora-modularity/meta-test-family/devel/mtf/metadata/examples/general-component/tests/simple.py"
somedir/metadata.yaml: "some link to another metadata"
clone_gits:
downloaded_git: "https://github.com/container-images/memcached.git"
import_tests:
/some/localfs/location/where/to/search/for/metadata
It will clone or dowload files locally as files or subdirectories
import_tests
as simplier solution how to not copy local files to another location and load these metedatasplease provide tool what will be able to transform various metadata format like.
if you have metadata in one single file, tranform them to files in directory structure, or vice versa, from separate simple files, generate one complex file.
it would also nice to have there some generator what for example generates basic metadata files recursively in directory structure,
could work like find . -name "runtest.sh" -print "some strucutre to this directiry."
it could create basic test metadata templates
According to documentation, test case identifiers should be unique within current fmf structure. However, it is possible to work with a configuration text file, that has duplicated identifiers.
Example:
/python/Sanity/sys:
component:
- python3
environment:
PYTHON: /usr/libexec/platform-python
PACKAGES: python3
/python/Sanity/sys:
component:
- python
Having that test.fmf
, fmf show
in the root directory shows
/Sanity/sys/test/CoreOS/python/Sanity/sys
component: python
so the first test was overwritten by the second one with duplicated identifier with no error returned.
As a user, I would like to be aware of the fact that I have duplicated test case identifiers in .fmf
file, either with a visible warning or error and abortion of execution.
Hi,
fmf ls hangs up and runs forever.
(venv)➜ empty git:(master_copr_build) ✗ mkdir empty
(venv)➜ empty git:(master_copr_build) ✗ cd empty
(venv)➜ empty git:(master_copr_build) ✗ fmf ls
10734 astepano 20 0 170704 20588 2368 R 50.3 2.0 0:18.81 fmf
Hi,
I am not sure if this is a wrong example, but it fails:
http://fmf.readthedocs.io/en/latest/examples.html
fmf
Traceback (most recent call last):
File "/usr/bin/fmf", line 35, in <module>
fmf.cli.main()
File "/usr/lib/python2.7/site-packages/fmf/cli.py", line 120, in main
tree = fmf.Tree(path)
File "/usr/lib/python2.7/site-packages/fmf/base.py", line 57, in __init__
self.grow(data)
File "/usr/lib/python2.7/site-packages/fmf/base.py", line 146, in grow
data = yaml.load(datafile)
File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 71, in load
return loader.get_single_data()
File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
node = self.get_single_node()
File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node
while not self.check_event(MappingEndEvent):
File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
if self.check_token(KeyToken):
File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 115, in check_token
while self.need_more_tokens():
File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 149, in need_more_tokens
self.stale_possible_simple_keys()
File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 289, in stale_possible_simple_keys
"could not found expected ':'", self.get_mark())
yaml.scanner.ScannerError: while scanning a simple key
in "/home/astepano/shared/osci/standard-test-roles/3.fmf", line 6, column 1
could not found expected ':'
in "/home/astepano/shared/osci/standard-test-roles/3.fmf", line 7, column 16
➜ standard-test-roles git:(master_copr_build) ✗ cat 3.fmf
description: Check basic download options
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min
/requirements
requirement: Various download options working correctly
priority: low
/get-file:
coverage: wget/download
/output-document:
coverage: wget/download
/continue:
/timestamping:
/tries:
/no-clobber:
coverage: wget/download
/progress:
/quota:
/server-response:
/bind-address:
/spider:
It would be useful to be able to merge multiple structures into a single metadata tree. An example use case can be merging custom configuration with default settings. As a quick hack we can do something like this:
tree = fmf.Tree("default")
tree.grow("custom")
For such cases it would be probably better to have possibility to initialize empty tree and then grow it multiple times:
tree = fmf.Tree()
tree.grow("default")
tree.grow("custom")
We could also add support for multiple trees directly in the constructor (which seems like a little bit more clean solution):
tree = fmf.Tree(['default', 'custom'])
However there are a few things which need to be solved: There are attributes which kind of expect a single tree only: root
, version
, original_data
. Also, repeated call of grow()
results in inherit()
being applied multiple times which might cause problems.
in https://github.com/sopos/tests:
$ fmf ls
/fapolicyd/Sanity/trust-db
/luksmeta/Regression/bz1461448-running-luksmeta-on-a-non-LUKS-device-returns-IO-error
/usbguard/Sanity/bz1772149-try-to-parse-match-all-rule
/usbguard/Sanity/config-sanity
/usbguard/Sanity/selinux
/usbguard/Sanity/service-sanity
$ fmf ls --filter 'tag: Fedora_missing_external_dependencies'
/usbguard/Sanity/selinux
$ fmf ls --filter 'tag: -Fedora_missing_external_dependencies'
/usbguard/Sanity/config-sanity
For the negative tag I would expect everything but /usbguard/Sanity/selinux to be listed.
When trying to run tmt init
as root in a container I get:
# tmt init
Traceback (most recent call last):
File "/usr/bin/tmt", line 11, in <module>
tmt.cli.main()
File "/usr/lib/python3.7/site-packages/click/core.py", line 763, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/core.py", line 716, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.7/site-packages/click/core.py", line 1136, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.7/site-packages/click/core.py", line 955, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.7/site-packages/click/core.py", line 554, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python3.7/site-packages/tmt/cli.py", line 747, in init
echo("Tree '{}' initialized.".format(tree.root))
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 472, in root
return self.tree.root
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 462, in tree
self._tree = fmf.Tree(self._path)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 84, in __init__
self.grow(data)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 291, in grow
dirpath, dirnames, filenames = next(os.walk(path))
StopIteration
# rpm -q tmt
tmt-0.8-1.fc31.noarch
# rpm -q python3-fmf
python3-fmf-0.10-1.fc31.noarch
Hi,
python2-fmf-0.5-1.fc27.noarch
In [1]: import fmf
In [3]: fmf.Tree(".")
---------------------------------------------------------------------------
FileError Traceback (most recent call last)
<ipython-input-3-b75dcdf15015> in <module>()
----> 1 fmf.Tree(".")
/usr/lib/python2.7/site-packages/fmf/base.pyc in __init__(self, data, name, parent)
62 self.name = "/"
63 if not isinstance(data, dict):
---> 64 self._initialize(path=data)
65 data = self.root
66 # Handle child node creation
/usr/lib/python2.7/site-packages/fmf/base.pyc in _initialize(self, path)
88 raise utils.FileError(
89 "Unable to find tree root for '{0}'.".format(
---> 90 os.path.abspath(path)))
91 root = os.path.abspath(os.path.join(root, os.pardir))
92 except StopIteration:
FileError: Unable to find tree root for '/root/file/tests'.
FMF is now a part of standard-test-roles.
Some tests do not uses FMF.
If I try to init fmf Tree it fails.
It would be nice to have an Exception of specific type for such case.
for one project I would like to see some semantics to be able to reference nodes via name (this simple reference should be enough)
see example code:
/maintainer_label:
test: script.py
tags: ["a", "b"]
/sanity
/from_tag_not_latest
test: anotherscript.py
/tests
/checks:
/check1@maintainer_label:
tags+: ["required"]
/check2@/sanity/from_tag_not_latest:
tags+: ["required"]
usable_targets: ["dockerfile"]
what will use special symbol @
to use this item name as sources
and data will be copied from this item instead of parent in tree structure.
so that list item /tests/checks/check1
will contain:
test: script.py
tags: ["a", "b", "required"]
does that make sense?
Possible issues
a@b@c
and allow also references to other refs itemsWhen exploring directories fmf
should probably ignore other metadata trees in the directory tree. Yesterday we've discussed the following real-life scenario which seems to support this approach:
Standard Test Roles support fetching tests from remote repositories. At the same time there can be tests stored directly in dist-git. When these two options are mixed we should still be able to gather reasonable metadata for both.
Here's a simple example directory structure:
.
├── fetched
│ ├── fetched1
│ ├── fetched2
│ ├── fetched3
│ ├── .fmf
│ │ └── version
│ └── main.fmf
├── .fmf
│ └── version
├── main.fmf
├── stored1
├── stored2
└── stored3
Current output in the main tests
directory (note the modified node identifiers):
/stored1
/stored2
/stored3
/fetched/fetched1
/fetched/fetched3
/fetched/fetched2
Current output for the fetched test cases in tests/fetched
:
/fetched1
/fetched3
/fetched2
Expected output for the main directory:
/stored1
/stored2
/stored3
Guys, what are your thoughts on this?
Hello.
tmt raises traceback when CWD lost rights to read/execute.
How to reproduce:
under root user:
dr-xr-x---. 4 root root 226 Nov 14 03:26 /root/
[root@koza-6 ~]# su rhack
[rhack@koza-6 root]$ tmt
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 96, in _initialize
while ".fmf" not in next(os.walk(root))[1]:
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/tmt", line 11, in
tmt.cli.main()
File "/usr/lib/python3.7/site-packages/click/core.py", line 763, in call
return self.main(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/core.py", line 716, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.7/site-packages/click/core.py", line 1113, in invoke
return Command.invoke(self, ctx)
File "/usr/lib/python3.7/site-packages/click/core.py", line 955, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.7/site-packages/click/core.py", line 554, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python3.7/site-packages/tmt/cli.py", line 159, in main
tmt.Test.overview(tree)
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 122, in overview
style(str(test), fg='red') for test in tree.tests()]
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 489, in tests
return [Test(test) for test in self.tree.prune(
File "/usr/lib/python3.7/site-packages/tmt/base.py", line 466, in tree
self._tree = fmf.Tree(self._path)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 74, in init
self._initialize(path=data)
File "/usr/lib/python3.7/site-packages/fmf/base.py", line 103, in _initialize
raise utils.FileError("Invalid directory path: {0}".format(root))
fmf.utils.FileError: Invalid directory path: /root
Something better than traceback would be nice :). Yeah, It's corner case but.. I happened! :)
$ fmf init
Traceback (most recent call last):
File "/usr/bin/fmf", line 4, in
import('pkg_resources').run_script('fmf==0.5.1', 'fmf')
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 658, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 1445, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/fmf-0.5.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 194, in main
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 73, in init
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 145, in command_init
File "/usr/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/home/jkrysl/tests/python-stqe/tests/.fmf'
I hit an easy to reproduce traceback when defining same leaf with different methods (using 2 files from same location). If I define it using main.fmf
and 1/main.fmf
, there is no issue. But doing the same thing using main.fmf
and 1.fmf
leads to following traceback:
# tree
.
├── 1.fmf
└── main.fmf
# cat 1.fmf
in_1: True
# cat main.fmf
/1:
in_main: True
# fmf
Traceback (most recent call last):
File "/usr/bin/fmf", line 4, in <module>
__import__('pkg_resources').run_script('fmf==0.4.1', 'fmf')
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 658, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1445, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/fmf-0.4.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in <module>
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 120, in main
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 57, in __init__
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 152, in grow
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 107, in child
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 124, in grow
AttributeError: 'dict' object has no attribute 'rstrip'
This is reproducible since the very beginning, so no regression:
# fmf
Traceback (most recent call last):
File "/usr/bin/fmf", line 4, in <module>
__import__('pkg_resources').run_script('fmf==0.1.1', 'fmf')
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 658, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1445, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/fmf-0.1.1-py2.7.egg/EGG-INFO/scripts/fmf", line 35, in <module>
File "build/bdist.linux-x86_64/egg/fmf/cli.py", line 96, in main
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 50, in __init__
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 115, in grow
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 82, in child
File "build/bdist.linux-x86_64/egg/fmf/base.py", line 90, in grow
AttributeError: 'dict' object has no attribute 'rstrip'
Image a node where I define a reference to another git repo where the branch continues.
I can image a reserved attribute
fmf_subtree:
url: https://github.com/...
reference: master
such repo would be pulled and the tree would be simply extended by this part.
It is still questionable how the inheritance should behave at such points where a new fmf tree begins. This might be controlled by another attribute of fmf_subtree, e.g. boolean inheritance.
fmf_subtree:
url: https://github.com/...
reference: master
inheritance: true
setup.py
says so (and, doesn't even mention Python 3.x). But, text/data/encoding issues exist when running on Python 2.7:
$ fmf > foo
Traceback (most recent call last):
File "/home/cleber/.local/bin/fmf", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/home/cleber/src/fmf/bin/fmf", line 35, in <module>
fmf.cli.main()
File "/home/cleber/src/fmf/fmf/cli.py", line 127, in main
print(show, end="")
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0160' in position 140: ordinal not in range(128)
$ git rev-parse HEAD
3d59b97b8beaa75035526fdf992530adef26f19a
As a User I want to be able to filter name by using fmf.filter so I have consistency with filter by attributes.
fmf ls --filter tier:1
but fmf ls --name /test/me
.
I want be able to use fmf ls --filter name:/test/me
as well
If you specify attributes on deeper layer than actual + 1, this gets interpreted as standalone leaf and does not get merged. To demonstrate this I created 2 entries that should get merged into one, as they appear to be on the same position in the tree. But the result is 2 different leaves, 'test -> test/directory' and 'test -> test -> directory' ('->' symbolizes branch of the tree).
[root@localhost test]# tree
.
├── main.fmf
└── test
└── directory
└── main.fmf
2 directories, 2 files
[root@localhost test]# fmf
test/test/directory
test: True
test/test/directory
test: True
[root@localhost test]# cat main.fmf
/test/directory:
test: True
[root@localhost test]# cat test/directory/main.fmf
test: True
Currently when searching for tests fmf returns relative paths:
# fmf /root/tests/ --brief --key test --filter component:libselinux
tests/upstream-tests/selinux/libselinux/setenforce
tests/upstream-tests/selinux/libselinux/getsebool
What I would like to see would be absolute paths:
# fmf /root/tests/ --brief --key test --filter component:libselinux --absolute_path
/root/tests/upstream-tests/selinux/libselinux/setenforce
/root/tests/upstream-tests/selinux/libselinux/getsebool
The Beakerlib metdatada file Makefile
contains, among other things, fields Requires
and Rhts-Requires
, which hold test dependencies. This is a yet another important piece of information, which should be stored along with the test itself, however, in its current form (in Beakerlib) it's pretty inflexible, as there is no way to split such dependencies for multiple versions of a distribution, let alone for different distributions.
That said, this real-life example:
@echo "Requires: systemd yum-utils network-scripts" >> $(METADATA)
is pretty problematic, as:
yum-utils
has been replaced by dnf-utils
on RHEL8, but there is no way to conditionally affect this during dependency resolutionyum-utils
may not be present on other distributions, so using these dependencies directly without any modifications is basically impossibleThis can be solved (and has been done so in many tests) by either placing both yum-utils
and dnf-utils
into the test dependencies and hope dnf/yum can handle it (and usually help it by using --skip-broken
), or edit the test itself to do something like:
if rlIsRHEL "<=7"; then
yum -q -y install yum-utils
else
dnf -q -y install dnf-utils
fi
which is ugly, unportable, and in many cases even unstable.
As we're going to discuss this more in depth next week, I'll just dump here some ideas which can be used as a starting point.
To make the dependency resolution more flexible, and in this case more dynamic, we could, theoretically, re-use part of the mechanism from relevancy resolution, as it already provides functionality necessary to resolve dependencies as well.
For example:
example: |
requires:
- "distro >= f-22 | distro >= rhel-8: systemd dnf-utils cryptsetup"
- "distro < f-21 | distro < rhel-8: systemd yum-utils cryptsetup"
The one of the disadvantages of this syntax is increased verbosity, as there is no way (at least known to me), to do something like distro > f-22: dnf-utils; else: yum-utils
.
On contrary to the first solution, this one is more passive, as it's the plain key-value mapping with a pre-defined distro format string.
requires: |
fedora-[12][01]:
- systemd yum-utils
fedora-[2-9].*:
- systemd dnf-utils
However, this is really confusing and error-prone format (and I'm not even sure if it's a valid YAML).
The Tree.node()
method fetches fmf metadata into ~/.cache/fmf
. This can grow in time. It would be nice to implement an easy way to clean up old/all cached stuff. There should be available both from python API and command line. Brainstorming command line usage:
fmf gc
fmf gc --all
fmf gc --old
As outlined in #43 it would be good to have an easy way to discover all metadata trees present under given directory on the filesystem. This would allow us to easily process them one-by-one. A new subcommand find
should be added which will list root directories of all subtrees found. Additional options can be provided later to optionally list the metadata as well. Example output:
> cd /usr/share/doc/fmf-0.5/
> fmf find .
./examples/child
./examples/scatter
./examples/deep
./examples/touch
./examples/wget
./examples/merge
It would be useful to be able to undefine an inherited attribute. For example having an extra directory with helper files which should not be considered as tests. Perhaps something like this?
test: None
Hi,
I think about using fmf command and how it deals with directories and root items.
dir structure like: a/b/c/main.fmf
and first main.fmf
is in c
directory could produce structure like:
c/fist ....
c/second ...
and root
item is a/b
and it could be invoked like
fmf --brief .
fmf --brief a
fmf --brief a/b
fmf --brief `pwd`
and everytime it will produce same output what I see very beneficial for referencing, than now it is very dependent on invocation of command.
then I can imagine, that you have tests for beakerlib and you have two components like bash
and wget
and you checkout these gits locally, call fmf
tool and it will produce consistent sructure not dependent on place of invocation.
imagine for example you have ~/git/wget
and ~/otherpath/git/bash
and invocation like fmf ~
will produce output like
wget/sanity
wget/second
bash/sanity
bash/regression
What do you think? maybe could be beneficial also for merging trees
Regards
Honza
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.