nexb / aboutcode-toolkit Goto Github PK

The current pre_process() is doing way to many things. We should break down these into couple of small functions instead.

When creating attribution, an error log should be created containing the warnings and errors
For example:
Mandatory fields are missing from the .ABOUT files.
The error log would show which components are missing the mandatory fields and specifically which mandatory fields are missing.

For now a simple .py script is best.
Later we need to have extra libs and the single file approach will no longer work.
We need to pick what packaging works knowing that target users may not be python developers. Some possibilities include:

• sdist on Pypi for Python developers. see #4
• pyz zip for general usage that requires a Python installation
• py2exe, pyinstaller or similar for Windows/Mac/Linux that could work without a Python installed

Should ABOUT be system-wide installed? with man pages? as .deb or .rpm packages?

For now we dump the python repr... we can do better

('download_url', '', 'This optional field has no value.'),('download_url', '', 'URL is either not in a valid format, or it is not reachable.')

If a field doesn't contains any value, we should not check for format.

For instance,

all the mandatory fields should be at the first few lines, and then probably all the vcs* should be grouped together etc.

1 - Overwrites the current ABOUT field value if existed

The current behavior is kinda like replace the whole ABOUT file.

For instance,
if my ABOUT file already have a version value but my input csv doesn't have any value in the version column, the version field value will be removed which is not correct.

WARNING: [Field: date-retrieved, Value: date-retrieved: 2013-01-16, Message: Field name contains invalid characters: '-': line ignored.]

The current design is to do nothing if the ABOUT file exist.

The tool should be able to look into the ABOUT file to see if there is anything that need to be updated if the ABOUT file exist.

Provide options, such as:

1. overwrite the current field value
2. keep the current field value and only add the "new" field and field value
3. replace the ABOUT file with the current generation

In addition, if no option is set, the tool will not touch the ABOUT file and prompt "ABOUT file already existed."

... we should move to unittest2 and and unitttest2 @expectedfailure

Create an error.txt if error exists.

For example:

description : Some description.

is silently ignored.

This is usually not a good thing to write files in the testdata directories that are under version control. Should use a temp file/dir instead

add list of supported SCM tools and raise warning if not found: cvs, svn, git, arch, hg, bzr, darcs, clearcase, perforce

The ABOUT spec states the paths are posix, so we should use the posixpath module for this. To be resolved to actual files the paths would need to converted to local os path (ie windows or posix conventions internally)

A test should easily show if we do it right first

Given ABOUT files or a list of ABOUT files in a CSV and a base directory, I want to generate an attribution notice for all these components:

• supported output should be text and HTML
• the output should be based on a template. Mustache or Jinja2 are a good candidates
• for this to work, there need to be some mandatory fields present

The argparse module allows easy implementation of optional and positional parameters and also handles the output formatting.

possible create another function called "process_input"

For instance,

self.assertTrue(about_file.check_url("http://www.google.com", True))

It will only return the request status code, such as 200, 404 etc instead of the boolean value.

Generate attribution from ABOUT files for a specific list of components. User should be able to pass a list of components they would specifically like to generate attribution for.

In other words, from a given set of about files and an input list (that contains component names which map to about file field "name") be able to generate attribution.

I would like to have a configurable command line option to pass a list of fields considered as mandatory overriding the basics... This is important because for a certain usage certain fields will be needed. For instance generating attribution or redistribution requires certain fields to be set which may be otherwise considered as optional.

If an engineering organization were to take up using ABOUT in a serious, ongoing way, it probably would be very helpful to provide a tool/utility that would generate a "scorecard" on a target codebase directory. Some of the scorecard points would include:

Percentage/counts of the directories that (1) have no ABOUT info, (2) have some ABOUT info, (3) appear to be fully-documented in ABOUT.

The value of this would be that as new code is checked into a codebase, the scorecard could help determine if it's time to review the ABOUT files.

The scorecard tool might also identify potentially obsolete ABOUT files, where the date (timestamp) on the ABOUT file is older (by a number of days, perhaps parameter-driven) that the file/directory that it describes.

The default should be to not check over the network if a URL exists.

The tool should be able to extract the license text based on the user license_url provided in the ABOUT file and save the license text in the user specific directory.

Given a directory with ABOUT files OR a csv and directory as an input, I want to collect all the source code archives that have the redistribute flag set , and bundle all these in a zip or tar

These for now are just ignored...
Should these be returned in the output?

The "..ABOUT" is not correct. If this is the case, maybe we want to use the "name" field as the filename?

This will also make implementing subcommands easier.

http://docs.python.org/2.7/howto/argparse.html
http://docs.python.org/2.7/library/argparse.html

Currently the tool will skip a directory without returning any error message if a directory is not readable

For intance, if I run the about.py on

tmp/t1/t2/test.ABOUT

The generated csv's about_file field will return
tmp/test.ABOUT

the t1/t2/ are pruned

as title

For instance,
if I run it like the following:
$ python about.py ../about-code-tool/testdata/ /cygdrive/z/tmp/t2.csv

The output of the about_file is
../about-code-tool/testdata/

which is incorrect.

user should be able to generate ABOUT files from a CSV as an input

Current proposals:

A. An interactive prompt.

For instance, user only needs to type

python about.py
Then the script will prompt:

1. Collect ABOUT files context
2. Generate ABOUT files
3. Exit
   Select:
   something like the above, and the ask for input/output and all kind of options.

B. Create another script, maybe, called generate_about_file.py

The syntax should be:

python genabout.py [input_csv] [generated location]

as title.

We should have a feature such that given a directory, it will parse ABOUT files, and ensure that all files in the directory are directly or indirectly (ie from a parent directory) documented or referenced in an ABOUT file

For instance, in the ABOUT file, user has specify the location of the license file. The tool should be able to copy this license file into a user specific location.

It would be nice to offer the possibility to log the warnings and errors in the stdout.

Probably a new option "--verbosity"

0 is default
1 is errors
2 is errors + warnings
So I can see the issues in my terminal without having to open the output file as I'm fixing the issues.

a -i and -o might be better command lines options than using implicit input and ouputs?

as title

Currently we are only checking that the about file field names are ascii.

The current design is to manipulate the code structure from the provided input and then generate the ABOUT files in the structure.

It may be a good idea to provide options for users to generate all the ABOUT files into the user specific directory.

The current validation method doesn't handle URL with spaces. Although it is very less likely there is any URL contains spaces, it happens.

Reference:
http://en.wikipedia.org/wiki/URL_normalization
https://pypi.python.org/pypi/urlnorm
http://code.google.com/p/google-url/

Weird behavior if the about_file value startswith('/')

One solution is the normalize it and make sure the about_file is not starting with '/'

replace the '/' with '.ABOUT' for directory.