to prevent issues like https://github.com/glottobank/lexibank-data/issues/115

Basic setup for logging in command line apps should be supported in clilib.ArgumentParser.

Add an apilib module providing basic boilerplate functionality for implementing API objects for data repositories. In particular, there should be

• commonly applicable validators for data objects based on the attr package,

def valid_enum_member(choices, instance, attribute, value):
    if value not in choices:
        raise ValueError(value)


def valid_range(min_, max_, instance, attribute, value):
    if value < min_ or value > max_:
        raise ValueError(value)

• a base class for API objects, taking a path to a directory as single argument.

class API(UnicodeMixin):
    def __init__(self, repos):
        self.repos = Path(repos)

    def __unicode__(self):
        return '<{0} repository {1} at {2}>'.format(self.repos.name, git_describe(self.repos), self.repos)

A function clldutils.path.readlines, which returns a list of lines read from a file would be useful, in particular as a place to add functionality like stripping of "commented" lines, which - combined with clldutils.dsv.reader's ability to read csv from lines - would go a long way towards replacing e.g. LingPy's "qlc" format.

badge.badge should accept all atomic parts of the label/URL it creates as input argument; such that it exposes all functionality of the shields.io service.

With argparse.FileType, I can easily create file objects. Passing such an object to configparser.ConfigParser is fine, but inifile.INI tries to convert it into a Path and then open it.

Similarly, this means I cannot just pass sys.stdout to .write.

To allow somewhat idempotent round-tripping, it must be possible to retain marker with empty values when parsing SFM files.

The ISO object should store the datestamp identifying the ISO 639-3 download version it provides access to.

A text wrapper which is aware of markdown tables (i.e. leaves them alone) would be useful (e.g. for glottolog).

To allow proper wrapping of without splitting command names the list of subcommands in the argument parser's help message should be separated with whitespace.

Since tempfile.TemporaryDirectory is not available in python 2.7, we should provide a temporary directory as context manager.

maybe, indicating in which line they occur?

The csvw sub-package should be turned into its own proper package (maybe requiring clldutils), since it contains a well-defined set of functionality that could be useful outside the context of clld apps.

ArgumentParser.main should return an integer suitable as exit code for sys.exit, e.g. 0 on success, 1 otherwise.

To make sure we use pathlib from the standard library for Python 3.5, the import order in clldutils.path should be switched to first try the standard lib module and fall back to pathlib2 on ImportError.

A context manager to update data stored as JSON would be nice:

>>> with jsonlib.update(fname) as obj:
...     obj['new'] = 2

The get_column method of a table schema

Currently, download and unpacking is baked into the same function iter_tables. Since the ISO object can be instantiated by pointing it to a local path with the zipped tables, it would be good if these could be downloaded easily.

Foreign key specifications do not have a .header (instead, they can be multiple columns, described by columnReference, if I understand http://w3c.github.io/csvw/metadata/#dfn-foreign-key-definition correctly). I think that one also says that other properties (.virtual) should not exist?

Aditionally, Table.write passes data writing to its columns' .write method. Foreign keys don't implement that method, and it might be hard to implement anyway because in the general case it outputs more than one column, if I get it correctly.

There should be support for identifying licenses using SPDX identifiers.
Maybe a scraped extract of the SPDX list should be included with clldutils.

Instead of raising KeyError when encountering an invalid command name (see below), clilib.ArgumentParser.main should print usage and an error message.

(pycldf)dlt5502178l:~/venvs/pycldf/pycldf$ cldf pycldf/tests/fixtures/ds1.csv
Traceback (most recent call last):
  File "/home/shh.mpg.de/forkel/venvs/pycldf/bin/cldf", line 9, in <module>
    load_entry_point('pycldf==0.5.2', 'console_scripts', 'cldf')()
  File "/home/shh.mpg.de/forkel/venvs/pycldf/pycldf/pycldf/cli.py", line 88, in main
    sys.exit(parser.main())
  File "/home/shh.mpg.de/forkel/venvs/pycldf/local/lib/python2.7/site-packages/clldutils/clilib.py", line 35, in main
    self.commands[args.command](args)
KeyError: 'pycldf/tests/fixtures/ds1.csv'

Midterm, we aim at switching all our code to using pytest for testing rather than the unittest/nose combo.
Thus, most of the code in testing will be obsolete or should be reimplemented to work in the pytest paradigm.

Instead of looping through all possible target rows for a foreign key for each source row, we should create an index of primary keys of the target table and then just check for membership for each foreign key.

See https://pymotw.com/3/decimal/#special-values for how things like Infinity of NaN are represented, and http://w3c.github.io/csvw/syntax/#h-formats-for-numeric-types for what must be recognized.

Since ISO 639-3 code tables - as of 2018-01-23 - in UTF-8 now contain a BOM, they must be read using the utf-8-sig encoding.

For several projects we want to be able to import custom code not within a python package. The most convenient (and portable) way to this seems to be via a context manager, adapting sys.path appropriately:

@contextmanager
def with_sys_path(d):
    p = d.as_posix()
    sys.path.append(p)
    yield
    if sys.path[-1] == p:
        sys.path.pop()

Alternatively, the complete functionality, i.e. importing a module by filesystem path could be provided:

        with with_sys_path(path.parent):
            return import_module(path.name)

Right now, subcommand names for clilib.ArgumentParser can only be function names. Thus, reserved words like class are impossible and names like list would cause shadowing of built-ins where the function is defined. So there should be a way to register callables as commands under custom names.

In clldutils.testing we should add functionality to capture stdout, along the lines of

import sys
from contextlib import contextmanager

from six import StringIO

@contextmanager
def capture(func, *args, **kw):
    out, sys.stdout = sys.stdout, StringIO()
    func(*args, **kw)
    sys.stdout.seek(0)
    yield sys.stdout.read()
    sys.stdout = out

See http://w3c.github.io/csvw/syntax/#parsing-cells

Some of the dependencies still listed in setup.py are obsolete now that csvw has moved to its own package.

Currently it is not possible to specify an encoding for ini files to be read in INI.from_file. On win, the encoding used seems to default to cp1252, which typically isn't what we want.

The regex defining valid SFM markers seems to be overly restrictive. It should also allow digits.

The regex passed as datatype.format is compiled

and then run using the match method, which will allow strings which start with a matching pattern to validate.

Some three-letter codes are reserved for local use by ISO 639-3. This information should be available programmatically, e.g. as a list:

from string import ascii_lowercase
local_use = ['q' + x + y for x in ascii_lowercase[:ascii_lowercase.index('t') + 1] for y in ascii_lowercase]

The constructor classmethod's must actually use the cls argument.

Rather than raising ValueError at the first data error when reading CSV files, logging (and skipping invalid rows) should be used to allow batch validation of complete datasets.

The method Table.iterdicts() has no docstring, but it's not entirely obvious what it does. I suggest the following.

"""Iterate over the rows of the table

Create an iterator that maps the information in each row to a `dict` whose keys are the column names of the table and whose values are the values in the corresponding table cells, or for virtual columns (which have no values) the valueUrl for that column. This includes columns not specified in the table specification.

Parameters
----------
log: Logger object (default None)
    The object that reports parsing errors. If none is given, parsing errors raise ValueError instead.
with_metadata: bool (default False)
    Also yield fname and lineno
fname: file-like, pathlib.Path, or str (default None)
    The file to be read. Defaults to inheriting from a parent object, if one exists.

Yields
------
fname: str (only if with_metadata)
lineno: int (only if with_metadata)
row: dict
""""

Currently, clldutils.testing.capture yields the captured output to stdout only. Often, though, it may be important to investigate this output together with the return value of the function call.

The JSON renderer in pyramid 1.6.1 passes the request object into an objects __json__ method as positional argument.

We want to allow foreign keys in tables to be optional, marked by a NULL value of the referencing column(s). The current behaviour can still be had by making the column(s) required.

The idea is to turn the metadata support into a mechanism to reformat csv files, e.g. to reformat dates one could do

>>> tg = TableGroup(...)
>>> items = list(tg.tables[0])
>>> tg.tables[0].schema.columns[0].datatype.format = 'yyyy-MM-dd'
>>> tg.tables[0].write(items)

It should be possible to read a csv file according to a dialect description provided as dict.

This is the 10th time that I had these issues: reading in a file, searching for a header, I see an error, and only in the end I find out, why the header (in this case "ID") was not found: The first character was \ufeffID.

This happened in the concepticon-api, which uses clldutils, and I wonder, given that this usually leads to a long debugging session, whether we should not remove this character immediately if it occurs as the header? Or are there other preferred ways how to deal with it?