While working on streitberggothic I encountered following issue:

# mismatch in col len
import pandas as pd
from pysem.glosses import to_concepticon

PATH = "Streitberg-1910-3659.tsv"

def main():
    dfgot = pd.read_csv(PATH, sep="\t").fillna("")
    glosses = [{"gloss": str(g), "pos": str(p)}
                for g, p in zip(dfgot.sense, dfgot.pos)]

    print(len(glosses),
          len(to_concepticon(glosses, language="de", pos_ref="pos",
                             max_matches=1)))

if __name__ == "__main__":
    main()

prints: 3645 3274

So there are somehow less output matches than input provided.

I thought this might be valuable information for the developers, even though eventually I found a workaround, like so:

def main():
    dfgot = pd.read_csv(PATH, sep="\t")

    conid, conglo = [], []
    for g, p in zip(dfgot.sense, dfgot.pos):
        gloss = [{"gloss": g, "pos": p}]
        out = list(to_concepticon(gloss, language="de",
                                  pos_ref="pos", max_matches=1).values())[0]
        if out:
            conid.append(out[0][0])
            conglo.append(out[0][1])
        else:
            conid.append(None)
            conglo.append(None)

    dfgot["CONCEPTICON_ID"], dfgot["CONCEPTICON_GLOSS"] = conid, conglo
    del dfgot["form"]
    dfgot.to_csv("concepts.tsv", index=False, encoding="utf-8", sep="\t")

if __name__ == "__main__":
    main()

The usage example contains this line:
to_concepticon([{"gloss": "Fuß", pos: "noun"}], language="de"}])

But, that seems to be a typo for this:
to_concepticon([{"gloss": "Fuß", "pos": "noun"}], language="de")

The dash regularly occurs in concepticon concept sets, to avoid that we loose the mapping, it should therefore not be defined as "marker" in the parse_gloss function.

pysem works with concepticon v2.5.0 but not with the latest version. I think it would be nice to be able to see somewhere which concepticon version the concepticon.zip file is based on, e.g. in the readme. Or maybe the file could be updated by the latest version or even replaced by an API?

this is not accounted for by now!

Currently, my script ids_lookup finds +100 concepts that are not found via the getcl lookup.

You suggested that this might come out of the fact that entries in Concepticon are not split. Do you mean the column spanish by this? Because this column gets indeed split in my script.

We should add some of these in a folder data/ for now, with additional information and code there. The resulting network file should then be accessible from within the pysem library.

For the dictionary linking procedure, we need a simple CLI that allows us to use the code to create a concept list from a dictionary (or another kind of file).

These don't seem to work by now.

The code is in theory available, but should be adjusted, so we can compute it from the CLICS data.