Giter VIP home page Giter VIP logo

cldflex's Introduction

cldflex

Convert FLEx data to CLDF-ready CSV.

Versions PyPI License

Many descriptive linguists have annotated language data in a FLEx (SIL's Fieldworks Lexical Explorer) database, which provides perhaps the most popular and accessible assisted segmentation and annotation workflow. However, a reasonably complete data export is only available in XML, which is not human-friendly, and is not readily converted to other data. A data format growing in popularity is the CLDF standard, a table-based approach with human-readable datasets, designed to be used in CLLD apps and easily processable by any software that can read CSV files, including R, pandas or spreadsheet applications. The goal of cldflex is to convert lexicon and corpus data stored in FLEx to CSV tables, primarily for use in CLDF datasets.

Installation

cldflex is available on PyPI:

pip install cldflex

Command line usage

At the moment, there are three commands: cldflex corpus for .flextext files; cldflex dictionary and cldflex wordlist for .lift files. All commands create a number of CSV files. One can either use cldfbench to create one's own CLDF datasets from these files, or add the --cldf argument to create a simple CLDF dataset. Project-specific configuration can be passed by --conf your/config.yaml, or creating a file cldflex.yaml

corpus

Basic usage:

cldflex corpus texts.flextext

Connect the corpus with the lexicon:

cldflex corpus texts.flextext --lexicon lexicon.lift

Create a CLDF dataset:

cldflex corpus texts.flextext --lexicon lexicon.lift --cldf

dictionary

Extract morphemes, morphs, and entries from lexicon.lift:

cldflex dictionary lexicon.lift

Create a CLDF dataset with a Dictionary module:

cldflex dictionary lexicon.lift --cldf

wordlist

Create a CLDF dataset with a Wordlist module:

cldflex wordlist lexicon.lift --cldf

API usage

The functions corresponding to the commands above are cldflex.corpus.convert() and cldflex.lift2csv.convert().

Configuration

There is no default configuration. Rather, cldflex will guess values for most of the parameters below and tell you what it's doing. It is suggested to start out configuration-free until something goes wrong or you want to change something. Create a YAML file for CLI usage, pass a dict to the convert methods.

  • obj_lg: the object language
  • gloss_lg: the language used for glossing / translation
  • msa_lg: the language used for storing POS information
  • lang_id: the value to be used in the created tables
  • glottocode: used to look up language metadata from glottolog
  • csv_cell_separator: if there are multiple values in a cell (allomorphs, polysemy...), they are by default separated by "; "
  • form_slices: set to false if you don't want form slices connecting morphs and word forms
  • mappings: a dictionary specifying name changes of columns in the created CSV files

cldflex's People

Contributors

fmatter avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

cldflex's Issues

be tolerant with segnum language

In line 420 of flex2csv.py you have

...
   (f"segnum_{conf['gloss_lg']}_phrase", "Sentence_Number"),
...

I do not know how often there is a different "segnum" per language in flex files. In my case, the segnum language was set to French, and the value was "3.1", so not particularly localized. The program failed hard and produced no output.

I can of course prepare some yaml file to take care of this, but maybe one could in the case of only one segnum being present for a given sentence simply default to that segnum instead of requiring the language to be specified.

~/.config/cldf/catalog.ini has no entry for glottolog'

I try to convert a file in script.

from cldflex.flex2csv import convert
flexinputfilepath = "ende.xml"
corpus = convert(flexinputfilepath)

works fine, as far as I can tell. But I want CLDF, so I say

from cldflex.flex2csv import convert
flexinputfilepath = "ende.xml"
corpus = convert(flexinputfilepath,cldf=True)

This gives me

KeyError: 'Config /home/snordhoff/.config/cldf/catalog.ini has no entry for glottolog'

https://github.com/fmatter/cldflex tells me "pass a dict to the convert methods." I have tried

convert(filename, conf={'Glottocode':'abcd1234'},cldf=True)
convert(filename, conf={'glottolog':'abcd1234'},cldf=True)

, to no avail.

I have also created a yaml file with the content Glottocode: abcd1234, as asked for by https://github.com/fmatter/cldflex. However, the script complains that it does not find an ini file, and that there is no entry for "glottolog". So I wonder whether yaml or ini is required. In any case, I have also created an ini file in ~/.config.cldf. I give the content below

[glottolog]
Glottocode = abcd1234
glottolog = abcd1234

This does not help either.
Could you share information how to provide the relevant glotto-information, or, otherwise, how to bypass the test (I do not need the glotto information for my use case).

enclitlics double

For the attached input XML file, enclitics are outputted twice in both the vernacular line (=da,=peyang) and the gloss line (=NOM,=COM).

ause=da=da	llɨg	kälsre=peyang=peyang
...
old woman=NOM=NOM	boy	small=COM=COM

Expected:

ause=da	llɨg	kälsre=peyang
old woman=NOM	boy	small=COM

In the XML file, we find two instances of "da", but only one "NOM" in uppercase, suggesting that indeed certain fields are outputted twice.

<morph type="enclitic" guid="d7f713e1-e8cf-11d3-9764-00c04f186933">
   <item type="txt" lang="kit">=da</item>
   <item type="cf" lang="kit">=da</item>
   <item type="gls" lang="en">=NOM</item>
   <item type="msa" lang="en">nominal enclitic</item>
</morph>
<item type="gls" lang="en">nom</item>
<item type="pos" lang="en">nominal enclitic</item>

test.xml.txt

check for None when splitting on Sentence_Number.Phrase_Number

In line 406 of flex2csv.py, you split an ID into sentence number (before period) and phrase number (after period). This fails if the sentence number had not been retrieved.

Be tolerant with records without sentence numbers. Users might still want to convert them.

As a quick fix, I used

    if "." in rec.get("Sentence_Number", "noperiod"):
        rec["Sentence_Number"], rec["Phrase_Number"] = rec["Sentence_Number"].split(".")
    return rec

but try/catch or checking for "Sentence_Number" in rec should also work

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.