This package is a part of a project at the University of Potsdam. The goal was to parse Twitter discourse in a PDTB style. This part of the project assumes as an input a constituency tree including POS-tags and tokens, together with the sentence-nested positions of the connectives themselves. From this input the script pipeline.py labels the argument spans that belong to the connective.

The ideas for the discourse parser are along the lines of Lin. et. al. https://www.comp.nus.edu.sg/~nght/pubs/nle13.pdf

For more information about the implementatin read the pdf in the article folder

At the current state it is possible to parse from end to end, but the results are still rather poor. It is expected, that they can still be improved with a better feature set.

To get all the data and packages you need run

git clone https://github.com/arksch/zwitscher.git
cd zwitscher
pip install -r requirements.txt

Test your setup by running the main script with default parameters

cd zwitscher
python pipeline.py

The script has the following options

$ python pipeline.py --help
Usage: pipeline.py [OPTIONS]

Running a pipeline to label argument spans in a syntax annotated discourse
with known positions of connectives

Options:
-s, --syntax_trees_xml_path TEXT
                                Path to a file with syntactic information in
                                TigerXML format
-c, --connective_positions_path TEXT
                                Path to a file with nested connective
                                positions in json format
                                E.g. '[[[14, 0],
                                [14, 5]], [[11, 0]], [[2, 0], [2, 1]]]'
-sc, --sent_dist_classification_path TEXT
                                Path to the sentence distance classifier
                                trained with the learn_sentdist.py script
-as, --argspan_classification_path TEXT
                                Path to the argspan classifier trained with
                                the learn_argspan.py script
--help                          Show this message and exit.

A discourse is typically a paragraph, but it can be of arbitrary size. If you want to parse other discourses, you will need TigerXML Syntax trees for each of them. Store each in a file. An example can be found in

zwitscher/data/test_pcc/syntax/maz-00001.xml

Since this parser assumes, that discourse connective tokens have already been identified, you will need to store this information as well for each discourse. Take care, that the tokenization matches the syntax trees. The positions of the discourse connectives are saved in a JSON file with sentencewise-nested positions. I.e. a list of connectives where each connective consists of token positions and each position is a sentence index with a token index in the given sentence. An example matching the syntax example above can be found in

zwitscher/data/test_pcc/maz-00001_nested_conn_indices.json

[ [[14, 0], [14, 5]], [[11, 0]], [[2, 0], [2, 1]], [[2, 22]], [[4, 5]], [[3, 2]], [[8, 10]], [[9, 0]], [[8, 14]] ]

To create these indices from PCC files you can run the script

python connective_positions_from_pcc.py -d data/test_pcc/connectors/maz-00001.xml -s data/test_pcc/syntax/maz-00001.xml
[[[14, 0], [14, 5]], [[11, 0]], [[2, 0], [2, 1]], [[2, 22]], [[4, 5]], [[3, 2]], [[8, 10]], [[9, 0]], [[8, 14]]]

python connective_positions_from_pcc.py --help
Usage: connective_positions_from_pcc.py [OPTIONS]

A script to print nested or unnested indices of connectives in the
sentences of a discourse from PCC files

Options:
-d, --discourse_path TEXT  Path to the discourse xml file. Defaults to
                          data/test_pcc/connectors/maz-00001.xml
-s, --syntax_path TEXT     Path to the syntax xml file. Defaults to
                          data/test_pcc/syntax/maz-00001.xml
-n, --nested               Output the nested or unnested positions
--help                     Show this message and exit.

A fellow project is developing a tool to extract discourse tokens from Twitter text. https://github.com/KaiserKlayton/DCIT

This project uses the DiMLex, a lexicon of German discourse connectives, created by:

Manfred Stede. "DiMLex: A Lexical Approach to Discourse Markers" In: A. Lenci, V. Di Tomaso (eds.): Exploring the Lexicon - Theory and Computation. Alessandria (Italy): Edizioni dell'Orso, 2002.

You can download a current public version of the DiMLex at https://github.com/discourse-lab/dimlex to replace the one copied to this repository.

You might want to train new classifiers. There are two scripts that also evaluate their results of the algorithm. You can start the scripts from the zwitscher/zwitscher folder with

$ python learn_sentdist.py
...similar output as below...
$ python learn_argspan.py

Learning classifier for finding the main nodes of connectors.
=============================================================
Loading data...
Unpickling gold data from data/PCC_disc.pickle
Loaded data
Cleaning data...
...
Training classifiers for arg0 labeling
======================================
Majority baseline: 0.881298
Cross validating Logistic regression classifier...
Cross validated Logistic Regression classifier
scores: [ 0.91257996  0.89434365  0.92102455  0.90918803  0.90277778]
mean score: 0.907983

Training classifiers for arg1 labeling
======================================
Majority baseline: 0.885781
Cross validating Logistic regression classifier...
Cross validated Logistic Regression classifier
scores: [ 0.88473853  0.88580576  0.88580576  0.88580576  0.88568376]
mean score: 0.885568
Learning classifiers on the whole data set...
Learned classifier on the whole data set
...done
Pickling sent_dist classifier to ed0f4620cd1c4402bfa582182be83b44_argspan_classification_dict.pickle

You can see the options of the scripts below. When you run them, they will create a new classification set for either sentence distance or argument span, that can be read by the pipeline.py script.

The evaluation of the scripts is printed to standard output. For this a classifiers is first trained on a part of the data and evaluated on the rest. The classifier that is saved is then trained on the complete data set.

If you want to use your own evaluation methods you will have to embed them inside the script. Currently no end-to-end evaluation is implemented. If you want to evaluate on a held out data set you will also have to implement that method.

By default both learning scripts load the pickled PCC. If you want to load new PCC data or you have changed some methods of the underlying objects, then you will have to change the unpickle_gold option to False.

The scripts have the following options:

$ python learn_argspan.py --help
Usage: learn_argspan.py [OPTIONS]

Learning and storing classifiers for argument spans of discourse
connectivesThe output file can be passed used by the pipeline with python
pipeline.py -as 123uuid_argspan_classification_dict.pickle

Options:
-f, --feature_list TEXT       A comma separated list of features. By default
                             connective_lexical,nr_of_siblings,path_to_node
                             ,node_cat
-lf, --label_features TEXT    A comma separated list of features that are
                             labels. By defaultconnective_lexical,path_to_n
                             ode,node_cat
-cf, --connector_folder TEXT  The folder to find the connectors from the
                             potsdam commentary corpusCan be left empty if
                             unpickle_gold is True
-pf, --pickle_folder TEXT     The folder for all the pickles
-uf, --unpickle_features      Unpickle precalculated features. Useful for
                             dev
-ug, --unpickle_gold          Unpickle gold connector data. Useful for dev
-pc, --pickle_classifier      Pickle the classifier for later use, e.g. in
                             the pipeline.Note that this adds an uuid to
                             the output, so it doesnt overwrite former
                             outputOtherwise this script will just print
                             the evaluation
--help                        Show this message and exit.

The sentence distance classification reaches an accuracy of over 90%, which is a reasonable result over the baseline of 57% majority labeling with both arguments in the same sentence. Note, that this does not yet use syntactic features.

On the contrary, the argument span labeling only reaches an f-score of 23% for the internal argument span and 25% for the external argument span. Both values were calculated by counting words that were labeled as in the ground truth as a true positive, and other labeled words as false positive/negative respectively.

Obviously, this package is still far from perfect. Feel free to contribute!

You can run tests with

py.test

• Better features (features.py)
  • Syntactic features for sentence distance
  • Use a dictionary for the words used in argument span labeling features
  • Use capitalization
  • Not only use the first connective word for argument span features
  • Faster feature calculation
• Better evaluation
  • Nested cross-validation, as the current evaluation is biased by selection of the best classifier
  • Evaluate the full pipeline, not only the sentence distance and argument span labeling separately
  • Wrap the argument span evaluation into a sklearn metric, so that it can be used directly in the classification training
  • More meaningful metrics, that allow to evaluate what actually goes wrong and are comparable to Lin et al (partial match of nouns and verbs, perfect match)
  • Do not count punctuation
  • Calculate some baselines and upper bounds
    • Full sentence labeling for different sentences
    • Labeling with the gold standard argument nodes
  • Evaluate the labeling for argument spans in different sentences
• Try other ways to create argument spans from the argument nodes (Lin et al use tree subtraction, but that might not be a good choice for PCC)
• Try other classification algorithms
  • Don't train the classifier on other cases than previous and same sentence
• Create command line arguments for the classifier training scripts
• Improve the parsing or fix inconsistencies in the PCC
• Fix one of the many ToDos in the code and refactor it to be readable