martinjewski / iaacalculator Goto Github PK

The IAACalculator calculates the iaa score for two or more annotator's using the Percentage-, RandolphKappa- or Bennett's agreement.

The basic tsv file that contains the data must have the following structure or it won't work:

[sentenceID verb sentence Annotator1 ...... AnnotatorN]

Furthermore it takes into account, that annotator's didn't annotate all sentences equally, which results in a "*" at the specific cell that should contain a sense number like "784421"(see below).

[id-s10-0 sagen Gesagt, alle Mann und Frau getan. 12345 * * * * * * 54321 * ]

The DKPro package is used to calculate the iaa scores, the "*" changes to null values, via parsing, to match the needs addItem function inside the CodingAnnotationStudy class.

[id-s10-0 sagen Gesagt, alle Mann und Frau getan. 12345 null null null null null null 54321 null ]

It should be mentioned that the Bennett's agreement works for only 2 Annotators.

the programm was written in a few days and thats why some outputs have pairwise agreement calculations for the first description row due to a sloppy coding, sorry for that!:)<<<

(tested on windows, with JavaSE-1.8) Input path should contain the name of the tsv file with .tsv at the end. in: ...\path\iaaFile.tsv out: ...\path\folder

for the method just write the mentioned method name.