make datapoints on graph clickable

I think we should have a readme for the core, where we explain "how to start using it", e.g. setting up DB, as well as its internal structure like evaluators, connections to DB, Learner etc.

Do you think Flywaydb might be of any use for us?
It also has a sbt plugin.

change "team name" to "configuration name" on add configuration page

Proposal:
Add bootstrap (https://www.playframework.com/documentation/2.4.x/AssetsLess) to the code and design a basic interface for the application.

Show page hierarchy as in:
http://getbootstrap.com/components/#breadcrumbs

Raw Text -> Span Splitting, Span Labeling
Sentence Boundaries -> Span Splitting, Span Labeling
Gold Token -> Span labeling

Instead of/in addition to this add calls to a real POS tagger. Here is one.

Would it make sense if we replace double score with EvaluationRecord?

Is there a way to sort the configurations shown in the landing page? Say by the last time they are used/updated?

proposal:
Add a class (in app/controllers with package name, say edu.illinois.cs.cogcomp.core) for the core.
The core is supposed to contain the interface of the solver and the evaluation system. The internal datastructure of the system is supposed to be a (TextAnnotation)[https://github.com/IllinoisCogComp/illinois-cogcomp-nlp/blob/master/core-utilities/src/main/java/edu/illinois/cs/cogcomp/core/datastructures/textannotation/TextAnnotation.java#L21]. So we need add core-utilities as dependency (example here)[https://github.com/IllinoisCogComp/saul/blob/master/build.sbt#L14].

The evaluation framework is supposed to send a TextAnnotation object to the solver, and will receive a modified TextAnnotation.

Retrieving and cleansing the TextAnnotations is very computationally expensive, so I propose we define a variable CHUNK_SIZE in a config file defining the number of TA's to send to the solver at a time.

We should be able to send the question in parallel to a solver.

overlapping f1, precision, and recall on the graph

Use progress bar while we are evaluating:
http://getbootstrap.com/components/#progress-basic

add a “Working…” page to display while the configuration is running

“Are you sure??” question to delete configuration. Or “All data will be lost”

get rid of the evaluators and use the ones in coreutils:
https://github.com/cogcomp-dev/illinois-cogcomp-nlp/tree/master/core-utilities/src/main/java/edu/illinois/cs/cogcomp/core/experiments/evaluators

Use btn-groups-toolbar to show extra configurations and runs:
http://getbootstrap.com/components/#btn-groups-toolbar

Add lemmatizer data and evaluate it

As of #54 we are using configuration_names to identify the configurations and assuming that they are unique. If so, when creating new configs, shall we check if the entered config id already exists in the db or not? (it will remove the previous one, if it already exist; right?)

fyi @dshine2 @paultgibbons

Priority:
Google Chrome
Firefox
Safari

Unfortunately:
Internet Explorer
Edge

@mssammon @christos-c @cogcomp-dev and all open-eval team!

We have discussed this a couple of times (beyond the open-eval) that how to access datasets programmatically, meanwhile not adding them directly to our code repositories. This is something that can be very useful for any project, but essential for open-eval, since we want to load the data programmatically inside the evaluation system.

I want to bring AI2's datastore to your attention. This package saves all data files/folders on Amazon S3 (which is very cheap). I did a little experimentation with this package (A deploy is here).

Using it is very easy; for example here I am uploading the dataset we use for training/testing POS tagging from my computer:

    Datastore.publishDirectory(
      "/home/daniel/ideaProjects/saul/data/POS",
      "edu.illinois.cs.cogcomp",
      "POS-tagging-data",
      1,
      false)

Later for accessing the data (on any machine, without having the data locally) I can have do:

    val dataPath: java.nio.file.Path =
      Datastore.directoryPath(
        "edu.illinois.cs.cogcomp",
        "POS-tagging-data",
        1)

Running the above will download the data from S3 to a folder (in home dir) and give its path. (running it next time it just reads it from cache). Building on top of the above, I can evaluate Saul POS tagger in few simple steps:

def testPOSTagger() = {
    val dataPath: java.nio.file.Path =
      Datastore.directoryPath(
        "edu.illinois.cs.cogcomp",
        "POS-tagging-data",
        1)

    /** Read your data from datastore. */
    lazy val testData = {
      val testDataReader = new PennTreebankPOSReader("testData")
      testDataReader.readFile(dataPath.toString + "/22-24.br")

      var sentenceId = 0
      testDataReader.getTextAnnotations.flatMap(p => {
        val cons = commonSensors.textAnnotationToTokens(p)
        sentenceId += 1
        //      Adding a dummy attribute so that hashCode is different for each constituent
        cons.foreach(c => c.addAttribute("SentenceId", sentenceId.toString))
        cons
      }).toList
    }

    /** Populate your data in the model */
    POSDataModel.tokens.populate(testData, train = false)

    /** Load the models for the POS classifier */
    POSClassifiers.loadModelsFromPackage()

    testPOSTagger(testData)
  }


  def testPOSTagger(testData: List[Constituent]): Unit = {
    val tester = new TestDiscrete
    val testReader = new LBJIteratorParserScala[Constituent](testData)
    testReader.reset()

    testReader.data.foreach(cons => {
      val gold = POSDataModel.POSLabel(cons)
      val predicted = POSClassifiers.POSClassifier(cons)
      tester.reportPrediction(predicted, gold)
    })

    tester.printPerformance(System.out)
  }

which gives me the follow results:

...
[info] VBG         91.483  92.240  91.860   1933   1949
[info] VBN         86.041  90.395  88.164   2707   2844
[info] VBP         93.894  91.374  92.617   1565   1523
[info] VBZ         96.941  96.059  96.498   2639   2615
[info] WDT         97.967  90.753  94.222    584    541
[info] WP          98.596  99.293  98.944    283    285
[info] WP$        100.000 100.000 100.000     37     37
[info] WRB         99.671  99.671  99.671    304    304
[info] ``         100.000 100.000 100.000   1074   1074
[info] ------------------------------------------------
[info] Accuracy    96.439    -       -      -    129654

So what do you think about adopting this project for our work?

Side note: here is repeating the same steps in Java:

java.nio.file.Path dataPath = Datastore$.MODULE$.directoryPath( "edu.illinois.cs.cogcomp", "POS-tagging-data", 1);

        PennTreebankPOSReader testDataReader = new PennTreebankPOSReader("testData");
        testDataReader.readFile(dataPath.toString() + "/22-24.br");

        java.util.List<Constituent> testData = testDataReader.getTextAnnotations()
                .get(0).getView(ViewNames.TOKENS).getConstituents();

        scala.collection.Iterable<Constituent> testDataInScalaCollection = scala.collection.JavaConversions.asScalaBuffer(testData);

        /* Populate your data in the model */
        POSDataModel.tokens().populate(testDataInScalaCollection, false);

        /* Load the models for the POS classifier */
        POSClassifiers.loadModelsFromPackage();

        /* Make prediction on the input instances */
        for(Constituent constituent : testData ) {
            String predicted = POSClassifiers.POSClassifier(constituent);
            System.out.println(constituent + "  ->  " +  predicted);
        }

When we deploy the project as jar files, the name will be learner_xx.jar which is not very intuitive.
Let's rename the module name to something else; say openeval-learner?

Following that, to keep the consistency, let's rename others as well, say to openeval-core.

Also let's make sure only the learner get's published not the other modules.

Show score type on configuration page, for per-record.

standardize naming (run vs. record?)

Change “Download our thin client” to Instructions on how to use the system" or sth like that.

Task -------------------------- Task Variant

Part of Speech Tagging -> Raw Text, Gold Token, Sentence Boundaries
Named Entity Recognition -> Raw Text, Gold Token, Sentence Boundaries
Parsing -> Raw Text, Gold Token, Sentence Boundaries
Co-reference -> Raw Text, Gold Token, Sentence Boundaries

We need to have an parameter file which contains important variables set from outside code; for example:

• address, port, usename, password for database

Since we are setting our learner to work based on POS tagging, let's import the POS tagging dataset into our DB, so that we can run our system against this data.
Since the data is propriety, I am not sharing it here. Instead I am share the data personally.

Here is how to read the data via a reader inside CoreUtils:

PennTreebankPOSReader testDataReader = new PennTreebankPOSReader("testData");
testDataReader.readFile(dataPath.toString() + "/22-24.br");

In the landing page, only show configuration name, and some small amount of details, and then on clicking them the details are shown.

Is it important to have tests for each of our routes?
Like this:
https://github.com/jamesward/play2torial/blob/master/JAVA.md#test-a-route