Conform Mockup

So they can potentially run on different servers. Depends on #3 and #7.

Edit: this includes multiple places where HTML is generated at the backend; at least the admin templates (invoked from ianalyzer.view) and the filter implementations in ianalyzer.filters.

We don't want to build up a giant collection of unused corpus definitions within this repository. Instead, the application should only define the abstraction, and a concrete corpus definition should be attached to an application by means of local configuration only (similar to Flask and Django configuration modules). This means that the existing UUDigitalHumanitieslab/timestextminer corpus will be reduced to the existing corpus definition for the Times, which in turn will disappear from this repository. The yet to be written corpus definition for the Dutch banking corpus (#1) will move to a new repository.

Edit: depends on #13.

Like #8 but without the dependencies.

Use something like https://github.com/troyanskiy/ngx-resource to communicate with the server. To reduce the amount of code repetition.

This is requirement for real REST. http://stateless.co/hal_specification.html

Plugins available for most web frameworks (including Flask and Backbone).

Currently as part of #3 a Flask and NG server have been created. It would be more ideal if these could be served from one single address.

Default Field
edit

When not explicitly specifying the field to search on in the query string syntax, the index.query.default_field will be used to derive which field to search on. It defaults to _all field.

If the _all field is disabled and no fields are specified in the request`, the simple_query_string query will automatically attempt to determine the existing fields in the index’s mapping that are queryable, and perform the search on those fields.

Conform Mockup

From-to in range pickers are now depicted as from:to, which is not intuitive to most people.
from-to might be better.
Also, a slider with low-high is not suited for all clients.
Some would prefer two fillable boxes denoting From and To.

Because those languages (almost) do not use spaces they should be handled differently than languages which do. For instance the highlighter (#40) splits using word boundaries. I'm sure there're also plenty of other interesting things which will explode on impact.

José reported that the slider is bit difficult to use. It would probably be most efficient to do this after #3.

There is a TODO screaming in ianalyzer.corpora.common.Corpus.json. The intention is that the Python-based corpus definition will remain the single source of truth about corpus structure, but other parts of the application (frontend, special services…) will be benefit from this same definition through the JSON interface.

Currently, the application does not have any migrations in place.

Using an arbitrary large number to set the upper bound of the range will put the useful values in only a small piece at the left side of the slider, which is not user-friendly.

Potential solutions:

• Create a log-based slider such that smaller values occupy more of the slider
• Preprocess the data to find a reasonable upper bound within some margin of the maximum value

There are parts of the code of the python-flask backend which are not used anymore (elasticsearch (other than indexing), interface). A cleanup to avoid unnecessary imports would be desirable.

This is recommended in their style guide. So instead of <search></search> we could use <ia-search></ia-search> this might especially be useful when these names might collide with actual HTML names or if some future project would import stuff from I-analyzer. This prefix can be set in .angular-cli.json.

Either using Alto (but this only makes sense if we also show the scanned image, which we don't do yet) or by applying a quick client-side search on the plain text (as suggested by @oktaal).

Currently different tests need to create these from scratch. Some of this could be shared or turned into a method. So instead of:

TestBed.configureTestingModule({
    imports: [RouterTestingModule.withRoutes([])],
    providers: [
        SearchService,
        { provide: ApiService, useValue: new ApiServiceMock() },
        { provide: ElasticSearchService, useValue: new ElasticSearchServiceMock() },
        LogService,
        QueryService,
        UserService,
        SessionService
    ]
});

It would be better if something like the following could be done:

TestBed.configureTestingModule(CommonTestingModule.createDef(/* settings */))

Possibly using Casper.

Conform Mockup

Term frequency per year, respecting the filters in the query.

Conform Mockup. Provides context sensitive information (e.g., corpus selection, search). Ideally, this would also provide corpus specific info, e.g. what filters can be used, some example queries, etc.
On the left, show menu to switch between different help topics.

The user can get to their previous searches by clicking on their name on the top of pages. Mockup of this is still to be designed (out of space for the free Moqups version).
Estimated time: 9.5 hours

It should be possible to secure access to the index (a username/password for the ElasticSearch DB is simply sent to the client as part of #9), and allow limiting access to the search interface.

The user creation form in the admin panel permits you to leave the password field blank. This is an undocumented feature; it is not clear what will happen if you do so. Specifically, will the user be able to login or not? This needs to be addressed.

Building on the framework of #3, add a search interface using the default corpus as defined in the configuration.

See 984cf2d.

Currently there is one unittest, which means that in theory, we have a unittest suite. Need to add more tests so we have something to fall back on when things break. Will be subject to available time.

Currently it depends on Flask-Admin. Need to eliminate this dependency. It would be fine to keep using Bootstrap, in fact it may be possible to embrace it more and (perhaps) use Sass as well.

The current installation at dutchbanking.hum.uu.nl requires two logins: one with a solis-ID over LDAP at the webserver level and one with a custom account at the application level. This is annoying for users. We need to change this to a single login with LDAP at the application level with a fallback option for users without a solis-ID, following the example in Microcontact.

Currently, if you add a new user, it is not possible to edit that user properly later (i.e., change their download limit): the user field will be empty when editing. Potentially, prefill fields such as download limit when creating a user to the default value. See also #37 , according to which the password field can even be left empty.

Might be addressed automatically as a side effect of #3, but making it explicit just in case.

When doing a search, the user gets a preview of five hits. This preview contains the right fields, but the fields do not always contain data, even though those fields are populated and present in the downloaded CSV file. In the case of the current DutchBanking index, all fields except for the document id are previewed as blank, even though all fields always contain data in the downloaded hits.

While downloading full search results still works, this does not make a good impression to the user.

Currently absolutely minimal support is in place in the search highlighter. I couldn't get the test to work with wildcards or quotation marks, because the mixing of those symbols cause confusion. At the very least for the developer. This could definitely be improved, maybe by using native symbols from those writing systems or adding some mode support for switching direction.

Show the search results of #32.

I wrote a user manual on the simple query string query language that is used for searching. However, I am not sure how certain top level parameters that influence usage/search results are set. E.g.:
analyze_wildcard, lenient, minimum_should_match, default field etc.
A list can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/query-dsl-simple-query-string-query.html
It would be nice if this could be documented so that I can use it for enhancement of the user manual.

We should make decisions about the name of the app soon so we can have a logo designed.

Currently, only five results are returned by the backend (default value). Should be resolved as a consequence of #9.

See Mockup

It could be done by assigning one on logon. Set and retain this in the ApiService and always add this property to the JSON data before sending.

We need to figure out what parts of the ES query string syntax are supported in the current setup, and how to change it.

This may be a non-issue, but there is no init method in the Corpus class. Should I add one, or is this left out for a reason?

When trying to serialize the corpus representation more than once an error is thrown. This can be reproduced using a shell from the (back-end) application root:

> from ianalyzer.corpora.dutchbanking import DutchBanking
> d = DutchBanking()
> print(d.json())

This goes well, now try to serialize again:

> d.json()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Users\Sheean\src\I-analyzer\ianalyzer\corpora\common.py", line 136, in json
    search_dict = field_dict['search_filter'].__dict__
AttributeError: 'dict' object has no attribute '__dict__'

So the config can be placed outside of the source tree and the corpus definitions can be agnostic about its location.

Conform Landing page mockup and selection mockup

As requested.

In particular, the query field of the search form will query all fields by default. It would be desirable to be able to restrict this, for example to the content field. This issue is somewhat related to #19.