On the new themes, there is no option to show more categories when there are more than a few. This is quite an important functionality to be broken.

Currently the dataspec file for a given type of data collection source- e.g. for LinkedIn is in: app/dataspec/dataspec-linkedin/dataset_details.json

This file contains values like "Path": "/home/username/Data/processed_version_error" and since this info changes from instance to instance- this should probably be moved to the instance-config.json file!

When users are creating an entity, sometimes it redirects before the entity exists and users get an error. The entity still saves correctly though and can be viewed on refresh, it just isn't there yet when users load the page. Check if the entity exists and wait before redirecting.

Users need a way to input a normalized term list input form. This form should output a hash/dictionary of the following format:

{
  "Canonical Name" : [
      "Name Variant 1",
      "Variant 2",
      "var 3"]
}

This is useful for normalizing multiple name variants or grouping documents with topically-related but different keywords. Users should be able to enter the keys they want to show up as the categories, and then for each key enter an unlimited # of terms documents should be checked for. Each of these lists of terms acts as a normal term list that Catalyst can extract, except instead of being tagged with the terms themselves it is tagged with the overall key associated with the terms.

In datasets where there is a field of display type Long Text but not of display type Description, the Long Text field shows up to the right and there is a large blank space on the left. When there is no Description type field, this should be single column.

This should work to check if there are description fields/get the num of description fields-
dataspec.field_info.select{|f| f["Display Type"]== "Description"}.length
You may need to use @dataspec or a different var depending on what dataspec is where this is called.

This is broken from my current UI updates, should be smallish once I examine the JS that shows and hides the differing form types again

Finish theme for PI instance

There are some buttons that don't do anything on the Catalyst job list page. We need to remove them for now. Then, separately, we can discuss if we should add functionality described on the buttons (such as rerunning Catalyst jobs).

The step where users filter the documents on the advanced search form for Catalyst is a bit confusing. The following should be done to make it clearer-

• Change "Search Name" to "Mining Job Name" or similar
• Move the Mining Job Name field above the field with the dropdown settings menu
• Divide the entities and documents/emails into two categories on dropdown (similarly to how fields are divided into text/category/date types on search form)

Go to a URL like this http://localhost:3000/search?utf8=%E2%9C%93&utf8=%E2%9C%93&q= when an empty query string and you will notice a nasty error page.

We should handle this a bit better and just redirect to the homepage or such!

This is spill over from NSADOCS work, needs to refactored for dataspec

Add search and display for DateTime parameters

Currently on the page where users select which entity they want to add there is a sidebar that lists field names of categories. This should be removed.

Users should be able to add their own fields to documents. This is already supported on the backend, but there needs to be an interface which collects the following details:

• Document type to add field on
• Field name (human readable name)
• Icon
• Field type

Basically a way for a human to "tag" one point of data in a given search result item as something of interest or an "anomaly" such as:

• Job at Defense Contractor
• Job at Pharmacy
• Job at Defense Contractor
• Job in Military
• Job at Defense Contractor

The thinking here being that once a user (of LookingGlass) flags something as an anomaly, they can then view other anomalies with the goal of perhaps finding patterns between anomalies. In the above case there might be "another person who worked at a similarly odd company in between defense contracting gigs during the same few year period"

This data visualization about People Killed by Police in America does some interesting composite overviews of all the data that is then filterable by Year, Weapon, Etc...

Currently the search dropdown menu looks like this:

This could be made more user friendly by grouping the types of search into sections using the <optgroup> tag, so something like:

<select name="ftypes">
  <optgroup label="Date Ranges">
    <option value="doc-date">Document Date</option>
  </optgroup>
  ...

Current this just shows the same word "Search" no matter the page you are on. This should be more dynamic and reflect specific pages and queries

This was lost in refactoring but should be added back! This should allow the user to show / hide certain fields in a search result items. The point being- to make it easier to comb through results visually

We should change our current Catalyst form to be an "advanced" form (much like the advanced search forms that are prevalent) so that we can keep much of the functionality, but default to a simpler form instead. This form should have the following defaults-

Search Settings: Don't allow filtering the documents at all. Just run over documents and emails (not entities). Run over all documents.

Choosing Catalyst Methods: This can be similar to the advanced settings.

Settings: The same defaults as in #75 should be used, but the user should not be able to modify them. That means that in many cases the user will only need to choose what input method they want to use (if there are no settings to choose, such as entity extraction) and nothing else at all on the whole form. In others, where the Catalyst method requires user input (such as term lists), the user will need to be prompted to enter these settings.

The current form should still be available so that the full range of configuration is there, just not as an obvious thing. We can write a tutorial on the configuration options on the advanced form once we aren't changing the UI too much anymore. The simple form should be usable by nontechnical users without instructions, ideally (or with minimal instructions when entering settings on some Catalyst methods).

There should be a way to set all documents to have a certain value for a specified categorical field. Both the frontend and backend are needed for this. The user interface needs to collect the following values-

• Document type (which determines which categorical fields to show)
• Field to set
• Value to set to (or if it should be cleared entirely)

Eventually we may want to support bulk setting of documents matching search values or bulk deletion, but for now let's just provide this to set/unset values for all documents.

Add the ability to rerun Catalyst jobs by clicking a button in LG.

Currently there is just a placeholder. Make it so users can pick which icon to use for which field.

When running rake assets:precompile in production, I get this error-
Sass::SyntaxError: Undefined variable: "$btn-border-radius-base".
(in /var/www/update/LookingGlass/app/assets/stylesheets/bootstrap-custom.scss:20)
/var/www/update/LookingGlass/vendor/bundle/ruby/2.2.0/gems/bootstrap-sass-3.3.5/assets/stylesheets/bootstrap/_buttons.scss:20:in `button-size'
/var/www/update/LookingGlass/vendor/bundle/ruby/2.2.0/gems/bootstrap-sass-3.3.5/assets/stylesheets/bootstrap/_buttons.scss:20
/var/www/update/LookingGlass/app/assets/stylesheets/bootstrap-custom.scss:17
I recall that we had this issue before. How can this be fixed?

Also, the logo in the upper left isn't showing up- what file/icon should be in the instance spec for that?

Thanks

Currently, the top navbar has some ugly bits and needs to be improved!

I made the diffing code work with the data loaded into LookingGlass. However, the ids for the diffing elements (like versions-container) are constant and hardcoded into the Javascript.

This is a problem because, due to the structure of the data where multiple items show per page, sometimes there need to be multiple diffing containers per page. This doesn't work with constant IDs. Is there a way to make them dynamic?

It seems like there are a couple ways this could be approached-

1. The html for diffing is now in app/views/docs/_changetracker.html.erb. Setting dynamic ids based on the doc ID would be quite easy here. But these need to be matched up in the JS somehow. Perhaps they could have a common class and the current JS could be run for each element in the class, but without the ID hardcoded?
2. The html for versions-diffing-type, versions-compute, and versions-diff could be rendered only once on the page and the div with the id versions-container could have a class of versions-container instead. Then all elements with this class could be added to the list to be diffed. These elements may require unique IDs themselves that correlate them to the appropriate doc (rather than just using the field name as the ID like they currently do).

The first solution seems preferable visually to me, but of course there may be other options too. What do you think?

To duplicate this issue, you should-

1. Download the new test data on the server from /mnt/disk/processed_test_data
2. Clone and use the current dataspec-linkedin repo
3. Change the path in the dataset_details in the dataspec-linkedin folder to point to the location of processed_test_data
4. Index the data with rails runner as normal
5. Open the search and look for Aziz Omar

The archive settings are managed by the ArchiveAdmin app, but there should be a link to edit them from LG.

This search visualization should take into account the following:

• Display basic Event items
• Display Document files and distinguish
• Color coding by categories / tags
• Sorting by: year, month, week, day
• Sort by: ascending / descending
• Switching between timeline / normal view
• Quick adding of new events

The site http://infogra.ph creates some pretty interesting little infographic cards that tell a story. It would be interesting if we could tease out information like this into widgets

Users need a way to input an array of terms for the Catalyst term list extraction method.

It appears we are currently pegged to ElasticSearch 1.5.2. I tested LookingGlass on 2.3.x, 1.6.x and 1.7.x with errors and negative results.

The reason the application is incompatible is because it relies heavily on Facets, which were already deprecated in the version that is being used. In ElasticSearch 2.x, they have been replaced with Aggregations. They have some useful docs on how to migrate from Facets to Aggregations.

Since there is lots of code that is affected here, it is quite a task to migrate to the latest ElasticSearch. But hopefully we can get it done eventually.

The following defaults should be set on the Catalyst advanced form:

• Icon: Already set. Keep using default even after picker added
• Field Name: Already set.
• Fields to Search: Should default to text/title/description on ArchiveDoc, Body/Subject/Attachment Text on EmailDoc, name/description or similar on the entities. Perhaps the defaults should be set in the dataspec for each document type.
• Parameters: In some cases (term lists) defaults won't be possible, but it should work with optional defaults, which should be specified in the Catalyst seed data

Reorganize the topbar on editable LG to have a dropdown with the different ways of adding content (currently a link to the upload form and a link to the entity form). Also try to make the wording clearer.

Currently you need to manually copy the dataspec-IndeedCrawl and dataspec-LinkdinCrawl to thhe app/dataspec folder.

This isn't mentioned in the readme.md

When users create an entity, ensure users can add values for categorical fields on the sidebar.

This is a special case of #83 where the document types (all of them), fields (to_publish), and values (Yes/No) are all preset. We just need "Set all documents to be published" and "Set all documents not to be published" buttons somewhere. This allows users to publish all documents by default and selectively filter out certain documents.

Might add more if I find them.

Updating jsontableschema
Fetching: jsontableschema-0.2.2.gem (100%)
  WARNING:   The 'jsontableschema' gem has been deprecated and will be replaced by the gem 'tableschema'.
             See: https://github.com/frictionlessdata/tableschema-rb

Post-install message from twitter-bootstrap-rails:
Important: You may need to add a javascript runtime to your Gemfile in order for bootstrap's LESS files to compile to CSS. 

Several of the active{*} or action{8} gems fail on less than 2.2.2, that's okay though since our production Ruby is now 2.4.1. I'm finding ~/.gemrc and ~/.bundle/config convenient to managing these environments.

Diffing now works if there is just a single item fields doc and the non item fields part that need to be diffed. But if multiple docs on a page need their item fields diffed, the box doesn't show up.

I've sent you an email with details on how to duplicate this on our test data instance. Could you please look into this?

Thanks

PDFs are suddenly not loading in the frame on the show view. This works in development, but not in production. The error in the web console is-
Load denied by X-Frame-Options: https://transparencytoolkit.org/sii_docs/brochure598.pdf does not permit framing.

If I remember correctly, we had this same problem back in June and managed to fix it somehow? @bnvk -do you remember how we fixed it?

From other datasets that get added such as LinkedIn and such

We need to add icons for the PI datspecs- both the company and materials dataspecs. These dataspecs can be found here- https://github.com/transparencytoolkit/dataspec-sii. Once these are added to LG, please also update the dataspec icon fields in the dataspec-sii repo with the appropriate images.

So everything is fine when viewing the ICWATCH site directly, but when visiting via Tor there's a nonspecific error that may be related to the CloudFlare settings. Here's a screenshot:

The current design of the interface (see below) as been decided to be "good enough" for now and thus implementing it in CSS is the next step.

I'm just opening an issue to track things I'm working on as it's my workflow for all my other projects, and it's more effort to not use Issues 😝

Make the document editing experience smoother by ensuring that users don't need to click the edit button many times to get to the edit interface.

Currently some of the Catalyst methods accept numerical input and just have dropdowns of numbers "One" to "Ten" as input. But this often does not make sense. Sometimes the input should be in the hundreds, or other times it may be between 0 and 1. The range that makes sense should be added to the Catalyst seed data and a slider should be used to select #s in that range (with the option for the user to type in the # directly).

Ensure delete document button works to remove document from Elasticsearch.

The following deprecation warnings are filling up the log and should be addressed at some point:

DEPRECATION WARNING: Calling URL helpers with string keys controller, action is deprecated. Use symbols instead. (called from prepNewPath at /var/www/sii/LookingGlass/app/helpers/category_link.rb:27)
DEPRECATION WARNING: Calling URL helpers with string keys action, controller is deprecated. Use symbols instead. (called from selected at /var/www/sii/LookingGlass/app/helpers/category_link.rb:38)
DEPRECATION WARNING: Calling URL helpers with string keys action, controller is deprecated. Use symbols instead. (called from getRemoveLink at /var/www/sii/LookingGlass/app/helpers/searched_format.rb:112)