Validate incoming data with JSON Schema.

Models Already in the OpenABM

• Artificial Anasazi
• Pumpa Irrigation Model
• Complex Model of Voter Turnout

Models not in OpenABM

• NetLogo model library
• https://github.com/Repast/repast.simphony.models

would be nice to be able to link Authors to Users / Profiles.

Add a nullable User OneToOne in Author but then we duplicate first_name and last_name in User - I think this is ok because the citation author string representation isn't always the same as User.first_name + User.last_name

• add additional ideas here @cpritcha

Need to keep track of if a user is an affiliate or a full member. Maybe a better idea for this to be an enum (Full Member, Affiliate, Guest) or to make special permission groups.

set up prod / dev deployment

• http://jdlm.info/articles/2016/03/06/lessons-building-node-app-docker.html

Models sometimes have references on the OpenABM website. These references are strings right now.

Eventually we may want to integrate these references with the catalog application. The import script currently leaves references as strings.

add form component to assign permissions on a codebase to other users

Single Page App

Pros

• No reloading of header and footer content between pages
• JSON rest API is better tested because we our application uses all of it
• JavaScript dependencies and code are neatly bundled. Frontend logic is all in JavaScript.

Cons

• Duplication of code for routing, data structures and validation
• Manual management of history
• More work to get Google Scholar and other services to index the site
• More work to integrate with Wagtail menu system
• Little to nothing works without JavaScript

Server Side Rendering + JavaScript

Pros

• Use built in Django functionality
• Simple pagination and display

Cons

• Harder to deal with streaming asynchronous updates (status notifications for model runs or see updates to the main page without a refresh)
• More difficult to determine to what is done by JavaScript and what is done by Django
• JavaScript dependency management is more manual

VueJS

https://vuejs.org/

CycleJS

https://cycle.js.org/

React + redux

http://redux.js.org/docs/basics/UsageWithReact.html

Possible components library: https://ant.design/docs/react/introduce

Should there be a checkbox to opt-in for reviewing models?

Could this Roles in the current OpenABM site?

Consistently named StreamField blocks will reduce deserialization code on the front end

General

• paragraph (TextBlock)
• heading (CharBlock)
• subheading (CharBlock)
• image (ImageChooserBlock)
• document (DocumentChooserBlock)
• page (PageChooserBlock)
• embed (EmbedBlock)

Components

• cardlist (ListBlock[CardBlock])
• card (CardBlock)
• carousel

Codebase landing page:

• codebase.root = /codebase/<codebase.identifier>
• base software package landing page includes all available versions + metadata for latest version

Codebase release landing page: `/<codebase.root>/releases/<release.identifier>

• release.identifier could be an ID or semver number, discuss
• metadata specific to that release: also show all available versions + metadata
• Impact, Citation, Alerts (if using an older version)
• expose metadata conforming to https://github.com/codemeta/codemeta schema or other standardized metadata schema

Relevant software citation working group discussion: force11/force11-scwg#157

• Django Filter
• http://www.django-rest-framework.org/api-guide/filtering/#djangofilterbackend

• mail transfer agent / smtp
• logging + logrotation (e.g., https://www.loggly.com/blog/how-to-implement-logging-in-docker-with-a-sidecar-approach/ but with rsyslog or some other)
• automated backups

https://docs.djangoproject.com/en/1.10/ref/contrib/gis/tutorial/

Validate and then display errors in form using validation styling

GitLab

• makes need for SSO more urgent
• large application + footprint
• tight coupling as we write glue code to their API
• provides a large swath of potentially useful dev tooling

Direct git interaction via python libraries

Build primitive abstractions for git repo management using a direct python lib:

• https://github.com/FriendCode/gittle
• https://github.com/gitpython-developers/GitPython
• https://github.com/libgit2/pygit2

no contributors:

https://test.comses.net/codebases/2227/

no parent model nids

2017-01-31 22:39:22 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid  for version 2333
2017-01-31 22:39:22 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid  for version 2339
2017-01-31 22:39:22 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid  for version 2352
2017-01-31 22:39:24 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid  for version 2405
2017-01-31 22:39:37 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid 4675 for version 4684
2017-01-31 22:39:37 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid 4675 for version 4685
2017-01-31 22:39:38 WARNING drupal_migrator.database_migration:_extract:378 Unable to locate parent model nid 4675 for version 4746

files exist on the filesystem, but cannot be found in the DB

• 3075 (Coherence based opinion dynamics), has files
• 3622 (no files)
• 3053 (multiple versions, no files)

2 phase registration, consider https://django-registration.readthedocs.io/en/2.2/

Could punt to Google Calendar

https://developers.google.com/api-client-library/python/apis/calendar/v3

https://developers.google.com/google-apps/calendar/

The Django backend should respond to any non API request without the the app bundle. Vue should then load the page corresponding to what page it is on.

consider building minimalist versioning email bot e.g. archive+<codebase.id>@comses.net

• parse documents and version as plaintext
• body: commit message with subject: header or allow override metadata (embedded in the body or as attached file)
• zipfile attachment interpreted as structured codebase (or codebase + provenance metadata file)
• error messages bubble back out as emails with links to web UI where they can be resolved faster

Options

• Use Wagtail
• Use a MarkDown editor. Could use SimpleMDE

Does not include styling

Jobs

• List
• Detail
• Create
• Update

Events

• List
• Detail
• Create
• Update

Codebase

• List
• Detail
• Create
• Update

CodebaseRelease

• List
• Detail
• Create
• Update

CodebaseReview

• List
• Detail
• Create
• Update

Discourse Integration

Show related forum entries on

• Codebase
• CodebaseRelease
• User Profile

Home Page

• Job summary
• Event summary
• Codebase summary
• Featured models
• Discourse forum summary

User Profile

• Mapping between Discourse users and CoMSES users
• List
• Detail
• Create
• Update

Adjust code in file_migration.py.

Store nid in secondary id hash field and all model + model version metadata in the rdbms. Still need idmapper or equivalent to link a model versions to its Authors, Users, Keywords

Need to figure out what permanent URL structure we want for referencing models.

At first pass, include model version file paths in an extra json field Django rdbms, we'll remove later. Then construct git repositories following conventions-based filesystem layout (cribbed from https://swcarpentry.github.io/good-enough-practices-in-scientific-computing/).

.
|-- CITATION
|-- README
|-- LICENSE
|-- requirements.txt
|-- data
|   -- birds_count_table.csv
|-- doc
|   -- notebook.md
|   -- manuscript.md
|   -- changelog.txt
|-- results
|   -- summarized_results.csv
|-- src
|   -- sightings_analysis.py
|   -- runall.py

https://cinhtau.net/wp/use-travis-ci-in-github-to-build-and-deploy-to-dockerhub/

https://docs.travis-ci.com/user/encryption-keys/

use https://github.com/FontCustom/fontcustom/ to transform SVGs into font icons

Options

• Use django-markupfield and bleach
• Subclass TextField and use bleach

Need to sync users between Discourse + Wagtail first.

Forum.json + Comments.json

Comments.json also has comments attached to other nodes (including replies to Forum posts in Forum.json), which we may want to import to discourse as well.

This should make it easier to edit descriptions and summaries on small screens

compare

• hashfs
• baggit

@cpritcha please feel free to update (with sources) if you still think modern filesystems can support many files without significant performance degradation.

At this point I'm considering a hybrid HashFS style implementation, using hashfs to store the submitted package, using the submitted package path as the parent directory and store Releases as bags under it. So something like
/repository/6a/e8/a7/5555209fd6c44157c0aed8016e763ff435a19cf186f76863140143ff72.tar.gz
would be the path of the original submitted archive and
/repository/6a/e8/a7/ would be the root directory for all CodebaseRelease archives.

HashFS lets us choose how to slice up the dir space, this one happens to be 4/2.

• http://serverfault.com/questions/343351/max-files-per-directory-in-ext4
• http://highscalability.com/blog/2012/6/20/ask-highscalability-how-do-i-organize-millions-of-images.html
• http://serverfault.com/questions/95444/storing-a-million-images-in-the-filesystem

npm run build is currently failing - we need to fix this for prod deployment.

https://vuejs.org/v2/guide/deployment.html

@cpritcha did Hypothesis find a bug?

ERROR: test_add_change_view (home.tests.test_views.EventViewSetTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/home/tests/test_views.py", line 112, in test_add_change_view
    @given(generate_event_data(), st.sampled_from(('change', 'add', 'view')))
  File "/usr/local/lib/python3.5/dist-packages/hypothesis/core.py", line 524, in wrapped_test
    print_example=True, is_final=True
  File "/usr/local/lib/python3.5/dist-packages/hypothesis/executors.py", line 78, in <lambda>
    lambda: function(data)
  File "/usr/local/lib/python3.5/dist-packages/hypothesis/executors.py", line 33, in execute
    return function()
  File "/usr/local/lib/python3.5/dist-packages/hypothesis/executors.py", line 78, in <lambda>
    lambda: function(data)
  File "/usr/local/lib/python3.5/dist-packages/hypothesis/core.py", line 111, in run
    return test(*args, **kwargs)
  File "/code/home/tests/test_views.py", line 117, in test_add_change_view
    self.check_authorization(action, owner, event)
  File "/code/wagtail_comses_net/test_helpers/view.py", line 133, in check_authorization
    self._check_authorization(user, obj, action, False)
  File "/code/wagtail_comses_net/test_helpers/view.py", line 95, in _check_authorization
    data = self._check_serialization_round_trip(obj)
  File "/code/wagtail_comses_net/test_helpers/view.py", line 89, in _check_serialization_round_trip
    if serializer.is_valid(raise_exception=True):
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/serializers.py", line 244, in is_valid
    raise ValidationError(self.errors)
rest_framework.exceptions.ValidationError: {'location': ['This field may not be blank.']}

Determine how the education and resources pages should be arranged in the new site. Wagtail would make it easy to generate breadcrumbs and manage content if we stick to a tree structure.

• implement simple version first, candidate pool only comses net full members (or reviewer)
• provide checklist, emphasize how fast it will be
• email-driven to start with
• need status flags to track state

Remaining tasks:

• invite candidate reviewers via email
• form for reviewer feedback (unique UUID URL or based on authenticated access under their user profile page)
• send emails to editor for reviewer acceptance / decline / feedback
• automatic generation of PeerReviewActions based on activity

https://meta.discourse.org/t/how-discourse-docker-script-manages-to-autostart-upon-host-machine-reboot/21469/6

https://blog.docker.com/2014/08/announcing-docker-1-2-0/

Explore our options

• Have VNC in Docker and allow users to connect through their own VNC viewer or in their browser (possibly using Avocado or noVNC)
• Integration with Jupyter. Could look into EMAworkbench which has support for running NetLogo models

OrCID Auth:

• https://members.orcid.org/api/integrate/orcid-sign-in#step1

Use Python Social Auth for

• Twitter
• FB
• GitHub
• Google

Integrate with Django:

• https://www.toptal.com/django/integrate-oauth-2-into-django-drf-back-end
• https://github.com/PhilipGarnero/django-rest-framework-social-oauth2
• https://github.com/GetBlimp/django-rest-framework-jwt
• https://github.com/Tivix/django-rest-auth

Add SSO endpoint for Discourse:

• https://meta.discourse.org/t/official-single-sign-on-for-discourse/13045
• https://meta.discourse.org/t/sso-example-for-django/14258

integrate with https://github.com/codemeta/codemeta schema, follow software citation principles from Force11

• force11/force11-scwg#157

On ValidationError responses the components display what made them invalid.

metadata for versioned dependencies to build / run models

Python

• requirements.txt

NetLogo

• Extensions

Repast Simphony

• Eclipse

Java

• Maven pom.xml / Ivy ivy.xml

A supervisor for managing cron and uwsgi.

Could use dumb-init

Hey @cpritcha can you add http://www.1001fonts.com/alte-din-1451-mittelschrift-font.html to our webpack build and make it available as a poor person's replacement to Fervor's FF Din font selection?

There's also https://fontlibrary.org/en/font/u-din-1451-mittelschrift but I would probably prefer to host it locally. I don't mind relying on Google for fonts but not sure how long fontlibrary.org will be around.

Roll our own bad implementation or integrate with and rebrand a self-hosted discourse.org Docker instance:

• install to forum.comses.net https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md
• implement SSO #36
• #37 migrate Drupal data: https://github.com/discourse/discourse/tree/master/script/import_scripts use create_post from https://github.com/discourse/discourse/blob/master/script/import_scripts/drupal.rb
• API docs discussion: https://meta.discourse.org/t/discourse-api-documentation/22706/167 http://docs.discourse.org/

create hashed URLs for read-only access to private codebase releases for external reviewers (exposed on the page)