onaio / kpi Goto Github PK

See comments here: #1 (review)

Figure out how kpi can be able to pull and edit a form initially published by ona

If you go to a formbuilder URL you don't have cookies set, or you do not have access, you see an infinite loading circle, but the JS throws a 404 error.

We should catch the 404:

• If the cookies are not set, redirect the user to zebra login with an after login redirect back to this page
• If the cookies are set we redirect to the project on zebra (and they'll get a permission error there)
  • For this we'll need the user name and the project ID, then we redirect to [protocol]://[zebra-host]/[username]/[project id], and zebra can take it from there

When patching assets, we trigger a deployment for the same asset, to update the form on our backend as well as the asset in Formbuilder.
The intent of this issue is to confirm that assets that are not of asset_type survey, should not trigger a deployment when patching the asset on Formbuilder. This is because there isn't an existing asset in onadata

Kobotoolbox who are the maintainers of KPI recently upgraded the package to using Django 2.

This issue will track efforts made in attempts to upgrade our own instance to run Django 2, following the work detailed here on how to upgrade

Proposed Implementation plan

• Rebase branch with Kobotoolbox two-databases_2.020.12b branch. Use this rebased branch to test

• Update the KPI ansible role to configure KPI deployments to run Ubuntu 18.04 with 2 Databases

• Update playbooks repo to configure KPI deployments to run Ubuntu 18.04 with 2 Databases

• Create a new KPI Postgres database running Postgresql v10.

• Using psql utility in the core servers, connect to postgres using the Onadata user

psql -p 5432 -h <db_host> -U onadata -d postgres -W

• Manually create KPI DB within the same Onadata postgres intallation

CREATE USER kpi WITH PASSWORD 'random_password';
CREATE DATABASE kpi;
GRANT ALL PRIVILEGES ON DATABASE "kpi" to kpi;

• Unfortunately, the database created is v9.6 as its created within the preview postgresql instance.

• Bring down the instance tracking the ona-custom-changes branch

• KPI service was down for 25 mins

• Migrate KPI tables and data following this plan while deploying kpi_2.0 alongside the current running branch ona_custom_changes. Time estimations for the actions performed in the data dump are detailed below:

• Data Dump (Approximately 40 mins)
  24/11/2020 from 17:55hrs to 18:35hrs
  25/11/2020 from 12:01hrs - 12:34hrs

• Restore data from the data dump to the KPI

• Data Restoration (Approximately 15 mins)
  24/11/2020 from 18:35hrs to 18:45hrs
  25/11/2020 from 12:34hrs - 12:48hrs

Notes

No downtime
Onadata, Enketo and Zebra services not interrupted.

**Actions performed while migrating tables and data**
- Updating email.
- Updating password
- Updating user profile image
- Uploading xlsform
- Open form in Enketo
- Create submission from enketo
- Could not create formbuilder form

• Deploy the kpi_2.0 branch.

• SSH into the newly deployed kpi_2.0 branch to confirm that new migrations received from after Kobo rebase are not yet applied

(kpi) root@preview-forms-ona-io:/home/kpi/app# ./manage.py showmigrations --settings=kobo.local_settings
DEBUG 2020-11-25 10:22:56,934 base 6559 140470675355456 Configuring Raven for host: https://sentry.onalabs.org
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [X] 0010_alter_group_name_max_length
 [X] 0011_update_proxy_permissions
authtoken
 [X] 0001_initial
 [X] 0002_auto_20160226_1747
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
database
 [X] 0001_initial
 [ ] 0002_auto_20190129_2304
django_celery_beat
 [ ] 0001_initial
 [ ] 0002_auto_20161118_0346
 [ ] 0003_auto_20161209_0049
 [ ] 0004_auto_20170221_0000
 [ ] 0005_add_solarschedule_events_choices
 [ ] 0006_auto_20180322_0932
 [ ] 0007_auto_20180521_0826
 [ ] 0008_auto_20180914_1922
 [ ] 0006_auto_20180210_1226
 [ ] 0006_periodictask_priority
 [ ] 0009_periodictask_headers
 [ ] 0010_auto_20190429_0326
 [ ] 0011_auto_20190508_0153
django_digest
 [X] 0001_initial
external_integrations
 [X] 0001_add_allowed_cors_origin_model
help
 [X] 0001_initial
hook
 [X] 0001_tables_creation
 [X] 0002_add_email_notifications_field
 [X] 0003_add_subset_fields_to_hook_model
 [ ] 0004_add_payload_template
 [ ] 0004_allow_blank_subset_fields
hub
 [X] 0001_initial
 [X] 0002_formbuilderpreference
 [X] 0003_auto_20160318_1808
 [X] 0004_configurationfile
 [X] 0005_perusersetting
 [ ] 0006_remove_formbuilder_preference_table
 [ ] 0007_alter_jsonfield_to_jsonbfield
kpi
 [X] 0001_initial
 [X] 0002_auto_20150619_0603
 [X] 0003_assetsnapshot
 [X] 0004_default_permissions_1910
 [X] 0005_taguid
 [X] 0006_importtask_messages
 [X] 0007_importtask_defaults
 [X] 0008_authorizedapplication
 [X] 0009_auto_20160315_0343
 [X] 0010_asset_deployment_data
 [X] 0011_explode_asset_deployments
 [X] 0012_onetimeauthenticationkey
 [X] 0013_uid_field
 [X] 0014_discoverable_subscribable_collections
 [X] 0015_assetversion
 [X] 0016_asset_settings
 [X] 0017_assetversion_uid_aliases_20170608
 [X] 0018_export_task
 [X] 0019_add_report_custom_field
 [X] 0020_add_validate_submissions_permission_to_asset
 [X] 0021_map-custom-styles
 [X] 0022_assetfile
 [ ] 0023_partial_permissions
 [ ] 0024_alter_jsonfield_to_jsonbfield
oauth2_provider
 [X] 0001_initial
 [X] 0002_08_updates
 [X] 0003_auto_20160316_1503
 [X] 0004_auto_20160525_1623
 [X] 0005_auto_20170514_1141
 [X] 0006_auto_20171214_2232
registration
 [X] 0001_initial
 [X] 0002_registrationprofile_activated
 [X] 0003_migrate_activatedstatus
 [X] 0004_supervisedregistrationprofile
reversion
 [X] 0001_squashed_0004_auto_20160611_1202 (4 squashed migrations)
sessions
 [X] 0001_initial
taggit
 [X] 0001_initial
 [X] 0002_auto_20150616_2121
 [X] 0003_taggeditem_add_unique_index

• Apply the new migrations(After rebase with Kobotoolbox)
• You get this error

DEBUG 2020-11-25 10:31:40,220 base 6795 140513684821824 Configuring Raven for host: https://sentry.onalabs.org
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0004_allow_blank_subset_fields, 0004_add_payload_template in hook).
- To fix them run 'python manage.py makemigrations --merge'

Run run 'python manage.py makemigrations --merge

• This runs successfully

(kpi) root@preview-forms-ona-io:/home/kpi/app# python manage.py makemigrations --merge
Merging hook
  Branch 0004_add_payload_template
    - Add field payload_template to hook
  Branch 0004_allow_blank_subset_fields
    - Alter field subset_fields on hook

Merging will only work if the operations printed above do not conflict
with each other (working on different fields or models)
Do you want to merge these migration branches? [y/N] y

Created new merge migration /home/kpi/app-versioned/kpi_2.0/kobo/apps/hook/migrations/0005_merge_20201125_1032.py

• Then migrate
  Migrations run successfully for 1 min or so

pi) root@preview-forms-ona-io:/home/kpi/app# ./manage.py migrate --settings=kobo.local_settings
DEBUG 2020-11-25 11:02:43,703 base 7433 139920873961280 Configuring Raven for host: https://sentry.onalabs.org
Operations to perform:
  Apply all migrations: admin, auth, authtoken, contenttypes, database, django_celery_beat, django_digest, external_integrations, help, hook, hub, kpi, oauth2_provider, registration, reversion, sessions, taggit
Running migrations:
  Applying database.0002_auto_20190129_2304... OK
  Applying django_celery_beat.0001_initial... OK
  Applying django_celery_beat.0002_auto_20161118_0346... OK
  Applying django_celery_beat.0003_auto_20161209_0049... OK
  Applying django_celery_beat.0004_auto_20170221_0000... OK
  Applying django_celery_beat.0005_add_solarschedule_events_choices... OK
  Applying django_celery_beat.0006_auto_20180322_0932... OK
  Applying django_celery_beat.0007_auto_20180521_0826... OK
  Applying django_celery_beat.0008_auto_20180914_1922... OK
  Applying django_celery_beat.0006_auto_20180210_1226... OK
  Applying django_celery_beat.0006_periodictask_priority... OK
  Applying django_celery_beat.0009_periodictask_headers... OK
  Applying django_celery_beat.0010_auto_20190429_0326... OK
  Applying django_celery_beat.0011_auto_20190508_0153... OK
  Applying hook.0004_allow_blank_subset_fields... OK
  Applying hook.0004_add_payload_template... OK
  Applying hook.0005_merge_20201125_1032... OK
  Applying hub.0006_remove_formbuilder_preference_table... OK
  Applying hub.0007_alter_jsonfield_to_jsonbfield... OK
  Applying kpi.0023_partial_permissions... OK
  Applying kpi.0024_alter_jsonfield_to_jsonbfield...
 Scanned                | # records need fixing
 Asset._deployment_data | 0
 Asset.content          | 0
 Asset.summary          | 0
 AssetSnapshot.details  | 0
 AssetSnapshot.source   | 0
 ExportTask.data        | 0
 ExportTask.messages    | 0
 ImportTask.data        | 0
 ImportTask.messages    | 0

    Database is ready.
    Proceeding with migration 0024 (text -> jsonb)

    This might take a while depending on your database size.
    If it's too slow, consider stopping this and instead running
        ./manage.py migrate_json_text_to_jsonb
    to avoid downtime.
    
 OK

• QA
  This has been tested on https://stage.ona.io here

• Update the terraform script that creates new databases for new Onadata deployments to also create KPI db if it doesnt esist already

• Remove KPI tables from Onadata Database

Describe the bug
Server 404 error when trying to download forms created using form builder.
"Failed to get permissions config!" and "failed to load environment data": When trying to edit form created using the form builder

Steps to replicate the behaviour

1. Create a test account on Ona - User A
2. Create a form using form builder logged in as a User A
3. Update email address on the test account created
4. While logged in as User A(with the updated email) try to download the xls form / Edit any form created before the email address was updated

Expected behaviour
When a user authors a form in the form builder, and changes their email address on the account settings, they should still be able to edit the form in KPI and also download the XLS format of the form.

Actual behavior
404 error while downloading forms created on form builder
"Failed to get permissions config!" and "failed to load environment data": When trying to edit form created using the form builder

More inforamation/context
https://onaio.slack.com/archives/C4S0D2S3H/p1614591107003400

When selecting the form layout, the read more link directs one to kobotoolbox site. See screenshot.

Option should be removed.

When running KPI via docker with the new setup, these are afew items that need to be updated

• Introduce KPI docker-compose file

KPI doesnt define this docker-compose file in their kobotoolbox/kpi repository. They have this instead defined in the kobotoolbox/kobo-docker repository.
The docker-compose file will now help us manage the KPI and postgres docker containers.

• Introduce KPI docker settings file

KPI doesnt define a settings file to work with Docker. They make use of this file, which points to the base setting file.
We are introducing this docker settings file in an effort not to alter the base setting file.

• Update docker/init.bash script

We need to update the init bash script here to handle migrations appropriately. These updates include:

1. Create KPI DB and user and grand persmissions for DB to user
2. Create Dump for KPI tables to new KPI DB
3. Run migrations after tables have migrated in dump

Resolves part of #26

For Kobotoolbox setup, there is some functionality included to make sure that data across the separate applications(KPI and Kobo) are linked on the Kobo end.

For our setup as Ona, data across these two applications are linked on the KPI end.
Forms created in Formbuilder are published in Onadata. We generate the identifier here to link the KPI asset to the Onadata form for future updates to the same form on Formbuilder(Synchronizing the two essentially.)

This error is generated currently when trying to update(redeploy) a form created in formbuilder

Resolves part of #26

New users on Onadata are not created on the KPI auth tables in the new KPI database.
Thus assigining of permissions fails, as the auth_user has not been created on the KPI Databse

Expected behavior

When new users are created on Onadata, we need to create user accounts for these users on KPI's new DB

Resolves part of #26

The tab still has some 'Kobo' reference. This should be removed

Environment (local, stage, preview, production)

Production

Expected behavior

When a mandatory response is unchecked for a question in form builder, the form is saved. Whenever the form is edited, the previous uncheck boxes should remain unchecked.

Actual behavior

• Mandatory response is unchecked for a question in form builder.
• The form is saved.
• Form is opened to edit and a question is added. The mandatory response is also unchecked.
• The form is saved.
• Form is opened to edit, and the question with the mandatory response uncheck in the first iteration of form edits now have the mandatory response rechecked

Steps to reproduce the behavior

1. Create a form using form builder where you uncheck the Mandatory Response check box for all questions.
2. Save the form.
3. Edit the saved form by adding a question. Uncheck the mandatory response for the added question.
4. Save the form.
5. Open the form to edit and check if the mandatory response is check for the first few questions created in the first iteration.

Implementation plan

Related kobotools issue:
kobotoolbox#1205
kobotoolbox/kobocat#348

Kobotool Support issue complaints:
http://support.kobotoolbox.org/customer/portal/questions/16746211-mandatory-response-

Discussion:

@pld
When does the user see a raw URL like this? Or does this never get to the user? If so we'll want to show them something prettier, or just redirect them.

@ivermac
They should get the page when they refresh the formbuilder page. The other thing I had thought but I hadn't implemented was to check if the authentication cookie exists before deploying (when a user clicks the save button) to onadata i.e if the auth cookie is not available, js would refresh page.

@pld
OK, since this is public we need to do something more user friendly than render text here

#1 (diff)

• Double duration of showing alerts
• Add helper text above the library items on the editor page

Kobotoolbox who are the maintainers of KPI recently upgraded the package to using Django 2.

This issue will track efforts made in attempts to upgrade our own instance to run Django 2, following the work detailed here on how to upgrade

We'd then add a boolean value in a user's extra details model to show whether the user can can publicize a collection or not

redirect forms.ona.io to ona.io