onaio / kpi Goto Github PK
View Code? Open in Web Editor NEWThis project forked from kobotoolbox/kpi
kpi: kobo-api
License: GNU Affero General Public License v3.0
This project forked from kobotoolbox/kpi
kpi: kobo-api
License: GNU Affero General Public License v3.0
See comments here: #1 (review)
Figure out how kpi can be able to pull and edit a form initially published by ona
If you go to a formbuilder URL you don't have cookies set, or you do not have access, you see an infinite loading circle, but the JS throws a 404 error.
We should catch the 404:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
utility.execute()
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/core/management/__init__.py", line 328, in execute
django.setup()
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
self.models_module = import_module(models_module_name)
File "/home/winny/.pyenv/versions/2.7.16/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/auth/models.py", line 41, in <module>
class Permission(models.Model):
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/db/models/base.py", line 139, in __new__
new_class.add_to_class('_meta', Options(meta, **kwargs))
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/db/models/base.py", line 324, in add_to_class
value.contribute_to_class(cls, name)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/db/models/options.py", line 250, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/db/__init__.py", line 36, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/db/utils.py", line 241, in __getitem__
backend = load_backend(db['ENGINE'])
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/db/utils.py", line 112, in load_backend
return import_module('%s.base' % backend_name)
File "/home/winny/.pyenv/versions/2.7.16/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 8, in <module>
from .features import DatabaseFeatures
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
from django.contrib.gis.db.models import aggregates
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/db/models/__init__.py", line 7, in <module>
from django.contrib.gis.geos import HAS_GEOS
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/geos/__init__.py", line 8, in <module>
from .libgeos import geos_version, geos_version_info # NOQA: flake8 detects only the last __all__
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/geos/libgeos.py", line 152, in <module>
_verinfo = geos_version_info()
File "/home/winny/.pyenv/versions/kpi_local/lib/python2.7/site-packages/django/contrib/gis/geos/libgeos.py", line 147, in geos_version_info
raise GEOSException('Could not parse version info string "%s"' % ver)
django.contrib.gis.geos.error.GEOSException: Could not parse version info string "3.6.2-CAPI-1.10.2 4d2925d6"
When patching assets, we trigger a deployment for the same asset, to update the form on our backend as well as the asset in Formbuilder.
The intent of this issue is to confirm that assets that are not of asset_type survey, should not trigger a deployment when patching the asset on Formbuilder. This is because there isn't an existing asset in onadata
Kobotoolbox who are the maintainers of KPI recently upgraded the package to using Django 2.
This issue will track efforts made in attempts to upgrade our own instance to run Django 2, following the work detailed here on how to upgrade
Rebase branch with Kobotoolbox two-databases_2.020.12b
branch. Use this rebased branch to test
Update the KPI ansible role to configure KPI deployments to run Ubuntu 18.04 with 2 Databases
Update playbooks repo to configure KPI deployments to run Ubuntu 18.04 with 2 Databases
Create a new KPI Postgres database running Postgresql v10.
Using psql utility in the core servers, connect to postgres using the Onadata user
psql -p 5432 -h <db_host> -U onadata -d postgres -W
CREATE USER kpi WITH PASSWORD 'random_password';
CREATE DATABASE kpi;
GRANT ALL PRIVILEGES ON DATABASE "kpi" to kpi;
Unfortunately, the database created is v9.6
as its created within the preview postgresql instance.
Bring down the instance tracking the ona-custom-changes branch
KPI service was down for 25 mins
Migrate KPI tables and data following this plan while deploying kpi_2.0
alongside the current running branch ona_custom_changes
. Time estimations for the actions performed in the data dump are detailed below:
Data Dump (Approximately 40 mins)
24/11/2020 from 17:55hrs to 18:35hrs
25/11/2020 from 12:01hrs - 12:34hrs
Restore data from the data dump to the KPI
Data Restoration (Approximately 15 mins)
24/11/2020 from 18:35hrs to 18:45hrs
25/11/2020 from 12:34hrs - 12:48hrs
Notes
No downtime
Onadata, Enketo and Zebra services not interrupted.
**Actions performed while migrating tables and data**
- Updating email.
- Updating password
- Updating user profile image
- Uploading xlsform
- Open form in Enketo
- Create submission from enketo
- Could not create formbuilder form
Deploy the kpi_2.0
branch.
SSH into the newly deployed kpi_2.0
branch to confirm that new migrations received from after Kobo rebase are not yet applied
(kpi) root@preview-forms-ona-io:/home/kpi/app# ./manage.py showmigrations --settings=kobo.local_settings
DEBUG 2020-11-25 10:22:56,934 base 6559 140470675355456 Configuring Raven for host: https://sentry.onalabs.org
admin
[X] 0001_initial
[X] 0002_logentry_remove_auto_add
[X] 0003_logentry_add_action_flag_choices
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
[X] 0007_alter_validators_add_error_messages
[X] 0008_alter_user_username_max_length
[X] 0009_alter_user_last_name_max_length
[X] 0010_alter_group_name_max_length
[X] 0011_update_proxy_permissions
authtoken
[X] 0001_initial
[X] 0002_auto_20160226_1747
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
database
[X] 0001_initial
[ ] 0002_auto_20190129_2304
django_celery_beat
[ ] 0001_initial
[ ] 0002_auto_20161118_0346
[ ] 0003_auto_20161209_0049
[ ] 0004_auto_20170221_0000
[ ] 0005_add_solarschedule_events_choices
[ ] 0006_auto_20180322_0932
[ ] 0007_auto_20180521_0826
[ ] 0008_auto_20180914_1922
[ ] 0006_auto_20180210_1226
[ ] 0006_periodictask_priority
[ ] 0009_periodictask_headers
[ ] 0010_auto_20190429_0326
[ ] 0011_auto_20190508_0153
django_digest
[X] 0001_initial
external_integrations
[X] 0001_add_allowed_cors_origin_model
help
[X] 0001_initial
hook
[X] 0001_tables_creation
[X] 0002_add_email_notifications_field
[X] 0003_add_subset_fields_to_hook_model
[ ] 0004_add_payload_template
[ ] 0004_allow_blank_subset_fields
hub
[X] 0001_initial
[X] 0002_formbuilderpreference
[X] 0003_auto_20160318_1808
[X] 0004_configurationfile
[X] 0005_perusersetting
[ ] 0006_remove_formbuilder_preference_table
[ ] 0007_alter_jsonfield_to_jsonbfield
kpi
[X] 0001_initial
[X] 0002_auto_20150619_0603
[X] 0003_assetsnapshot
[X] 0004_default_permissions_1910
[X] 0005_taguid
[X] 0006_importtask_messages
[X] 0007_importtask_defaults
[X] 0008_authorizedapplication
[X] 0009_auto_20160315_0343
[X] 0010_asset_deployment_data
[X] 0011_explode_asset_deployments
[X] 0012_onetimeauthenticationkey
[X] 0013_uid_field
[X] 0014_discoverable_subscribable_collections
[X] 0015_assetversion
[X] 0016_asset_settings
[X] 0017_assetversion_uid_aliases_20170608
[X] 0018_export_task
[X] 0019_add_report_custom_field
[X] 0020_add_validate_submissions_permission_to_asset
[X] 0021_map-custom-styles
[X] 0022_assetfile
[ ] 0023_partial_permissions
[ ] 0024_alter_jsonfield_to_jsonbfield
oauth2_provider
[X] 0001_initial
[X] 0002_08_updates
[X] 0003_auto_20160316_1503
[X] 0004_auto_20160525_1623
[X] 0005_auto_20170514_1141
[X] 0006_auto_20171214_2232
registration
[X] 0001_initial
[X] 0002_registrationprofile_activated
[X] 0003_migrate_activatedstatus
[X] 0004_supervisedregistrationprofile
reversion
[X] 0001_squashed_0004_auto_20160611_1202 (4 squashed migrations)
sessions
[X] 0001_initial
taggit
[X] 0001_initial
[X] 0002_auto_20150616_2121
[X] 0003_taggeditem_add_unique_index
DEBUG 2020-11-25 10:31:40,220 base 6795 140513684821824 Configuring Raven for host: https://sentry.onalabs.org
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0004_allow_blank_subset_fields, 0004_add_payload_template in hook).
- To fix them run 'python manage.py makemigrations --merge'
Run run 'python manage.py makemigrations --merge
(kpi) root@preview-forms-ona-io:/home/kpi/app# python manage.py makemigrations --merge
Merging hook
Branch 0004_add_payload_template
- Add field payload_template to hook
Branch 0004_allow_blank_subset_fields
- Alter field subset_fields on hook
Merging will only work if the operations printed above do not conflict
with each other (working on different fields or models)
Do you want to merge these migration branches? [y/N] y
Created new merge migration /home/kpi/app-versioned/kpi_2.0/kobo/apps/hook/migrations/0005_merge_20201125_1032.py
pi) root@preview-forms-ona-io:/home/kpi/app# ./manage.py migrate --settings=kobo.local_settings
DEBUG 2020-11-25 11:02:43,703 base 7433 139920873961280 Configuring Raven for host: https://sentry.onalabs.org
Operations to perform:
Apply all migrations: admin, auth, authtoken, contenttypes, database, django_celery_beat, django_digest, external_integrations, help, hook, hub, kpi, oauth2_provider, registration, reversion, sessions, taggit
Running migrations:
Applying database.0002_auto_20190129_2304... OK
Applying django_celery_beat.0001_initial... OK
Applying django_celery_beat.0002_auto_20161118_0346... OK
Applying django_celery_beat.0003_auto_20161209_0049... OK
Applying django_celery_beat.0004_auto_20170221_0000... OK
Applying django_celery_beat.0005_add_solarschedule_events_choices... OK
Applying django_celery_beat.0006_auto_20180322_0932... OK
Applying django_celery_beat.0007_auto_20180521_0826... OK
Applying django_celery_beat.0008_auto_20180914_1922... OK
Applying django_celery_beat.0006_auto_20180210_1226... OK
Applying django_celery_beat.0006_periodictask_priority... OK
Applying django_celery_beat.0009_periodictask_headers... OK
Applying django_celery_beat.0010_auto_20190429_0326... OK
Applying django_celery_beat.0011_auto_20190508_0153... OK
Applying hook.0004_allow_blank_subset_fields... OK
Applying hook.0004_add_payload_template... OK
Applying hook.0005_merge_20201125_1032... OK
Applying hub.0006_remove_formbuilder_preference_table... OK
Applying hub.0007_alter_jsonfield_to_jsonbfield... OK
Applying kpi.0023_partial_permissions... OK
Applying kpi.0024_alter_jsonfield_to_jsonbfield...
Scanned | # records need fixing
Asset._deployment_data | 0
Asset.content | 0
Asset.summary | 0
AssetSnapshot.details | 0
AssetSnapshot.source | 0
ExportTask.data | 0
ExportTask.messages | 0
ImportTask.data | 0
ImportTask.messages | 0
Database is ready.
Proceeding with migration 0024 (text -> jsonb)
This might take a while depending on your database size.
If it's too slow, consider stopping this and instead running
./manage.py migrate_json_text_to_jsonb
to avoid downtime.
OK
QA
This has been tested on https://stage.ona.io
here
Update the terraform script that creates new databases for new Onadata deployments to also create KPI db if it doesnt esist already
Remove KPI tables from Onadata Database
Describe the bug
Server 404 error when trying to download forms created using form builder.
"Failed to get permissions config!" and "failed to load environment data"
: When trying to edit form created using the form builder
Steps to replicate the behaviour
Expected behaviour
When a user authors a form in the form builder, and changes their email address on the account settings, they should still be able to edit the form in KPI and also download the XLS format of the form.
Actual behavior
404 error while downloading forms created on form builder
"Failed to get permissions config!" and "failed to load environment data"
: When trying to edit form created using the form builder
More inforamation/context
https://onaio.slack.com/archives/C4S0D2S3H/p1614591107003400
When running KPI via docker with the new setup, these are afew items that need to be updated
KPI doesnt define this docker-compose file in their kobotoolbox/kpi repository. They have this instead defined in the kobotoolbox/kobo-docker repository.
The docker-compose file will now help us manage the KPI and postgres docker containers.
KPI doesnt define a settings file to work with Docker. They make use of this file, which points to the base setting file.
We are introducing this docker settings file in an effort not to alter the base setting file.
We need to update the init bash script here to handle migrations appropriately. These updates include:
Resolves part of #26
For Kobotoolbox setup, there is some functionality included to make sure that data across the separate applications(KPI and Kobo) are linked on the Kobo end.
For our setup as Ona, data across these two applications are linked on the KPI end.
Forms created in Formbuilder are published in Onadata. We generate the identifier here to link the KPI asset to the Onadata form for future updates to the same form on Formbuilder(Synchronizing the two essentially.)
This error is generated currently when trying to update(redeploy) a form created in formbuilder
Resolves part of #26
New users on Onadata are not created on the KPI auth tables in the new KPI database.
Thus assigining of permissions fails, as the auth_user has not been created on the KPI Databse
When new users are created on Onadata, we need to create user accounts for these users on KPI's new DB
Resolves part of #26
Production
When a mandatory response is unchecked for a question in form builder, the form is saved. Whenever the form is edited, the previous uncheck boxes should remain unchecked.
Related kobotools issue:
kobotoolbox#1205
kobotoolbox/kobocat#348
Kobotool Support issue complaints:
http://support.kobotoolbox.org/customer/portal/questions/16746211-mandatory-response-
Discussion:
@pld
When does the user see a raw URL like this? Or does this never get to the user? If so we'll want to show them something prettier, or just redirect them.
@ivermac
They should get the page when they refresh the formbuilder page. The other thing I had thought but I hadn't implemented was to check if the authentication cookie exists before deploying (when a user clicks the save button) to onadata i.e if the auth cookie is not available, js would refresh page.
@pld
OK, since this is public we need to do something more user friendly than render text here
We'd then add a boolean value in a user's extra details model to show whether the user can can publicize a collection or not
redirect forms.ona.io to ona.io
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.