chairemobilite / evolution Goto Github PK

License: MIT License

JavaScript 14.57% SCSS 1.39% TypeScript 80.45% HTML 0.04% Dockerfile 0.02% Python 3.54%

evolution's Introduction

Evolution

Evolution is a survey platform for travel survey. Its originality resides in the support of travel diaries, where participants in the survey can enter all the trips they did in a day for example. But it allows to develop flexible questionnaires, in multiple arbitrary sections, with complex conditions, validations, labels and choices, which can all be scripted to use any of the previous answers.

Surveys that use this platform are complete applications. They are scripted: questions and sections are defined in javascript or typescript.

Typically, a survey application is split in 2 separate websites, one where participants can fill the questionnaire directly and another one for administering, monitoring, validating surveys, as well as for technical support to participants and phone interviewers.

This repo contains an example travel survey, in the example/demo_survey folder. It is possible to copy-paste this directory and start editing the survey.

Install dependencies for Linux

The following instructions explain how to install dependencies for Linux.

For Ubuntu 20.04 or 22.04 users, use:

sudo apt-get install postgresql postgis lua5.3 liblua5.3-dev postgresql-postgis postgresql-postgis-scripts

Install dependencies for Windows

The following instructions explain how to install dependencies for Windows.

Download PostgreSQL at https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
Install PostgreSQL. During the installation, make sure you provide a password for your postgres database superuser, and that you install the components pgAdmin 4 and Stack Builder. You can use port 5432
psql --version: Verify your PostgreSQL version inside Powershell
Find and run the “Stack Builder” program that was installed with the database. Select the “Spatial Extensions” section and choose the latest “PostGIS Bundle” option. Accept the defaults and install

Prepare and compile the application

The following instructions explain how to prepare and compile the application

Create a .env file in the project root directory (you can copy the .env.example file) and setup the project
Update the .env with the Postgres connection string and create a new Google map API key if you need the Google map in the project

PG_CONNECTION_STRING_PREFIX=postgres://postgres:<Password>@localhost:5432/ 
GOOGLE_API_KEY= MyGoogleApiKey

git submodule init && git submodule update will get the Transition repository
yarn install or just yarn: Will download the packages required by the application
yarn compile: Convert the typescript files to javascript
yarn setup: Run this command to setup the database for the current project
yarn migrate: Update the database schema with latest changes. This can be run whenever the source code is updated
Optionally yarn create-user: Run this task to create a new user in the database. The user will be able to login to the web interface. This command can be run entirely in a non-interactive mode with the following parameters: yarn create-user --username <username> --email <email> --password <clearTextPassword> [--first_name <firstName> --last_name <lastName> --[no-]admin --[no-]valid --[no-]confirmed --prefs <jsonStringOfPreferences>]. For example, to create and administrator user with the english language as preference, run the following command yarn create-user --username admin --email [email protected] --password MyAdminPassword --admin --prefs '{ "lang": "en" }'

Run the example application

The example application contains 2 distinct application. For local development, we will run the participant app on port 8080 (the default port) and the administrative app on port 8082. Each application needs to build the client app and run the server.

To build and run the participant application:

yarn build:dev or yarn build:prod to build the client app, respectively in development mode (with complete code for debug purposes), or production mode (with minified code for better performances)
yarn start will start the server and listen on port 8080

The participant application can be reached at http://localhost:8080.

To build and run the administrative application:

yarn build:admin:dev or yarn build:admin:prod to build the admin app, respectively in develpment adn production modes.
HOST=http://localhost:8082 yarn start:admin --port 8082 will start the server on port 8082, while overwriting the HOST environment variable to match the admin URL.

Update the application

When updating the application, or switching to a branch that may have changes to the transition submodule, run the following instructions to ensure the application is properly up to date

# Pull the branch
git checkout main
git pull origin main

# Update the applicaiton
yarn reset-submodules
yarn
yarn compile
yarn migrate

Run UI tests for the application

Evolution supports running UI tests with playwright. Surveys need to implement their own tests, but evolution-frontend provider a library in the tests/ui-testing folder.

See the examples/demo_survey/tests folder for examples UI testing

To run the tests for the demo_survey application, follow the following steps:

Copy the configuration file in the repository to test and change the build to your needs

cp packages/evolution-frontend/playwright-example.config.ts survey/playwright.config.ts

Install the dependencies and browsers to use for playwright by running yarn test:ui:install-dependencies. This will install all playwright browsers and dependencies. It is possible to fine-tune the browsers to install. See the playwright documentation for more information (https://playwright.dev/docs/browsers).

For example, to run the tests on firefox, use

npx playwright install --with-deps firefox

You need to start the application as you would to run it:

yarn build:dev or yarn build:prod
yarn start

Run the UI tests

yarn test:survey

Notes: In the test:survey script to define in the project, add the LOCALE_DIR environment variable, to register the translations for the current project. For example, in the demo_survey project, the script is defined as follows:

"test:survey": "LOCALE_DIR=$(pwd)/locales npx playwright test"

Each test defined needs to get its own context for the test execution. The following gives and example of how to start a UI test for an application:

import { test } from '@playwright/test';
import * as testHelpers from 'evolution-frontend/tests/ui-testing/testHelpers';
import * as surveyTestHelpers from 'evolution-frontend/tests/ui-testing/surveyTestHelpers';
import { SurveyObjectDetector } from 'evolution-frontend/tests/ui-testing/SurveyObjectDetectors';

const context = {
    page: null as any,
    objectDetector: new SurveyObjectDetector(),
    title: '',
    widgetTestCounters: {}
}

// Configure the tests to run in serial mode (one after the other)
test.describe.configure({ mode: 'serial' });

// Initialize the test page and add it to the context
test.beforeAll(async ({ browser }) => {
    context.page = await testHelpers.initializeTestPage(browser, context.objectDetector);
});

// Open the page and login
surveyTestHelpers.startAndLoginAnonymously({ context, title: 'Déplacements de longue distance au Québec', hasUser: false });

// TODO Add tests here

// Logout from the survey at the end
surveyTestHelpers.logout({ context });

evolution's People

Contributors

Stargazers

Watchers

Forkers

tahini greenscientist wanadianguy davidmurray samuel-duhaime frost000 kaligrafy millette

evolution's Issues

evolution: devMode should be reset at logout

Now if a user with proper privileges sets dev mode to see all questions, then logs out, the dev mode is still active when the person comes back in.

accessibility: radio and checkboxes should be within fieldsets

And their labels should rather be the legend of the fieldset. See https://www.w3.org/WAI/tutorials/forms/grouping/

Given what the Question.js file looks like now, it's not trivial at all to change. But oh joy! The Question.js class is ready to be moved to typescript!

evolution: Show a diff of the validated interviews

It should be possible, in the interview summary window to easily show what was changed by the validator. We could use a diff-like syntax. Some package can diff json objects. We have the original responses, as well as the validated ones, so it would be easy to display any change.

evolution: Connect to call center API

Participants can call a call center by phone and a telephoniste will fill the questionnaire. The token can be entered by the participant on the phone and the survey will open in front of the telephoniste.

We'll need to get the specification of the call center API. This issue is pending this information.

evolution: Remove unused fields in db

The is_started, start_at and end_at are unused. The data that is used is in the responses field.

We should decide whether to use those fields and drop the corresponding responses objects, or drop those fields altogether

evolution: when a select menu includes choices with conditional, it must empty the value when the selected choice is being hidden

Example: Occupation: age is 40, you select partTimeWorker, then you change age for 4, and partTimeWorker is hidden now, but it stasy as the value, so the occupation field is still valid, but should not. It should be set to null whenever the conditional selected choice is made hidden.

evolution: Use one set of responses in most function/parsers

In many places right now, we need to send both validated_data and responses fields, sometimes called responses and _responses respectively. This can be confusing as responses have different meaning and we need to absolutely make sure the right set of responses is sent to the right parameter.

The main reason for this is... the language of the interview! That is set somehow, but it needs to remain the language the interviewee selected and not be updated by a validator who might have another language choice. If the language is correctly managed between roles, it will be easy to update all parsers and calls to use only a single set of responses sent in parameter

evolution: URLs with // in it are not equivalent to /

for example https://<site>//interviews/byCode/<someCode> should be equivalent to https://<site>/interviews/byCode/<someCode>, but the first leads to a blank page, while the second is the expected page.

evolution: InputMapFindPlace: after dragend, show a button for refreshing places on map around center, like "Search this area"

evolution: Allow to remove the validation status of the interview

For any button adding a validation status, it should be possible to click again on the button to remove the validation status

evolution: Have an anti-spam mechanism for login links and forgot password page

To avoid somebody spamming an email account with the forgot password or passwordless email, we should provide some mechanism to stop sending such emails after a while.

evolution: Add a confirm dialog before resetting validation data

When the validators clicks on the reset interview button, there should be a confirmation message, as this will remove all validations done so far.

Also, should resetting the interview also reset the is_valid and is_completed and is_validated flags?

accessibility: make sure the slider widget is accessible

Replace by radio buttons or drop-down menu with labels values and maybe add more precise half values.

evolution: Assign validations to individual validators

Exact requirements are to be discussed before implementing, but as I understand today

In general, an admin ro super-validator should be able to filter/search surveys, select them and assign them to individual validators. An interview can be assigned to more than one validator.

accessibility: Inputs should always be below the labels

To help with screen readers et le grossissement d'écran, the inputs should always be below the labels, like in the mobile version.

But given the current look & feel of the forms, we should review our design? It would be a thin form because texts shouldn't be too wide...

evolution: rename audits to auditsDetails or validationResults

there is an ambiguity between audits (count per error) and validation results (acutel error for each validation). We call them audits everywhere, but they should not be called the same.

evolution: add a privacy policy page accessible from any page

evolution: Determine how to log interview flow

Now, each field change is saved in a field in the database. We want to be able to analyze the flow of interviews, the time taken in each field, etc, without necessarily saving the data itself. How is that best done? To be determined and investigated...

evolution: enforce rate limit for people entering access code

We want to prohibit brut forcing the access code (received via a letter), so if someone try to enter several access code, it should be blocked or get a Recaptcha

evolution: Find a solution to keep track of users who edited an interview

The quick&easy solution is to add a field in responses with an array of user ids who edited the interview. But this exposes the IDs of users with higher privileges than participants. That is not good. It should be handled only server side and never make it to the client.

evolution: Migrate survey section to typescript

This includes migrating a few components from the src/components/survey section:

evolution: Implement a separate application for participants and other users

Independent of where the users vs participants are stored, a separate "user" login page can be configured the same way as the main application's login (see the auth configuration options).

In the backend, allow to specify which auth options to use for the various login pages
Since auth options are different, it would make sense to have them served by different auth routes, so we should be able to set a prefix to the routes
In the frontend, all pages regarding an authentication method (for example login/register/forgotPwd/reset/etc for localLogin) should have the URL links configurable through the props.

evolution: Let the `findPlace` widget support multiple geocoders

The widget config of the find place should have a geocoder field to indicate which geocoder to use. That field should be an array of geocoder. If there are multiple geocoders, queries will be done in order (position 0 has precedence) and results from the later geocoders will be added to the initial ones according to certain criteria.

The use case for this is that the openstreetmap data is not complete for some regions, so returned choices may not be complete. In those cases, using a google query will be more accurate, but we still want to have an idea of how often an openstreetmap choice was clicked.

evolution: Cleanup/rename or refactor the interview flags in db

In the database, there are the following boolean fields that need precision:

is_valid/is_validated: it's like 2 different levels of validity. is_valid is the first level of validation, like "there's no error, I checked", is_validated is the ultimate state the interview can be in: it's done over with, this interview is good to go! In between is the is_completed flag, which here means "ok, I'm an authority on validation and I checked that the validation was done correctly, but the interview still needs to be checked, it's not completely validated yet". It does not mean that the interview is completed by the participant! Confused? I sure am!

2 possibilities to fix:

rename those flags and make sure their meaning is obvious and well-documented
Since they all correspond to a state the interview is in, replace with a single enum field that contains the current state of the interview. Boolean fields could mismatch (what if an interview gets to a state with is_validated is true and is_valid is false?), but the state field can't be wrong. We can define a state machine to define the conditions to pass from one state to the next.

use evolution custom inputs for login/register and other authentication forms

It will help for accessibility and will add clearer labels.

evolution: convert all icons to svg

evolution: Support token authentication for surveys

Participants receive a token by email or written mail and can enter it to start a survey.

Make sure it works and the participants can come back to their survey

evolution: Let the passwordless auth method also support SMS

For now, passwordless supports email, but we could also support SMS, with some public service API for sending

evolution: allow auto formatting of fields

Example: accept phone number with our without spaces and/or dashes or access codes with or without dashes

evolution: Allow an anonymous user to change his mind

If someone logged in anonymously, there is no way to go back to the survey once the session expires. But what if the user realizes the survey is longer than she thought and wants to do something else and come back? It should be possible for an anonymous user to click on a link to add an email address, which will be associated with the current account. Then, next time, she can use the passwordless authentication.

evolution: Anonimize some info for validators level 1

Those validators don't need to have access to personal information of the participant. This information should be removed from the interviews sent to the browser, without impacting the copy of the data, so the client should typically assume it does not have the entirety of the responses (meaning the backend should be responsible to copy the responses field to the validated_data and frontend actions should never sent whole arrays for these fields).

While easy to somewhat-ish implement this, testing it to make sure no code path does this will be hard...

evolution: Validation audit and reset should be done server side

In the validation code client side, often, both the responses and validated_data are sent to the client, as _responses and responses. That means the respondent's original response are there, unanonymized on the client, and there's even a console.log displaying them!

To avoid code duplication and avoid sending all this data to the client, resetting the interview should be done server-side only, and only one set of responses sent to the client, the cleaned one.

Not trivial to implement though, as it may have many side-effects. Let's continue refactoring the code before tackling this.

survey • SegmentsSection custom template

Create custom template for the segments (modes) section:

evolution: add widget parameter for random conditional, with which object to use as random

Example: if randomProbability = 0.25 and randomObjectPath = "household.persons.{_activePersonId}", for this person, the widget should appear 25% of the time, but the decision to make it appear or not should be set only once for the person object.
Another example: if randomProbability = 0.33 and randomObjectPath = "household" but the widget path is for a person, the widget should appear for 33% of the households, but if it appear, it should for every household members.

evolution: Refactor server/client validations to avoid custom code paths

When refactoring the interview update, thought should be put on how to streamline server and client validations. Now, the server validations come back from the updateInterview server call and are stored in the survey state, while client validations are... in the Question widget I think?

It should be easy and straightforward to add validations server-side and client-side and should not require too much custom code like the ones to fix chairemobilite/transition-legacy#1183 and chairemobilite/transition-legacy#1244

evolution: Use interview cache results for interview by household size

The new custom widgets of od_mtl_2023 added a monitoring cache, where for each interview a few attributes are saved to build various widgets. This cache should have a base in the main evolution code, with projects specifying the fields to use to group by. And the interview by household size should use this data instead.

The refresh button to save this cache should also be in the main evolution code base.

evolution: Consider not using redux-async-queue

The package is 5 years old and it is not possible to empty the queue. It is used for survey updates.

evolution: Add a filter on accessCode

There should be a filter on whether an accessCode is null or not, that differenciates household survey from person survey

evolution: remove comments for old ornament on login pages and Survey component

Add a project config to enable/disabled ornament on the top of every survey pages and remove old comments

evolution: Highlight the currently selected interview in validation list

When a interview summary is selected and being viewed, the corresponding interview should be highlighted in the list, for easier identification.

survey • tests • surveyHelper

Test all survey helper functions

Add checkboxes: "remember me" + "I use a public computer" and make cookie expiration period longer for transition users, survey interviewers and admins when remember me is checked and not a public computer

It is a pain to have to login again every day when administrating a survey.

forgot password: add the application/survey title to the forgot password email

so the user can understand from which app it comes

survey • CompletedSection

Create a completed section:

Thank you text with info on number of days until the interview will be frozen
Total distance (bird distance) for household
Distance moyenne par membre du ménage s'étant déplacé
Nombre de déplacements totaux
Nombre de déplacements par membre du ménage
Lien vers ARTM
Quit button (logout)
(nice to have): map of all trips by all members with different color per person

evolution: Avoid centering text on main page

Hi,

I believe that left-aligned text looks more professional. Comparison:

Thanks.
David

evolution: add a better failsafe or fallback for parseStrings

If a value sent as an argument to parseString does not contain the key for the actual locale, it should fallback to a default value or trigger an error.

accessibility: Make sure the survey can be filled without scripting enabled!!

That would be a big piece, but from https://webaim.org/techniques/formvalidation/ it says

The form must also submit to the server if client-side scripting is unavailable. Forms that rely solely on script functions or event handlers should be avoided. Instead, use a true URL action value for the form. Client-side validation can still be invoked, and it would be processed first if scripting is enabled.

evolution: auto-set users with interviewer/validator/admin/super-validators permissions own interviews to isValid = false

interviewers/admin/validator/super-validator interviews should not count in stats and therefore must be set invalid

evolution: Support the photon geocoder in typescript

Now, the InputMapFindPlacePhoton is a different widget. We just need to have a proper geocoding API and support the photon geocoding functions in the new InputMap* widgets.

evolution: Migrate redux actions to typescript

The backend routes are now well typed. Since the redux actions are used to set and maintain the application's state, used throughout the survey components, it would be good if it is also typed and safe, even if the components themselves are not yet typed. It will document the state at the same time.