jspsych / datapipe Goto Github PK

Would it be worth adding information about what version(s) of jsPsych the code is compatible with? I guess for the Javascript function option, sending data should be compatible with basically any JsPsych version, is that right? What about the jsPsych-specific code?

🤔

Just a simple counter that ticks by 1 every time a new file is written through the API call.

Setup testing environment to test the cloud functions in the app

The general uploading a file process worked great! This is amazing! I think it'll be very intuitive for regular users with some UI/UX tweaks.

I wish the server prepended the timestamp to the filename, this would prevent malicious actors from overwriting existing data.

I ran into one bug. When validation is on, but the data does not conform to the specification, the server errors out instead of returning error information gracefully. In this case, the server creates a component but did not upload the data.

Here are the response headers in that case. Note that it also does not include the appropriate cross-origin headers.

HTTP/3 500 Internal Server Error
cache-control: private
content-encoding: gzip
content-type: text/plain; charset=utf-8
server: Google Frontend
x-cloud-trace-context: f98af38c066dd5a7141ef86c54ac6d29
x-content-type-options: nosniff
accept-ranges: bytes
date: Wed, 14 Dec 2022 22:33:39 GMT
x-served-by: cache-msp11872-MSP
x-cache: MISS
x-cache-hits: 0
x-timer: S1671057218.690842,VS0,VE1744
vary: x-fh-requested-host, accept-encoding
alt-svc: h3=":443";ma=86400,h3-29=":443";ma=86400,h3-27=":443";ma=86400

Additional info if needed:

I'm running this Javascript code from my personal domain:

fetch("https://pipe.jspsych.org/api/data/", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Accept: "*/*",
  },
  body: JSON.stringify({
    experimentID: "VbtDBjDAOin2",
    filename: "PID.json",
    data: '{"name": "Testing"}',
  }),
});

Since this is a cross-domain request, Firefox (or any browser I believe) sends an OPTIONS request and tests for the appropriate Access-Control-Allow-Origin header in the response.

Here were the appropriate OPTIONS response headers

HTTP/2 204 No Content
access-control-allow-headers: content-type
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-allow-origin: https://vijaymarupudi.com

The table is kind of clunky in its current form.

Also need to think longer term about pagination if there are lots of experiments.

Plugins like html-audio-response generate base64 encoded audio data. We could have a separate API endpoint that takes this data, decodes it, and dumps it to OSF as an audio file.

Only big concern is how much this would cost to run at scale.

If people were going to use this to host an experiment, they would probably need to see a privacy policy specifically for DataPipe, so that they have something to show IRB folks that the data will not be used for any purposes and that it will be simpled passed through.

Providing information about OSF's storage limits https://help.osf.io/article/137-osf-storage-caps and privacy policy https://github.com/CenterForOpenScience/cos.io/blob/master/PRIVACY_POLICY.md. I'm not sure if OSF guarantees that they will not retain the files even after they are deleted.

This information could probably be consolidated to an IRB FAQ or privacy page.

From #46

To help protect from abuse

https://firebase.google.com/docs/app-check/web/recaptcha-provider?authuser=1&hl=en

Hi folks,

Just wanted to alert those of you who are testing this system out right now that I just updated the jsPsychPipe plugin to v0.2. When I wrote it I named a parameter data, which was a silly mistake because jsPsych has a default data parameter and this created a weird conflict. In v0.2 the parameter is now called data_string. I'm in the process of updating the sample code in the app to reflect this change.

Thanks for all of the testing and bug reports!

The admin/[experiment_id].js page has a feature to show the code necessary to communicate with the server. I used Chakra's basic <Code> component, but it lacks a lot of features that would be nice to have, like syntax highlighting and a copy to clipboard button. It's also difficult to get the formatting right. Let's figure out a way to fix this...

Perhaps for <= tablet size we should use a dropdown menu instead?

The homepage could include the workflow of adding code to the experiment to make the data upload work. It's so simple that I think showing that immediately to people would make them more likely to use it and understand what to expect.

From #46 suggestion.

The data validation settings should clarify that the JSON property is checked on every item in the array sent back, not the object sent back.

From #46

The way this is currently checked is to see whether at least one item in the array has each of the required fields. The idea is that in a jsPsych experiment not every trial will have every key in the data object, but you may want to list the keys that the experiment should contain, such that when you convert JSON to, e.g., an R data frame, you have listed the columns you expect to see in the data.

Hello everybody!

I'm testing the platform, and to start with I've tried to create a new experiment. To do so, I set up my osf token and I have already been able to access the board where my experiments are displayed. I have created a new project in osf to be able to connect the experiment, but when I try to click on "create" it loads briefly and nothing changes. I have tried to modify some settings (enable disable data validation, create multiple conditions, etc) but the result is always the same.

I don't know if there has been an error on my part or if I'm running into a potential bug. In any case I report the issue to see what could be going on.

A separate API call that returns the next number in a sequence from 1 to n, where n is configurable in the experiment parameters.

Big Q is whether this can work with firestore as the backend, or is that too slow to update?

Log number of requests per experiment for each API endpoint for some basic health monitoring. This should help us detect where a problem is if there is a sudden surge in spam, for example.

This is just a small thing - the code uses subject_id defined in the first example, but it is omitted in the second example.

Because some basic things like loading the plugin are mentioned, people may expect the code is a complete working example. In other words, the example should include:

const subject_id = jsPsych.randomization.randomID(10);

We could allow people to set a cap per experiment so that large amounts of spam can't be generated. Service would reject new posts if rate limit is reached.

Some questions that we might consider answering

How do I use this service?

Will this host my experiment?

No, you'll need to use a different service to make the experiment available online. The benefit of using this service is that you won't need to configure any of the backend/server components of an experiment, so you can use a provider like GitHub Pages to host the experiment for free. Link to tutorial on how to do this?

Will this store my data?

No. Pipe My Data will send your data directly to the OSF and will not store a copy of the data.

How much does this cost?

Pipe My Data is free to use.

Why is this service free?

The expensive parts of hosting an experiment are providing storage and bandwidth for the experiment files and data. Fortunately there are providers who are willing to do both of these things for free. GitHub (and others) will host a website for free and the Open Science Framework will store data for free. Unfortunately these providers are not directly connected to each other, so that's what we are trying to solve. Pipe My Data is a very lightweight (i.e., cheap) service that makes it easy to link a hosting provider with a data storage provider.

How much does this service cost to run? (Goal here is to provide transparent info about our costs in the event that we need to solicit donations to keep it running)

We host Pipe My Data using Google's Firebase framework, so the cost of Pipe My Data depends on how much usage it gets. Currently our resource consumption is less than $1 per month. Once we have been up and running for a while we will post more information about how much it costs to run the service here. We have funding reserves in the Open Collective account for jsPsych development (link). Our goal is to provide transparent in formation about our costs and our available funds to run the service so you can determine whether we are likely to be able to keep the service running. We are grateful for donations to help keep the service running. You can donate here (link).

Who can see the data that I collect using this service?

What are the risks of using this service and how can I mitigate them?

There are a few risks that you should be aware of before using Pipe My Data.

First, in order to use this service you must provide us with an OSF authorization token so that we can write data to your OSF account on your behalf. This key enables full write access, so if we suffer a data breach it would be possible for someone who got access to the token to make malicious changes to your OSF account. To mitigate this risk, you should create an OSF token that is just for this service so that you can revoke authorization when you are done using the service. The strongest security would be to use an active token only when you need to collect data through this service.

Second, this service does allow a technically savvy user to potentially write fake data to your OSF project. This is almost always a risk with online experiments because the data are usually recorded on the participant's computer before being sent to the server. It is possible for a malicious user to change or create the data before sending it to the service. It is also possible that a user could spam data to your OSF account or could send files that are not actually experiment data. We provide tools to mitigate these risks by allowing you to specify validation rules for the data that is sent and to rate limit the amount of data you are receiving.

Third, this service is not a commercial venture with a dedicated user support team. If something goes wrong, we may not be able to respond quickly. However, the code that runs this service is open source (link) and thoroughly tested. [something about benefits of open source for reliability?]. The service is hosted using the Google Cloud, so we get the benefit of Google's infrastructure to make sure the service keeps running.

Here's the code that it takes to send data, at least as of right now.

          fetch('https://us-central1-osf-relay.cloudfunctions.net/api', {
            method: 'POST',
            headers: {
              'Accept': '*/*',
              'Content-Type': 'application/json'
            },
            body: JSON.stringify({
              experimentID: "Qfp0WCaH8tat",
              filename: `${jsPsych.randomization.randomID(10)}.json`,
              data: JSON.stringify(jsPsych.data.get().values())
            })
          })

Goal would be to encapsulate this in a very simple plugin to get something like:

const saveData = {
	type: jsPsychSaveDataOsfRelay,
	experimentID: "Qfp0WCaH8tat",
	filename: `${jsPsych.randomization.randomID(10)}.json`,
        data: jsPsych.data.get().json()
}

data could have a default value of jsPsych.data.get().json(). filename could also have a default value that generates a random ID.

This is a great idea, I had no idea OSF provided free bucket storage. I wonder what the storage quota is.

I was stuck at finding the graphical process for getting an OSF OAuth Token. When I create a new experiment, It tells me that I have to connect to an OSF account, where a button then takes me to the settings page where I can change the password or enter the token. But I'm not sure how to get the token.

I created a first demo (based on jspsych-html-button-response) and upload works nice!

However, I am facing a problem with validation: when I just turn the validation on and use the default value ("trial_type"), the code returns "result": "Data are not valid" in the final JSON shown on the screen.

My understanding was that that the validation should pass if trial_type was present in each element in the array (which is here). Maybe it is not an error in code and I just misspecified the validation criteria, but I guess using the default value will be one of the first things people will try.

There was no error in the console, I replicated the fail in both FF and Chrome, and both running the jsPsych example on local server and on a github page.

Including testing security rules on firestore

On an experiment page (/pages/admin/[experiment_id].js) we can add a code snippet that includes a jsPsych trial to save code. Eventually this could be expanded to include a generic JS statement (e.g., fetch(...)) if we allow not jsPsych usage.

This way if a token becomes invalid we can avoid an API request until it is updated.

see #66 for some context.

this looks like a useful guide: https://stackoverflow.com/questions/50741936/rate-limiting-for-google-firebase-cloud-functions

I have started to test an experiment that I am just finishing programming with jspsych. When I tried to test the experiment I realised that it did not detect the datapipe plugin. What it says that you have to put the instructions to load the plugin is the following tag: <script src="https://unpkg.com/@jspsych-contrib/jspsych-pipe"></script>

To see if the problem was in the tag, I went to the jspsych-contrib repo and looked for the instructions to load the plugin. In that repo, this is the tag they say you have to use: <script src="https://unpkg.com/@jspsych-contrib/[email protected]"></script>

I think the plugin may have been updated but the data pipe tag that is said to be copied is not. I am posting this as an issue in case there is a way to automatically update this.

It would be nice to be able to add conditions after an experiment has been created, but this might not be a high priority for many users.

From #46

Currently a bit of a mess 😅

Hi!

I made some small UX (and DX) improvements

• add README with dev instructions
• unify sign in and sign up views
• upgrade chakra to 2.4.1 (for Card component)
• change Navbar logic
• for unauth users show only tabs:
  • Home
  • Help
  • Sign In
  • Sign Up (main call-to-action button)
• for auth users show tabs:
  • Home
  • Help
  • Experiments
  • New Experiment (call to action button)
  • Account

In /functions/index.js there is no validation of incoming data. Anything sent will be posted to OSF. We should do some basic checking here to ensure that the data is from an (the?) experiment.

• Could insist on JSON/CSV data, but a nice use case of this service would also be to save data files that are generated by recording audio/video. Maybe this is an option that can be set on the experiment page.
• We could allow some form of validation implementation per experiment. Select what kinds of data is expected. Provide some kind of data structure info. etc.

See #47 for motivation.

• Allow create to work even if someone posts in the full URL for OSF repo
• Handle bad OSF URLs cleanly with error message.

Links to GH repo, Sponsor/Donate, Help, FAQ?

This is a riskier API call because it is hard to validate. Should be turned off by default.

We could send email alerts if something fails, like OSF rate limit hit for a user.

I think this will require modifying the send data API to support including a condition number.

Probably a subdomain of jspsych.org ? Or something more open ended since this can be used by non-jspsych projects?

For cloud functions
https://firebase.google.com/docs/hosting/functions#direct-requests-to-function

/pages/signin.js is not implemented.