Giter VIP home page Giter VIP logo

aind-data-transfer-service's People

Contributors

github-actions[bot] avatar helen-m-lin avatar jtyoung84 avatar mekhlakapoor avatar yosefmaru avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

aind-data-transfer-service's Issues

Add button to cancel job on job status page

Is your feature request related to a problem? Please describe.
Sometimes a user wants to cancel a job that has been submitted.

Describe the solution you'd like
Add a button on the jobs status page to cancel a running job.

Describe alternatives you've considered
Currently, an admin needs to be contacted to cancel a running job.

Additional context
Add any other context or screenshots about the feature request here.

Let service upload to aind-open-data bucket

User story

As a user, I want the server to have aws credentials to the open-data- bucket, so I can write to aind-open-data

Acceptance criteria

  • When a user submits a job request with aind-open-data bucket set, then it will upload data there

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Add option define sources by an index value

Is your feature request related to a problem? Please describe.
I usually do multiple recordings from the same animal within a given day, so the only difference in the asset name between them right now is the timestamp. There's a lot of redundancy in one of the columns.

Describe the solution you'd like
Would it be possible down the road to have some other column in the upload .csv to designate some unique aspect of an experiment, like an index value or something?

Describe alternatives you've considered
Generating the csv file via a python script

Additional context
Add any other context or screenshots about the feature request here.

behavior-videos modality is not being recognized

Describe the bug
Even though aind-data-schema is pinned to 0.26.5, "behavior-videos isn't being recognized

To Reproduce
Steps to reproduce the behavior:

  1. Create a csv file with behavior-videos as one of the modalities and load it as csv
  2. Click Preview
  3. See error

Expected behavior
behavior-videos should be recognized as a valid modality

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Publish to docker registry

User story

As a user, I want this published to a docker registry, so I can use deploy it easily to a k8s cluster.

Acceptance criteria

  • When a pr is merged into main, a docker image will be deployed to a registry

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Add metadata_dir field to excel template

Is your feature request related to a problem? Please describe.
It's not entirely clear how users point to a directory where metadata files associated with a data asset is located.

Describe the solution you'd like
Add a column with metadata_dir field

Describe alternatives you've considered
At some point, we should have a fillable form

Additional context
Add any other context or screenshots about the feature request here.

support datetime format xlsx uses

Is your feature request related to a problem? Please describe.
The job template lets people put acquisition datetime in human readable form. This should be a supported format.

Add qos=production to production job configs

User story

As a user, I want my jobs to execute with a higher priority, so I don't have to wait as long in the queue.

Acceptance criteria

  • Apparently, job requests tagged with qos=production are given higher priority in the slurm queue

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Dynamically generate job template

User story

As a developer, I want the excel template automatically generated, so I can avoid maintaining the template in a sharepoint folder.

Acceptance criteria

  • Given the service is started or re-started, then a python script will run that automatically generates an xlsx upload template and saves it to a variable. It's generated fast enough that it can just do so at every request for now.
  • Given a user hits an endpoint, job_upload_template, then the upload template will be returned
  • Given a user clicks the JobUploadTemplate link in the ui, then instead of redirecting them to sharepoint, the xlsx will be downloaded to their local computer. (A dialogue may optionally be present)
  • (Follow up about template schema...)

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Original issue

David wrote a capsule: https://codeocean.allenneuraldynamics.org/capsule/8742749/tree

Add an API endpoint that runs this and returns the template for download as an xlsx file

Add instructions to main page

Is your feature request related to a problem? Please describe.
There is very little explanation for how to use the site. This makes it hard for users to know how to submit jobs effectively.

Describe the solution you'd like
On the main page, a clear indication of:

  • A description of what the service is for
  • What sources of metadata the service is pulling information from
  • What the columns in the job template mean and where users can look for controlled vocabularies
    • specifically the metadata field, and how user-supplied JSONs are handled w.r.t. automatically fetched ones.
  • How to submit a job in the UI
  • How to monitor the job in the status page
  • How to submit a job through API instead of through the UI
  • What to do when jobs go wrong (e.g. where to find logs, who to ask for help, how to request that things get deleted).

Notes

  • Maybe this should be stored in a markdown file that is also easy to read when viewed on GitHub?

Use different endpoint to retrieve jobs' status

User story

As a user, I want to see the status of submitted jobs longer, so I can check the job status a few days I've submitted one.

Acceptance criteria

  • When a user checks the job status page, then they can see the status of their job for a few days after it's been completed

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

We can get this info from a different endpoint: "api/slurmdb/v0.0.37/jobs"

Add github action to publish docker image

User story

As a devops engineer, I want to pull the docker image from a registry, so I can use it in a k8s environment easily.

Acceptance criteria

  • When a pull request is merged into main, a docker image will be published to githubs docker image registry

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Get rid of the `preview` button

User story

As a user, I want to see the validation results as soon as a file is selected, so I don't have to click too many buttons.

Acceptance criteria

  • Given a user has selected a file, then the preview or validation errors list will be triggered automatically. Currently, a user selects a file, then they need to click the preview button. If the file is valid, then the user clicks the submit button to submit the request. We'd like to remove the second step if possible.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Original issue

On uploading the sheet, automatically show the preview.

Get rid of the preview button.

Remove Job Button

User story

As a user, I want a button that can remove a specific job from the list of jobs.

Acceptance criteria

  • Given a list of jobs, add a "remove job" button.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Update aind-data-schema version

User story

As a user, I want to use the latest version of aind-data-schema, so I can submit jobs with the latest modality values.

Acceptance criteria

  • Upgrade aind-data-schema to version 0.26.5
  • This may require refactoring pydantic models in the settings class

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Support xlsx job templates

User story

As a user, I want to be able to upload an xlsx file, so I can avoid having to convert an xlsx file to csv before using the service.

Acceptance criteria

  • Given a user uploads a file to the validate_csv endpoint, then it will check the file extension and convert an xlsx sheet to a csv file before validation.
  • Given a user looks at the UI, then it will be clear that they can submit either a csv file or an xlsx sheet.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Authentication?

User story

As an admin, I want users to authenticate when access the app, so I can ensure only trusted actors can submit jobs to the hpc.

Acceptance criteria

  • A user needs to authenticate when accessing the url.
  • (Stretch) Ideally, this should be SSO with allen account

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Add caching to /api/job_upload_template

Is your feature request related to a problem? Please describe.
As a developer, I want to the /api/job_upload_template endpoint to be more efficient so that a new template file is not generated every time.

Describe the solution you'd like
Implement caching for /api/job_upload_template endpoint (and any others as appropriate).

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Note: this feature may not be needed/ not priority since the current implementation is quite fast already.

Add filter by service and upload jobs on job status page

User story

As a user, I want to have the option to filter by service and upload jobs from the status page, so I can better select/view specific jobs

Acceptance criteria

  • Add filter by service and upload jobs

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Rename repo to aind-data-transfer-service

User story

As a developer, I want the repo to be renamed to aind-data-transfer-service, so I can more accurately convey the purpose of the repo.

Acceptance criteria

  • The repository is renamed to aind-data-transfer-service

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Data transfer job status page

When a user submits a job, they need to be able to return to see the status of the job.

Acceptance criteria:

  • A new page called "Job Status", URL http://<service-url>/jobs
  • A link from the submission/home page to the job status page titled "Job Status"
  • On the new jobs page, a table of existing jobs. Columns are:
    • Asset Name = JobStatus.name <-- include asset name in job name please
    • Status = JobStatus.job_state
    • Submit Time = JobStatus.submit_time, formatted in human readable form, local time zone
    • Start Time = "JobStatus.start_time`, formatted in human readable form, local time zone
    • End Time = "JobStatus.end_time`, formatted in human readable form, local time zone
  • By default, show all jobs that are pending, running, or complete within the last week.
  • Table is sorted by submission time, most recent on top
  • Above the table, a line of text that says "Jobs currently running: <N>"

Job Definition Panel

User story

As a user, I want the GUI to have a panel to input a single job definition

Acceptance criteria

  • Given the parameters in aind-data-transfer, create a panel for a single job definition

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Allow user to make themselves owner of data asset on Code Ocean

Is your feature request related to a problem? Please describe.
A service account user registers the data asset on Code Ocean and makes the asset viewable to everyone. However, it will be helpful to add an option to make the person who initiated the upload an owner in case the data asset needs to be archived.

Describe the solution you'd like
Add an option to make the person who initiated the upload an owner of the data asset on Code Ocean

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add api endpoint to allow someone to post hpc configs

User story

As a user, I want to define hpc configs, so I can modify the hpc resources better.

Acceptance criteria

  • There is an endpoint that a user can post json hpc configs.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Job Validation Button

User story

As a user, I want a button that will add a job to a list if it's valid.

Acceptance criteria

  • Write some validation check for a job.
  • Given a valid job, add job to list.
  • Given an invalid job, report error to user.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Improve display of error messages and preview table

Is your feature request related to a problem? Please describe.

  1. As a user, it is confusing when the preview table is not cleared when I upload a new file to validate.
  2. As a user, it is confusing when previous error messages are not cleared when I upload a new correct file to validate.
  3. As a user, it is confusing to understand error messages when they are all displayed in 1 line bunched together.

Example:
image

Describe the solution you'd like

  1. A new table should be displayed rather than rows appended.
  2. Previous error messages should be cleared.
  3. Error messages should be displayed line by line.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Submit Button

User story

As a user, I want to return serialized response when I hit submit.

Acceptance criteria

  • Given a csv file with a list of jobs, when the submit button is clicked, the GUI should return a the serialized csv file.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

`/api/validate_csv` changes broke workflow

Describe the bug
Submitting jobs to the service was working fine at the end of last year, now I'm encountering a problem with /api/validate_csv.

To Reproduce

As per one of the tests:

def test_validate_csv(self):
"""Tests that valid csv file is returned."""
with TestClient(app) as client:
with open(SAMPLE_CSV, "rb") as f:
files = {
"file": f,
}
response = client.post(url="/api/validate_csv", files=files)
expected_jobs = [
j.model_dump_json() for j in self.expected_job_configs
]
expected_response = {
"message": "Valid Data",
"data": {"jobs": expected_jobs, "errors": []},
}
self.assertEqual(response.status_code, 200)
self.assertEqual(expected_response, response.json())

with the following csv:

modality0.source,modality0,s3-bucket,subject-id,platform,modality1.source,modality1,acq-datetime

//allen/programs/mindscope/workgroups/np-exp/codeocean/DRpilot_676909_20231214/ephys,ecephys,aind-ephys-data,676909,ecephys,//allen/programs/mindscope/workgroups/np-exp/codeocean/DRpilot_676909_20231214/behavior-videos,behavior-videos,2023-12-14 12:43:11
import pathlib

import requests


def _raise_for_status(response: requests.Response) -> None:
    """pydantic validation errors are returned as strings that can be eval'd
    to get the real error class + message."""
    if response.status_code != 200:
        try:
            raise eval(response.json()['data']['errors'][0])
        except (KeyError, IndexError, requests.exceptions.JSONDecodeError, SyntaxError) as exc1:
            try:
                response.raise_for_status()
            except requests.exceptions.HTTPError as exc2:
                raise exc2 from exc1

csv_path = pathlib.Path("//allen/programs/mindscope/workgroups/np-exp/codeocean/DRpilot_676909_20231214/upload.csv")
url = "http://aind-data-transfer-service/api/validate_csv"
validate_csv_response = requests.post(url=url, files=dict(file=csv_path.read_bytes()))

_raise_for_status(validate_csv_response)

output:

Traceback (most recent call last):
  File "<stdin>", line 6, in _raise_for_status
  File "<string>", line 1
    Invalid input file type
            ^^^^^
SyntaxError: invalid syntax

The above exception was the direct cause of the following exception: 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 11, in _raise_for_status
  File "<stdin>", line 9, in _raise_for_status
  File "C:\Users\ben.hardcastle\github\np_codeocean\.venv\Lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 406 Client Error: Not Acceptable for url: http://aind-data-transfer-service/api/validate_csv

which seems to originate from here, suggesting a file path should now be provided instead of file contents:

if not form["file"].filename.endswith((".csv", ".xlsx")):
errors.append("Invalid input file type")

However, I'm not sure what the format of the request should be. The following gave a 500 error:

validate_csv_response = requests.post(url=url, json=dict(file=csv_path.as_posix()))

Send a REST request to submit upload job through the HPC

User story

As a user, I want to submit a rest request to the HPC, so I can run a data compression and upload job.

Acceptance criteria

  • When a user uploads a csv file and hits submit, then a rest request will be sent to the HPC cluster to run a compression and upload job.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Github workflow to publish docker image to registry

User story

As a developer, I want the docker image registered, so I can pull it down and run it in a k8s cluster.

Acceptance criteria

  • When a PR is merged into main, then a github action will build a docker image and register it.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Add REST API to handle csv validation and job submission

User story

As a user, I want to submit a csv file for validation and submit jobs via json, so I can use a REST API to run these operations.

Acceptance criteria

  • There are endpoints to submit a csv file and to submit json

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Edit Job Button

User story

As a user, I want to be able to modify an existing job from a list.

Acceptance criteria

  • Given a list of jobs, add a button that will allow the user to modify a job.

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Support CSV import

Entering many jobs will be hard. Let users upload a CSV file and parse it into Json
Acceptance Criteria:

  • "Import Jobs from CSV" button exists
  • Jobs are displayed in the table once imported
  • Any validation/parsing errors show up in a div flagged by row.

Data transfer job preview

After a user uploads a CSV and before they submit a job, they should be able to review the jobs and see any validation error messages.

Acceptance criteria:

  • After a user uploads a csv, display a JSON preview of the jobs in a container/div titled "Job Preview"
  • After a user uploads a csv, if there are errors, display an error message in a separate container titled "Errors"
  • If there are errors, disable the "Submit" button

Dockerfile to run flask app

User story

As a devops engineer, I want to build the app in a docker file, so I can deploy it to a k8s cluster.

Acceptance criteria

  • A dockerfile that will build the app in a docker image
    - [ ] A github action that will publish the docker image to a docker registry We should wait to publish until the app is a better state. Punting this to #4

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Convert `JobUpdateTemplate` class to pydantic model

Is your feature request related to a problem? Please describe.
As a developer, I want all config classes to be consistent and use pydantic models when possible. The JobUpdateTemplate class currently configures and creates the job upload template using a static method. This can be refactored to be a pydantic model and can also ease future conversion to form-based templating.

Describe the solution you'd like
Refactor /configs/job_upload_template.py as a pydantic model.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Prevent duplicate upload jobs

Is your feature request related to a problem? Please describe.
By accident, we managed to submit duplicate upload jobs twice this week: once through the website (not sure how this happened) and once via HTTP request (ran the same request twice).

We then weren't able to cancel the duplicate jobs - for one of them we didn't even realize until Jon alerted us.

In the end, running two uploads of the same data simultaneously didn't cause an issue: the second operation on each file must have seen that the data already existed and skipped it. A duplicate run of the sorting capsule was started after upload, and would have been wasteful had Jon not canceled it.

I can't think of a case where someone would want to upload the same session multiple times simultaneously, so I propose that the server could prevent this from happening.

Describe the solution you'd like
Before allowing a new upload job to be submitted, check there isn't already an upload job for that session in progress.

If there is a reason to allow multiple uploads with the same session ID, then compare csv/job upload parameters instead.

Describe alternatives you've considered

  • user can check on site or via HTTP request.
    Problems with this are:
    • user has to be aware that duplicate upload jobs are possible
    • user has to remember to check every time, or implement their own check in code (resulting in multiple people solving the same problem)
  • "submit job" helper code could be created that has this check built in; everybody uses that.
    Potential problem with this: may trade customizability of current API for usability with a subset of features, and then the full API will be used anyway when extra features are needed

Upgrade aind-data-schema

User story

As a user, I want to use the latest aind-data-schema version

Acceptance criteria

  • Uses latest aind-data-schema version in pyproject.toml file

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Issues submitting csv file from certain environments

Describe the bug
There is an occasional error uploading a csv file generated from excel

To Reproduce
Steps to reproduce the behavior:

  1. On windows, use the template to generate a csv file.
  2. Try to upload the file and hit preview
  3. Occasionally, errors will appears where certain fields contain invisible unicode characters such as \ufeff
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
We can probably fix this by adding an encoding such as f = open('file', mode='r', encoding='utf-8-sig')

More than 2 modalities not supported?

Describe the bug
Trying to upload a session with 3 modalities specified in job csv: ecephys, behavior, behavior-videos

To Reproduce
At http://aind-data-transfer-service/, manually attach the following files:

  • with modality0, modality1:
    • \allen\programs\mindscope\workgroups\np-exp\codeocean\DRpilot_676909_20231214\upload.csv
    • valid
  • with modality0, modality1, modality2:
    • \allen\programs\mindscope\workgroups\np-exp\codeocean\DRpilot_702131_20240226\upload.csv
    • invalid

Support developer consistency across OS/ Windows

Is your feature request related to a problem? Please describe.
As a developer, I want to be able to run the unit tests and have the same experience on Windows compared to if I develop on Linux.

Describe the solution you'd like
Some issues with unit tests have already been resolved in previous PR #62 for path comparisons.

One remaining issue is that the error messages displayed from /api/validate_csv are slightly different depending on OS, which breaks the unit tests if running on Windows. We can pull out the error message text returned by /api/validate_csv using e.Exception or other format rather than directly returning repre(e), and update all unit tests affected.

Example:
Currently expected on Linux: "AttributeError('WRONG_MODALITY_HERE')"
Current expected on Windows: "AttributeError(\"type object 'Modality' has no attribute 'WRONG_MODALITY_HERE'\")"

We should check if there are any other issues related for running on different OS.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

FastAPI App to define data-transfer jobs

User story

As a user, I want to input data using a browser-based GUI, so I can more easily define data-transfer jobs

Acceptance criteria

  • Write FastAPI app for users to input data-transfer jobs
    - [ ] A panel where users can input a single job definition (check aind-data-transfer repo for parameters)
    - [ ] A button that can add job to a list if it's valid
    - [ ] A button that can remove a job from a list
    - [ ] A button that modify an existing job from a list
    - [ ] A submit button that can will log a rest request (we will modify this in the future to send the request somewhere)

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Add Dockerfile to containerize the service

User story

As a developer, I want to containerize the service, so I can run it on a k8s cluster.

Acceptance criteria

  • There should be a Dockerfile in the root directory that runs the service in a container

Sprint Ready Checklist

  • 1. Acceptance criteria defined
  • 2. Team understands acceptance criteria
  • 3. Team has defined solution / steps to satisfy acceptance criteria
  • 4. Acceptance criteria is verifiable / testable
  • 5. External / 3rd Party dependencies identified
  • 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.