Acceptance criteria:

• get_data_assets
• a method called search_data_assets
• exposes all parameters of the data asset API
• if start and limit are passed, return the requested number of assets
• if start and limit are not passed, return all matching assets (make multiple requests)

Notes:
Returning all the data assets will be handled in #14

Describe the bug
Code Ocean made some significant changes regarding how data assets are registered. The arguments for aws access keys have been removed and will return Internal Server Errors when using the current version of aind-codeocean-api.

To Reproduce
Steps to reproduce the behavior:

1. In the test env, try to register a data asset from a private bucket.
2. Notice the error message in the response.

Expected behavior
We should be able to use the REST API to register data assets.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

User story

As a user, I want to use different versions of pydantic, so I can install what is required.

Acceptance criteria

• Unpin pydantic

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

There is now a custom_metadata field in the data asset API. Let's update a couple methods:

1. register_result_as_data_asset
2. update_data_asset

User story

As a developer, I want to use enums instead of strings, so I can avoid writing strings and ensure that the requests are being sent properly.

Acceptance criteria

• The API fields that code ocean uses (such as 'has_more' and 'results' etc.) should be defined using an Enum class.
• The URL strings should be defined using an Enum class.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to ensure connections are being closed, so that resources are being used efficiently.

Acceptance criteria

• All the requests responses should be wrapped into a with statement

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

Describe the bug
Occasionally, duplicate results may return. The search_all_data_assets method uses pagination to return responses in chunks of 1000. We may need to double-check that.

To Reproduce
Hard to reproduce since it isn't consistent. Steps to reproduce the behavior:

1. Create a CodeOceanClient, co_client
2. Run response = co_client.search_all_data_assets()
3. Check the response
4. Can set log_level to info to see the exact queries being sent to the code ocean server.

Expected behavior
Each data asset in the results should be unique.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

User story

As a user, I want to specify a capsule version to run, so I can run against stable versions more easily.

Acceptance criteria

• There should be an optional version field in the run_capsule method

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Example curl:

curl -H "Content-Type: application/json" -u ${CUSTOM_KEY}: -X POST https://codeocean.acme.com/api/v1/computations --data-raw '{
    "capsule_id":"123456-abcdef",
    "version": 1,
    "data_assets": [{"id": "78910-ghifh", "mount": "a_mount_location"}]
}'

There has been a change to Update Data Asset Permissions endpoint which requires reconfiguration of current implementations of this endpoint: the “Everyone” parameter changed from true/false to viewer/none.

Update this

Is your feature request related to a problem? Please describe.
I'm fixing some results assets which contain some wrong metadata using a capsule that generates the new corrected results.
I'd like to push capture the new asset with the same name as the previous one, and automatically archive/delete the old asset.

I'm doing this fix for ~100 assets, so it would be nice to have a way to script this.

Describe the solution you'd like
A co_client.archive(data_asset_id) function that:

1. checks if the asset is attached to any capsule and
2. archives it if not attached, raise if attached (so it could be detached manually)

Describe alternatives you've considered
Doing this manually, but I feel that we might need this in the future for similar interventions

Additional context
None

Describe the bug
Several warnings show up in Pycharm:

• Function 'register_result_as_data_asset' does not have a parameter 'keep_on_external_storage'
• Function 'register_result_as_data_asset' does not have a parameter 'index_data'
• Default argument value is mutable
• Default argument value is mutable

To Reproduce
Steps to reproduce the behavior:

1. Open file in Pycharm
2. See warnings

Expected behavior
The code should be clean as possible and the warnings should be fixed.

Additional context
All of these should be easy to fix.

I just issued a search_data_assets query and notice that the actual URI used was:

https://codeocean.allenneuraldynamics.org/api/v1/data_assets?self=%3Caind_codeocean_api.codeocean.CodeOceanClient+object+at+0x7f2e79763f70%3E&start=0&limit=10&ownership=owner

Expected Behavior

self should not appear in the URI.

Is your feature request related to a problem? Please describe.
This repo is using outdated github workflows

Describe the solution you'd like
Use the new reusable workflows to control versioning and linter

Describe alternatives you've considered
Leave as is; same frustrating version check and same old linter

User story

As a user, I want to call a single method to return all data assets matching a query, so I don't have to call a method multiple times.

Acceptance criteria

• There should be a method that makes multiple requests (if necessary) to the code ocean backend to retrieve all data assets using the search data asset api.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Should wait until #10 is finished first to make this easier

User story

As a user, I want the option to create a credentials file, so I can retrieve my credentials easily.

Acceptance criteria

• A runnable python module that prompts the user in the command prompt to input location (with a default to $HOME/.codeocean/credentials.json), domain, and token.
• A json file with domain and token is created at that location.
• A user can load the credentials using the CodeOceanCredentials class

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to to store/retrieve credentials in aws secrets manager, so I don't have to worry about storing them on my local machine.

Acceptance criteria

• Add a method to optionally construct the CodeOceanCredentials class from a secret store in aws secrets manager.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a DevOps engineer, I want to create a test suite, so I can check/test codeocean's api after each upgrade

Acceptance criteria

• Ideally a python script that can be run and checks api integrations

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

parellel to #50

User story

As a user, I want to have a readme, so I can understand how to install and run the software.

Acceptance criteria

• Clean-up the Readme file
• Run sphinx to generate doc strings.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want an api to change permissions on a data asset, so I can avoid manually changing permissions in the ui.

Acceptance criteria

• There should be method in the python client called update_permissions that alters the permissions of a data asset.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Can use the following examples:

#!/usr/bin/env bash

# curl -X POST \
# 'https://acmecorp-edge.codeocean.com/api/v1/data_assets/abc-123/permissions' \
# -u $API_KEY: -H 'Content-Type: application/json' \
# --data-raw '{
# 	"users": [ {"email": "[email protected]", "role": "owner"} ] ,
# 	"groups": [ {"group": "ad-group", "role": "viewer"} ] ,
# 	"everyone": true
# }'

#role: owner or viewer
curl -X POST \
'https://acmecorp-edge.codeocean.com/api/v1/data_assets/abc-123/permissions' \
-u $API_KEY: -H 'Content-Type: application/json' \
--data-raw '{
	"users": [ {"email": "[email protected]", "role": "owner"} ] ,
	"everyone": true
}'

When the client issues a query, add a logging message to print it to the console.

User story

As a developer, I want github actions to run, so I can catch bugs, tag versions, and publish to pypi.

Acceptance criteria

• When a PR is opened, then a github action to check the version number, run tests, and linters will execute.
• When a PR is merged into main, then a new tag is created, and the package is published to pypi.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a developer, I want to have one location for the codeocean python api, so I can maintain it better.

Acceptance criteria

• The code/tests from nd-data-transfer is moved into this repo.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want a cleaner interface, so I can implement the classes easier.

Acceptance criteria

• There should be a central client class instead of multiple classes.

Sprint Ready Checklist

1. • Acceptance criteria defined
2. • Team understands acceptance criteria
3. • Team has defined solution / steps to satisfy acceptance criteria
4. • Acceptance criteria is verifiable / testable
5. • External / 3rd Party dependencies identified
6. • Ticket is prioritized and sized

Notes

Add any helpful notes here.

This is primarily to enable us to move data from a private codeocean bucket to our open data bucket.

Acceptance Criteria:

• move_data_asset
• Moves data to a new bucket
• New prefix/path is the same as the name of the asset
• No assets are created or deleted in the process
• No provenance is lost on the data asset record
• Asset is marked as "external"

Note: it's possible this does not work with the existing API.

Code Ocean is building this feature, blocking for now

This works:

domain = "https://some.domain.org"
co_client = CodeOceanClient(domain=domain, token=token)

This does not:

domain = "https://some.domain.org/"
co_client = CodeOceanClient(domain=domain, token=token)

The error doesn't manifest until you actually query (e.g. co_client.search_all_data_assets()) and the error is inscrutable.

Describe the bug
I'm trying to use search_all_data_assets to find all of the assets with a certain name prefix. This is leading to a bug.

To Reproduce

for result in co_client.search_all_data_assets(query="name:SmartSPIM"):
    print(result)

yields

Traceback (most recent call last):
  File "/home/david.feng/projects/aind-data-migration-scripts/scripts/2023-03-13/update_asset_permissions.py", line 8, in <module>
    for result in co_client.search_all_data_assets(query="name:SmartSPIM"):
  File "/home/david.feng/miniconda3/envs/wrangle/lib/python3.10/site-packages/requests/models.py", line 828, in generate
    chunk = self.raw.read(chunk_size)
AttributeError: 'NoneType' object has no attribute 'read'

Expected behavior
This should iterate through the results and print them. This search works with the co_client.search_data_assets method, so I would expect to be able to copy-paste it to search_all_data_assets.