User story

As a user, I want to be able to download credentials from aws, so I don't have to manage it locally.

Acceptance criteria

• The credentials should try to pull from AWS secrets manager if they are not explicitly set or found in env vars.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a db maintainer, I want a method to remove records, so I can maintain the database.

Acceptance criteria

• Given a user creates a docdb client, then there is a method that can be invoked where the user can delete a record by its id.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want write requests to be chunked, so I can write a long list of records without any errors.

Acceptance criteria

• When a user upserts a long list of records, the writes will be chunked

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to be able to install doc store and rds dependencies separately, so I can have a smaller package footprint.

Acceptance criteria

• A user can run pip install aind-data-access-api[doc-store-client] to just install pymongo db etc.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want an api to connect to the document db, so I can read and write there.

Acceptance criteria

• Given a user has the proper credentials, when they use this api, then they should be able to read and maybe write to the document db.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to be able to write to a relational database, so I can build front-end apps to view the data easily.

Acceptance criteria

• A client that can read/write to a relational database (Postgres to start)
• A method that will write a pandas data-frame out as a table.
• If the table does not exist, it will create the table by inferring the schema from the pandas schema.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want the option to paginate read results, so I can avoid resource api gateway resource limits

Acceptance criteria

• A user can pass start, limit, paginate, and record_count into the get_records method
• The retrieve_data_asset_records pulls down records in chunks

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

when I pip install, I am missing pydantic, pymongo, sqlalchemy, ...

User story

As a user, I want to create a boto3 client using a session token, so I can use the api with a token instead of credentials.

Acceptance criteria

• If a user supplies a session token, then the boto3 client will be created using the token instead of aws credentials.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

When I try to pip install aind-data-access-api[full] s3fs, it takes over an hour attempting to resolve dependencies, and unclear if it will finish. I suspect how we are handling boto is what makes this difficult.

User story

As a user, I want to use the latest version of pydantic, so I can use this alongside other packages that require >=2.0

Acceptance criteria

• Update pydantic dependency to >=2.0

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to be able to read/write from a document store, so I can process metadata stored there.

Acceptance criteria

• A package with a client that connects to a mongodb.
• A method that can retrieve a collection of documents with some filters.
• Appropriate tests and docs

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want an api to connect to the redshift db, so I can read and write there.

Acceptance criteria

• Given a user has the proper credentials, when they use this api, then they should be able to read and maybe write to the redshift db.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a developer, I want a single repository managing our data access layer, so I can easily import the packages where they're needed.

Acceptance criteria

• API to list/update data assets in s3
• API to list/update data assets registered on CodeOcean
• API to list/update data assets registered to DocumentDB

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want a method that allows me to run a generic sql query, so I can have finer control over things if I want.

Acceptance criteria

• There should be a convenient method to run a generic sql query.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to better error handlers, so I can know what went wrong with a request.

Acceptance criteria

• Add better error handling in the clients.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to pull from pypi, so I can easily install the package.

Acceptance criteria

• Update docs with installation and usage instructions
• Add github action to publish to pypi

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want a library to access secrets, so I can import it in other projects.

Acceptance criteria

• A user can call a get_secret(secret_name) method that will retrieve a secret if the user has the proper aws permissions
• A user can call a get_parameter(parameter_name) method that will retrieve a parameter if the user has the proper aws permissions (not as strict as the permissions required for a secret)
• The dependencies should be self-contained, i.e., the user can run pip install aind-data-access-api[secrets] to just import the external library boto3 essentially.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

User story

As a user, I want to retrieve schemas from the document store, so I can use them.

Acceptance criteria

• There should be a method called retrieve_schema_records

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Maybe this isn't necessary? This can be achieved by:

doc_store_creds = DocumentStoreCredentials(aws_secrets_name="*****")
doc_store_client = DsClient(credentials=doc_store_creds, collection_name="procedures")
proc_schemas = list(doc_store_client.collection.find())

User story

As a user, I want the MongoDB client to optionally set the retryWrites to False, so I can write to AWS DocDB.

Acceptance criteria

• A user can set the option of retryWrites to False when constructing the MongoDB client.

Sprint Ready Checklist

• 1. Acceptance criteria defined
• 2. Team understands acceptance criteria
• 3. Team has defined solution / steps to satisfy acceptance criteria
• 4. Acceptance criteria is verifiable / testable
• 5. External / 3rd Party dependencies identified
• 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.