This way we save the trouble of downloading a corrupted file, by not fetching associated blocks from storage backend every-time, before failing at a certain point. Thereby, we also lower bandwidth incurred.

Should generate a 400 level error that the file was not found

We should be validating the content length while making requests, so that we dont accept arbitrarily large data for processing. This can be accomplished by adding rules through stoplight for the falcon request object.

The fix should also incorporate tests.

Add webhooks for capturing certain information into other services.

Primary use-case: Validation Service, register/unregister Vaults on creation/deletion

We need to be able to list non-finalized files. Files should also have a ref-modified field on them to support this for the Validation & Cleanup Service variant

Use Cases:

• Client-side cleanup of non-finalized files
• Validation & Cleanup Service: cleanup of non-finalized files

PostgresSQL is releasing functionality that enables master ↔️ master synchronization so that it can behave much like Cassandra and MongoDB but still provide the resilience of a standard SQL database. This configuration of PostgresSQL should be examined to see if it can be used successfully in a cross-dc environment, and specifically whether it would reduce the costs of running Deuce when compared to using Cassandra.

Lets make sure that any repeated calls to the databases are memoized, so that the same queries dont get executed twice.

Its also important that we un-memoize on change of states.

This should give us a nice speed boost

Add the ability to asynchronously delete items

• Add DELETE verb support to the Blocks/Files/StorageBlocks/Vaults end points that takes a JSON list of the items (blockid/fileid/storage block id/vault name respectively) to be deleted.
• All deletions must meet the same requirements as the current non-asynchronous versions

Goal is to speed up certain operations such as multi-block and multi-file deletions; clean-up operations. Primarily thinking of the clean-up service (Deuce-Valere) which could utilize the multi-block/storage block deletions to speed up the cleanup operations.

I came across an issue where doing the following would result in data getting cleaned up improperly:

1) Create a vault
2) Add a file (create, assign, upload, finalize)
3) Run the Valere validation & cleanup (command-line) to get a reference point (should have 1 file, 1 block, both be 'current')
4) Delete the file and block
5) Delete the Vault
6) Repeat Steps 1-3; the current data will be incorrect

My test script (available here: https://github.com/BenjamenMeyer/deuce-tools/blob/master/deucevalere/build_env_exhibit_issue.sh) uploads two files, and creates four orphaned blocks as well. It also uploads a third file and deletes only the file. Output should match the following:

Initial Tally

Post Clean-up

When I first found the issue, the current blocks in step #3 would be zero, despite having 2 existing files that were suppose to have referenced them.

We thought that adding the consistency settings other than Consistency Level One would resolve this; however, that does not seem to be the case.

Areas to explore:

• See if it's an issue in Cassandra itself:
  • Build a series of queries that match up with what the script does
  • Turn on Trace mode in the CQL Shell
  • Run the queries
• See if #272 will solve the problem (it should as it makes each vault uniquely named)
• Figure out if there is something in Deuce that is causing the problem

This is the description of the error when trying to finalize a file without sending the File Size X-File-Length:

""('Missing header', 'The \"x-file-length\" header is required.')""

Looks like we escape around x-file-length more than necessary.

Add a new table/column family to Cassandra to separate the Vault names from internal references. Internally, Vaults should be referenced by a UUID value generated when the Vault is created.

Suggestion:

Vault Name (user specified)
Vault ID (UUID)
Vault Storage Name (some combined form of the other two)

I suggest the storage name so that we can track it easily to the storage layer without necessarily having to query Cassandra.

Current documentation is on Apiary (http://docs.deuce.apiary.io/).
Should documentation be on ReadTheDocs and integrated with the source for better maintenance?

How do Cassandra Schema changes get made in deployments?

os.environ['HOME'] is not valid for certain scenarios when building distributables, namely Python Wheels. Code should be setup to support such distributables.

This would be based upon x-ref-modified header, and you would potentially add a TTL to this, which can be left configurable from the config.ini and validated and transformed into an integer in configspec.ini
Example: x-ref-modified: 4

our TTL per config.ini is 2

then x-expires-at : 6

Remember, this header is only to be returned if the reference count is zero.

The cleanup and validation service should pick the block up after that time and delete it.

Explore how Deuce works with encrypted files using AES CBC mode. We should determine if Deuce can successfully de-duplicate databases encrypted using the scheme in the agent.

Create an endpoint, that would list all the bad blocks per file.

GET v1/vaults/{vaultid}/files/{fileid}/blocks?bad_only=True

pagination should be handled for this and it would be preferable if this information is stored in a separate column family ( since it would be easier to paginate, and also cache)

This is an intermittent issue that only happens when the API Tests are run using Python 3 (3.4).
The test:
test_upload_storage_block (tests.api.tests.test_block_storage.TestBlockUploaded)

fails when trying to verify the headers returned.
The test tries to perform an illegal operation on the Block Storage Endpoint (try to upload/PUT a block).

The response obtained is:

RESPONSE RECEIVED

response status..: <Response [405]>
response time....: 0.14439654350280762
response headers.: {'Content-Length': '0', 'Date': 'Fri, 23 Jan 2015 18:32:31 GMT', 'Via': '1.0 570080-ATL1WWSG01.secops.rackspace.com', 'Connection': 'close', 'Age': '1'}
response body....: b''

Keep in mind that it is an intermittent issue that is only happening when the API Tests runs with Python 3.

Cleanup whitespace from the JSON responses to minimize network traffic

The SQLite Metadata Driver is designed as a reference driver; however, if deploying with it and using multi-processing the deployment will run into issues as the client will improperly get errors, namely due to locks in access of the in-memory SQLite Database.

One method of solving this is to add another SQL-based Database Metadata driver, e.g PostgresSQL.
Intent was deployment on a NoSQL backend.

1. Define the topology of the layout.
2. Create a cluster of metadata servers (Mongo, Cassandra).
3. Create a cluster of Deuce web heads and Load Balancers
4. Run stress tests and find any potential coding faults.

The models and controllers have mixed logic. Logic should be in the models, supports, and drivers as appropriate, not in the controllers.

Check the drivers to ensure that all errors are appropriately generated.

For instance, there are some cases where the Metadata drivers do not raise exceptions when they should.