tomharvey / aws-s3cp-bulk Goto Github PK

In https://github.com/tomharvey/aws-s3cp-bulk/blob/master/lib/csv-processor.js#L5 the region is hard coded. Read this from user defaults.

Each invoked copy function will only process one file but is configured with 128MB of RAM, the minimum amount configurable for a Lambda function. To copy a small file, we do not make full use of this available RAM

If each invocation of this function could process multiple files concurrently, then we would make better use of the 128MB of RAM - without dramatically increasing the running time for the function.

This will create a more cost efficient mechanism.

The desired concurrency will depend on the size of the files being processed, and will certainly be far lower than the tens of thousands that the main function is designed to handle.

Add a configureable file_concurrency variable to configure how many files each copy command will handle.

https://github.com/tomharvey/aws-s3cp-bulk/blob/master/lib/invoke.js#L11

We should abstract this out to use the actual stage or default to dev

The tests folder contains a mixture of each. Separate these out and move integration testing to only run on circle-ci.

Add some integration test setup functions, to populate the operational bucket with fixtures in a ./tests/fixtures directory, which is not AWS::AccountId or AWS::Region specific.

Missing functionality in report-summary.write_report_to_s3