mdibl / biocore_utils Goto Github PK

Test the pipeline with different configurations

• General purpose machine
• Memory Optimized machine

Configure a new EC2 Instance with S3 mounts

Write a Bicore-S3 data migration package

Create mounts to S3 buckets from local servers

1. Configure a new EC2 Instance with S3 mounts
2. Create EC2 images with different EBS volumes
3. Create mounts to S3 buckets from local servers
4. Write a Bicore-S3 data migration package
5. Schedule Bicore-S3 buckets data migration to run on Jenkins
6. Test the pipeline with different configurations

• General purpose machine
• Memory Optimized machine

@RoyMcMorranMDIBL , please add this to your requests list - We need a new powerful machine which primary task is to serve as a file server - mapping both local file systems mount points and the AWS file systems mount points - This is where data migration scripts will run from to sync data and software between biocore and the cloud. In addition to your usual installs, I would like the following software installed: AWS Command Line Interface (AWS CLI), s3fs, and EFS

Schedule Bicore-S3 buckets data migration to run on Jenkins

Steps to setup MDIBL's Jenkins master with On-demand and Spot fleet slaves

See:

1. Setup Jenkins Server
2. Launch an Elastic Beanstalk Environment
3. Install s3fs file system software on your Ec2 instance

Create instances of regendb on the cloud - prod, public, and dev.
Create the /data and /opt/software mounts that map
to our s3/biocore-data and s3/biocore-software mounts respectively

See:

1. Create and Connect to a PostgreSQL Database
2. Install phpgadmin:
   a) See: https://www.postgresql.org/download/
   b) See: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html

See:

1. http://calculator.s3.amazonaws.com/index.html
2. https://aws.amazon.com/ebs/
3. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html
4. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-platforms.html

Create EC2 images with different EBS volumes