General 1st Interaction Feedback about cmda HOT 31 CLOSED

You're saying this works:

cmda sh -c 'echo hello > /mnt/efs/someremotefile.txt'
cmda ls -l /mnt/efs/someremotefile.txt

But this doesn't:

echo hello > hello.txt
cmda upload hello.txt /mnt/efs/
cmda ls -l /mnt/efs/hello.txt

from cmda.

Interesting! I wonder where the Rate Exceeded is coming from.

Mine's in Bursting Mode too – and haven't run into this yet.

Can you copy files at all? Like even tiny ones?

from cmda.

I wonder if it's coming from Lambda? But that would suggest you're invoking the function a lot

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-troubleshoot-throttling/

What's the output of cmda info btw?

from cmda.

Ah, interesting – I've encountered a scenario where it gets into a bit of a loop with large files and retries. I think it's related to this: aws/aws-sdk-js#3311

Will play around a little more

from cmda.

So I tried setting the concurrency to 5 and that just made things worse and slower with nothing happening. I took it back down to 1 again and tried a single file ~130KB and that's not going well now either. But I'll keep debugging with something small and go from there so I can report back.

from cmda.

A few details about my setup that may be useful to know. I am using both the AWS_PROFILE and CMDA_FUNCTION environment variable. Maybe CMDA_FUNCTION env is not hooked up in all places?

from cmda.

Ok, try installing version v1.0.1 of the CLI and see how that goes (shouldn't need to update your stack or anything)

(npm install -g cmda@latest if you've installed it globally)

from cmda.

I suspect it's an issue to do with the function retrying the upload too many times if it gets corrupted. There's something up with the aws-sdk where it doesn't fully finish uploads in certain cases (which is why I opened that issue). That means that the tarball is corrupt, and cmda retries when it encounters such situations.

In my original testing it usually worked on the second or third upload, so I didn't have any max retries in there – I suspect you were getting throttled because it kept retrying over and over.

Have now added max retries, and also updated to my patched version of the aws sdk, so hopefully that fixes your issue. Otherwise it's something else and we need to keep exploring!

from cmda.

I did notice when I downloaded the tarball from S3 to my Mac I could not decompress it. Trying 1.0.1 now.

from cmda.

Updated and verified with cmda info. Still not working, my code is.

echo hello > hello.txt
cmda upload ./hello.txt /mnt/mydir/

I have verified that the file on S3 can be downloaded and uncompressed and is what I expect. Another aspect of my setup is that I am using the lambci/lambda:build-nodejs12.x container with shared volumes of .:/var/task:delegated and ~/.aws:/root/.aws:delegated which I do often. So far uploading is the only issue. I'll try a download.

from cmda.

Wait, what? You're doing this in docker?

from cmda.

Oh I see, not running the Lambda function in docker – just using the build container as a sandbox to install and run the cmda cli?

from cmda.

Correct! Keeps developers and CI/CD pipelines from having to use ad-hoc platform tools. BTW, I just verified that a download of a created text file via echo does not work. Never even makes the file on S3.

from cmda.

How did you create your cmda stack? And what version is it? functionVersion in info

from cmda.

Btw, when you say "still not working" do you mean you're still getting the same "Rate Exceeded" error?

from cmda.

ClickOps via the app service catalog. serverlessrepo:semanticVersion | 1.0.0 Guess I should tear that down and reset it up?

No, not working is just hanging here. Invoking Lambda to copy files remotely..

from cmda.

I want to hook up some sort of feedback mechanism, but it's tricky without creating some sort of websocket or something to communicate with – I think best strategy is tailing the logs and looking for a shared identifier, but would still be behind by a bit.

So just to reiterate where we're at – you're still unable to upload any files, even tiny ones?

from cmda.

Btw, the Invoking Lambda to copy files remotely message can hang for quite a while if it's a number of files, especially if there are symlinks. EFS is slowwwww in such cases, and it just takes a while to write them

from cmda.

Correct to the above on what works and what does not.

Btw, the Invoking Lambda to copy files remotely message can hang for quite a while if it's a number of files

So this is just for the one hello.txt like the example above. I just deleted the cmda and am reinstalling it from the serverless app repo. Still looks like v1.0.0 on there. Is only the npm package 1.0.1?

from cmda.

Sorry, not making progress, the simplest upload times out after 5 minutes.

echo hello > hello.txt
cmda upload ./hello.txt /mnt/mydir/

Invoking Lambda to copy files remotely...Rate Exceeded.

from cmda.

Damn. Wish I could reproduce. No idea what that Rate Exceeded could be coming from.

Could it be a permissions thing maybe?

The files are being uploaded to the S3 bucket with upload, yeah? But you're saying they don't even make it to S3 with download?

I just uploaded 1.0.1 to SAR as well – just in case it makes a difference (can't see how it would for upload, but it might for download)

from cmda.

The files are being uploaded to the S3 bucket with upload, yeah?

Yup.

But you're saying they don't even make it to S3 with download?

Correct.

I'm at a loss of things to try. I did fiddle with my security group some more. It did have ALL on since it was a dev test. Not sure what else to try. I know I don't want to use an EC2 instance and DataSync, so I might just write a quick Ruby Lambda that takes all files from an S3 bucket and syncs to EFS for this pipeline I'm doing. Would be willing to try other things if you had thoughts too.

from cmda.

Ah, it could be a VPC issue – ie, the Lambda can't access S3!

That would make sense I think because I've only tested from VPCs that have a NAT Gateway 🤦‍♂️

I'll see how to add a VPC endpoint for Lambda -> S3... hopefully the CFN isn't too painful

from cmda.

So I think you can do it like this:

aws ec2 create-vpc-endpoint --vpc-id vpc-1a2b3c4d --route-table-ids rtb-11aa22bb --service-name com.amazonaws.us-east-1.s3

I'll need to setup a new VPC to verify, but if you want to try that (just try using the main route table id for the VPC)

from cmda.

You can get the main route table like this:

aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-1a2b3c4d Name=association.main,Values=true --query 'RouteTables[].RouteTableId'

from cmda.

Hmmm, maybe the main route table isn't the right one – maybe you have to use the route tables associated with the vpc subnets connected to the Lambda. God why is this stuff so complicated

from cmda.

Thanks. I would not put too much stock into this. I think I can fix my VPC.

from cmda.

Still, would be nice to catch this and warn the user – and add documentation around it. At the least.

from cmda.

Perhaps. I'm just bad at networking and using EFS just showed me that. My work initially used the public subnets which did not have a route to S3. I've since switched to private subnets and got things working. Good call out too on that corrupt stream data too. I certainly saw that at one time.

Thanks again. I'll close this out now.

from cmda.

Cheers – I've published v1.0.2 that displays a nice error if it detects the Lambda can't communicate with S3.

Next step is looking to see if I can create a command that can automatically give your VPC access if you don't have it already.

from cmda.

Published v1.1.0 with a command to add a VPC endpoint (cmda create-vpc-endpoint) that should give the VPC access to the S3 bucket. This is also suggested if cmda detects that S3 isn't reachable by the Lambda function

from cmda.