example deploy fails `Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)` about datajob HOT 7 OPEN

As mentioned in the README, I clicked "Start Execution" in the GUI and this has run successfully :-)

In the logs of the 3 AWS Glue (1.0) Jobs, I do see the "hello world" message, nice

from datajob.

The None has a capital letter which is invalid.

I ran:

export AWS_DEFAULT_ACCOUNT=_____________29
export AWS_PROFILE=my-profile
export AWS_DEFAULT_REGION=your-region # e.g. eu-west-1

<..>/datajob/examples/data_pipeline_simple$ datajob deploy --config datajob_stack.py 
cdk command: cdk deploy --app  "python <..>/datajob/examples/data_pipeline_simple/datajob_stack.py"  -c stage=None
jsii.errors.JavaScriptError: 
  Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)
  Bucket name must only contain lowercase characters and the symbols, period (.) and dash (-) (offset: 21)

thanks Peter, I just noticed it today that this crashes with my latest changes. if you explicitly pass the stage argument it should work.

datajob deploy --config datajob_stack.py --stage dev

i plan to fix this tomorrow evening.

from datajob.

CloudFormation was successfully deployed. Now trying to run it ...

from datajob.

fyi:

datajob execute --state-machine <your state machine name

to trigger the pipeline should work now

from datajob.

The "destroy" (of the CloudFormation stack and IAM Role) also worked successfully, including the destruction of the CustomCDKBucketDeployment....

Only the CloudWatch log groups are still present. That might be OK.

Not sure if a finite Retention of 3 months or so would be better ?

/aws/lambda/data-pipeline-simple-dev-CustomCDKBucketDeployment-5N8SZ8C1NN9 | Never expire | - | - | -
-- | -- | -- | -- | --
  | /aws/lambda/data-pipeline-simple-dev-deployment-bucketBackend | Never expire | - | - | -
  | /aws/lambda/data-pipeline-simple-devBackend | Never expire | - | - | -

I will stop typing in this Issue.

I suggest to close this issue after either the README is fixed with the --stage dev explanation or some stage (e.g. dev) is set as default.

from datajob.

good point, thanks Peter. we should create the cloudwatch logs as part of the stack

from datajob.

Before you do that (create snd destroy CloudWatch logs as part of CF (CloudFormation) stack): what is the life time of this CF stack? Would you want to see the Glue Job logs longer than the lifetime of this CF stack?

from datajob.