awslabs / fargatecli Goto Github PK

what

• It would be great if all commands supported a --wait flag

why

• Allow the operator to chain commands together (E.g. via a Makefile)

example

fargate certificate request $(DOMAIN) --alias $(HOSTNAME) --wait
fargate certificate validate $(DOMAIN) --wait
fargate lb create atlantis \
		--cluster $(CLUSTER) \
		--certificate $(DOMAIN) \
		--port HTTPS:443 \
                 --wait
fargate lb alias atlantis $(HOSTNAME) --wait
fargate service create atlantis \
		--cluster $(CLUSTER) \
		--lb atlantis \
		--num 1 \
		--cpu 256 \
		--memory 2048 \
		--port "HTTP:80" \
		--rule "PATH=/*" \
		--env "TEST=123" \
		--image nginx:latest \
                 --wait

anyway to add custom health checks configurations to loadbalancer create cmd?

We are using graphql so everything is under /graphql vs /

Testing out this CLI interface for Fargate, and from following along with the video there are a couple of errors I don't understand.

For starters, I am specifying the exact command line from the video but get this error:

$ fargate service create app --port HTTP:8080 --lb test-app --num 2
[!] Invalid load balancer and protocol
network load balancer web-app only supports TCP

This is a Docker instance launched from the private ECR registry:

$ fargate task run --image 123456.dkr.ecr.us-east-1.amazonaws.com/test-app/test-app --subnet-id subnet-123456a --subnet-id subnet-123456b -v

I am also getting different errors based upon the ports I am specifying, for example:

$  fargate lb create test-app --port 8080 --certificate example.com
[i] Created load balancer test-app
$ fargate lb destroy test-app
[i] Destroyed load balancer test-app
$  fargate lb create test-app --port 443 --certificate example.com
[!] Could not create ELB load balancer
ValidationError: At least two subnets in two different Availability Zones must be specified
        status code: 400, request id: 1234-567-8912345

The test-app was launched with two subnets attached to different availability zones...?

Also, how does one go about launching two separate tasks within separate availability zones? I would assume that would be a requirement for the application load balancer? My goal is being able to use an alias so that https://example.com -> application load balancer -> Fargate tasks running web application

Thanks in advance, your project will be a big help if I can get it working properly!

We need the elb to be internal vs internet-facing. Could there be a way to specify the scheme in this cmd fargate lb create $ELB_NAME --scheme 'internal' ?

If you destroy an lb before destroying a service that is configured within that lb, it'll orphan the service's target group which causes the CLI to throw an exception when you attempt to destroy the service.

This CLI looks pretty cool! However, if I was to think of or do a google search for Fargate, I'd probably end up at the AWS product (to which this speaks) rather than this very useful CLI. Would you consider renaming the project to something more distinctive or at least descriptive (fargatecli?) to reduce confusion with the "actual" Fargate?

Right now, fargate service restart will create a new task definition to force a deployment of the service. UpdateService has a mechanism to force a new deployment and this seems much cleaner than bumping the task definition with no changes.

I have a service running, created with this command:
fargate service create web -v --lb fargate --port 3000

$ fargate service list
NAME	IMAGE								CPU	MEMORY	LOAD BALANCER	DESIRED	RUNNING	PENDING
web	651426287273.dkr.ecr.us-east-1.amazonaws.com/web:20180102224503	256	512	fargate		1	1	0

Notice the image tag is date based, rather than being git-sha based.

yet I'm in a git repo:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

See https://twitter.com/brandonmblack/status/948664769971662849 for use case.

I was looking forward to share data volumes between the fargate cluster. So far I've been able to find this, but have you tried to employ the same on your fargate-cli.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html

Ok based on your previous response I was able to successfully create a load balancer on port 8080 and then create a service with an alias that mapped http://example.com:8080 -> lb -> two Fargate tasks. Awesome!

So then I destroyed the load balancer and the service, and then went back to create the load balancer again but this time specifying port 443, and got this error message:

$ fargate lb create testapp --certificate example.com --port 443 --verbose
[d] Creating ECS cluster
[d] Created ECS cluster fargate
[d] Creating ELB load balancer
[d] Creating ELB target group
[!] Could not create ELB target group
DuplicateTargetGroupName: A target group with the same name 'testapp-default' exists, but with different settings
        status code: 400, request id: 1234-2345-345678

My current architecture is just a Docker image on the backend listening on port 8080 (no SSL currently). I am trying to create a load balancer that will accept connections on 443 with our Amazon Route 53 SSL certs, and then forward those requests to the Fargate Docker tasks that have exposed port 8080.

Is this a possible scenario or have anything to do with this error message above? Do the external application load balancer ports have to map to the Fargate task ports with a 1:1 mapping, e.g. lb:443 -> task:443?

Is it possible to create/deploy a service with multiple ports exposed?
I think it would have to add additional targets with the addition ports to the target group that is created.

Until sidecar containers are supported, or even after they're supported, it would be useful to enable X-Ray integration using a flag. This could be something during service create command.

Link points to -> https://github.com/jpignata/fargate/releases/download/v0.1.1/fargate-0.1.1-darwin-amd64.zip

Should point to -> https://github.com/jpignata/fargate/releases/download/v0.1.1/fargate--darwin-amd64.zip

I'm trying to create a LB for my ECS Service, but I'm stuck here:

fargate lb create name --port 8000 --subnet-id subnet-a218a38d
[!] Could not find EC2 security group

fargate lb create name --port 8000 --security-group-id sg-d1d0cca5 --subnet-id subnet-a218a38d
[!] Security groups can only be specified for HTTP/HTTPS load balancers

I've read somewhere that LBs require two subnets, but providing them in those commands doesn't help either.

Am I missing something? Is there a workaround?
Thanks!

If we enter:

fargate service env set myapp --env MY_VAR=has,comma

The result is:

[!] Invalid environment variable
comma must be in the form of KEY=value

This is an issue because I'm trying to supply a mongodb string, ie:

MONGO_URL=mongodb://user:pass@host1,host2/db?auth=123&key=456

Can the value portion be escaped somehow?

Yet another question :)

I have an ECR container that handles document processing, and the goal is to have an S3 upload of a document trigger a Lambda function, which in turn launches a parameterized Fargate task using the name of the document uploaded to S3.

I have an Amazon Linux development AMI setup where I could compile and include the fargate CLI within the Lambda bundle. So the million dollar question is, could your fargate CLI be used to launch a parameterized Fargate task using the name of the file uploaded to S3, perhaps using environment variables such as those in the advanced configuration options for Fargate task definitions?

Having issues with automatic installation on CI/CD platform.
I can download the zip file via curl but am unable to successful extract it End-of-central-directory signature not found.
So i had to download it through the UI and check in the executable in order to get it successfully installed on CI/CD platform.

It would be super awesome if you could provide an "official" homebrew tap for fargate.

This way it would be very nice and simple to install fargate on macOS without having to have know anything about how to compile go or downloading things from GitHub releases.

You can do that e.g. by using https://github.com/goreleaser/goreleaser. Many options are available and I'd be happy to help if that's something you'd consider, @jpignata.

Encapsulate the "happy path" of request and validate in one command

On the first run got this problem:

[!] Could not run ECS task
InvalidParameterException: Unable to assume the service linked role. Please verify that the ECS service linked role exists.
status code: 400, request id: 83d5f2a7-12bf-11e8-8f9d-3fc6a73cfa3d

Fixed using the guide bellow:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_IAM_role.html

ps: Nice project. Congratz! 🎉

According to https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ there are 4 regions where fargate is supported now:
Northern Virginia | Ohio | Oregon | Ireland
I made changes locally and everything works for me. I can share my changes - diff is attached.
diff --git a/README.md b/README.md
index 3ba3401..ab67bb1 100644
--- a/README.md
+++ b/README.md
@@ -13,8 +13,8 @@

Region

-By default, fargate uses us-east-1. Also available us-west-2, us-east-2, eu-west-1.
-The CLI accepts a --region parameter for future use and
+By default, fargate uses us-east-1 as this is the single region where AWS
+Fargate is available. The CLI accepts a --region parameter for future use and
will honor AWS_REGION and AWS_DEFAULT_REGION environment settings. Note that
specifying a region where all required services aren't available will return an
error.
diff --git a/cmd/root.go b/cmd/root.go
index bca26b2..994ed22 100644
--- a/cmd/root.go
+++ b/cmd/root.go
@@ -45,7 +45,7 @@ CPU (CPU Units) Memory (MiB)
4096 8192 through 30720 in 1GiB increments
`)

-var validRegions = []string{"us-east-1", "us-west-2", "us-east-2", "eu-west-1"}
+var validRegions = []string{"us-east-1"}

var (
clusterName string
@@ -105,16 +105,12 @@ CloudWatch Logs, and Amazon Route 53 into an easy-to-use CLI.`,
}
}

•           region_is_valid := false
            for _, validRegion := range validRegions {
                    if region == validRegion {
  

•                           region_is_valid = true
                            break
                    }
  

•           }
  

•           if !region_is_valid {
  

•                   console.IssueExit("Invalid region: '%s' [valid regions: %s]", region, strings.Join(validRegions, ", "))
  

•                   console.IssueExit("Invalid region: %s [valid regions: %s]", region, strings.Join(validRegions, ", "))
            }
  
            config := &aws.Config{
  

Digging around the code, I could not find a way to override this setting when creating services.

Is it possible to deploy services without a public IP?

Awesome tool! JSON output would be really useful for scripting.

Hello,

Thank you for the CLI. It seems super awesome so far.

Question.. Do I have to have the cluster already defined/created if I do not want to use the default Fargate cluster?

I am currently receiving ClusterNotFoundException if I do
fargate --cluster new_cluster task run long_running

Thanks
G

what

• Last release was cut in january

why

• One month ago, cli was updated to support new regions; current release does not support us-west-2
• Most users will use the binary release rather than build their own from source

references

• #62 (comment)

I ran fargate task list.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x166dafa]

goroutine 1 [running]:
github.com/jpignata/fargate/ecs.(*ECS).CreateCluster(0xc0000bd220, 0x1a7f1f0, 0x37, 0x0, 0x0)
        /Users/john/go/src/github.com/jpignata/fargate/ecs/cluster.go:15 +0xaa
github.com/jpignata/fargate/cmd.glob..func12(0x22fa6e0, 0x231e190, 0x0, 0x0)
        /Users/john/go/src/github.com/jpignata/fargate/cmd/root.go:153 +0x3ff
github.com/spf13/cobra.(*Command).execute(0x22fa6e0, 0x231e190, 0x0, 0x0, 0x22fa6e0, 0x231e190)
        /Users/john/go/src/github.com/spf13/cobra/command.go:746 +0x242
github.com/spf13/cobra.(*Command).ExecuteC(0x22f7c20, 0xc00013ff88, 0x10071f0, 0xc000090058)
        /Users/john/go/src/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x22f7c20, 0x0, 0x0)
        /Users/john/go/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/jpignata/fargate/cmd.Execute()
        /Users/john/go/src/github.com/jpignata/fargate/cmd/root.go:166 +0x56
main.main()
        /Users/john/git/bitfield/fargate/main.go:8 +0x20

I have been successfully using fargate for a while now, an awesome tool by the way!

However, yesterday it started throwing an error as if my credentials were not set up right:

[!] Could not create Cloudwatch Logs log group
SubscriptionRequiredException: The AWS Access Key Id needs a subscription for the service
	status code: 400, request id: 7f6464c9-16fe-11e8-b99d-2d585cda5bb0

I checked that the permissions on the AWS account did not change and the environment variables are available (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) in the shell.

Since I have not changed anything in the environment where I ran the commands, I'm not sure what went wrong. Do you have any idea where else could the issue lie?

Thanks for making this great tool. I have a question I'm hoping will have a simple answer. I'm experimenting with this tool trying to set up a number of HTTP REST APIs. I am able to make requests from a client through the load balancer just fine, using the DNS name from the load balancer, but I'm not sure how I can make requests between services.

I have services (servicea, serviceb) and their containers are exposing ports 8001 and 8002. I would like servicea to be able to make HTTP requests to serviceb without having to go through the load balancer. I've tried using e.g. http://localhost:8002 and http://serviceb:8002 and neither one of those work (ENOTFOUND) is there any way to set it up so that the services can reach each other over HTTP using information that is available at the time the containers are built?

Thanks for your time!

So we don't have a "default" VPC. (deleted it)
Need the ability to input the "awsvpc networking" as a command line option.

Hi, Can you post your IAM roles required for running a task by a user. I seem to have caught an error. Heres the error

[!] Couldn't register ECS task definition
AccessDeniedException: User: arn:aws:iam::*******:user/<username> is not author                                                                                                                     ized to perform: iam:PassRole on resource: arn:aws:iam::*********:role/ecsTas                                                                                                                     kExecutionRole
        status code: 400, request id: f31c6334-03e8-11e8-aaf9-1f6bfe37bc80

I am sure there will be more errors after i give the required IAMroles. Since, I'm not admin. I would like to know all the required IAM roles for running a docker on fargate.

BTW, great project! loved it

Basic task run:
InvalidParameterException: subnets can not be empty.
Basic task run with subnets:
InvalidParameterException: security group cannot be blank

From @nathanpeck:

I notice that whenever I try to vertically scale a service using the CLI I get a response like this:

fargate service update locations --cpu 1024 --memory 2048
[!] Could not describe ECS task definition
InvalidParameterException: Task Definition can not be blank.
	status code: 400, request id: 7884accc-f729-11e7-ba7a-c7393d0e382f

I got that error when using AWS_PROFILE variable:
[!] Could not find your AWS credentials

aws cli working as expected.

FarGate is currently expensive compared to EC2, so we can expect that, after getting something configured correctly using FarGate, a user might want to migrate it to EC2.

Are you aware of any scripts for doing this auto-magically, and if not, might this be within the scope of this project? thanks :-)

Got some basic zsh completion working using babysnakes PR (spf13/cobra#646)

https://github.com/rsteube/fargate-completion

It would be great to enable service auto-scaling based upon CloudWatch events. For example, CPU or memory usage reaching a certain threshold.

See this post for implementation details.

Hey @jpignata, excellent work on this project! Thank you.

Question: what would you think about adding the ability to deploy application images and environment variables from a docker-compose.yml file? This would make it a bit more declarative and easier to use. You could also docker-compose up to run locally and then fargate deploy -f docker-compose.yml to be confident that you deployed the same config that you tested locally. Would you accept a PR for this?

version: "2"
services:
  my-app:
    build: .
    image: 618440173281.dkr.ecr.us-east-1.amazonaws.com/my-app:1.0
    ports:
    - 80:3000
    environment:
      FOO: bar
      BAR: baz

fargate --cluster fargate-poc service deploy -f docker-compose.yml

It would be nice if we could schedule a task via this cli?

I have a task which is made up of 2 containers.
From the task I can see the container id for each container.
Is it possible to get docker stats for a container within a task?
CloudWatch only shows me CPU utilization at the service level.

I apologize in advance if this is not the right forum for questions, I didn't see any reference to a mailing list on your github.

I have a Continuous Integration pipeline in place for automating the build process for our Golang-based Docker images. Basically whenever there is any type of a code commit to the web application framework, new Go binaries are built followed by an updated Dockerfile and with all of those changes pushed to the Amazon private ECR where our Fargate tasks are cancelled and then relaunched with the latest Docker ECR image.

Does your fargate CLI have any support yet for detecting when a Fargate task image has been updated, so that those running tasks can be cancelled and then re-launched (preferably one at a time within a load balancer so as not to disrupt the application)?

Or should I just integrate the fargate CLI into the task reload process once the updated Docker image has been pushed to ECR?

Thanks in advance!

Ok here's another question. Each time I run a task:

$ fargate task run testapp --verbose --image 1234567890.dkr.ecr.us-east-1.amazonaws.com/testapp

...the task definition for testapp is incrementing with each run (e.g. testapp:1, testapp:2 etc).

Is this standard behavior to create a new task definition with each call? Is there any way to just use testapp:1 for each invocation of fargate task run?

When running fargate service env list -- i.e. missing the name of the service that I wanted to list the env variables for -- I got the following error:

$ fargate service env list
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/jpignata/fargate/cmd.glob..func16(0x1f424c0, 0x1f68320, 0x0, 0x0)
	/Users/jp/workspace/go/src/github.com/jpignata/fargate/cmd/service_env_list.go:18 +0x6c
github.com/spf13/cobra.(*Command).execute(0x1f424c0, 0x1f68320, 0x0, 0x0, 0x1f424c0, 0x1f68320)
	/Users/jp/workspace/go/src/github.com/spf13/cobra/command.go:750 +0x2c1
github.com/spf13/cobra.(*Command).ExecuteC(0x1f41740, 0xc42001e0b8, 0x0, 0xc420014f70)
	/Users/jp/workspace/go/src/github.com/spf13/cobra/command.go:831 +0x30e
github.com/spf13/cobra.(*Command).Execute(0x1f41740, 0x0, 0x0)
	/Users/jp/workspace/go/src/github.com/spf13/cobra/command.go:784 +0x2b
github.com/jpignata/fargate/cmd.Execute()
	/Users/jp/workspace/go/src/github.com/jpignata/fargate/cmd/root.go:149 +0x5e
main.main()
	/Users/jp/workspace/go/src/github.com/jpignata/fargate/main.go:8 +0x20

In comparison, running fargate service info (again missing service name), gives this much more helpful error message:

Error: accepts 1 arg(s), received 0
Usage:
  fargate service info <service-name> [flags]

Flags:
  -h, --help   help for info

Global Flags:
      --cluster string   ECS cluster name (default "fargate")
      --no-color         Disable color output
      --region string    AWS region (default "us-east-1")
  -v, --verbose          Verbose output

Would be good if the first was the same as the second.

Please allow the cli to update the service to increase the desired task number after the service has been created.
Right now, we can only set the desired task numbers upon creation for services.

what

• fargate certificate validate picks wrong zone

why

• Presumably it picks the first zone matching the substring, rather than picking the longest matching zone based on substring