Giter VIP home page Giter VIP logo

Comments (12)

cristim avatar cristim commented on May 12, 2024 1

Yes, that's expected.

You can go ahead and set that tag on the spot requests, autospotting should then take over their instances, however I will try to fix the root cause of the problem so it's not going to happen again.

from autospotting.

machbio avatar machbio commented on May 12, 2024 1

Need to learn, how GO package system works. Compile worked. Thanks.

from autospotting.

roeyazroel avatar roeyazroel commented on May 12, 2024

Maybe wrapping the following code with 'waitForRequestToFulfilled' or some timeout will work:

func (a *autoScalingGroup) bidForSpotInstance(
ls *ec2.RequestSpotLaunchSpecification,
price float64) {

svc := a.region.services.ec2

resp, err := svc.RequestSpotInstances(&ec2.RequestSpotInstancesInput{
    SpotPrice:           aws.String(strconv.FormatFloat(price, 'f', -1, 64)),
    LaunchSpecification: ls,
})

if err != nil {
    logger.Println("Failed to create spot instance request for",
        a.name, err.Error(), ls)
    return
}

spotRequest := resp.SpotInstanceRequests[0]
spotRequestID := spotRequest.SpotInstanceRequestId

logger.Println(a.name, "Created spot instance request", *spotRequestID)

// THIS LINE SHOULD BE WRAP WITH SOME TIMEOUT OR WAITFOR
a.tagSpotInstanceRequest(*spotRequestID)


spotInstanceID := a.waitForSpotInstance(spotRequest)

if spotInstanceID != nil {
    logger.Println(a.name, "found new spot instance", *spotInstanceID,
        "\nTagging it to match the other instances from the group")
    a.region.tagInstance(spotInstanceID, a.filterInstanceTags())
}
}

from autospotting.

cristim avatar cristim commented on May 12, 2024

Good catch, I haven't seen this kind of error before, thanks a lot for reporting it.

But I really can't explain why wouldn't the spot instance request be found.

I just read about the ec2.WaitUntilSpotInstanceRequestFulfilled waiter, I wasn't aware this existed, but using that would make a lot of sense. I'll try to come up with a patch soon.

from autospotting.

roeyazroel avatar roeyazroel commented on May 12, 2024

BTW, the outcome of this issue is that the instance is been created, and gets the right launch configuration of the ASG, but not join to the ASG. So what i have is ECS cluster with 8 instances, the ASG size is 6, and 2 extra that are part of the cluster but not part of the ASG.

from autospotting.

cristim avatar cristim commented on May 12, 2024

@roeyazroel please check and confirm if this problem is fixed for you, since I don't have such a large environment available for testing.

You can use the Travis build number 50 or any later.

from autospotting.

machbio avatar machbio commented on May 12, 2024 

panic: runtime error: invalid memory address or nil pointer dereference

19:18:10
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7ff8a9fe3c48]

19:18:10
goroutine 53 [running]:

19:18:10
github.com/cristim/autospotting/core.(*autoScalingGroup).havingReadyToAttachSpotInstance(0xc42026a2d0, 0xc420689e01, 0x2)

19:18:10
/go/src/github.com/cristim/autospotting/core/autoscaling.go:471 +0xa08

19:18:10
github.com/cristim/autospotting/core.(*autoScalingGroup).process(0xc42026a2d0)

19:18:10
/go/src/github.com/cristim/autospotting/core/autoscaling.go:188 +0x360

19:18:10
github.com/cristim/autospotting/core.(*region).processEnabledAutoScalingGroups.func1(0xc42027dc00, 0xc420333a40, 0xc420329d40, 0x35, 0xc42027dc00, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

19:18:10
/go/src/github.com/cristim/autospotting/core/region.go:321 +0x6f

19:18:10
created by github.com/cristim/autospotting/core.(*region).processEnabledAutoScalingGroups

19:18:10
/go/src/github.com/cristim/autospotting/core/region.go:323 +0x10c


from autospotting.

cristim avatar cristim commented on May 12, 2024

Reopening, it turns out this is still happening to @machbio even with the latest version, as reported on gitter.

from autospotting.

derdrachen avatar derdrachen commented on May 12, 2024

We are having the same issues. here is a log snippet:

2017/06/20 00:38:06 autoscaling.go:548: Bidding for spot instance for use1-prod-ecs-lc-00da598e70d5250d109f08551f-asg
2017/06/20 00:38:06 autoscaling.go:585: use1-prod-ecs-lc-00da598e70d5250d109f08551f-asg Created spot instance request sir-5nbr5xyh
2017/06/20 00:38:06 spot_instance_request.go:89: use1-prod-ecs-lc-00da598e70d5250d109f08551f-asg Failed to create tags for the spot instance request InvalidSpotInstanceRequestID.NotFound: The spot instance request ID 'sir-5nbr5xyh' does not exist
status code: 400, request id: 8acb8e7a-9344-497a-b89a-3817acb8f63b
2017/06/20 00:38:06 autoscaling.go:594: use1-prod-ecs-lc-00da598e70d5250d109f08551f-asg Can't tag spot instance request InvalidSpotInstanceRequestID.NotFound: The spot instance request ID 'sir-5nbr5xyh' does not exist
status code: 400, request id: 8acb8e7a-9344-497a-b89a-3817acb8f63b
2017/06/20 00:38:06.653885 Execution completed, nothing left to do

from autospotting.

cristim avatar cristim commented on May 12, 2024

Looks like the spot request is not yet ready to be tagged,and we should implement a retry logic that would not just quit after the first tagging failure.

from autospotting.

machbio avatar machbio commented on May 12, 2024

Lambda Build Error exists for the Pull request -

$ make -f Makefile.lambda
docker run --rm						\
		-e HANDLER=handler				\
		-e PACKAGE=handler	 			\
		-e BUILD_NUMBER='' 	\
		-v /Users/xxx/godev:/go					\
		-v /Users/xxx/code/projects/project-name/external/autospotting:/tmp			 		\
		-w /tmp					 	\
		eawsy/aws-lambda-go-shim:latest make -f Makefile.lambda all
go build \
	-buildmode=plugin \
	-ldflags="-w -s -pluginpath lambda -X lambda.Version=" \
	-o handler.so
# _/tmp
./autospotting.go:36: cannot use conf.Config (type *autospotting.Config) as type autospotting.Config in argument to autospotting.Run
./autospotting.go:56: unknown field 'SleepMultiplier' in struct literal of type autospotting.Config
make: *** [build] Error 2
make: *** [docker] Error 2

from autospotting.

cristim avatar cristim commented on May 12, 2024

Thanks for reporting this, I will try to reproduce but it may be just because you didn't update the code in your gopath. Try to run go get -u./...

from autospotting.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.