how does ec2-fleet resolve the known_hosts issue? about ec2-fleet-plugin HOT 5 CLOSED

I disable known hosts verification. The instances are fired up in EC2 in a private VPC and picked up only if they are part of the EC2 fleet, they run a custom cloud-init script (user-data) that installs the Jenkins public key.

You could also create a custom image in AWS and use that for launching which already has the Jenkins user and key installed.

from ec2-fleet-plugin.

Got it, thanks @bertjwregeer . I think I still prefer being able to verify, but I don't see how. It would be nice in that respect if ssh worked like full ssl, with an x509 cert signed by a common authority. Then I wouldn't need the actual host pubkey in known_hosts, just the cert of the signing authority. Oh well.

It actually has taken me a bit to "turn my head around" on this. Normally, when deploying scaling apps on aws, I use an autoscaling group, and have new "worker" nodes self-join a cluster. In the Jenkins case, I would use ASG (or possibly spot fleet) to spin up new workers and have them join Jenkins.

The challenges I found with that approach are:

1. It isn't sufficient to just "connect"; I need to "add" the new node using the Jenkins CLI or REST API as a "permanent" node, and then either retrieve the jnlp agent-specific secret and join, or use ssh (which creates the same issue with host key, although possibly more easily solvable if I am initiating from the host which already knows its host key). Solvable.
2. Deletion of hosts isn't automatic. If I have a node created by an ASG, it then might be terminated. Instead of Jenkins removing it (as happens, e.g. with Kubernetes and cloud-provider set to AWS), it lingers as an unreachable host. I could create a clean-up job, but that is just more custom work.
3. Deletion of hosts isn't "Jenkins-aware", i.e. it might delete a host that has a running job. Of course, the same thing can happen with spot fleet as well, so not as challenging.

The ec2-fleet plugin pretty much solves 2 & 3, and I can set the userdata or custom AMI (or use docker with custom OCI image) to resolve 1, as you mentioned.

Still and all, doing it from the Jenkins Server via a plugin, rather than from ASG, is a little different.

Any way to use jnlp instead of ssh for the spot fleet nodes?

from ec2-fleet-plugin.

@deitch SSH CAN work like that though. See https://jameshfisher.com/2018/03/16/how-to-create-an-ssh-certificate-authority.html for a guide on how to setup/configure that.

from ec2-fleet-plugin.

SSH CAN work like that though

Quite! (and I am facepalming myself for not thinking of it...; thanks @bertjwregeer :-) )

Does the ssh implementation on the server side in jenkins ssh-slave support it? Or is it just exec-ing out to the on-host ssh client anyways?

from ec2-fleet-plugin.

Mind, there is not-insignificant work to do to automate this process: agent starts up, generates a host key, that in turn needs to be signed, which means it needs some way to reach a CA and be trusted that it is valid, and around again...

Last time I did this was (as mentioned above) for k8s workers. They have a "pre-kube" startup phase that uses a secret key injected via userdata to reach a CA and request signing. Not perfect, but it works.

from ec2-fleet-plugin.