AWS Docker Autoscaler: error with Docker Autoscaler Executor and EC2 Instance Connect

[vpotap]

I am encountering an occasional issue while using the Docker Autoscaler executor. Sometimes, the following errors are logged, indicating problems with EC2 Instance Connect and the state of the EC2 instance:

Preparing the "docker-autoscaler" executor
ERROR: Failed to remove network for build
ERROR: Preparation failed: error during connect: Get "http://internel.tunnel.invalid/v1.24/info": preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: <here-id>, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance. (docker.go:779:0s)
Will be retried in 3s ...
ERROR: Failed to remove network for build
ERROR: Preparation failed: error during connect: Get "http://internel.tunnel.invalid/v1.24/info": preparing environment: getting instance connect info: refreshing connect info: rpc error: code = Unknown desc = sending ssh key: operation error EC2 Instance Connect: SendSSHPublicKey, https response error StatusCode: 400, RequestID: <here-id>, EC2InstanceStateInvalidException: Unable to connect to the instance. EC2 Instance Connect is not supported on a terminated instance. (docker.go:779:0s)

The issue occurs occasionally, and re-running the job typically resolves it.

[Tiduster]

I can confirm we have the same errors occasionally on our own setup.

The bug is on docker autoscaler side unfortunately, nothing that can be done from this module from my understanding of this issue.

Relevant issue on the official project repo: https://gitlab.com/gitlab-org/fleeting/plugins/aws/-/issues/52