-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
check instance lifecycle before starting the work #75
Conversation
Co-authored-by: Zeke Hunter-Green <zeke.huntergreen@guardian.co.uk>
Co-authored-by: Zeke Hunter-Green <zeke.huntergreen@guardian.co.uk>
(i) => i.InstanceId === instanceId, | ||
)?.LifecycleState; | ||
if (lifecycleState === undefined) | ||
throw new Error('Could not find instance lifecycle state!'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe include curly braces https://softwareengineering.stackexchange.com/questions/16528/single-statement-if-block-braces-or-no
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed - this felt wrong
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work!
paired with @zekehuntergreen
What does this change?
We found several instances that had been launched several days previously without being scaled in. This delayed subsequent transcription jobs because the ASG capacity manager wouldn't update desired capacity.
The related logs for these instances were showing that the service had gone into failed state because of exception when trying to update scaling protection.
AutoScalingServiceException [ValidationError]: The instance <instanse-id> is not in InService or EnteringStandby or Standby.
Looking at AWS documentation, the instance can only go into
Standby
&EnteringStandby
states through user interaction. So most probably, the state of the instance in our case wasPending
.In this change
InService
, it won't continue the work, but rather waits for 30 seconds.How to test