Skip to content

Commit

Permalink
✨ Detect and report pending pod unschedulable. (#712)
Browse files Browse the repository at this point in the history
Report event with reason why pending pod cannot be scheduled.
Not sure if this should be merged (or not). Troubleshooting why k8s
won't run a pod seems best investigated by snooping around the cluster.
After all ... pods waiting to be scheduled is anticipated (normal) thing
on a busy cluster.
This PR opened only to (potentially) help with troubleshooting broken
CI. It would be better for the CI test to fetch and log task pods stuck
at pending.

Signed-off-by: Jeff Ortel <jortel@redhat.com>
  • Loading branch information
jortel committed Jul 9, 2024
1 parent e674440 commit 8bfd399
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions task/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ const (
ImageError = "ImageError"
PodNotFound = "PodNotFound"
PodCreated = "PodCreated"
PodPending = "PodPending"
PodRunning = "PodRunning"
Preempted = "Preempted"
PodSucceeded = "PodSucceeded"
Expand Down Expand Up @@ -1271,6 +1272,13 @@ func (r *Task) podPending(pod *core.Pod) {
status,
pod.Status.ContainerStatuses...)
started := 0
for _, cnd := range pod.Status.Conditions {
if cnd.Type == core.PodScheduled &&
cnd.Reason == core.PodReasonUnschedulable {
r.Event(PodPending, cnd.Message)
return
}
}
for _, status := range status {
state := status.State
if state.Waiting != nil {
Expand All @@ -1287,6 +1295,8 @@ func (r *Task) podPending(pod *core.Pod) {
r.Event(ImageError, waiting.Reason)
r.State = Failed
return
} else {
r.Event(PodPending, waiting.Reason)
}
}
if status.Started == nil {
Expand Down

0 comments on commit 8bfd399

Please sign in to comment.