Remove scheduler waits to speed up recovery time

[pierDipi]

Currently, the scheduler and autoscaler are single threads and use a lock to prevent multiple scheduling and autoscaling decision from happening in parallel; this is not a problem for our use cases, however, the multiple wait currently present are slowing down recovery time.

From my testing, if I delete and recreate the Kafka control plane and data plane (sort of simulates an upgrade), without this patch it takes hours to recover when there are 400 triggers or 20 minutes when there are 100 triggers; with the patch it is immediate (only a 2/3 minutes with 400 triggers).

• Remove waits from state builder and autoscaler
• Add additional debug logs
• Use logger provided through the context as opposed to gloabal loggers in each individual component to preserve knative/pkg resource aware log keys.

---

Before with 200 triggers:

Before with 1024 triggers, we see a very high work queue depth for 3 hours (knative controller queue size)

---

After with various number of triggers

Work queue depth is not high for long periods

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

Attention: Patch coverage is 66.40625% with 43 lines in your changes missing coverage. Please review.

Project coverage is 66.56%. Comparing base (e79f3b6) to head (03fd6ae).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/scheduler/statefulset/autoscaler.go	48.83%	19 Missing and 3 partials ⚠️
pkg/scheduler/state/state.go	56.25%	9 Missing and 5 partials ⚠️
pkg/scheduler/statefulset/scheduler.go	89.58%	5 Missing ⚠️
pkg/scheduler/state/helpers.go	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8200      +/-   ##
==========================================
- Coverage   67.47%   66.56%   -0.91%     
==========================================
  Files         371      371              
  Lines       18036    18271     +235     
==========================================
- Hits        12169    12162       -7     
- Misses       5088     5324     +236     
- Partials      779      785       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pierDipi]

/test unit-tests

[matzew]

/lgtm

I like the extra added logging here as well

[pierDipi]

/test reconciler-tests

[pierDipi]

/cherry-pick release-1.15

[knative-prow-robot]

@pierDipi: once the present PR merges, I will cherry-pick it on top of release-1.15 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pierDipi]

/cherry-pick release-1.14

[knative-prow-robot]

@pierDipi: once the present PR merges, I will cherry-pick it on top of release-1.14 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[knative-prow-robot]

@pierDipi: #8200 failed to apply on top of branch "release-1.14":

Applying: Remove scheduler `wait`s to speed up recovery time
Using index info to reconstruct a base tree...
M	pkg/scheduler/scheduler.go
M	pkg/scheduler/scheduler_test.go
M	pkg/scheduler/state/helpers.go
M	pkg/scheduler/state/state.go
M	pkg/scheduler/state/state_test.go
M	pkg/scheduler/statefulset/autoscaler.go
M	pkg/scheduler/statefulset/autoscaler_test.go
M	pkg/scheduler/statefulset/scheduler.go
M	pkg/scheduler/statefulset/scheduler_test.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/scheduler/statefulset/scheduler_test.go
Auto-merging pkg/scheduler/statefulset/scheduler.go
CONFLICT (content): Merge conflict in pkg/scheduler/statefulset/scheduler.go
Auto-merging pkg/scheduler/statefulset/autoscaler_test.go
Auto-merging pkg/scheduler/statefulset/autoscaler.go
Auto-merging pkg/scheduler/state/state_test.go
Auto-merging pkg/scheduler/state/state.go
Auto-merging pkg/scheduler/state/helpers.go
Auto-merging pkg/scheduler/scheduler_test.go
Auto-merging pkg/scheduler/scheduler.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Remove scheduler `wait`s to speed up recovery time
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[knative-prow-robot]

@pierDipi: #8200 failed to apply on top of branch "release-1.15":

Applying: Remove scheduler `wait`s to speed up recovery time
Using index info to reconstruct a base tree...
M	pkg/scheduler/statefulset/scheduler.go
M	pkg/scheduler/statefulset/scheduler_test.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/scheduler/statefulset/scheduler_test.go
Auto-merging pkg/scheduler/statefulset/scheduler.go
CONFLICT (content): Merge conflict in pkg/scheduler/statefulset/scheduler.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Remove scheduler `wait`s to speed up recovery time
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.