[BUG] Investigate memory leak in single-binary deployment

[eapolinario]

Describe the bug

A user in the OSS slack reported that single-binary pods keep getting killed due to OOM errors. Here's a graph of memory consumption in this case (different colors refer to different pods):

They are running stock Flyte v1.8.1 release.

Expected behavior

Memory consumption should remain relatively constant.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[wild-endeavor]

look at prometheus metrics, educated guess from dan - every time a task runs, prometheus metrics in propeller might be contributing to memory usage.

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏

[cpaulik]

This looks a lot like what I'm seeing as well. See https://flyte-org.slack.com/archives/CP2HDHKE1/p1723819039330009 for more discussion.

Is there a way to disable prometheus metrics to test this? I haven't found a configuration option yet...

would deploying via flyte-core help here? Since this issue is specifically for single-binary deployment.

[eapolinario]

After some investigation, it turns out that accumulating metrics data is the expected behavior of the prometheus golang client, as per prometheus/client_golang#920. We emit high-cardinality metrics in flytepropeller, as an example you can see here the emitted metrics after running a few hundred workflow executions. Note how we maintain a lot of metrics per execution id. This is the symptom described in the prometheus client gh issue.

One could argue that emitting metrics per-execution id is the wrong granularity, but that's exactly what we do in the case of single-binary.

would deploying via flyte-core help here?

Knowing what we know now, the set of default metrics emitted by flytepropeller in the default cause does not cause the issue. Similar argument can be made to flyteadmin. In other words, this was a single-binary issue-only.

Here's the PR to remove that label from single binary metrics: #5704

[wild-endeavor]

Related to #5606

[divyank000]

This has solved the issue at a shorter time horizon. However I still see the issue on the longer time horizon.
initially for us, it was around few hours on production workload. Now it is around 2 days on production workload.