New Observability Architecture for Tracing/Metrics Events

[karlem]

Epic: Implement New Observability Architecture with Top-Down and Bottom-Up Tracing/Metrics Events

Description

We need to implement a new observability architecture that leverages top-down and bottom-up tracing and metrics events, as outlined in the observability spec draft. This enhancement aims to improve our system's monitoring and debugging capabilities by providing more detailed and structured insights.

This epic will track the implementation through several smaller issues, ensuring a structured and efficient development process.

Tasks Breakdown

1. Emit Top-Down Traces #1054

• Design and implement the top-down tracing logic.
• Ensure all relevant events are captured and emitted correctly.
• Test that the subscriber works as expected.

2. Emit Bottom-Up Traces #1056

• Design and implement the bottom-up tracing logic.
• Ensure all relevant events are captured and emitted correctly.
• Test that the subscriber works as expected.

3. Introduce Subscriber Configuration #1057

• Implement the configuration for the subscriber.
• Ensure dynamic reconfiguration is supported.
• Test the configuration thoroughly.

4. Pamper the Config API #1058

• Properly implement the configuration API.
• Design and implement filter mechanisms for domains and specific events.
• Test the filtering functionality to ensure it works as intended.

5. Finalize Remaining Emits #1059

• Complete the implementation of all remaining emits as outlined in the spec.
• Ensure consistency and accuracy across all tracing and metrics events.

6. Test and Validate the New Architecture

• Perform comprehensive testing to validate the new observability features.
• Ensure both the journal and metrics sinks receive data as expected.
• Validate the performance impact of the new observability logic.

Acceptance Criteria

• All top-down and bottom-up traces are implemented and emitted correctly.
• The subscriber configuration is implemented and supports dynamic changes.
• The configuration API is robust and supports necessary filtering mechanisms.
• Comprehensive testing is completed, ensuring the new observability features work as intended.
• Documentation is updated to reflect changes in observability methods and tools.

This epic is part of a larger observability enhancement initiative. Each task will be tracked as a separate issue to ensure focused development and thorough testing.

[raulk]

The bulk of this is done, and we have a bunch of follow-ups in various issues and PRs like #1058 #1090.