api: support native histograms in api/v1/metrics

[jan--f]

Another piece for #515.

An example return:

[beorn7]

This looks perfect.

Could you amend the test in api_test.go so that it exercises a native histogram (or maybe even two, an integer one and a float one).

[beorn7]

A marginally related thing: The protobuf format supports MetricType GAUGE_HISTOGRAM by now, for which we never added support to the PGW. (There can be classic histograms that are gauge histograms.)

It should be easy to add the support for both classic and native histograms, now that you are on it…

[beorn7]

And another thought: IIRC this API is only used for the UI. (There might be users that use it for something else, but I would rather expect them to use the /metrics output directly.) In the UI, I think we should aim for a representation like in the Prometheus UI, which is an explicit description of the individual buckets with absolute population, upper and lower bound. We could argue that we don't need to do this conversion in the UI code because we already have an API exclusively serving the UI, so that API could already render this representation, in the most extreme case just as a string, as created by the histogram's String() method. This would simplify the UI code.

A less extreme approach would be to let the API render the buckets individually, but with upper/lower bound and absolute population (similar to the Prometheus query API). This would still allow a reasonably simple UI code, while the API output would still lend itself to some data processing.

The approach so far in this PR is also fine, but it would mean that the UI would show the "internal" histogram representation (with delta population, schema, spans), or you would need to recreate a lot of histogram logic in the UI code.

[beorn7]

After more thinking:

I keep talking about the UI above, because I have forgotten again that we still don't use the API to render the UI in the PGW (cf. #596).

While we do have a vague plan to eventually use the API to render the UI, the current usage pattern of the API are more relevant for the decision how to format native histograms. However, I'm not sure what those usage patterns are. If users want a representation close to the exposition format or close to the internal representation in the TSDB, the current approach in this PR is a good fit. However, if users use the API to render some kind of UI themselves, a representation closer to what the Prometheus query API is doing would be much better. The latter would also help with that future plan of rendering the PGW UI based on the API response. My gut feeling is we should do the latter, but I could easily be convinced otherwise.

For reference, this is the format in the Prometheus query API: https://prometheus.io/docs/prometheus/latest/querying/api/#native-histograms

[jan--f]

@beorn7 I added code for human readable histograms.

While I fix the tests, I was wondering about a few things:

• I'm not sure about the naming of the histogram package in pushgateway, happy to rename this to something more sensible.
• The func GetReadable[Float]Buckets(h *model.Histogram) []string functions could live in prometheus/model/histogram, went with this for expediency.
• For the api endpoint I needed to add the buckets field (since we're in JSON land). The Prometheus code doesn't have this, as it formats a string directly.

[beorn7]

Thank you very much.

I think the naming or layout of the histogram package is not critical. It can be easily refactored if needed. I think what you did serves its purpose.

However, the string formatting for a whole bucket in GetReadable is rubbing me the wrong way. That would be an entirely new way of an API return. I would prefer to follow what the Prometheus query API is doing, as linked above.

Here is a truncated example from a Prometheus query API response:

            "buckets": [
              [
                1,
                "-0.0000152587890625",
                "-0.00000762939453125",
                "42"
              ],
              [
                1,
                "-0.000003814697265625",
                "-0.0000019073486328125",
                "2"
              ],
              [
                0,
                "9.5367431640625e-07",
                "0.0000019073486328125",
                "145"
              ],
              [
                0,
                "0.000003814697265625",
                "0.00000762939453125",
                "11"
              ],
            ]

I think this is more useful than your "full string" approach because it allows users that are interested in the data (rather than just rendering a UI) to easily access bucket boundaries and bucket counts. (For example, Grafana is using this format to render heatmaps based on Prometheus histograms.) At the same time, it will be very easy to render a UI from that. With a few line of JavaScript or whatever you are using in your UI, you can render what GetReadable is returning now. And if you want to be fancier, you could also render a graphical representation of the histogram etc. (which is hard from the opaque string that GetReadable creates).

You can look at the code that lives over in prometheus/prometheus to see how this is rendered:
https://github.com/prometheus/prometheus/blob/aba007148057c1947122b18b2ad606883cc27220/util/jsonutil/marshal.go#L66-L137

(It's using jsoniter, so not directly re-usable, but it can still serve as inspiration.)

[jan--f]

Ah my apologies, I totally missed that aspect. Will update, thank you!

[jan--f]

Apologies for the long lag yet again.
The API endpoint now returns buckets like the Prometheus query API.

      "test_hist": {
        "time_stamp": "2024-03-15T16:16:28.843966226+01:00",
        "type": "HISTOGRAM",
        "help": "Just a histogram test",
        "metrics": [
          {
            "buckets": [
              [
                0,
                "17.448123722644123",
                "19.027313840043536",
                "139"
              ],
              [
                0,
                "19.027313840043536",
                "20.749432874416154",
                "85"
              ],
              [
                0,
                "20.749432874416154",
                "22.62741699796952",
                "70"
              ],

[jan--f]

This and GetAPIFloatBuckets are exported function since I intend to use them for the web ui changes.

[beorn7]

Looks great at first glance. I'll have a detailed look in the coming week.

[beorn7]

Looks great, thank you.