FLINK PROMETHEUS SOURCE

STATUS

WHAT IS THIS FOR?

The aim of this project is to create a source which can receive prometheus metrics with prometheus standard, transform into a model and move them to a Flink stream which is used on your ETL process.

Up to now the way to receive such metrics is through an endpoint which is listening in Task Manager and body content is compressed in snappy.

Wikipedia - Snappy

So, a pre-requisite to use this source is to expose a port in which this service attends requests.

HOW TO USE IT

Note	This dependency is still under development and it is not deployed in any repository to include into your proyect, so to use it, you have to download this repository and install it in your local repo as mvn clean install from directory prometheus-source-flink.

In order to use this dependency into your flink app, you have to include it in your pom.xml as:

<dependency>
    <groupId>com.github.ivanas93</groupId>
    <artifactId>prometheus-source-flink</artifactId>
    <version>1.0.0</version>
</dependency>

Also in your configuration params you have to specify which port and path you want to use to attend requests with metrics.

app.source.remote-write.port=9000
app.source.remote-write.path=/api/v1/remote_write

where app.source.remote-write may vary as this is a configurable prefix used when the source is created.

DataStream<TimeSeries> timeSeriesStream = env.addSource(new RemoteReadSource("app.source.remote-write", config));

As a result the DataStream<TimeSeries> contains the metrics with TimeSeries model, which is what you can adapt to your pipe.

This TimeSeries looks like:

public class TimeSeries implements Serializable {
    private List<Label> labels;
    private List<Sample> samples;

    //....
}

Where the Sample model contains each of those metric sample lines in the metric, with its timestamp.

public class Sample implements Serializable {
    private String metricName;
    private Map<String, String> labels;
    private Double sample;
    private long timestamp;

    //...
}

WHY TO USE IT

Prometheus metrics work with a specific model which has been depict in proto and this project has made use of these resources to build models and it might be a convoluted task to develop and test each place where they are required.

HOW TO CONTRIBUTE

We are more than welcome with help in any aspect: by opening issues, reviewing documentation, giving feedbacks to improve in any way or resolving any of the open issues.

Also an example project is next to this project implementation to have a close insight about how to use it. Feel free to add the use-cases which might help or explain a behaviour.

DISCOURAGE

This is still under development, but we are using it in a real project and so far so good.