- Overview
- Installing the Gomon Data Source
- Employing Prometheus, Loki, and Grafana
- Putting it all together
- Visualize
- Notices
The Gomon Data Source measures the state of the system, including its CPU, memory, filesystems, network interfaces, and processes. The Data Source also observes system logs and the system's management of files and processes. Hence, data source processing consists of two fundamental operations: measurement and observation.
The measurements of the system are performed through kernel interfaces that report the state of resources of the system: the CPU, the memory, the filesystems, the network interfaces, and the processes.
The observations of the system are events captured by the logging, file management, and process management subsystems. While not observing these events directly, the Data Source assumes that subsystem reporting is timely. A potential enhancement would be for the data source to initiate its own log, file, and process events to measure periodically the lag between initiation and reporting of these events.
The Gomon Data Source streams the measurements to Prometheus and the observations to Loki to record. In turn, Grafana references Prometheus and Loki to chart the measurements and report the observations. The data source assets folder includes a Grafana Dashboard to graph the streams to illustrate system conditions.
The data source can also format and present the system's inter-process connections in a Grafana Node Graph Panel. Alternatively, if Graphviz is installed, the data source can render the inter-process connections node graph in a web view.
For more information about backend plugins, refer to the documentation on Grafana Backend Plugins.
Building the frontend component depends on the Node runtime. Download the Node installer for your system and install.
Building the backend component depends on Go. Download Go's installer for your system and install.
The plugin’s build and deployment use the Mage build tool. Follow the instructions on the site to install Mage.
Download the Gomon Data Source
git clone https://github.com/zosmac/gomon-datasource
cd gomon-datasourcemage -v frontendNote: backend is the default build.
mage -v [backend]The data source backend plugin is built into the dist subdirectory. Copy the contents to the Grafana plugins directory. The backend’s query of process connections requires root authority. Therefore, the backend module must be owned by root with its setuid permission set.
cp -R dist/ ${PLUGINS_DIR}/zosmac-gomon-datasource
cd ${PLUGINS_DIR}/zosmac-gomon-datasource
sudo chown 0:0 gomon-datasource_$(go env GOOS)_$(go env GOARCH)
sudo chmod u+s gomon-datasource_$(go env GOOS)_$(go env GOARCH)Follow these steps for deploying the three servers that record measurements (Prometheus) and observations (Loki) to facilitate visualization (Grafana).
Prometheus is an open-source systems monitoring server for scraping and storing time-series data. At startup, the data source opens a /metrics endpoint for Prometheus to collect the measurements stream.
To install Prometheus, select an appropriate binary from the Prometheus download page for your Operating System and Architecture, download, and install.
To enable collection, open the prometheus.yml configuration file for the Prometheus deployment and add the following to scrape_configs (a sample configuration file is in assets/prometheus.yml):
- job_name: "gomon-datasource"
scheme: https
tls_config:
ca_file: /etc/ssh/cert.pem
static_configs:
- targets: ["localhost:1234"]Loki is an open-source log aggregation server. Via the data source's HTTP POSTs to the /loki/api/v1/push endpoint, Loki receives the data source observations. To install Loki, create a loki folder, select appropriate loki and promtail binaries from the list of Assets on the Loki releases page for your platform, and download. Each binary also requires a configuration file; follow the instructions on the Loki installation page to copy these to the loki folder.
cd ${LOKI_DIR}
unzip =(curl -L "https://github.com/grafana/loki/releases/latest/download/loki-$(go env GOOS)-$(go env GOARCH).zip")
unzip =(curl -L "https://github.com/grafana/loki/releases/latest/download/promtail-$(go env GOOS)-$(go env GOARCH).zip")
chmod a+x loki-$(go env GOOS)-$(go env GOARCH) promtail-$(go env GOOS)-$(go env GOARCH)
curl -O -L "https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml"
curl -O -L "https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml"To install Grafana, select an appropriate binary from the Grafana download page for your platform, download, and install.
cd ${PROMETHEUS_DIR}
./prometheus >>prometheus.log 2>&1 &cd ${LOKI_DIR}
./loki-$(go env GOOS)-$(go env GOARCH) -config.file loki-local-config.yaml >>loki.log 2>&1 &cd ${GRAFANA_DIR}
GF_PATHS_PLUGINS=${PLUGINS_DIR} GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=zosmac-gomon-datasource bin/grafana-server web >>grafana.log 2>&1 &Add Prometheus and Loki Data Sources to Grafana
Note: while the Data Source's Settings Panel shows the default data source URL, you must still type it into the URL field. Then, scroll down and select Save & test.
Note: while the Data Source's Settings Panel shows the default data source URL, you must still type it into the URL field. Then, scroll down and select Save & test.
Install the Gomon Data Source Dashboard to Grafana
curl ${GRAFANA_CRED} -X POST -i -w "\n" -H "Content-Type: application/json" -T ./assets/dashboard.json "http://localhost:3000/api/dashboards/db"
If Graphviz is installed, Gomon can render a node graph of the inter-process and remote host connections via the /gomon endpoint:
To download and install Graphviz, select a stable release, download its tar file, build, and install.
Note: gomon-datasource specifies -Tsvgz to the dot command. Ensure that the zlib development library is installed on your system, e.g. on Ubuntu sudo apt install zlib1g-dev, on Fedora sudo yum install zlib devel.
tar xzvf =(curl -L "https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/7.1.0/graphviz-7.1.0.tar.gz")
cd graphviz-7.1.0
./configure
make
sudo make installNote: installing from the git repository (i.e. git clone https://gitlab.com/graphviz/graphviz/) requires pre-configuration with autogen.sh, which in turn requires GNU autoconf, automake, and libtool. Find details at https://graphviz.org/download/source/#git-repos
Copyright © 2021-2023 The Gomon Project.