Dashboards

SCSB application has the following dashboards for monitoring

1. SCSB Applications General Metrics

2. SCSB Docker Container Metrics

3. SCSB System monitoring Metrics

Architecture

Open

SCSB monitoring requires 3 components in this monitoring system setup:

1.SCSB applications running as containers

2.Prometheus (gather application’s metrics)

3.Grafana (for visualisation of metrics data)

Logical Flow

Open

Prometheus is a pull-based system unlike the ELK stack where its a push-based system. Pull based means Prometheus actively pulls data from your application (by scrapping of your configured HTTP endpoint). In case of a Spring Boot app this would be your /metrics endpoint that comes as part of Spring Boot's Actuator. So you can see how easily Prometheus works really well with a Spring Boot app.

Prometheus stores the scrapped data in its own TSDB (Time Series Database).

Lastly, Prometheus connects up to Grafana a visualisation tool (like Kibana from the ELK stack) for viewing dashboard of your gathered metrics data.

From there, you can view and analyse the performance of your Spring Boot web apps.

Setup your application to be able to expose the metrics so Prometheus can scrape it (i.e. provide metrics Prometheus requires) With a Spring Boot application this is very easy as all you need to do is add Spring Boot Actuator starter pack and configure in your configuration file to expose those data on the actuator endpoints (/metrics) thereby allowing Prometheus to pull from it (i.e. scrape it)

Very first step is to add in the required dependencies to your application:

1. Spring Boot Actuator

2. Micrometer-Registry-Prometheus

Prometheus: Metrics Collector

Prometheus is a free and open source system monitoring tool that stores all its data in a time series database. You can easily fine-tune the definition of your metrics and generate more accurate reports as well because it offers a powerful query language and multidimensional data. It helps you graphing the resulting data on dashboards. Then Grafana can visualize them in various graphs and charts. To query the matrixes from Prometheus, it provides a new query language called PromQL. We can define the PromQL queries with Grafana to get data from Prometheus.

Grafana is a leading graph and dashboard builder visualizing time series infrastructure and application metrics. It allows you to create alerts, notifications, and ad-hoc filters for your data.

The Prometheus server is responsible for collecting and storing metrics as well as processing expression queries and generating alerts. There are so many patterns available for organizing the storage of metrics. SCSB use the Docker image’s default behavior of using a Docker data volume to store the metrics.

Node Exporter

Promethus provides a way to monitory third party applications and services with Exporters. Exporters act as side-car to third party application/services. They collect data from third party applications/services and expose them with HTTP endpoint which Prometheus can scrape. There are various exporters available with Prometheus. The available exporters can be find from here. The most common exporter which is used by SCSB is node exporter, which can be installed on every server to read system level metrics such as cpu, memory, file system etc.

 SCSB also uses cAdvisor (Container Advisor) which provides  users an understanding of the resource usage and performance characteristics of the running containers in the server. It is a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container, it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide. cAdvisor has native support for Docker containers and should support just about any other container type out of the box.

Adding Prometheus as a Grafana Data Source

We will now add Prometheus to Grafana data source, importing the Prometheus stats dashboard and analyzing the data.

Let’s configure Grafana to access Prometheus server as the data source for creating graphs. If in case you have more than one Prometheus server running then you will need to configure this for every Prometheus server because each Prometheus server is a separate data source. Open up your web browser and visit http://YourServerIP:<grafana-port>/ and login to your account.

To visualize data on Grafana, first need to add data source. In this case Prometheus is the data source. Following are the steps to add the data source in Grafana.

Once you logged in, click on the Grafana icon in the top left corner to show the main menu. Select Data Sources to navigate to the data source list page. Click Add New in the top navigation bar. You should see the following:

Create a new data source using the following values.

1.Name: Prometheus

2.Type: Prometheus

3.URL: http://promethus-host:xxxx (default port is 9090)

4.Access: Browser

5.Basic Auth: According to Your Server

Finally, click on Add button to add your data source, and then click Test Connection to verify everything is working properly.