Monitor Spring Boot App with Micrometer and Prometheus

Modern distributed applications typically have multiple microservices working together. Ability to monitor and manage aspects like health, metrics, logging etc. are key to run the applications in production. In this article, we will explore the concepts around Spring Actuator, Micrometer and Prometheus to Monitor Spring Boot applications.

Spring Boot Actuator

Reference: Spring Production-ready Features

Spring Boot includes a number of features to help monitor and manage applications in production via HTTP endpoints or JMX. Auditing, health, and metrics gathering can be automatically applied to applications.

The spring-boot-actuator module provides all of Spring Boot’s production-ready features. The recommended way to enable the features is to add a dependency on the spring-boot-starter-actuator “Starter”.

To add the actuator to a Maven-based project, add the following ‘Starter’ dependency to pom.xml.

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
</dependencies>

Actuator endpoints let you monitor and interact with your application. Spring Boot includes a number of built-in endpoints. For example, the health endpoint provides basic application health information. Most applications choose exposure over HTTP, where the ID of the endpoint and a prefix of /actuator is mapped to a URL. For example, by default, the health endpoint is mapped to /actuator/health.

An endpoint is considered to be available when it is both enabled and exposed. To configure the enablement of an endpoint, use its management.endpoint.<id>.enabled property in application.properties. By default, all endpoints except for shutdown are enabled.

management.endpoint.info.enabled=true

Since Endpoints may contain sensitive information, you should carefully consider when to expose them. For example, to stop exposing all endpoints over HTTP and only expose the health, metrics and loggers endpoints, use the following property in application.properties:

management.endpoints.web.exposure.include=health,metrics,loggers

Let us look at few examples to understand the Spring Boot actuator.

Configure log level of Spring Boot app at runtime

To change the log level of a Spring Boot application during run time, we can use the endpoint "/actuator/loggers". This is useful when you want to collect more detailed logs to debug production issues for a specific period of time.

To change the log level of root logger:

curl --location --request POST 'http://localhost:8080/actuator/loggers/ROOT' \
--header 'Content-Type: application/json' \
--data-raw '{
    "configuredLevel": "DEBUG"
}'

To view the list of all loggers and current configuration in use:

curl -X GET http://localhost:8080/actuator/loggers

Query the Spring Boot app metrics

To query the metrics of application use the endpoint "/actuator/metrics". For example, to get the process cpu usage:

curl -X GET http://localhost:8080/actuator/metrics/process.cpu.usage

Returns the process.cpu.usage metric.

{
    "name": "process.cpu.usage",
    "description": "The \"recent cpu usage\" for the Java Virtual Machine process",
    "baseUnit": null,
    "measurements": [
        {
            "statistic": "VALUE",
            "value": 0.0
        }
    ],
    "availableTags": []
}

To see list of all available metrics use:

curl -X GET http://localhost:8080/actuator/metrics

Query the Spring Boot app health information

To query the health status of application use the endpoint "/actuator/health".

curl -X GET http://localhost:8080/actuator/health

Returns the status of the application.

{"status":"UP"}

Introduction to Micrometer

Reference: Micrometer

Micrometer is a metrics instrumentation library powering the delivery of application metrics from Spring Boot applications. Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in.

As an instrumentation facade, Micrometer allows you to instrument your code with dimensional metrics with a vendor-neutral interface and decide on the monitoring system as a last step. Instrumenting your core library code with Micrometer allows the libraries to be included in applications that ship metrics to different backends.

Contains built-in support for AppOptics, Azure Monitor, Netflix Atlas, CloudWatch, Datadog, Dynatrace, Elastic, Ganglia, Graphite, Humio, Influx/Telegraf, JMX, KairosDB, New Relic, Prometheus, SignalFx, Google Stackdriver, StatsD, and Wavefront.

Micrometer Concepts

Let us review key concepts around Micrometer.

• Meter Meter is the interface for collecting a set of measurements (which we individually call metrics) about the application
• MeterRegistry Meters in Micrometer are created from and held in a MeterRegistry. Each supported monitoring system has an implementation of MeterRegistry.
• SimpleMeterRegistry Micrometer includes a SimpleMeterRegistry that holds the latest value of each meter in memory and does not export the data anywhere. A SimpleMeterRegistry is autowired in a Spring Boot application.

  MeterRegistry registry = new SimpleMeterRegistry();
  registry.counter("books");

• CompositeMeterRegistry Micrometer provides a CompositeMeterRegistry to which you can add multiple registries, letting you publish metrics to more than one monitoring system simultaneously.

  CompositeMeterRegistry composite = new CompositeMeterRegistry();
  SimpleMeterRegistry simple = new SimpleMeterRegistry();
  composite.add(simple); 

• Meter primitives Micrometer supports a set of Meter primitives, including Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounter, FunctionTimer, and TimeGauge. Different meter types result in a different number of time series metrics.
• Tags A meter is uniquely identified by its name and dimensions/ tags. Dimensions/ Tags let a particular named metric be sliced to drill down and reason about the data. In the example below, if we select database.calls, we can see the total number of calls to all databases. Then we can group by or select by db to drill down further or perform comparative analysis on the contribution of calls to each database.

  registry.counter("database.calls", "db", "users")
  registry.counter("http.requests", "uri", "/api/users")

• @Timed annotation The micrometer-core module contains a @Timed annotation that frameworks can use to add timing support to either specific types of methods such as those serving web request endpoints or, more generally, to all methods. In the example below, we have added @Timed annotation to the API method which exposes timing metrics on the endpoint.

  @PostMapping("/movies")
  @Timed("movies.api")
  public String orderMovie() {
     return bookService.orderMovie();
  }

• Supported Metrics and Meters Spring Boot provides automatic meter registration for a wide variety of technologies. In most situations, the defaults provide sensible metrics that can be published to any of the supported monitoring systems. Few examples:
  • JVM Metrics Auto-configuration enables JVM Metrics by using core Micrometer classes. JVM metrics are published under the jvm. meter name. E.g. jvm.threads.live, jvm.memory.max
  • System metrics Auto-configuration enables system metrics by using core Micrometer classes. System metrics are published under the system., process., and disk. meter names. E.g system.cpu.usage
  • Application Startup Metrics Auto-configuration exposes application startup time metrics. E.g. application.started.time, application.ready.time
  Refer to this link for all supported metrics.

Introduction to Prometheus

Reference: Prometheus
Let us review key concepts around Prometheus.

• Prometheus is an open-source systems monitoring and alerting toolkit.
• Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.
• In Prometheus, time series collection happens via a pull model over HTTP.
• Supports PromQL, a flexible query language
• Prometheus has an alertmanager to handle alerts
• Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.

Monitor Spring Boot App with Micrometer and Prometheus

Let us work on an example to make the concepts clear. In this example, we will do the following:

• Implement a Spring Boot app and instrument micrometer and prometheus registry
• Publish custom metrics from Spring Boot app to count orders placed
• Use the @Timed annotation to find the API timing metrics
• Use docker compose to run the demo app and prometheus as containers
• Query the metrics from Prometheus and validate

Implement the Spring Boot app

Start with a Spring Boot starter application and include the actuator (spring-boot-starter-actuator) and micrometer prometheus registry (micrometer-registry-prometheus) dependencies.

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
	</dependency>	
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-actuator</artifactId>
	</dependency>
	<dependency>
		<groupId>io.micrometer</groupId>
		<artifactId>micrometer-registry-prometheus</artifactId>
	</dependency>    
</dependencies>

Update the application.properties file to expose the endpoints. Pls note that all of these endpoints are enabled by default.

management.endpoints.web.exposure.include=health,metrics,prometheus,loggers

Next we will define the application and rest controller. We are exposing 2 APIs to order books and movies. Pls note the usage of @Timed annotation which we enabled to query the API timing metrics.

package com.stackstalk;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

import io.micrometer.core.annotation.Timed;

@SpringBootApplication
@RestController
@Timed
public class DemoMonitoringAppApplication {

	@Autowired
	ItemService itemService;
	
	public static void main(String[] args) {
		SpringApplication.run(DemoMonitoringAppApplication.class, args);
	}

	@PostMapping("/books")
	@Timed("books.api")
	public String orderBook() {
		return itemService.orderBook();
	}
	
	@PostMapping("/movies")
	@Timed("movies.api")
	public String orderMovie() {
		return itemService.orderMovie();
	}
}

By default Spring Boot applications are Autowired with SimpleMeterRegistry. In this example, we define the configuration to create a CompositeMeterRegistry.

package com.stackstalk;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.composite.CompositeMeterRegistry;

@Configuration
public class DemoMonitorAppConfig {

	@Bean
	public MeterRegistry getMeterRegistry() {
		CompositeMeterRegistry meterRegistry = new CompositeMeterRegistry();
		return meterRegistry;
	}
}

Finally in the component class we create 2 counter metrics to track the number of orders being placed.

package com.stackstalk;

import org.springframework.stereotype.Component;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.composite.CompositeMeterRegistry;

@Component
public class ItemService {

	private static int bookOrderId = 0;
	private static int movieOrderId = 0;
	private Counter bookCounter = null;
	private Counter movieCounter = null;
	
	public ItemService(CompositeMeterRegistry meterRegistry) {		
		bookCounter = meterRegistry.counter("order.books");
		movieCounter = meterRegistry.counter("order.movies");
	}
	
	public String orderBook() {		
		bookOrderId += 1;
		bookCounter.increment();
		return new String("Ordered Book with id = " + bookOrderId);
	}
	
	public String orderMovie() {		
		movieOrderId += 1;
		movieCounter.increment();
		return new String("Ordered Movie with id = " + movieOrderId);
	}
}

Get the full source code in GitHub.

Create Docker container for the Spring Boot app

Create a docker image for the demo application. Use a simple Dockerfile which package and run the application JAR.

FROM openjdk:17-alpine 
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

To create the Docker image use:

docker build -t demoapp:1 -f Dockerfile .

Run the demoapp and Promethues as Docker containers

We will run demoapp and prometheus as docker containers. Sample prometheus.yml file to scrape the demoapp metrics.

global:
  scrape_interval:     15s

scrape_configs:
  - job_name: "demoapp_metrics"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["demoapp:8080"]

Create a docker-compose.yml and include the demoapp and prometheus as services. We also mount the prometheus.yml to the prometheus container.

services:
  demoapp:
    image: demoapp:1
    ports:
      - "8080:8080"
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

Start the services using the docker compose command:

docker-compose up

Query the metrics from Prometheus

Now let us query Prometheus to see the application metrics we have introduced. We see the counters for books and movies. Since we have introduced the @Timed annotation we also see the metrics around the APIs.

curl -X GET http://localhost:8080/actuator/prometheus

# HELP order_books_total  
# TYPE order_books_total counter
order_books_total 1.0
# HELP books_api_seconds_max  
# TYPE books_api_seconds_max gauge
books_api_seconds_max{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/books",} 0.0
# HELP books_api_seconds  
# TYPE books_api_seconds summary
books_api_seconds_count{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/books",} 1.0
books_api_seconds_sum{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/books",} 0.05435327

# HELP order_movies_total  
# TYPE order_movies_total counter
order_movies_total 1.0
# HELP movies_api_seconds  
# TYPE movies_api_seconds summary
movies_api_seconds_count{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/movies",} 1.0
movies_api_seconds_sum{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/movies",} 0.023019645
# HELP movies_api_seconds_max  
# TYPE movies_api_seconds_max gauge
movies_api_seconds_max{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/movies",} 0.023019645

Another approach is to use the Promethus Dashboard at "http://localhost:9090/" to query and visualize the metrics.

Conclusion

Ability to monitor and manage aspects like health, metrics, logging etc. are key to run the applications in production. As discussed, leveraging Spring Actuator, Micrometer and Prometheus is a simple approach to Monitor Spring Boot applications. Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Refer to more Spring Boot articles in this collection.