Application monitoring with Graphite: an example how to integrate Dropwizard Metrics in a Spring Boot application

Registering our metrics & configuring our reporters

The starting point for Metrics is the MetricRegistry class, which contains a collection of all the metrics for our application. The environment specific configuration classes described above extend AbstractMonitoringConfiguration which contains an instance of this MetricRegistry (lines 30-31):

package com.craftsmen.graphite.monitoring.example.infrastructure;
 
import java.net.InetSocketAddress;
import java.util.concurrent.TimeUnit;
import javax.inject.Inject;
import org.springframework.beans.factory.annotation.Value;
import com.codahale.metrics.JmxReporter;
import com.codahale.metrics.MetricFilter;
import com.codahale.metrics.MetricRegistry;
import com.codahale.metrics.graphite.Graphite;
import com.codahale.metrics.graphite.GraphiteReporter;
import com.codahale.metrics.graphite.GraphiteReporter.Builder;
import com.codahale.metrics.jvm.GarbageCollectorMetricSet;
import com.codahale.metrics.jvm.MemoryUsageGaugeSet;
import com.codahale.metrics.jvm.ThreadStatesGaugeSet;
import com.ryantenney.metrics.spring.config.annotation.MetricsConfigurerAdapter;
 
public abstract class AbstractMonitoringConfiguration
  extends MetricsConfigurerAdapter {
 
  @Value("${graphite.host}")
  private String graphiteHost;
 
  @Value("${graphite.port}")
  private int graphitePort;
 
  @Value("${graphite.amount.of.time.between.polls}")
  private long graphiteAmountOfTimeBetweenPolls;
  
  @Inject
  private MetricRegistry registry;
 
  private String graphitePrefix;
 
  abstract protected void configureReporters();
 
  protected void configureReporters(String graphitePrefix) {
    this.graphitePrefix = graphitePrefix;
    configureReporters(registry);
  }
 
  @Override
  public void configureReporters(MetricRegistry metricRegistry) {
    registerReporter(JmxReporter.forRegistry(metricRegistry)
      .build()).start();
    GraphiteReporter graphiteReporter = 
      getGraphiteReporterBuilder(metricRegistry)
      .build(getGraphite());
    registerReporter(graphiteReporter);
    graphiteReporter.start(graphiteAmountOfTimeBetweenPolls,
      TimeUnit.MILLISECONDS);
  }
 
  private Builder getGraphiteReporterBuilder(MetricRegistry 
    metricRegistry) {
    metricRegistry.register("gc", new GarbageCollectorMetricSet());
    metricRegistry.register("memory", new MemoryUsageGaugeSet());
    metricRegistry.register("threads", new ThreadStatesGaugeSet());
    metricRegistry.register("os", new OperatingSystemGaugeSet());
    return GraphiteReporter.forRegistry(metricRegistry)
      .convertRatesTo(TimeUnit.SECONDS)
      .convertDurationsTo(TimeUnit.MILLISECONDS)
      .filter(MetricFilter.ALL)
      .prefixedWith(graphitePrefix);
  }
 
  private Graphite getGraphite() {
    return new Graphite(new InetSocketAddress(graphiteHost,
      graphitePort));
  }
}

So what basically happens is that when the environment specific configuration calls the configureReporters method (lines 37-40) we call the configureReporters(MetricRegistry metricRegistry) method which we implemented in (lines 42-52). Here we register different reporters for our metrics like JmxReporter (which is out of scope here) and GraphiteReporter. We register them through the registerReporter method located in the MetricsConfigurerAdapter class. We use this method since it allows the MetricsConfigurerAdapter to shut down the reporter when the Spring context is closed.

To register a GraphiteReporter we first need to create a GraphiteReporter Builder (lines 54-65). First we add the metrics that we are interested in to the metricRegistry by registering them (lines 56-59). There are different kinds of metrics that we can add like Meters, Gauges, Counters, Histograms and Timers which are explained in more detail on the Dropwizard Metrics Getting Started page. Then we create the GraphiteReporter Builder by calling the static forRegistry method, passing in our metricRegistry, which contains the metrics we just added, and we set some conversion rates/durations and filters (lines 60-64). Important to note here is that we also set the graphitePrefix.

Secondly we get a GraphiteReporter by calling the build method on the GraphiteReporter Builder that we have just created. This method takes as input a Graphite client instance which is constructed in lines 67-70, using the graphiteHost and graphitePort instance variables.

Once we have the GraphiteReporter we can register it (line 49) and then start it (lines 50-51).

An interesting thing to point out is the OperatingSystemGaugeSet that is added in line 59. This one is not part of the Dropwizard Metrics library. Besides garbage collection metrics I was looking for a way to measure some more OS specific metrics like process CPU time and total/free physical memory (size). In my search on how to do this I came across an interesting post about logging JVM and OS metrics in a spring application. The OperatingSystemGaugeSet comes from this blogpost and I have added this to the demo application.

In order for the metrics that we registered to be pushed to Graphite, we need to ensure that we add the @EnableMetrics annotation at the top of our environment specific classes (MonitoringConfigurationTestEnvironment and MonitoringConfigurationProductionEnvironment) mentioned earlier. Note that in some cases the @EnableMetrics does not work correctly if it implements any interfaces. In this case Spring AOP attempts to use a Java proxy, instead of a CGLIB Proxy, which would cause problems. The solution would be to eliminate the interface, or to set the attribute proxyTargetClass=true in the @EnableMetrics annotation as described here. Another issue using the @EnableMetrics can occur when using it in a Spring Boot auto configuration class. A solution how to solve that is described here.

Now that everything has been setup, our metrics have been registered and our reporters have been configured, we can start measuring our methods. In my demo application I have created the following REST controller:

package com.craftsmen.graphite.monitoring.example.controller.customer;
 
import static org.springframework.hateoas.mvc.ControllerLinkBuilder.linkTo;
import java.util.List;
import java.util.NoSuchElementException;
import javax.inject.Inject;
import javax.servlet.http.HttpServletRequest;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import com.codahale.metrics.annotation.ExceptionMetered;
import com.codahale.metrics.annotation.Timed;
import com.craftsmen.graphite.monitoring.example.controller.AbstractHttpController;
import com.craftsmen.graphite.monitoring.example.entities.Customer;
import com.craftsmen.graphite.monitoring.example.service.CustomerService;
import com.ryantenney.metrics.spring.config.annotation.EnableMetrics;
 
@RestController
public class CustomerController
  extends AbstractHttpController {
 
  @Inject
  private CustomerService customerService;
 
  @Inject
  private CustomerToJsonMapper customerToJsonMapper;
 
  @Inject
  private JsonToCustomerMapper jsonToCustomerMapper;
 
  @Inject
  private RequestValidator requestValidator;
 
  /**
   * This method will retrieve the {@link Customer}
   * for the id provided.
   *
   * If the {@link Customer} cannot be found,
   * a {@link NoSuchElementException} will be thrown
   * and a {@link ResponseEntity} with
   * {@link HttpStatus.NO_CONTENT} will be returned.
   * If the {@link Customer} exists, it will be
   * returned in the {@link ResponseEntity}.
   *
   * @param id
   *   the unique identifier for this customer
   * @return {@link ResponseEntity} containing
   *   {@link Customer} retrieved if found,
   *   the HTTP headers and the HTTP status
   *   code
   */
  @RequestMapping(
    value = "/customer/{id}",
    method = RequestMethod.GET,
    produces = JSON_MIME_TYPE)
  @ResponseBody
  @Timed
  @ExceptionMetered
  public ResponseEntity get(@PathVariable Long id,
    HttpServletRequest request) {
    logApiCalls(request);
    String requestUrl = linkTo(CustomerController.class)
      .withSelfRel()
      .getHref();
    Customer customer = customerService.findById(id);
    String response = customerToJsonMapper
      .mapCustomerToJson(customer, requestUrl);
    return new ResponseEntity(response,
      createHeaders(request.getMethod()), HttpStatus.OK);
  }
 
  // More REST-endpoints here, but left out in this
  // example.
}

Since we included the metrics-spring dependency in our Maven configuration, which transitively provides us with the metrics-annotations dependency, we can just add any of the following annotations to our methods: @CachedGauge, @Counted, @ExceptionMetered, @Gauge, @Metered, @Metric and @Timed

Here I have chosen to add the @Timed and @ExceptionMetered annotations. The general core concepts of Metrics are as follows:

1. A Histogram is a statistical distribution of a stream of data.
2. A Meter measures rates of events.
3. A Timer is a Histogram plus a Meter. More specifically a Histogram of the timings of a particular call, and a Meter of the rate at which that call is invoked.
4. Calls that are @Timed do not need to be @Metered because an invocation meter is already included in the timer.
5. If you would only be interested in the call rate you would use @Metered.
6. If you would also be interested in the timing you would use @Timed (but this also gives you the call rate).

Adding the @ExceptionMetered annotation to the method will create a Meter of the unhandled exceptions thrown out of this method. This will allow us to track the error rates for this specific method. It can be used together with the @Timed or @Metered annotations, or entirely by itself. It will create a name in the registry that will not collide with the @Timed or @Metered metrics.

When I run the demo application for the first time it makes some rest calls and posts the data to Graphite:

As you can see here all our data has been pushed to collectd/graphite-monitoring-example/test and is neatly organised per class and method. For the get method we can see that 1000 requests were made and 0 exceptions occurred. The exceptions are organised in a separate folder for each method if we used the @ExceptionMetered annotation.

We can also see that each HTTP status code is monitored for every endpoint we marked with @Timed: