Application monitoring with Graphite: an example how to integrate Dropwizard Metrics in a Spring Boot application 1


Introduction

For a Java Spring Boot project I was working on a while ago I had to build in some application monitoring for measuring the number of requests on REST endpoints and also for monitoring JVM and OS metrics. These metrics needed to be send to Graphite. Graphite is a great open source tool for handling visualizations and metrics and it offers a powerful querying API.

Graphite itself does not collect any metrics though. It requires metrics to be sent to it in a Graphite format. There are many metric collection tools available for a various number of languages. For Java Dropwizard Metrics is a popular and powerful library that provides us with ways to measure the behavior of critical components in our production environment and it offers a good visibility on the metrics of these components. It also offers a Graphite reporter that can send all this data to our Graphite deployment. It is a solid foundation for using Graphite with Java.

This blog however is not about installing Graphite nor is it about how to create dashboards and nice graphs in Graphite. If you are interested in how Graphite works and which components it exists of, you can read this article. InfoQ also has a nice article about installing graphite and about creating graphs in graphite.

So what is this blog about then? What I will describe is how I integrated Dropwizard Metrics within my Spring Boot application. A full working demo application (a REST API for maintaining customers) and a docker image with Graphite are available on Github for those that are interested to try it out themselves. Feel free to clone these projects to play around with.

Setup Maven dependencies

First of all we need to include some maven dependencies in our project. Instead of including the metrics-core dependency (as the Dropwizard Metrics getting started mentions) we can include the following dependency:

which will transitively provide us with the metrics-core dependency and it will also provide us with two other useful dependencies: metrics-healthchecks and metrics-annotations. Especially the latter is interesting. It will allow us to simply add annotations like @Metered, @Timed, @Counted and @ExceptionMetered to our methods.

Furthermore we need to include the following dependency:

which provides us with a GraphiteReporter that we can use to send our metrics to a Graphite server.

The last metrics related dependency we need is:

Environment specific configuration

For my application I want to be able to monitor multiple environments and push the data to Graphite under a separate location. We can do this by creating a prefix. For monitoring the application in the test environment, we could add the following configuration class:

and we could add another similar configuration class for monitoring the application in a production environment like so:

In these environment specific configuration classes we need to implement the configureReporters method (lines 16-19). Here we call the configureReporters(String graphitePrefix) method in the AbstractMonitoringConfiguration class so that, based on the active profile, metrics data is pushed to the correct prefix (i.e. location) in Graphite.

Registering our metrics & configuring our reporters

The starting point for Metrics is the MetricRegistry class, which contains a collection of all the metrics for our application. The environment specific configuration classes described above extend AbstractMonitoringConfiguration which contains an instance of this MetricRegistry (lines 30-31):

So what basically happens is that when the environment specific configuration calls the configureReporters method (lines 37-40) we call the configureReporters(MetricRegistry metricRegistry) method which we implemented in (lines 42-52). Here we register different reporters for our metrics like JmxReporter (which is out of scope here) and GraphiteReporter. We register them through the registerReporter method located in the MetricsConfigurerAdapter class. We use this method since it allows the MetricsConfigurerAdapter to shut down the reporter when the Spring context is closed.

To register a GraphiteReporter we first need to create a GraphiteReporter Builder (lines 54-65). First we add the metrics that we are interested in to the metricRegistry by registering them (lines 56-59). There are different kinds of metrics that we can add like Meters, Gauges, Counters, Histograms and Timers which are explained in more detail on the Dropwizard Metrics Getting Started page. Then we create the GraphiteReporter Builder by calling the static forRegistry method, passing in our metricRegistry, which contains the metrics we just added, and we set some conversion rates/durations and filters (lines 60-64). Important to note here is that we also set the graphitePrefix.

Secondly we get a GraphiteReporter by calling the build method on the GraphiteReporter Builder that we have just created. This method takes as input a Graphite client instance which is constructed in lines 67-70, using the graphiteHost and graphitePort instance variables.

Once we have the GraphiteReporter we can register it (line 49) and then start it (lines 50-51).

An interesting thing to point out is the OperatingSystemGaugeSet that is added in line 59. This one is not part of the Dropwizard Metrics library. Besides garbage collection metrics I was looking for a way to measure some more OS specific metrics like process CPU time and total/free physical memory (size). In my search on how to do this I came across an interesting post about logging JVM and OS metrics in a spring application. The OperatingSystemGaugeSet comes from this blogpost and I have added this to the demo application.

In order for the metrics that we registered to be pushed to Graphite, we need to ensure that we add the @EnableMetrics annotation at the top of our environment specific classes (MonitoringConfigurationTestEnvironment and MonitoringConfigurationProductionEnvironment) mentioned earlier. Note that in some cases the @EnableMetrics does not work correctly if it implements any interfaces. In this case Spring AOP attempts to use a Java proxy, instead of a CGLIB Proxy, which would cause problems. The solution would be to eliminate the interface, or to set the attribute proxyTargetClass=true in the @EnableMetrics annotation as described here. Another issue using the @EnableMetrics can occur when using it in a Spring Boot auto configuration class. A solution how to solve that is described here.

Start measuring

Now that everything has been setup, our metrics have been registered and our reporters have been configured, we can start measuring our methods. In my demo application I have created the following REST controller:

Since we included the metrics-spring dependency in our Maven configuration, which transitively provides us with the metrics-annotations dependency, we can just add any of the following annotations to our methods: @CachedGauge, @Counted, @ExceptionMetered, @Gauge, @Metered, @Metric and @Timed

Here I have chosen to add the @Timed and @ExceptionMetered annotations. The general core concepts of Metrics are as follows:

  1. A Histogram is a statistical distribution of a stream of data.
  2. A Meter measures rates of events.
  3. A Timer is a Histogram plus a Meter. More specifically a Histogram of the timings of a particular call, and a Meter of the rate at which that call is invoked.
  4. Calls that are @Timed do not need to be @Metered because an invocation meter is already included in the timer.
  5. If you would only be interested in the call rate you would use @Metered.
  6. If you would also be interested in the timing you would use @Timed (but this also gives you the call rate).

Adding the @ExceptionMetered annotation to the method will create a Meter of the unhandled exceptions thrown out of this method. This will allow us to track the error rates for this specific method. It can be used together with the @Timed or @Metered annotations, or entirely by itself. It will create a name in the registry that will not collide with the @Timed or @Metered metrics.

When I run the demo application for the first time it makes some rest calls and posts the data to Graphite:

As you can see here all our data has been pushed to collectd/graphite-monitoring-example/test and is neatly organised per class and method. For the get method we can see that 1000 requests were made and 0 exceptions occurred. The exceptions are organised in a separate folder for each method if we used the @ExceptionMetered annotation.

We can also see that each HTTP status code is monitored for every endpoint we marked with @Timed:

Furthermore we can see that the response times for the different endpoints are being monitored:

And we also find back our metrics that we registered in the AbstractMonitoringConfiguration class:

 

Conclusion

In this article I have shown how we can integrate Dropwizard Metrics within a Spring Boot application using Java annotation based configuration. I described which Maven dependencies need to be configured and discussed how we can setup configurations for multiple environments and push the data to Graphite under specific locations. Next I gave an example how metrics and reporters can be registered and finally I have shown an example of a REST controller where we can add Metrics annotations to our methods.

The purpose of gathering all the metrics is to be able to visualize them over a certain timespan. This is what we can use Graphite-web for which allows us to visualize and play with our metrics (as seen in the screenshots above). It provides a fairly solid amount of capabilities in terms of what we can do with our data, but some argue that the visualization component is not exactly beautiful.

But since Graphite has a great extensive API, there are a huge number of alternative dashboards out there which we can interact with. One of the biggest alternative visualization tool for Graphite nowadays is Grafana which is definitely worth looking at.

Resources


Een reactie plaatsen

Het e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *

Een gedachte over “Application monitoring with Graphite: an example how to integrate Dropwizard Metrics in a Spring Boot application

  • Julien

    Hi,

    Thank you for this article
    I read it and dig in the source code of your application but I don’t understand how you get the counters of the HTTP response code
    Could you tell me where these counters are registered?

    Thank you