Friday, January 27, 2017

Streaming WebSphere Liberty Profile Log Entries to Splunk

Log data is an important source of information containing insight into the operation and health of your application.  Being able to index, search and alert on events in near realtime increases your awareness of potential interruptions to service.  Ultimately log analytics can reduce Mean Time to Recovery (MTTR) and may give you the ability to correct an issue before it impacts your end users.

IBM's WebSphere Liberty Profile is a great option for running containerized Java EE micro-services.  IBM provides a fully compliant Java EE 7 runtime using a light weight application server along with a number of features (plugins) which can be added to provide additional functionality.  One of the features provided by the Liberty Profile streams log entries to Logstash.  In terms of end to end log aggregation and analytics, Logstash is a forwarding service which will gather log entries and forward them to a log analytics engine. You can read more about Logstash here.

The first question you might ask is why use this feature...  why not just use a standard Docker logging driver like syslog or journald... or even the Splunk driver directly?  Why do i need another layer in my log aggregation framework.  A Liberty Profile server produces four log files; console.log, messages.log, trace.log and the files emitted by the First Failure Data Capture (FFDC) service.  A Liberty Profile server can be configured to emit a large portion of the data from these different log files to stdout within the container which in turn could then be handled by a standard log driver provided by Docker.  The FFDC log files are the exception to this rule.  FFDC log files are written to their own directory and each file contains the details of a single error that occurred in the server or application.  The difficultly is in getting to the FFDC log files within the container.... which in my experience are very helpful in the first round of problem determination.

 This is where the Logstash feature the Liberty Profile provides is beneficial.  Not only does the feature emit the logs normally printed to message.log and trace.log, but it will also emits the FFDC events.  As an added bonus it emits these log events in a JSON format so no additional log entry parsing is required before each entry is forwarded to your log analytics engine.  Logstash is available as an official Docker image on Dockerhub making the adoption in a containerized environment quick and easy.

 Note: IBM provides detailed documentation on how to configure logging within a server instance as well as the Logstash feature to which I included links in the references section below.

 What does the flow of log entries look like from the Liberty Profile to Splunk using the tools outlined above?  The diagram below is a simple illustration of the flow of data from one or more Liberty Profile containers to Splunk using Logstash.

The Logstash feature in the Liberty Profile sends the data to the Logstash container using Logstash's Lumberjack protocol.  Logstash then formats the log entry using the HTTP Output plugin and sends the entry to Splunk's HTTP Event Collector.  Once received, Splunk indexes the entry and makes it available for querying and visualization.

 I've created a GitHub repository with a example of how to configure each component.  The instructions to run the example are in the read me file.  Once the container stack is running, you will see log entries in Spunk within a few seconds.  From there you can query and visualize the data into custom dashboards and alerts.  As an example, below is screen shot of a log entry that originated from a Liberty Profile server queried from Splunk.


The JSON format emitted by the Liberty Profile feature basically eliminations the need to parse the log entry into fields.  In addition environment variable can be set in the Liberty Profile containers to control the values of the hostName and serverName fields in each entry.  These fields can be used to denote the host machine the container is running on as well as the micro-service running within the container.

The entries emitted from the FFDC service are also provided in a JSON format easily consumed by Splunk.  As an example:



Using the the Liberty Profile Logstash feature and Logstash container in this way opens up the possibility of sending the data to other analytics services.  Logstash provides a number of output plugins for open source solutions like Elasticsearch a well as other SaaS solutions like New Relic, Datadog and Loggly.  As this example shows, the generic output plugins like the HTTP plugin can provide further integration to any service providing a HTTP based API.  While Logstash may look like an extra component, it provides a level of flexibility when choosing your analytics engine that a direct connection would greatly limit.

Below is a list of references I used to create the example application:

Posted by at
Labels: , , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)