GLF Logs Statistics

Problem:

Tens or hundreds of files, gigabytes in size, millions of lines, and thousands of error messages of varying severity across multiple processes and hosts, with clock skew between hosts.

Some errors are harmless, some are from unrelated workflows, and some are results of the original problem.

The challenge is to find the root cause.

Solution:

There are a few GLF viewers available, and some of them are decent.

However, they all lack certain essential functionalities or/and convenience features — such as workflow views, graphical representations/tables/trees/full text views, show/hide options for extended text/stacks, global stack view, filters, search, statistical analysis, and mining (flexible data extraction with conversion to tables for further analysis).

Example:

It took only couple minutes after opening GLF logs in BiWhy and checking the statistics table, which groups log messages by removing unique information from them.

The first most frequent error turned out to be irrelevant, but the second was our root cause.