a large body of water with a mountain in the background

Logan: Logfile Analysis

Established: October 12, 2015

Understanding Techniques and Tools for More Effective Telemetry and Log Data Analysis.

Increasingly, business processes require data-driven real-time feedback based on large quantities of log data and customer telemetry from multiple sources. The Logan Project takes a broad approach to understanding the specific needs of consumers of telemetry and log data, focusing on giving them better support for extracting the data they need, cleaning it, and creating queries against it.

To understand the needs of real users, we conduct both broad-based email surveys and in-person interviews to determine the major challenges that data scientists and consumers of telemetry data face. We work closely with internal users to understand requirements that tool-based automation need to support.

Based on these findings, we have implemented a proof-of-concept for visually building queries involving sequences of events, phrased as visual regular expressions. Our most recent research focuses on cohort comparison and data cleaning.

We are also exploring programming by example, with a prototype for filtering data streams and forming nested queries. We are just beginning to scratch the surface of ways to allow both programmers and non-programmers to extract, filter, and query telemetry data sets comfortably.

Learn more about event-oriented sequential analysis at the IEEE Vis 2016 Event Event (opens in new tab).

People

Portrait of Kael Rowan

Kael Rowan

Principal Research Software Development Engineer (RSDE)

Portrait of Mary Czerwinski

Mary Czerwinski

Partner Researcher and Research Manager

Portrait of Steven Drucker

Steven Drucker

Partner Research Manager

Portrait of Sumit Gulwani

Sumit Gulwani

Distinguished Scientist and Vice President

Portrait of Ben Zorn

Ben Zorn

Partner Researcher