Apache Metron in the Real World

Apache Metron is a storage and analytic platform specialized in cyber security. This talk was about demonstrating the usages and capabilities of Apache Metron in the real world. The presentation was led by Dave Russell, Principal Solutions Engineer – EMEA + APAC at Hortonworks, at the Dataworks Summit 2018 (Berlin).

Presentation

Apache Metron is a cyber security application framework that provides organizations the ability to ingest, process and store diverse security data feeds in order to detect cyber anomalies and enable them to rapidly respond.

It provides a scalable advanced security analytics framework which is built with Hadoop technologies and is specifically designed to monitor network traffic and machine logs within an organization by continuously consuming live flowing data from a lot of “data in motion” sources.

Apache Metron overview

Metron has a clear and intuitive interface.

Apache Metron interface

For each input we have some useful informations from Metron and we can filter on our own data too.

  • A score to evaluate the level of the threat
  • A timestamp
  • The alert status
  • The threat reason (for eg. “The distinct number of machines that user U22 attempted to login to (2) is more than 5 standard deviations (0.29) from the median (1.00)”)
  • An associated user

Which response does Metron bring ?

Currently, data retention time is much lower than the detection time of a breach, the average data retention duration is 6 months while for breach detection it’s 8 months. So we need a system that stores huge amounts of data over several years and that’s where Metron comes in!

Sometime in the next few years we’re going to have out first category-one cyber-incident; one that will need a national response
Ian Levy, Technical Director of National Cyber Security Center

Metron also come with algorithmic parts to detect threats.

Profiling by time

Sizing considerations

For cluster sizing there are several points to consider:

  • Events per second (average and peak)
  • Retention time for Hot/Warm/Cold zones
  • Enrichments
  • Node sizing
  • I/O Considerations
  • PCAP (API for capturing network traffic)

The sizing of a cluster must be progressive:

Data sheet

Metron offers many different solutions to each problem:

Ingest

Parsers

Enrichment and threat feeds

Analytic features

  • Profiler and statistical baselining engine
  • Model Services for advanced ML
  • Threat Triage rules and scoring engine

Index and search features

Data science features

Forensic features

Deploying Metron

Like sizing, deploying a Metron cluster must be progressive.

A fully deployed Apache Metron ecosystem

For example, a 3 phases deployment:

Let’s try it!

Sources

By |2018-06-07T13:38:15+00:00May 29th, 2018|Categories: Cyber Security, DataWorks Summit 2018, Events|Tags: , , |0 Comments

About the Author:

Michael HATOUM is a Big Data consultant with 4 years of experience. He designed, built and operated ingestion workflows and real-time services while helping his clients define their needs and implement them. He has experience in planning, architecture design and cluster deployment. Excellent Java developer, he is confortable in the prototyping and the industrialization of applications, in collaboration with the different teams of the client. He is also very versatile and able to carry out the various tasks entrusted to him. Passionate, he maintains an active watch in several fields of computer science and is autonomous in the acquision of know-how.

Leave A Comment