Ultra-short guide to Big Data and Security

If you are looking to navigate your way through the complexities of Big Data and its use in Security, here are some links to get you off the ground:

Big Data Basics

  1. What is Big Data
  2. Which are the major technologies used for Big Data
    1. Hadoop (Apache’s open-source implementation of Google’s MapReduce algo)
    2. Elastic Search, Logstash, Kibana (ELK)
  3. Major implementations of Hadoop – Cloudera and Hortonworks
  4. How it started to become mainstream – Google publishes its MapReduce algorithm


Big Data For Security?

How can we leverage this capability to deliver security intelligence? There are a number of very interesting implementations that have been done:

  1. Here’s a very good take by Splunk on Big Data for Security 
  2. Using HP Arcsight for APT Detection
  3. 10 Steps to a Better Big Data Security Strategy 
  4. How are Cisco and Verizon leveraging ELK for security intelligence


Getting started

Begin by playing with your own ELK setup on AWS or Azure and then see if you can do anomaly detection using ELK. If you need a large set of sample logs, hop over here. Or better yet go straight and get Apache Metron up and running!

K K Mookhey

K. K. Mookhey (PCI QSA, CISA, CISSP, CISM) is the Founder Director at Network Intelligence (www.niiconsulting.com) as well as the Founder of The Institute of Information Security (www.iisecurity.in). He is an internationally well-regarded expert in the field of IT governance, information risk management, forensic fraud investigations, compliance, and business continuity. He has more than a decade of experience in this field, having worked with prestigious clients in India such as the top 4 private bank, the top 4 public sector banks, the top 5 IT companies, and some of the largest industrial conglomerates. Internationally, he has done consulting and audit engagements for United Nations organizations, numerous Banks and manufacturing firms in the Middle East, as well as various government entities. He has published numerous articles, two books, presented at numerous conferences such as Blackhat, OWASP Asia, ISACA, Interop, and Nullcon.


  1. your blog is really helpful and thus it is very well unique , i have came to know lot of information it is really nice thanks for sharing these valuable information.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.