Analyze VPC Flow Log using ElasticSearch and Kibana

Please visit my website and subscribe to my youtube channel for more articles

https://devops4solutions.com/

Image for post
Image for post

Tools/Task:

  1. Create Flow Log in VPC
  2. IAM Role for VPC Flow Log
  3. Cloudwatch Log Group
  4. ElasticSearch
  5. IAM Lambda role

Create Flow Log on VPC

  1. Open VPC
  2. Click on Create Flow Log

3. For Role — Click on Set up Permissions, it will create a role and then select it

4. For Destination Log Group — this is the cloud watch log group name

The name of the CloudWatch Logs log group to which the flow log will be published. A log stream will be created in this log group for each network interface being monitored.

Flow Log is created successfully.

You can check the flow logs being generated

ElasticSearch Service

  1. Create domain

All default setting then

Click Confirm

It will take around 10 minutes to create a elasticsearch service

Now go to cloudwatch

Click on Actions -> Stream to Amazon ElasticSearchService.

Select this Account and your ES cluster

CloudWatch Logs uses Lambda to deliver log data to Amazon ES. You must specify an IAM role that grants Lambda permission to make calls to Amazon ES. You can choose an existing role or create an IAM role that automatically has the required permissions. To deliver log data to another account, you must specify the Elasticsearch Domain ARN and Elasticsearch Endpoint of other account and ensure permissions are granted to be able to publish to that ARN.

Select — Click on Create new role

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“logs:CreateLogGroup”,
“logs:CreateLogStream”,
“logs:PutLogEvents”
],
“Resource”: [
“arn:aws:logs:*:*:*”
]
},
{
“Effect”: “Allow”,
“Action”: “es:ESHttpPost”,
“Resource”: “arn:aws:es:*:*:*”
}
]
}

You will see all the fields

Click on Discover, you will see the graph based on timestamp

Create a visualization as per your requirement

Create a Dashboard, add the visualization which you have created above to show it on your dashboard.

To do it using script, please follow the below link, it will work as it is.

References

CLick Here

Written by

Devops Automation Enginneer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store