Cloud Custodian
Please visit my website and subscribe to my youtube channel for more articles
Cloud Custodian is an open source framework which is used to monitor your AWS infrastruture and run the policies which are written in yaml against your whole AWS account/regions. The policies is like a rule which is put in the yaml file to find out if something is not compilant.
Basic Conecpts:
- Policy —
c7n.policy
Defined in yaml, specifies a set of filters and actions to take on a given AWS resource type. - Resource — Any resource on AWS which you want to monitor
- Mode —
c7n.policy
(yes,policy
)Defines how the policy will execute (lambda, config rule, poll, etc). Policies run inpull
mode by default.Modes are responding to cloudwatch events and that can be the reason of running any policy locally or manually.If you want to pull all resources of a specific type based on your filter, you can remove the mode section to run it manually.
TIPS:
generally, think of the mode: cloudtrail policies as real-time remediation policies, if you have less urgent/more resource intensive policies you should run those on a cron outside of lambda
For ex —
- I need to see if my whole EC2 instances are properly tagged or not.
- All RDS instances should not be publicly available
- IF any EC2 instance is idle for 10 days then terminate it
Accounts Used
- security account where all installation and configuration of cloud custodian will reside
- Other accounts like sandbox-02 which will get monitored using this
Installation
Prerequisite:
Python
Check which python version is installed, if not then install python first$ virtualenv --python=python2.7 custodian
$ source custodian/bin/activate
(custodian) $ pip install c7n
Installation of cloud custodian is completed successfully.
Configure AWS CLI
You need python-pip on that machine to install awscli
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpmsudo yum install python-pipsudo pip install awscli --upgrade pipRun the below command after logging to some user, for eg -ec2-user not as a root useraws configure
Once you configured your keys using aws configure, everything is stored under
cd ~/.aws/
pwd
/home/ec2-user/.aws
cat credentials
Write your first policy:
Create a file custodian.yml
vi custodian.yml
Put the below content in your file. This will fetch all the instances which are in running state.
policies:
- name: my-first-policy
resource: ec2
filters:
- "State.Name": running
Validate your policy
custodian validate custodian.yml
Dry run : This will show you what your policy will do before actually running it
custodian run --dryrun -s . custodian.yml
Now run your yaml file
custodian run --output-dir=. custodian.yml

Now we will modify the policy and rerun the same command and you will see in output that count is 0.
policies:
- name: my-first-policy
resource: ec2
filters:
- "State.Name": running
- "tag:Custodian":present

This is how we can write the rules as per your need.
Cloud Custodian Sample Examples
Click here . Here some predefined roles provided by the cloud custodian. You can use them directly or customize it as per your need.
I am trying few use cases as below
Example 1 : Account — Login From Invalid IP Address
This example needs you to create an IAM role with AWSLambdaFullAccess and we will assume that role in our yaml configuration. If you will not create this role you will get an error “ assert role, “Lambda function role must be specified”
AssertionError: Lambda function role must be specified”
Step 1 Create IAM Role
Go to AWS Console -> IAM ->Create Role -> Select Lambda Access -> AWSLambdaFullAccess -> Create Policy (cloud-custodian-role)


Step 2: Create yaml file
policies:
- name: invalid-ip-address-login-detected
resource: account
description: |
Notifies on invalid external IP console logins
mode:
type: cloudtrail
role:yourrolearn
events:
- ConsoleLogin
filters:
- not:
- type: event
key: 'detail.sourceIPAddress'
value: |
'^((158\.103\.|142\.179\.|187\.39\.)([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))|(12\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))$'
op: regex
Step 3 Run the yaml file
if you are not in custodian folder you always need to run the below command
source custodian/bin/activate
(custodian) custodian run --output-dir=. policy/ipaddress.ymlOutput of this command2018-06-14 13:50:08,544: custodian.policy:INFO Provisioning policy lambda invalid-ip-address-login-detected
2018-06-14 13:50:08,636: custodian.lambda:INFO Publishing custodian policy lambda function custodian-invalid-ip-address-login-detected

Example 2
Root Account login
Step 1 Create rootaccount.yml file
policies:
- name: root-user-login-detected
resource: account
description: |
Notifies Security and Cloud Admins teams on any AWS root user console logins
mode:
type: cloudtrail
role: yourrolearn
events:
- ConsoleLogin
filters:
- type: event
key: "detail.userIdentity.type"
value_type: swap
op: in
value: Root
Step 2 Run the yaml file
if you are not in custodian folder you always need to run the below command
source custodian/bin/activate
(custodian) custodian run --profile=local --output-dir=. policy/rootlogin.ymlOutput of this command
2018-06-14 14:16:24,339: custodian.policy:INFO Provisioning policy lambda root-user-login-detected
2018-06-14 14:16:24,459: custodian.lambda:INFO Publishing custodian policy lambda function custodian-root-user-login-detected
Lambda function will get created for both the examples on your AWS console.

To test the lambda function manually for the root login policy, you can use this json
Example 3: Policy to find ec2 instances which doesn’t have tag owner and as part of action add the tag automatically. Create the policy tag.yml
policies:
- name: tagging-owner
resource: ec2
filters:
- "tag:owner": absent
actions:
- type: tag
key: owner
value: "Tagging example"
Now run it
custodian run --profile=local --output-dir=. policy/tag.yml
Cloud Custodian for multiple accounts
c7n-org is a tool to run custodian against multiple AWS accounts, Azure subscriptions, or GCP projects in parallel.
Installation
pip install c7n-org
It will get installed at the location /home/ec2-user/custodian/bin
In order to run c7n-org against multiple accounts, a config file must first be created containing information about all the accounts with regions.
Example of this config file , so we need to create an IAM role on those account that primary account can assume like this
accounts:
- account_id: '123123123123'
name: account-1
regions:
- us-east-1
- us-west-2
role: arn:aws:iam::123123123123:role/CloudCustodian
Let’s create IAM role on account (sandbox02) which you want to monitor
On sandbox2 account,I am creating a role “cloudcustodian-role”, select lambda access with Administrative permissions


How to assume this role on the cloud custodian account
Now on cloudcustodian account ->
create a new file called accounts.yml
accounts:
- account_id: '123123123123'
name: account-1
regions:
- us-east-1
- us-west-2
role: arn:aws:iam::123123123123:role/CloudCustodiansource custodian/bin/activatec7n-org run -c policy/accounts.yml -s output -u policy/custodian.yml --dryrunc7n-org run -c policy/accounts.yml -s output -u policy/custodian.yml
After running above command, you will get the list of EC2 instances which are in running state on that account. It will create the output directory and with the directory structure as mentioned below

Cross Account Access
I have two accounts (sandbox02 and security account)
Requirement
- I want to run some scripts from security account to sandbox02 account
- Need to create one role on sandbox account with permissions(for now i have given administrative permissions) to another account (security account)
- Security account will assume that role.
- Attach this role to EC2 Instance
- Need to remove the AWS default credentials which are present in ~/.aws
On Sandbox account ->
- Create a role “cloudcustodian-role”, put the security account number

Trust Relationship

Policy — Administrative access.
On security account
I created one role “cloudcustodian-crossaccountrole” and one policy
cloud-custodian-crossaccount-policy which assume the role which we have created in the sandbox-02 account
Trust relationship of the role is below
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com",
"lambda.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
Policy as mention below
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::aacountnumber ofsandbox:role/cloudcustodian-role"
}
]
}
Now in your accounts.yaml file, put the role which we have created on sandbox-02 account and create new yaml file ,just remove the hard coded role details from there.
cd /home/ec2-user/custodian/bin
c7n-org run -c policy/accounts.yml -s output -u policy/rootlogin_cross.ymlc7n-org run -c policy/accounts.yml -s output -u policy/ipaddress_cross.yml

After this you will see two lamda function created successfully on sandbox02 account

Install cloud custodian emailer
Click Here for more details
Create a directory
$ mkdir mailer
Clone the repository:
$ git clone https://github.com/capitalone/cloud-custodian
Install dependencies (with virtualenv):
$ virtualenv c7n_mailer
$ source c7n_mailer/bin/activate
$ cd /home/ec2-user/mailer/cloud-custodian/tools/c7n_mailer
$ pip install -r requirements.txt
Install the extensions:
python setup.py develop
Create SQS queue
Go to AWS Console -> SQS -> Create QuickQueue
Create IAM role which has read access to this queue ( I have given administrative access also you can refine as per your need )

Now you have lambda as a trusted entity

You have to verify your email address also, go to AWS Console ->SES -> Identity Management -> Email Address -> Verify.
You have to verify all email addresses (all recipients list) and this is region specific.
Create a mailer.yml and put the below content
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
from_address: you@example.com
Now let’s make a Custodian policy to populate your mailer queue. Create a test-policy.yml
: Change the emailid and queue name
policies:
- name: c7n-mailer-test
resource: sqs
filters:
- "tag:MailerTest": absent
actions:
- type: notify
template: default
priority_header: '2'
subject: testing the c7n mailer
to:
- you@example.com
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
Now run
source c7n_mailer/bin/activatec7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .Output:
2018-06-18 15:13:15,738: custodian.policy:INFO policy: c7n-mailer-test resource:sqs region:us-west-2 count:1 time:0.00
2018-06-18 15:13:16,232: custodian.actions:INFO sent message:a7ff12b2-3625-4c56-9df5-79930cf5c7c1 policy:c7n-mailer-test template:default count:1
2018-06-18 15:13:16,232: custodian.policy:INFO policy: c7n-mailer-test action: notify resources: 1 execution_time: 0.49Below command is to send any email which are in queue
c7n-mailer --run -c mailer.ymlOutput:(c7n_mailer) [ec2-user@ip-172-31-24-18 mailer]$ c7n-mailer --run -c mailer.yml
2018-06-18 15:13:57,748 - custodian-mailer - INFO - Downloading messages from the SQS queue.
2018-06-18 15:13:58,159 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:13:58,291 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:13:58,510 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:13:58,768 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:14:08,783 - custodian-mailer - INFO - No sqs_messages left on the queue, exiting c7n_mailer.If there are no messages in the queue, the following output will get displayed(c7n_mailer) [ec2-user@ip-172-31-24-18 mailer]$ c7n-mailer --run -c mailer.yml
2018-06-18 15:16:02,243 - custodian-mailer - INFO - Downloading messages from the SQS queue.
2018-06-18 15:16:12,333 - custodian-mailer - INFO - No sqs_messages left on the queue, exiting c7n_mailer.