Cloud Custodian

Nidhi

Jun 12, 2018·9 min read

Please visit my website and subscribe to my youtube channel for more articles

https://devops4solutions.com/

Cloud Custodian is an open source framework which is used to monitor your AWS infrastruture and run the policies which are written in yaml against your whole AWS account/regions. The policies is like a rule which is put in the yaml file to find out if something is not compilant.

Basic Conecpts:

1. Policy — c7n.policyDefined in yaml, specifies a set of filters and actions to take on a given AWS resource type.
2. Resource — Any resource on AWS which you want to monitor
3. Mode — c7n.policy (yes, policy)Defines how the policy will execute (lambda, config rule, poll, etc). Policies run in pull mode by default.Modes are responding to cloudwatch events and that can be the reason of running any policy locally or manually.If you want to pull all resources of a specific type based on your filter, you can remove the mode section to run it manually.

TIPS:

generally, think of the mode: cloudtrail policies as real-time remediation policies, if you have less urgent/more resource intensive policies you should run those on a cron outside of lambda

For ex —

1. I need to see if my whole EC2 instances are properly tagged or not.
2. All RDS instances should not be publicly available
3. IF any EC2 instance is idle for 10 days then terminate it

Accounts Used

1. security account where all installation and configuration of cloud custodian will reside
2. Other accounts like sandbox-02 which will get monitored using this

Installation

Prerequisite:

Python

Check which python version is installed, if not then install python first$ virtualenv --python=python2.7 custodian
$ source custodian/bin/activate
(custodian) $ pip install c7n

Installation of cloud custodian is completed successfully.

Configure AWS CLI

You need python-pip on that machine to install awscli

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpmsudo yum install python-pipsudo pip install awscli --upgrade pipRun the below command after logging to some user, for eg -ec2-user not as a root useraws configure

Once you configured your keys using aws configure, everything is stored under

cd ~/.aws/
pwd
/home/ec2-user/.aws
cat credentials

Write your first policy:

Create a file custodian.yml

vi custodian.yml

Put the below content in your file. This will fetch all the instances which are in running state.

policies:
  - name: my-first-policy
    resource: ec2
    filters:
      - "State.Name": running

Validate your policy

custodian validate custodian.yml

Dry run : This will show you what your policy will do before actually running it

custodian run --dryrun -s . custodian.yml

Now run your yaml file

custodian run --output-dir=. custodian.yml

Now we will modify the policy and rerun the same command and you will see in output that count is 0.

policies:
  - name: my-first-policy
    resource: ec2
    filters:
      - "State.Name": running
      - "tag:Custodian":present

This is how we can write the rules as per your need.

Cloud Custodian Sample Examples

Click here . Here some predefined roles provided by the cloud custodian. You can use them directly or customize it as per your need.

I am trying few use cases as below

Example 1 : Account — Login From Invalid IP Address

This example needs you to create an IAM role with AWSLambdaFullAccess and we will assume that role in our yaml configuration. If you will not create this role you will get an error “ assert role, “Lambda function role must be specified”
AssertionError: Lambda function role must be specified”

Step 1 Create IAM Role

Go to AWS Console -> IAM ->Create Role -> Select Lambda Access -> AWSLambdaFullAccess -> Create Policy (cloud-custodian-role)

Step 2: Create yaml file

policies:

  - name: invalid-ip-address-login-detected
    resource: account
    description: |
      Notifies on invalid external IP console logins
    mode:
       type: cloudtrail
       role:yourrolearn
       events:
          - ConsoleLogin
    filters:
      - not:
          - type: event
            key: 'detail.sourceIPAddress'
            value: |
               '^((158\.103\.|142\.179\.|187\.39\.)([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
               \.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))|(12\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
               \.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))$'
            op: regex

Step 3 Run the yaml file

if you are not in custodian folder you always need to run the below command
source custodian/bin/activate
(custodian)  custodian run --output-dir=. policy/ipaddress.ymlOutput of this command2018-06-14 13:50:08,544: custodian.policy:INFO Provisioning policy lambda invalid-ip-address-login-detected
2018-06-14 13:50:08,636: custodian.lambda:INFO Publishing custodian policy lambda function custodian-invalid-ip-address-login-detected

Example 2

Root Account login

Step 1 Create rootaccount.yml file

policies:

  - name: root-user-login-detected
    resource: account
    description: |
      Notifies Security and Cloud Admins teams on any AWS root user console logins
    mode:
       type: cloudtrail
       role: yourrolearn       
       events:
          - ConsoleLogin
    filters:
       - type: event
         key: "detail.userIdentity.type"
         value_type: swap
         op: in
         value: Root

Step 2 Run the yaml file

if you are not in custodian folder you always need to run the below command
source custodian/bin/activate
(custodian)  custodian run --profile=local --output-dir=. policy/rootlogin.ymlOutput of this command
2018-06-14 14:16:24,339: custodian.policy:INFO Provisioning policy lambda root-user-login-detected
2018-06-14 14:16:24,459: custodian.lambda:INFO Publishing custodian policy lambda function custodian-root-user-login-detected

Lambda function will get created for both the examples on your AWS console.

To test the lambda function manually for the root login policy, you can use this json

Example 3: Policy to find ec2 instances which doesn’t have tag owner and as part of action add the tag automatically. Create the policy tag.yml

policies:
  - name: tagging-owner
    resource: ec2
    filters:
      - "tag:owner": absent
    actions:
      - type: tag
        key: owner
        value: "Tagging example"

Now run it

custodian run --profile=local --output-dir=. policy/tag.yml

Cloud Custodian for multiple accounts

Click here

c7n-org is a tool to run custodian against multiple AWS accounts, Azure subscriptions, or GCP projects in parallel.

Installation

pip install c7n-org

It will get installed at the location /home/ec2-user/custodian/bin

In order to run c7n-org against multiple accounts, a config file must first be created containing information about all the accounts with regions.

Example of this config file , so we need to create an IAM role on those account that primary account can assume like this

accounts:
- account_id: '123123123123'
  name: account-1
  regions:
  - us-east-1
  - us-west-2
  role: arn:aws:iam::123123123123:role/CloudCustodian
 

Let’s create IAM role on account (sandbox02) which you want to monitor

On sandbox2 account,I am creating a role “cloudcustodian-role”, select lambda access with Administrative permissions

How to assume this role on the cloud custodian account

Now on cloudcustodian account ->

create a new file called accounts.yml

accounts:
- account_id: '123123123123'
  name: account-1
  regions:
  - us-east-1
  - us-west-2
  role: arn:aws:iam::123123123123:role/CloudCustodiansource custodian/bin/activatec7n-org run -c policy/accounts.yml -s output -u policy/custodian.yml --dryrunc7n-org run -c policy/accounts.yml -s output -u policy/custodian.yml 

After running above command, you will get the list of EC2 instances which are in running state on that account. It will create the output directory and with the directory structure as mentioned below

Cross Account Access

I have two accounts (sandbox02 and security account)

Requirement

1. I want to run some scripts from security account to sandbox02 account
2. Need to create one role on sandbox account with permissions(for now i have given administrative permissions) to another account (security account)
3. Security account will assume that role.
4. Attach this role to EC2 Instance
5. Need to remove the AWS default credentials which are present in ~/.aws

On Sandbox account ->

1. Create a role “cloudcustodian-role”, put the security account number

Trust Relationship

Policy — Administrative access.

On security account

I created one role “cloudcustodian-crossaccountrole” and one policy

cloud-custodian-crossaccount-policy which assume the role which we have created in the sandbox-02 account

Trust relationship of the role is below

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ec2.amazonaws.com",
          "lambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Policy as mention below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::aacountnumber ofsandbox:role/cloudcustodian-role"
        }
    ]
}

Now in your accounts.yaml file, put the role which we have created on sandbox-02 account and create new yaml file ,just remove the hard coded role details from there.

cd /home/ec2-user/custodian/bin
c7n-org run -c policy/accounts.yml -s output -u policy/rootlogin_cross.ymlc7n-org run -c policy/accounts.yml -s output -u policy/ipaddress_cross.yml

After this you will see two lamda function created successfully on sandbox02 account

Install cloud custodian emailer

Click Here for more details

Create a directory

$ mkdir mailer

Clone the repository:

$ git clone https://github.com/capitalone/cloud-custodian

Install dependencies (with virtualenv):

$ virtualenv c7n_mailer
$ source c7n_mailer/bin/activate
$ cd /home/ec2-user/mailer/cloud-custodian/tools/c7n_mailer
$ pip install -r requirements.txt

Install the extensions:

python setup.py develop

Create SQS queue

Go to AWS Console -> SQS -> Create QuickQueue

Create IAM role which has read access to this queue ( I have given administrative access also you can refine as per your need )

Now you have lambda as a trusted entity

You have to verify your email address also, go to AWS Console ->SES -> Identity Management -> Email Address -> Verify.

You have to verify all email addresses (all recipients list) and this is region specific.

Create a mailer.yml and put the below content

queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
from_address: you@example.com

Now let’s make a Custodian policy to populate your mailer queue. Create a test-policy.yml: Change the emailid and queue name

policies:
  - name: c7n-mailer-test
    resource: sqs
    filters:
     - "tag:MailerTest": absent
    actions:
      - type: notify
        template: default
        priority_header: '2'
        subject: testing the c7n mailer
        to:
          - you@example.com
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test

Now run

source c7n_mailer/bin/activatec7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .Output:
2018-06-18 15:13:15,738: custodian.policy:INFO policy: c7n-mailer-test resource:sqs region:us-west-2 count:1 time:0.00
2018-06-18 15:13:16,232: custodian.actions:INFO sent message:a7ff12b2-3625-4c56-9df5-79930cf5c7c1 policy:c7n-mailer-test template:default count:1
2018-06-18 15:13:16,232: custodian.policy:INFO policy: c7n-mailer-test action: notify resources: 1 execution_time: 0.49Below command is to send any email which are in queue
c7n-mailer --run -c mailer.ymlOutput:(c7n_mailer) [ec2-user@ip-172-31-24-18 mailer]$ c7n-mailer --run -c mailer.yml
2018-06-18 15:13:57,748 - custodian-mailer - INFO - Downloading messages from the SQS queue.
2018-06-18 15:13:58,159 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:13:58,291 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:13:58,510 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:13:58,768 - custodian-mailer - INFO - Sending account:cydaptivlabs-secops policy:c7n-mailer-test sqs:1 email:default to [u'khandelwal12nidhi@gmail.com']
2018-06-18 15:14:08,783 - custodian-mailer - INFO - No sqs_messages left on the queue, exiting c7n_mailer.If there are no messages in the queue, the following output will get displayed(c7n_mailer) [ec2-user@ip-172-31-24-18 mailer]$ c7n-mailer --run -c mailer.yml
2018-06-18 15:16:02,243 - custodian-mailer - INFO - Downloading messages from the SQS queue.
2018-06-18 15:16:12,333 - custodian-mailer - INFO - No sqs_messages left on the queue, exiting c7n_mailer.