Integrate API Gateway with Kinesis Firehose using CloudFormation

24 August 2017

Integrating API Gateway with other AWS Services can be pretty important to increase the scope of an API into other services.

What I wanted to achieve was a cheaper upload mechanism for S3. The easiest way to allow upload through API gateway is to call a Lambda for every API call and then upload the payload into an S3 bucket. But this is rather costly if you increase the throughput from a few single call to a few hundred calls a second.

So what I came up with, was to directly invoke the Kinesis Firehose from an API Gateway. That way I could avoid the cost of the Lambda and the cost of a S3 request per API call.

So here an overview picture of what I am about to build. API Gateway to Kinesis Firehose

The easiest part in CloudFormation is the S3 bucket. This one does not require any specific configuration for this to work.

AWSTemplateFormatVersion: '2010-09-09'
Description: Firehose sample
Resources:
  DataBucket:
    Type: AWS::S3::Bucket

Now I configured a Firehose. As a prerequisite I needed a role which would allow the Firehose to actually write into the S3 bucket.

KinesisRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - firehose.amazonaws.com
        Action:
        - sts:AssumeRole
    Path: "/"
    Policies:
      - PolicyName: KinesisRolePolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
              - s3:AbortMultipartUpload
              - s3:GetBucketLocation
              - s3:GetObject
              - s3:ListBucket
              - s3:ListBucketMultipartUploads
              - s3:PutObject
            Resource:
              - Fn::GetAtt: [ DataBucket , Arn ]
              - Fn::Join:
                - ""
                - - Fn::GetAtt: [ DataBucket , Arn ]
                  - "/*"

Now the Firehose configuration.

EventFirehose:
  Type: AWS::KinesisFirehose::DeliveryStream
  Properties:
    S3DestinationConfiguration:
      BucketARN:
        Fn::GetAtt: [ DataBucket, Arn ]
      BufferingHints:
        IntervalInSeconds: 60
        SizeInMBs: 10
      CompressionFormat: GZIP
      Prefix: events/
      RoleARN:
        Fn::GetAtt: [ KinesisRole, Arn ]

The API Gateway only consists of one method (without any resources), to keep the sample as simple as possible. Here the API Gateway definition.

ApiGatewayRestApi:
  Type: AWS::ApiGateway::RestApi
  Properties:
    Name:
      Fn::Join:
        - ""
        - - Ref: AWS::StackName
          - "-api"

Now the tricky part for all of this lies in the method definition (specifically the input mapping).

ApiGatewayPostMethod:
  Type: AWS::ApiGateway::Method
  Properties:
    ApiKeyRequired: true #to secure my API I used a simple API key. Otherwise my Firehose would be open to the internet.
    AuthorizationType: NONE
    HttpMethod: POST
    Integration:
      Type: AWS #signal that we want to use an internal AWS service
      Credentials:
        Fn::GetAtt: [ GatewayRole, Arn ] #role for the API to actually invoke the firehose
      Uri:
        Fn::Join:
          - ""
          - - "arn:aws:apigateway:"
            - Ref: AWS::Region
            - ":firehose:action/PutRecord" #this URI basically describes the service and action I want to invoke.
      IntegrationHttpMethod: POST #for kinesis using POST is required
      RequestTemplates:
        application/json: #now the mapping template for an incoming JSON
          Fn::Join:
            - ""
            - - "#set( $key = $context.identity.apiKey )\n" #assign the API key to local variable
              - "#set( $keyname = \"apiKey\" )\n"
              - "#set( $traceidval = $input.params().get(\"header\").get(\"X-Amzn-Trace-Id\"))" #get the trace id to later extract a timestamp of the incoming request
              - "#set( $bodyname = \"body\" )\n"
              - "#set( $traceid = \"traceid\")\n"
              - "#set( $body = $input.body )\n" #assign the request payload to variable
              - "#set( $quote = '\"' )\n"
              - "#set( $b64 = $util.base64Encode(\"{$quote$keyname$quote:$quote$key$quote,$quote$traceid$quote:$quote$traceidval$quote,$quote$bodyname$quote:$body}\") )\n"
              #now encode the payload in base64 to form a valid Firehose request
              - "{\n" #begin of the Firehose request json
              - "\"DeliveryStreamName\": \""
              - Ref: EventFirehose
              - "\",\n"
              - " \"Record\": { \"Data\": \"$b64\" }\n}" #end of the Firehose request json
      RequestParameters: #Firehose requires the content type to not be json, but amz-json
        integration.request.header.Content-Type: "'application/x-amz-json-1.1'"
      IntegrationResponses:
        - StatusCode: 200 #create a default response for the caller
          ResponseTemplates:
            application/json: '{"status":"OK"}'
    MethodResponses:
      - StatusCode: 200
    ResourceId:
      Fn::GetAtt: [ ApiGatewayRestApi , RootResourceId ]
    RestApiId: !Ref ApiGatewayRestApi

Now we still need the Gateway role to be defined.

GatewayRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - apigateway.amazonaws.com
        Action:
        - sts:AssumeRole
    Path: "/"
    Policies:
      - PolicyName: GatewayRolePolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
              - firehose:PutRecord
            Resource: "*"

I added a Deployment, so the API would automatically get deployed after it is created.

ApiGatewayDeployment:
  Type: AWS::ApiGateway::Deployment
  DependsOn:
    - ApiGatewayPostMethod
  Properties:
    RestApiId: !Ref ApiGatewayRestApi
    StageName: prod

Here is the overall CloudFormation template:

api-to-firehose-cf.yml template