Enable usage logs on AWS API Gateway

Usage logs are how Apiable sees what flows through your Amazon API Gateway. You deploy a Kinesis Firehose stream that ships your gateway's access logs to an S3 bucket, point the gateway stage at that stream, and paste a one-line JSON log format. Apiable then meters every call per plan and per subscription. This is the path that turns on usage-based billing and per-subscription analytics.

What do usage logs on AWS do for Apiable?

They give Apiable a per-request record of every API call. Apiable reads those records to attribute usage to the right plan and subscription, which is what usage-based billing and developer analytics run on.

A subscription on AWS already carries an API key and a rate limit. Usage logs add the missing piece: the actual call-by-call traffic, tagged with the plan and subscription it belongs to. Apiable ingests the delivered log files from your bucket and loads them so reports and metered billing can use them.

How do AWS access logs reach Apiable?

Through Amazon Kinesis Firehose. Your gateway stage writes access logs to a Firehose delivery stream, Firehose writes them to an S3 bucket, and Apiable ingests the files from that bucket.

The flow is:

1. Your API Gateway stage emits one access log line per request.
2. A Firehose delivery stream receives those lines and batches them.
3. Firehose writes the batched files to an S3 bucket under the prefix apiable/aws/logs/.
4. Apiable reads new files from that prefix, attributes each row to a plan and subscription, and loads them for billing and analytics.

How do you deploy the logs bucket and Firehose stream with one click?

Apiable publishes both as CloudFormation templates. You create two stacks from the AWS console, in order: the bucket first, then the stream that writes to it.

1. Open the CloudFormation console in the region your API Gateway runs in.
2. Choose Create stack, then With new resources.
3. Select Amazon S3 URL and paste the template URL for the stack you are creating.
4. Fill in the parameters, then create the stack.

Stack 1: the logs bucket

https://apiable-launchstack-templates.s3.amazonaws.com/apiable-logs-bucket/1.0.0/template.yaml

When the stack completes, open its Outputs tab and copy the bucket name, the bucket ARN, and the role ARN. You need the bucket ARN for the next stack, and all three go to Apiable.

Stack 2: the Firehose delivery stream

https://apiable-launchstack-templates.s3.amazonaws.com/apiable-usagelogs-stream/1.0.0/template.yaml

Copy the delivery stream ARN from the stack's Outputs tab. You paste it into the gateway stage in the next section.

How do you deploy the same components with the CDK?

Use this route only if you already manage the account with the CDK. It produces the same bucket and stream as the one-click stacks.

Run the two scripts from the apiable/cdk repository against your AWS account: first the logs bucket, then the Firehose stream that targets it. Bootstrap CDK once first if this is a new account or region.

1. Clone apiable/cdk and install the AWS CDK toolkit.
2. Bootstrap CDK for your account and region, for example ./cdk-bootstrap.sh 123456789012 eu-central-1.
3. Deploy the S3 logs bucket. Skip this if you already have one.
4. Deploy the Firehose stream, passing the bucket ARN from the previous step.

Deploy the S3 logs bucket

deploy-logs-bucket.sh takes your AWS account ID, region, and a stack name. The stack name becomes part of the bucket, so a stack name of dev produces a bucket named apiable-logs-dev.

./deploy-logs-bucket.sh 123456789012 eu-central-1 dev

This creates the stack apiable-<stackname>-logs-bucket and a bucket whose ARN looks like arn:aws:s3:::apiable-logs-<stackname>. Note the bucket name, its ARN, and the region. You will pass the ARN to the next step and send all three to Apiable.

Deploy the Firehose stream

deploy-usagelogs-stream.sh takes your AWS account ID, region, the logs bucket ARN, and a stack name. It creates a Firehose delivery stream that writes to the bucket under the apiable/aws/logs/ prefix.

./deploy-usagelogs-stream.sh 123456789012 eu-central-1 arn:aws:s3:::apiable-logs-dev dev

The stream is named amazon-apigateway-<name>. The name must start with amazon-apigateway-, which is an API Gateway requirement for access log destinations. The stream's ARN is written to cdk-outputs.json as the usagelogs-...-firehose-arn output. Copy that ARN. You paste it into the gateway stage in the next section.

How do you turn on access logging in the API Gateway console?

In the API Gateway console, open the stage, edit its Logs and Tracing settings, enable custom access logging, paste the Firehose ARN as the destination, and paste the one-line log format. Then save.

1. Open the API Gateway console and select your API.
2. Go to Stages and select the stage you want to meter.
3. Open Logs and Tracing and choose to edit.
4. Enable Custom access logging.
5. In Access log destination ARN, paste the Firehose stream ARN from cdk-outputs.json.
6. In Log Format, paste the one-line JSON format below.
7. Save the changes.

What is the exact log format to paste?

Use this JSON object as a single line. The keys are fixed. Apiable reads them by name, so paste the format exactly as it appears here even if some values will not resolve on your gateway.

{"api_id": "$context.apiId","api_key": "$context.identity.apiKey","key_id": "$context.identity.apiKeyId","ip": "$context.identity.sourceIp","method": "$context.httpMethod","uri": "$context.path","response_size": "$context.responseLength","response_status": "$context.status","resource_id": "$context.resourceId","request_id": "$context.requestId","request_latency": "$context.responseLatency","request_time": "$context.requestTimeEpoch","stage": "$context.stage","plan_id": "$context.authorizer.apiable_plan_id","subscription_id": "$context.authorizer.apiable_subscription_id"}

When does usage start showing up?

After the stream is delivering and Apiable has wired up your bucket, usage appears once calls flow through the metered stage. Firehose batches before it writes, so allow a short delay between a call and the record landing.

The Firehose stream buffers records before writing to S3, so a single call is not delivered instantly. Once files land under apiable/aws/logs/, Apiable ingests them every 15 minutes and the usage becomes available to billing and analytics. Allow for both delays when you test. Make a few test calls through the stage, wait for the next ingest, then check developer analytics for the subscription.

Troubleshooting

Match what you see to the fix.

Where to next