Enable usage logs on AWS API Gateway

Usage logs are how Apiable sees what flows through your Amazon API Gateway. You deploy a Kinesis Firehose stream that ships your gateway's access logs to an S3 bucket, point the gateway stage at that stream, and paste a one-line JSON log format. Apiable then meters every call per plan and per subscription. This is the path that turns on usage-based billing and per-subscription analytics.

What do usage logs on AWS do for Apiable?

They give Apiable a per-request record of every API call. Apiable reads those records to attribute usage to the right plan and subscription, which is what usage-based billing and developer analytics run on.

A subscription on AWS already carries an API key and a rate limit. Usage logs add the missing piece: the actual call-by-call traffic, tagged with the plan and subscription it belongs to. Apiable ingests the delivered log files from your bucket and loads them so reports and metered billing can use them.

How do AWS access logs reach Apiable?

Through Amazon Kinesis Firehose. Your gateway stage writes access logs to a Firehose delivery stream, Firehose writes them to an S3 bucket, and Apiable ingests the files from that bucket.

The flow is:

1. Your API Gateway stage emits one access log line per request.
2. A Firehose delivery stream receives those lines and batches them. The stage writes to the stream directly, or through a CloudWatch Logs subscription filter if its access logs already go to CloudWatch.
3. Firehose writes the batched files to an S3 bucket under the prefix apiable/aws/logs/.
4. Apiable reads new files from that prefix, attributes each row to a plan and subscription, and loads them for billing and analytics.

Access logs or execution logs?

Access logs only. This distinction matters to security reviewers, so it is worth being precise about it.

Amazon API Gateway produces two different kinds of log, and they are configured separately on the stage:

Everything on this page configures Custom access logging only. Apiable never enables execution logging, never reads a CloudWatch execution log group, and has no permission to do either. The gateway role grants API Gateway actions only, with no CloudWatch Logs access at all.

The practical consequence is worth stating plainly: nothing reaches the bucket that is not a key in the log format you paste. Request bodies, response bodies, and headers are not among those keys, so they are never delivered. If your security team wants to review exactly what leaves the account, the log format table below is the complete list. There is no other channel.

How do you deploy the logs bucket and Firehose stream with one click?

Apiable publishes both as CloudFormation templates. You create two stacks from the AWS console, in order: the bucket first, then the stream that writes to it.

1. Open the CloudFormation console in the region your API Gateway runs in.
2. Choose Create stack, then With new resources.
3. Select Amazon S3 URL and paste the template URL for the stack you are creating.
4. Fill in the parameters, then create the stack.

Stack 1: the logs bucket

https://apiable-launchstack-templates.s3.amazonaws.com/apiable-logs-bucket/1.0.0/template.yaml

When the stack completes, open its Outputs tab and copy the bucket name, the bucket ARN, and the role ARN. You need the bucket ARN for the next stack, and all three go to Apiable.

Stack 2: the Firehose delivery stream

https://apiable-launchstack-templates.s3.amazonaws.com/apiable-usagelogs-stream/1.1.0/template.yaml

Copy the delivery stream ARN from the stack's Outputs tab, where it is listed as FirehoseArn. You paste it into the gateway stage when you turn on access logging.

How do you deploy the same components with Terraform?

Use the Apiable Terraform modules if you manage this account with Terraform. They create the same bucket and stream as the one-click stacks. Each module is published as a zip archive beside its template, so terraform init downloads it with no registry and no credentials.

module "apiable_logs_bucket" {
  source = "https://apiable-launchstack-templates.s3.amazonaws.com/apiable-logs-bucket/1.0.0/terraform.zip"
  name   = "your-tenant-name"
}

module "apiable_usagelogs_stream" {
  source          = "https://apiable-launchstack-templates.s3.amazonaws.com/apiable-usagelogs-stream/1.1.0/terraform.zip"
  name            = "usagelogs"
  logs_bucket_arn = module.apiable_logs_bucket.bucket_arn
}

After terraform apply, the bucket module outputs bucket_name, bucket_arn, and s3_assume_role_arn, and the stream module outputs firehose_arn. Send the bucket values to Apiable, and paste firehose_arn into the gateway stage.

If you already have an Apiable logs bucket, do not apply the bucket module for it. Import the existing bucket into Terraform first. Each module's README, inside the archive, lists the import commands.

How do you deploy the same components with the CDK?

Use this route only if you already manage the account with the CDK. It produces the same bucket and stream as the one-click stacks, on the direct path.

Run the two scripts from the public apiable/cdk repository against your AWS account: first the logs bucket, then the Firehose stream that targets it. Bootstrap CDK once first if this is a new account or region.

1. Clone apiable/cdk and install the AWS CDK toolkit.
2. Bootstrap CDK for your account and region, for example ./cdk-bootstrap.sh 123456789012 eu-central-1.
3. Deploy the S3 logs bucket. Skip this if you already have one.
4. Deploy the Firehose stream, passing the bucket ARN from the previous step.

Deploy the S3 logs bucket

deploy-logs-bucket.sh takes your AWS account ID, region, and a stack name. The stack name becomes part of the bucket, so a stack name of dev produces a bucket named apiable-logs-dev.

./deploy-logs-bucket.sh 123456789012 eu-central-1 dev

This creates the stack apiable-<stackname>-logs-bucket and a bucket whose ARN looks like arn:aws:s3:::apiable-logs-<stackname>. Note the bucket name, its ARN, the role ARN, and the region. You will pass the bucket ARN to the next step and send the rest to Apiable.

Deploy the Firehose stream

deploy-usagelogs-stream.sh takes your AWS account ID, region, the logs bucket ARN, and a stack name. It creates a Firehose delivery stream that writes to the bucket under the apiable/aws/logs/ prefix.

./deploy-usagelogs-stream.sh 123456789012 eu-central-1 arn:aws:s3:::apiable-logs-dev dev

The stream is named amazon-apigateway-usagelogs-<stackname>, so a stack name of dev produces amazon-apigateway-usagelogs-dev. The name must start with amazon-apigateway-, which is an API Gateway requirement for access log destinations. The script writes the stream's ARN to cdk-outputs.json. Copy that ARN for the gateway stage.

The CDK scripts build the direct path only. For the CloudWatch Logs path, use the one-click stack or the Terraform module.

How do you turn on access logging in the API Gateway console?

In the API Gateway console, open the stage, edit its Logs and Tracing settings, enable custom access logging, paste the Firehose ARN as the destination, and paste the one-line log format. Then save.

1. Open the API Gateway console and select your API.
2. Go to Stages and select the stage you want to meter.
3. Open Logs and Tracing and choose to edit.
4. Enable Custom access logging.
5. In Access log destination ARN, paste the Firehose stream ARN from the stack outputs, the Terraform output, or cdk-outputs.json.
6. In Log Format, paste the one-line JSON format below.
7. Save the changes.

How do you feed the stream from CloudWatch Logs?

Create the stream with LogSource set to cloudwatch_logs, keep the stage's access logs going to their CloudWatch log group, and add a subscription filter on that group that sends to the stream. Firehose unwraps the CloudWatch records, so the bucket receives the same rows as on the direct path.

1. Create the stream with LogSource set to cloudwatch_logs in the one-click stack, or log_source = "cloudwatch_logs" in the Terraform module.
2. On the stage, keep Custom access logging pointed at your CloudWatch log group, and set Log Format to the one-line format below. The format is the same on both paths.
3. Create an IAM role that CloudWatch Logs can assume, with permission to call firehose:PutRecord on the stream.
4. Create a subscription filter on the access-log group, with the Firehose stream as the destination and the role from step 3. Leave the filter pattern empty so every line is forwarded.

What is the exact log format to paste?

Use this JSON object as a single line. The keys are fixed. Apiable reads them by name, so paste the format exactly as it appears here even if some values will not resolve on your gateway.

{"api_id": "$context.apiId","api_key": "$context.identity.apiKey","key_id": "$context.identity.apiKeyId","ip": "$context.identity.sourceIp","method": "$context.httpMethod","uri": "$context.path","response_size": "$context.responseLength","response_status": "$context.status","resource_id": "$context.resourceId","request_id": "$context.requestId","request_latency": "$context.responseLatency","request_time": "$context.requestTimeEpoch","stage": "$context.stage","plan_id": "$context.authorizer.apiable_plan_id","subscription_id": "$context.authorizer.apiable_subscription_id"}

When does usage start showing up?

After the stream is delivering and Apiable has wired up your bucket, usage appears once calls flow through the metered stage. Firehose batches before it writes, so allow a short delay between a call and the record landing.

The Firehose stream buffers records for up to five minutes, or until 5 MB accumulate, before writing to S3. Once files land under apiable/aws/logs/, Apiable ingests them about once an hour and the usage becomes available to billing and analytics. Allow for both delays when you test, so usage can take a little over an hour to appear. Make a few test calls through the stage, wait for the next ingest, then check Developer Analytics for the subscription.

Troubleshooting

Match what you see to the fix.

Where to next