Amazon Kinesis Data Streams

The problem

Acme Shop wants to see what shoppers do while they are still doing it. Every page view, search and add-to-cart in the web shop and the mobile app becomes a click event, about 2,000 a second at busy times and around 1 KB each. Several teams want the same events: a live dashboard needs conversion and error rates within seconds, the data team wants every click in the S3 data lake, and a stream-processing job groups clicks into sessions to spot bots and abandoned carts. When that job gets a bug fix, its team will want to run it over last week's clicks again.

Calling each consumer from the web servers ties the shop to every consumer's speed and uptime. A queue hands each message to one consumer and deletes it, so every reader would need its own copy and nobody could go back in time. Writing the clicks into a database table, only to scan them out again, costs more than it should at this rate. What fits is a log: write each event once, let every reader work through it at its own pace, and keep it for a while.

Amazon Kinesis Data Streams is AWS's managed log of that kind. A stream is divided into shards; producers add records over an HTTPS API, consumers read each shard in order, and records stay for the retention period, 24 hours by default and up to 365 days, whether anyone has read them or not. AWS runs the servers and replicates every record synchronously across three Availability Zones. Acme still decides how much capacity to buy, which key to partition by and how long to keep the data.

How it works

Streams, shards and records

• A stream is a set of shards, and a shard is both an ordered sequence of records and a unit of capacity. In provisioned mode each shard accepts up to 1 MB/s or 1,000 records a second of writes, partition keys included, and serves up to 2 MB/s of reads through at most 5 GetRecords calls a second. Acme's stream clicks has 4 shards: 4 MB/s or 4,000 records a second in and 8 MB/s out, and its 2,000 clicks a second use half of that.
• A record is a partition key, a data blob and a sequence number. Kinesis never looks inside the blob. A record can be at most 1 MiB by default; since October 2025 a stream can be configured for records of up to 10 MiB (UpdateMaxRecordSize). Large records borrow burst capacity from their shard, whose limits stay the same, so AWS means them for occasional payloads and suggests keeping them under 2% of the traffic.
• Shards have IDs such as shardId-000000000001; the diagram calls Acme's four shards 0 to 3. A record can typically be read less than a second after it was written.

Partition keys, hash keys and sequence numbers

Every record carries a partition key, a Unicode string of up to 256 characters. Kinesis computes the MD5 digest of the key and reads it as a 128-bit unsigned integer, the hash key. Each open shard owns a contiguous hash key range, and together the ranges cover every value from 0 to 2^128 − 1. A new stream divides that space evenly, so each of Acme's 4 shards owns a quarter, and a record goes to the shard whose range holds its hash key. Every record with the same key therefore lands in the same shard. A producer that wants to pick the shard itself sends an ExplicitHashKey instead.

The numbers in the animation are real. The MD5 of s-41 starts with 6e19676a, which is in the second quarter, so session s-41 always lands in shard 1; the scraper's session s-90 (b772d1fa…) lands in shard 2:

import hashlib

def hash_key(partition_key: str) -> int:
    # Kinesis reads the MD5 digest of the key as a 128-bit unsigned integer
    return int.from_bytes(hashlib.md5(partition_key.encode("utf-8")).digest(), "big")

def shard_of(partition_key: str, shards: int = 4) -> int:
    # valid while the shards split the key space evenly, as after CreateStream
    return hash_key(partition_key) * shards >> 128

for key in ["s-41", "s-90"]:
    print(key, f"{hash_key(key):032x}"[:8], shard_of(key))
# s-41 6e19676a 1
# s-90 b772d1fa 2

list-shards reports the ranges in decimal:

aws kinesis list-shards --stream-name clicks --output text \
  --query 'Shards[].[ShardId, HashKeyRange.StartingHashKey, HashKeyRange.EndingHashKey]'
# shardId-000000000000  0                                        85070591730234615865843651857942052863
# shardId-000000000001  85070591730234615865843651857942052864   170141183460469231731687303715884105727
# shardId-000000000002  170141183460469231731687303715884105728  255211775190703847597530955573826158591
# shardId-000000000003  255211775190703847597530955573826158592  340282366920938463463374607431768211455

When a shard stores a record it assigns a sequence number, a long decimal string that grows over time within the shard, and the reply to PutRecord or PutRecords returns it with the shard ID. Unlike Kafka offsets, sequence numbers are not consecutive: they work as positions ("start after this record"), not as a count of records. The diagram shortens them to three digits.

Capacity modes

• Provisioned. You choose the number of shards and pay for each shard-hour whether the shard is busy or not, plus a charge per 25 KB of data written. You scale with UpdateShardCount or by splitting and merging single shards (see below). AWS suggests this mode for predictable traffic and when you want control over which hash keys go to which shard; its sizing rule is the larger of the write rate divided by 1 MB/s and the read rate of all consumers together divided by 2 MB/s.
• On-demand Standard. You set no shard count. A new stream takes 4 MB/s of writes; Kinesis then adds shards so that a stream can absorb twice the highest write rate of the previous 30 days, and splits a shard within about 15 minutes once it receives more than 500 KB/s. Traffic that more than doubles the previous peak within 15 minutes can still be throttled until the stream catches up. A stream can grow to 10 GB/s of writes and 20 GB/s of reads in US East (N. Virginia), US West (Oregon) and Europe (Ireland), and to 200 MB/s and 400 MB/s in other Regions unless AWS Support raises it, and an account can have 50 on-demand streams by default. You pay per GB written and read, plus a fixed charge per stream-hour.
• On-demand Advantage (November 2025) is a setting for all the on-demand streams of an account in one Region. It drops the per-stream charge, makes writes, reads and longer retention at least 60% cheaper per GB, charges enhanced fan-out reads like ordinary ones, raises the number of enhanced fan-out consumers per stream from 20 to 50 (in the Regions the documentation lists), and lets you set warm throughput so that a stream is ready for a burst before it arrives; since July 2026 warm throughput can also shrink a stream again. In return the account commits to at least 25 MB/s of writes and 25 MB/s of reads across its on-demand streams, billed as a shortfall when it uses less, and the setting can't be turned off for 24 hours after it is turned on. AWS positions it for accounts that write at least 10 MB/s, fan out to more than two consumer applications or run many streams.
• A stream can switch between provisioned and on-demand twice in 24 hours without interrupting producers or consumers. It keeps its shards, and from then on either you or Kinesis manages them.
• Service-managed partition keys (September 2026) are a setting for on-demand streams: with the record distribution strategy AUTO instead of the default USER_PARTITION_KEY, Kinesis ignores partition keys and spreads records over the shards itself, at no extra charge. Hot keys go away, and so does ordering. AWS suggests it for logs, metrics, telemetry and clickstreams that don't need order, and keeps partition keys for change data capture, per-account transactions and session analytics; Acme's Flink job builds sessions, so it keeps the session ID as its key.

Writing records

• PutRecord writes one record. PutRecords writes up to 500 records in a request of up to 10 MiB including partition keys (5 MiB before October 2025), and the records can go to different shards.
• PutRecords is not all-or-nothing. The reply lists every record in request order, each with either its ShardId and SequenceNumber or an ErrorCode (ProvisionedThroughputExceededException or InternalFailure), and FailedRecordCount says how many failed. A producer that doesn't resend the failed records, with exponential backoff, loses them. Because each record is handled on its own, PutRecords doesn't promise to keep the order of a request; a producer that needs strict order for a key sends one PutRecord at a time and passes the previous record's sequence number in SequenceNumberForOrdering.
• Retries write duplicates. If a call times out after Kinesis stored the record, the retry stores it again under a new sequence number. Acme gives every click a clickId, so readers can drop the copy.
• The Kinesis Producer Library (KPL), a Java library, batches, retries and rate-limits for you. It can also aggregate many small user records into one Kinesis record, which matters when records are smaller than 1 KB, since then the 1,000 records a second per shard is the limit reached first. Readers have to unpack them again: the KCL does it, Firehose does it automatically, and Lambda functions use AWS's de-aggregation modules. KPL 1.x (December 2024) is built on the AWS SDK for Java 2.x, KPL 0.x reached end of support on 30 January 2026, and aggregation doesn't work with service-managed partition keys.
• AWS services write to streams too: the Kinesis Agent tails log files, and CloudWatch Logs, AWS IoT Core, EventBridge, CloudFront real-time logs and DynamoDB (see Where it fits) can deliver into a stream.

From the CLI, with the records in a file (raw-in-base64-out sends the data as written instead of expecting base64):

aws kinesis put-records --stream-name clicks \
  --cli-binary-format raw-in-base64-out --records file://clicks.json

[
  { "Data": "{\"clickId\":\"c-90211\",\"page\":\"/cart\"}", "PartitionKey": "s-41" }
]

Reading: shared throughput or enhanced fan-out

• A shared-throughput consumer, the default kind, asks GetShardIterator for a starting point: the oldest retained record (TRIM_HORIZON), the newest (LATEST), a sequence number (AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER) or a time (AT_TIMESTAMP). It then calls GetRecords in a loop; one call returns up to 10,000 records or 10 MB, and an iterator expires after 5 minutes. All such consumers of a shard share its 5 calls and 2 MB a second, and after a call that returns 10 MB the calls in the next 5 seconds are throttled. AWS gives an average delay of about 200 ms from write to read with one consumer, rising to about 1 second with five.
• An enhanced fan-out (EFO) consumer is registered on the stream with RegisterStreamConsumer. It calls SubscribeToShard, and Kinesis then pushes records to it over an HTTP/2 connection for up to 5 minutes, after which the consumer subscribes again. Each registered consumer gets its own 2 MB/s per shard, whatever the others do, with a typical delay of about 70 ms. A stream can have 20 registered consumers, or 50 under On-demand Advantage. In provisioned mode EFO adds a charge per consumer-shard hour and per GB read.
• For on-demand streams AWS recommends a single shared-throughput application, so that it has room to catch up after an outage, and enhanced fan-out for every other one.

The usual consumers

• AWS Lambda. An event source mapping polls each shard about once a second (sharing it with the other pollers) or reads through an EFO consumer, and invokes the function synchronously with a batch from one shard: 100 records by default and up to 10,000, collected for up to MaximumBatchingWindowInSeconds (300 at most) and up to a 6 MB payload. It works on one batch per shard at a time, in order; ParallelizationFactor (1 to 10) runs several batches of a shard at once and still keeps the records of each partition key in order. Start new mappings at TRIM_HORIZON: with LATEST, a mapping can miss records written while it is being created or updated.
• When the function fails, Lambda by default retries the batch until its records expire from the stream, and the rest of the shard waits behind it. Bound that with MaximumRetryAttempts and MaximumRecordAgeInSeconds; let BisectBatchOnFunctionError split a failing batch in two, which doesn't count against the retries, to corner the bad record; and send what is finally given up to an on-failure destination, an SQS queue, SNS topic or S3 bucket. With ReportBatchItemFailures the function returns the sequence number of the first record it couldn't process, and Lambda retries from there instead of from the start of the batch. Delivery is at least once, so the function must be idempotent.

aws lambda create-event-source-mapping --function-name live-metrics \
  --event-source-arn arn:aws:kinesis:us-east-1:111122223333:stream/clicks \
  --starting-position TRIM_HORIZON --batch-size 100 --parallelization-factor 2 \
  --bisect-batch-on-function-error --maximum-retry-attempts 3 \
  --maximum-record-age-in-seconds 3600 --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:111122223333:live-metrics-failed"}}'

• Amazon Data Firehose, called Kinesis Data Firehose until February 2024, reads a stream from LATEST with GetRecords, once a second per shard (twice with full backup turned on), and counts against the shard's shared limits like any other poller. It de-aggregates KPL records, buffers them per destination (for S3, 1 to 128 MiB and 0 to 900 seconds, 5 MiB and 300 s by default, whichever is reached first) and then writes an object; it can also convert records to Parquet or ORC and partition them by content. With a stream as its source, Firehose retries a failed S3 delivery for as long as the stream still holds the records.
• Amazon Managed Service for Apache Flink, called Kinesis Data Analytics until August 2023, runs Apache Flink applications. Flink's Kinesis source polls by default; set its reader type to EFO and give it a consumer name, and the connector registers that consumer and reads through it. Flink keeps its shard positions in its own checkpoints rather than in Kinesis or DynamoDB, so it restores its state and its positions together after a failure, and it reads a parent shard to the end before its children.
• Kinesis Client Library (KCL) applications run on EC2, ECS, EKS or anywhere else. The KCL is a Java library (other languages go through its MultiLangDaemon) that gives each shard to one worker under a lease and stores each lease's checkpoint, the position of the last record processed, in a DynamoDB lease table named after the application. KCL 3.x (November 2024) adds a worker-metrics table and a coordinator-state table and moves leases away from workers whose CPU is busiest; since KCL 3.5 all three can live in one table. Consumers built with KCL 2.0 or later use enhanced fan-out by default. KCL 1.x reached end of support on 30 January 2026.
• Since August 2026 Kinesis can also deliver a stream by itself, with no consumer to run, to a general-purpose S3 bucket in the source format or to Apache Iceberg tables in Amazon S3 Tables (streaming tables). Both need an on-demand stream and are billed per GB delivered. EventBridge Pipes, AWS Glue streaming jobs, Spark on Amazon EMR, Amazon Redshift streaming ingestion and Amazon OpenSearch Ingestion read streams as well.

Order and duplicates

• Order holds within a shard. Records with the same partition key go to the same shard, and every reader sees a shard's records in sequence-number order; across shards there is no order. Order can still be lost on the way in: PutRecords doesn't keep the order of a request, and a retried record lands after the ones sent in the meantime.
• Resharding keeps order only if readers cooperate. After a split or a merge the old (parent) shard stops taking writes but keeps its records, so a reader has to finish the parent before it starts on the children. The KCL, Lambda and Flink's connector do this.
• Delivery is at least once on both sides. Producer retries store the same data twice under two sequence numbers, and consumers see records again after a restart, a deployment, a lease moving to another worker, a reshard or a retried batch; AWS notes that consumer retries cause most duplicates. Carry a unique ID in each record and make consumers idempotent.
• Item changes that DynamoDB writes to a stream can also arrive out of order and more than once; their ApproximateCreationDateTime attribute tells the real order and exposes duplicates.

Resharding, hot shards and hot keys

• SplitShard divides one shard's hash key range in two at a hash key you choose, and MergeShards joins two shards whose ranges are adjacent; each operation takes one or two shards. The parent shard becomes CLOSED (no new records, the old ones still readable) and then EXPIRED once the retention period has passed, and the children get new shard IDs.
• UpdateShardCount resizes a provisioned stream evenly (UNIFORM_SCALING) by doing the splits and merges for you. By default a stream can be scaled 10 times in a rolling 24 hours, to at most double or at least half of its current shard count per call and to at most 10,000 shards; AWS recommends targets in steps of 25% of the current count, which complete faster.
• A hot shard receives more than its share. AWS suggests finding it with shard-level metrics or by logging the shard ID that every put returns, then splitting just that shard. Acme splits shard 2 in the middle of its range, so the bot's key ends up in a child shard with half of shard 2's normal traffic: 250 + 600 = 850 records a second, under the limit.

aws kinesis split-shard --stream-name clicks \
  --shard-to-split shardId-000000000002 \
  --new-starting-hash-key 212676479325586539664609129644855132160   # 2^127 + 2^125, hex a000…

• A hot key can't be split. One key has one hash key and so lives in one shard, which never takes more than 1 MB/s or 1,000 records a second, in any capacity mode; on-demand splits busy shards, but it doesn't separate the keys inside them. The cures are a different key (for example the session ID plus a suffix for known heavy senders, giving up their order), stopping the source, or service-managed partition keys for a stream that doesn't need order.

Retention

• Records stay 24 hours by default. IncreaseStreamRetentionPeriod raises the period to at most 8,760 hours (365 days), and records that hadn't expired yet stay; DecreaseStreamRetentionPeriod lowers it to no less than 24 hours and makes older records unreadable almost at once.
• Acme keeps 7 days, so a fixed Flink job can replay last week:

aws kinesis increase-stream-retention-period --stream-name clicks --retention-period-hours 168

• The retention period is also the deadline for a stalled reader: whatever it hasn't read by then is gone. AWS suggests an alarm on the maximum of GetRecords.IteratorAgeMilliseconds that fires well before it reaches half the retention period.

Encryption and access

• Server-side encryption uses an AWS KMS key: either the AWS managed key aws/kinesis, which costs nothing although Kinesis's calls to KMS are billed, or a customer managed key, which producers and consumers need permission to use. It applies to records written after you turn it on (StartStreamEncryption); records already in the stream stay unencrypted.
• Clients connect over TLS (1.2 required, 1.3 recommended). IAM policies separate producers (kinesis:PutRecord, kinesis:PutRecords), consumers and the operators who reshard. A resource-based policy on a stream or a consumer shares it with another account, for example with a Lambda function there; sharing an encrypted stream that way needs a customer managed key. Interface VPC endpoints keep the traffic inside AWS's network (see Private Endpoints).

Metrics to watch

• IncomingBytes and IncomingRecords against the shard limits; WriteProvisionedThroughputExceeded and PutRecords.ThrottledRecords for throttled writes.
• GetRecords.IteratorAgeMilliseconds, the age of the last record read through GetRecords, which shows how far the slowest polling reader is behind (Lambda reports its own IteratorAge per function); SubscribeToShardEvent.MillisBehindLatest does the same for enhanced fan-out readers; ReadProvisionedThroughputExceeded counts throttled reads.
• Stream-level metrics are free. Shard-level metrics, which name the hot shard, cost extra and are switched on per stream with EnableEnhancedMonitoring.

Pricing

Prices in US East (N. Virginia), from the AWS price list of September 2026:

For Acme's stream in a 30-day month: $43.20 for 4 shards, $72.58 for 5,184 million payload units (2,000 records a second, each under 25 KB), $57.60 for 7 days of retention and, for Flink's enhanced fan-out, $43.20 of consumer-shard hours plus about $64 for 4,944 GB read. That is about $281, before Lambda, Firehose and Flink charge for their own work. The same traffic in On-demand Standard would cost about $1,166 ($396 to write, $396 for the two polling readers, $247 for Flink, $29 for the stream and $99 for retention), and On-demand Advantage would bill its minimum, about $3,040 a month. Steady, predictable traffic is where provisioned shards pay off.

Data between Kinesis and producers or consumers in the same Region is not charged, and Kinesis isn't part of the AWS Free Tier. Kinesis's own S3 delivery is billed on top of the stream: in US East (N. Virginia), $0.0275 per GB delivered from an On-demand Standard stream and $0.011 under On-demand Advantage.

Where it fits

• Solutions: clickstream and product analytics, as here; telemetry pipelines that collect logs, metrics and IoT readings; change data capture from DynamoDB tables, or from relational databases through AWS DMS or Debezium; live dashboards, fraud and anomaly detection; landing raw events in an S3 data lake, the bronze layer of a medallion architecture.
• Patterns it implements or supports: Publish-Subscribe with replay, since every reader gets every record; the event backbone of an event-driven architecture; sharding by hash range, with the same hot-key problem; competing consumers at the granularity of a shard, as KCL workers divide a stream's shards between them; idempotent consumers to absorb redelivery; CQRS read models and materialized views kept current by consumers.
• Usual neighbours: producers on ECS, EC2 or Lambda, the Kinesis Agent, CloudFront real-time logs, IoT Core and DynamoDB; consumers such as Lambda, Data Firehose, Managed Service for Apache Flink, KCL applications and Redshift streaming ingestion; S3, Glue and Athena downstream; CloudWatch, KMS and IAM around it.
• Managed offerings: Kinesis Data Streams is itself the managed service, and it exists only on AWS. AWS's other managed log is Amazon MSK, which runs Apache Kafka. On other clouds the closest services are Azure Event Hubs and Google Cloud Pub/Sub.

When to use it

Choose Kinesis Data Streams when several applications on AWS need the same events, in order per key, and may need to read them again; when the volume needs partitioning but nobody wants to run brokers; and when the readers are AWS services that consume streams directly, such as Lambda, Data Firehose and Managed Service for Apache Flink. Look elsewhere when each message should go to one worker (SQS), when a message must reach many endpoints at once (SNS), when events should be routed by their content (EventBridge), when the data only has to land in storage (Data Firehose with direct puts, or Kinesis's own S3 delivery), or when the team wants Kafka's protocol, connectors and stream-processing libraries (Amazon MSK).

Figures from the AWS documentation in October 2026 and from the pages of this catalog that the table links to.

Trade-offs

• Capacity comes per shard, and per key. A shard's 1 MB/s and 1,000 records a second are hard limits, and a single key can never use more than one shard. A skewed key throttles writes while the stream as a whole has room, as in step 4: one shard can't lend its spare capacity to another.
• Order only within a shard, delivery at least once. Producers and consumers both create duplicates, and keeping order across shards, or through retries, is the application's job.
• Polling readers share 2 MB/s per shard. Two or three polling applications fill a shard's read capacity, so each further reader usually needs enhanced fan-out, which costs extra outside On-demand Advantage.
• Resharding is coarse. Splits and merges handle one or two shards at a time, UpdateShardCount allows 10 changes a day of at most double or half, and readers have to drain parent shards first.
• Small records, short memory by default. 1 MiB per record (10 MiB if raised, for occasional use) and 24 hours of retention; every extra day costs money.
• You pay for capacity, not only for use, per shard-hour in provisioned mode and per stream-hour in On-demand Standard; On-demand Advantage swaps those for an account-wide minimum.
• AWS only. The API and the client libraries are Kinesis's own; moving to Kafka or another cloud means new producer and consumer code.

Implementation notes

• Choose the partition key for order and spread. It should keep together what must stay in order (a session, an account, a device) and have far more values than there are shards, none much busier than the rest. Look out for keys that can turn hot: a bot, a very large merchant, a default value such as unknown.
• Size provisioned streams for the peak, with headroom: the larger of writes ÷ 1 MB/s (or records ÷ 1,000) and polling reads ÷ 2 MB/s, rounded up, plus room for bursts and for readers catching up. When traffic is unpredictable, start on-demand and switch once the pattern is known.
• Handle partial failures in producers. Check FailedRecordCount on every PutRecords reply and resend only the failed records with exponential backoff and jitter, or let the KPL do it.
• Make every consumer idempotent with an ID carried in the record, and checkpoint only after the work is done.
• Give Lambda mappings a failure policy: bounded MaximumRetryAttempts and MaximumRecordAgeInSeconds, BisectBatchOnFunctionError or ReportBatchItemFailures, and an on-failure destination, so one bad record can't stall its shard until it expires.
• Alarm on lag and throttling: the maximum of GetRecords.IteratorAgeMilliseconds (and Lambda's IteratorAge) well before half the retention period, WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded.
• Keep KCL applications tidy. Give each application a unique name, because its lease table is named after it, and delete its DynamoDB tables when you retire it; the KCL leaves them behind.
• Encrypt from the start and split permissions. Turn on server-side encryption before the first record is written, since older records stay unencrypted, and give resharding rights to an operator role rather than to producers and consumers.
• Rehearse throttling. Since October 2025, AWS Fault Injection Service has Kinesis actions that inject ProvisionedThroughputExceededException and ExpiredIteratorException errors, so you can see how producers and consumers cope before a real hot shard shows you.