What is Amazon Kinesis Data Streams?

A stream is a retained ordered log

Unlike a queue, a consumer reading a stream does not remove records for other consumers. Each application tracks its own position and can process the same retained data for different purposes such as analytics, detection, enrichment, or archival.

Ordering exists within a shard, not across the whole stream. A partition key determines how producer records are grouped and therefore affects order, throughput distribution, and consumer parallelism.

Design partitioning and capacity together

A heavily concentrated partition key can create a hot shard even when total stream traffic looks acceptable. Choose keys with enough distribution while keeping together the records whose relative order matters.

On-demand mode manages capacity for variable traffic, while provisioned mode exposes shard planning. In both modes, understand service limits, scaling response, record size, producer retries, and the traffic expected during recovery.

Give each consumer an explicit contract

Consumers can use shared throughput or enhanced fan-out depending on latency and isolation needs. Lambda event source mappings can manage polling and batch invocation for function consumers.

Consumers must handle duplicates, schema versions, partial batches, poison records, and iterator progress. Treat replay as a deliberate operational capability, including how old code interprets retained records.

Watch lag, skew, and data loss risk

Monitor write throttles, read throughput, iterator age, consumer errors, and shard distribution. Iterator age is a direct signal that processing is falling behind the head of the stream.

Set retention long enough for realistic incident recovery and document replay procedures. If recovery time can exceed retention, copy important raw data to a durable store such as Amazon S3.