500 million records a day into OpenSearch, without losing any

A Magento 2 estate was producing more event and log data than its search layer could take. Queries that used to return instantly were timing out, and the ingestion path dropped records whenever a sale doubled traffic. The requirement was blunt: take everything, lose nothing, keep search under a second.

What was breaking

Applications wrote to the search cluster directly. That works until traffic spikes, at which point the cluster applies back-pressure, the application times out, and the record is gone. There was no buffer between producer and consumer, so the fastest producer set the pace for everyone.

The pipeline

• Producers write to Kinesis Firehose, which absorbs bursts and hands downstream a rate it can cope with
• Lambda transforms records in flight: normalising fields, dropping malformed entries into a separate stream rather than the main one, and enriching with the identifiers search needs
• Firehose delivers to OpenSearch Service, and in parallel writes the raw batch to S3
• Aurora holds the reference data the transform needs, cached in the Lambda execution environment

The parallel S3 write is what makes "lose nothing" true. If OpenSearch rejects a batch, the raw records are still in S3 and can be replayed. Without it, a rejected batch is a lost batch.

Keeping search fast

• Time-based index rotation, so a query for last week touches last week's indices and not the entire history
• Shard count sized against actual document counts rather than the default, which had produced thousands of tiny shards
• Index State Management moving older indices to warm nodes and eventually out of the cluster
• Field mappings defined explicitly instead of relying on dynamic mapping, which had been indexing fields nobody queried

The shard fix alone accounted for most of the query improvement. It usually does.

Storage

Batches land in S3 gzipped and partitioned by date. A lifecycle policy moves partitions past the hot window to infrequent access and then to Glacier tiers. The data stays queryable through Athena, which covers the occasional historical question without keeping years of records on search-cluster storage.

Where it landed

• 500M+ records ingested per day with no dropped entries in the failure modes we can reproduce
• Sub-second search over billions of documents
• About 60% lower storage cost from compression and lifecycle tiering
• Spikes now queue instead of failing, so a sale is a graph shape rather than an incident

What was hard

Mapping conflicts. Years of dynamic mapping had produced fields typed differently in different indices, and reindexing that history without a search outage took longer than building the pipeline did.

What we still do

Cluster sizing review each quarter, ingestion monitoring, and the alerting on Firehose delivery failures.