Observability Architecture at 6 TB/Day
At small-team scale, the entire LGTM stack is three single binaries pointed at one S3 bucket. I wrote that build up separately (LGTM for small teams), and for two to ten engineers it’s the right answer — Grafana’s own Loki docs cap monolithic mode at roughly 20 GB a day (Grafana Loki install docs, accessed June 2026). This post is the other end of that line. The stack here moves on the order of 6 TB of telemetry a day across 15 departments, and at that volume the single binary is no longer an option. Every backend splits into its component microservices, each scaled on its own axis, sharded across object storage, with retention and cardinality enforced as hard limits rather than good intentions. That is roughly three hundred times the small-team volume, and almost nothing about the small-team build survives the jump intact. The reason it was worth building at all — an 85% cut against the SaaS bill — is the Datadog alternative teardown; this post is purely the how.
The thing worth saying up front: the difference between the two builds is not “more of the same.” It is a different architecture. Monolithic mode runs every internal component of Loki, Tempo, and Mimir inside one process because at low volume the process boundaries cost more than they buy. At 6 TB a day the process boundaries are the entire point — they are what let you put forty ingesters on the write path and four queriers on the read path and scale the two numbers independently as the load on each moves. If you take one thing from this post, take that: at scale, the unit of scaling is the component, not the backend. You stop thinking “I need more Loki” and start thinking “I need more distributors but the ingesters are fine.”
What follows is the architecture I’d defend in a design review. The Grafana-documented mechanics — microservices mode, the hash ring, shuffle-sharding, the store-gateway, retention via the compactor — are sourced inline and dated. The specifics of the real build — distributed mode across all three backends, replication factor 3, AWS S3 with a 90-day hot tier and Glacier Deep Archive beyond that, a memcached cache tier at ~80% hit rate, per-tenant limits in the distributor — are the ones I ran. Where I don’t have an exact internal figure in front of me (a per-ingester memory number, a store-gateway shard size), I’ve left it out rather than invent one.
Why monolithic mode stops working
Monolithic mode — Grafana also calls it single-binary — runs all of a backend’s internal components in one process and scales by running more identical copies behind a load balancer (Grafana Loki deployment modes, accessed June 2026). That works because at low volume the write path and the read path put roughly comparable load on the box, so scaling them together wastes nothing.
At 6 TB a day that assumption breaks in both directions. The write path — accepting, validating, replicating, and flushing telemetry — runs hot all day at a fairly steady rate set by how much your fleet emits. The read path — engineers running queries, dashboards refreshing, alerts evaluating — is spiky and bursty and concentrated in working hours and incidents. Scale them together and you either over-provision the read path to survive the write load or starve the write path to afford the read burst. Neither is acceptable when the data is 6 TB a day and the readers are 15 departments. So you split.
The intermediate option, simple scalable deployment (SSD), splits Loki into just two targets — a write path and a read path (plus a backend target in current versions) — and is Grafana’s recommended default up to a few terabytes a day (Grafana Loki deployment modes, accessed June 2026). SSD is a real and underrated middle ground; if the build were 2 TB a day I might stop there. But at 6 TB across 15 tenants, with different departments wanting different retention and different query loads, you want each component scaled and isolated on its own — and that is full microservices mode, where every component runs as its own process with its own -target flag (Grafana Loki components, accessed June 2026). The real build runs all three backends — Loki, Mimir, and Tempo — in distributed (microservices) mode, not SSD.
The write path: distributors and ingesters
Every one of the three backends has the same write-path shape, and once you see it in Loki you see it in all of them.
Distributors are the front door. In Loki, the distributor “is responsible for handling incoming push requests from clients” — it validates each stream for correctness and tenant limits, then sends it to n ingesters in parallel where n is the replication factor, choosing ingesters by consistent hashing (Grafana Loki components, accessed June 2026). Distributors are stateless. That is the most important property on the write path: because they hold no data, you scale them purely on incoming request volume and CPU, and you can add or remove them freely (Grafana Mimir scaling out, accessed June 2026). Mimir and Tempo distributors do the same job — validate, shard, replicate — with Tempo sharding traces by trace ID (Grafana Tempo architecture, accessed June 2026).
Ingesters are where it gets careful. The ingester holds recently received data in memory, builds it into chunks (Loki) or blocks (Mimir/Tempo), and periodically flushes those to object storage; until the flush happens, it also serves that recent data to queries (Grafana Loki components; Grafana Mimir architecture, both accessed June 2026). Ingesters are stateful, and that one word changes how you operate them. You cannot just kill an ingester — it holds in-memory data that hasn’t been flushed yet. Grafana’s classic scale-down procedure is to call the /ingester/shutdown API endpoint so the ingester flushes its in-memory series to long-term storage before it leaves (Grafana Mimir scaling out, accessed June 2026).
Ingesters sit in a consistent hash ring and are replicated. The replication factor — typically 3 — means each stream is written to three ingesters, so losing one ingester loses no data (Grafana Loki consistent hash rings, accessed June 2026). The ring is coordinated by a gossip protocol; Grafana’s guidance is to “use the memberlist key-value store type unless there is a compelling reason to use a different type,” because memberlist propagates ring state to all nodes for eventual consistency without an external dependency like Consul or etcd (Grafana Loki hash rings, accessed June 2026). In the real build the ingesters replicate 3x, so losing a single ingester loses no data. At 6 TB a day the ingester tier is the largest and most expensive part of the write path, because it has to hold the working set in memory across the replication factor.
The operational rule that falls out of this: distributors scale on traffic and are disposable; ingesters scale on retained-in-memory volume and must be drained on the way down. Two different components, two different scaling signals, which is exactly why you split them.
The read path: query-frontend, scheduler, querier, store-gateway
The read path is where the architecture earns its keep, because a query at 6 TB-a-day scale can touch a lot of data, and a naïve read path will either time out or fall over.
Query-frontend sits in front of the queriers. It is an optional service that “accelerates the read path” by queuing queries, splitting a large request into parallel sub-queries, and caching results (Grafana Loki components, accessed June 2026). The split is the magic: a query over 24 hours becomes 24 one-hour sub-queries that run in parallel across many queriers, so wall-clock time drops even though total work is the same.
Query-scheduler is the next layer of fairness. It provides queuing “beyond the query frontend,” maintaining separate per-tenant queues so one department’s heavy query can’t starve another’s, and lets queriers pull jobs as workers (Grafana Loki components, accessed June 2026). With 15 departments sharing the cluster, per-tenant queue fairness is not optional — it is the thing that stops one team’s runaway dashboard from taking down everyone’s alerts.
Queriers do the actual work. The Loki querier “is responsible for executing LogQL queries,” fetching data from both ingesters (for recent, not-yet-flushed data) and long-term storage, then deduplicating (Grafana Loki components, accessed June 2026). The Mimir querier is stateless and “evaluates PromQL expressions by fetching time series and labels,” using the store-gateway for long-term storage and the ingester for recent data (Grafana Mimir querier, accessed June 2026). Queriers are stateless, so you scale them on query load and concurrency — which, because read load is spiky, is the component you most want to autoscale.
Store-gateway is the Mimir component that makes querying object storage at scale survivable, and it’s worth its own paragraph because it’s the clearest example of sharding done right. The store-gateway “queries blocks from long-term storage,” loading the index-header of each block in its shard so queriers can hit it (Grafana Mimir store-gateway, accessed June 2026). It uses shuffle-sharding to divide each tenant’s blocks across a subset of store-gateway instances rather than all of them, set by -store-gateway.tenant-shard-size — which isolates blast radius, because only a subset of instances ever load a given tenant’s blocks (Grafana Mimir store-gateway, accessed June 2026). Blocks are also sharded across the store-gateway hash ring and replicated with -store-gateway.sharding-ring.replication-factor (default 3) so a single instance failure doesn’t make blocks unqueryable (Grafana Mimir store-gateway, accessed June 2026). The store-gateway is stateful in the careful sense: Grafana’s explicit guidance is to “scale down no more than two store-gateways at the same time” so you never drop below the replication factor for any block (Grafana Mimir scaling out, accessed June 2026). The real build runs a memcached caching tier — chunk, index, and results caches, with NVMe extstore for overflow — in front of object storage, at roughly an 80% hit rate, so most queries never touch S3.
Sharding, replication, and the hash ring
Step back and the same primitive runs through all of it: a consistent hash ring, used three different ways.
On the write path the ring shards incoming streams across ingesters and replicates each one to n of them. On the read path the ring shards stored blocks across store-gateways and replicates each block for availability. And shuffle-sharding restricts each tenant to a subset of the ring so 15 departments don’t all contend for the same instances (Grafana Mimir store-gateway, accessed June 2026). The ring is the load-bearing abstraction for the whole scale build — it’s how the cluster decides who owns what without a central coordinator, and memberlist gossip keeps every node’s view of it eventually consistent (Grafana Loki hash rings, accessed June 2026).
Two ring-related settings matter disproportionately at this scale. Zone-awareness (-store-gateway.sharding-ring.zone-awareness-enabled) spreads replicas across availability zones so an AZ failure doesn’t take out all copies of a block (Grafana Mimir store-gateway, accessed June 2026). And wait-stability on startup (-store-gateway.sharding-ring.wait-stability-min-duration) prevents a cold-starting cluster from churning block assignments while instances are still joining (Grafana Mimir store-gateway, accessed June 2026). Neither matters at small-team scale. Both are the difference between a clean failover and a thundering-herd outage at 6 TB a day.
S3 is the floor everything stands on
At every scale, object storage is the durable backend — but at small-team scale it’s a convenience, and at 6 TB a day it’s the load-bearing wall. All three backends store their long-term data as blocks in S3-compatible object storage: AWS S3, GCS, Azure Blob, or self-hosted MinIO (Grafana Mimir architecture, accessed June 2026). Mimir’s storage format stores each tenant’s series into its own TSDB, persisted as on-disk blocks with a two-hour range by default, and the store-gateway keeps a view of the bucket current by periodically downloading a bucket index (Grafana Mimir store-gateway, accessed June 2026).
The reason this matters more at scale is arithmetic. 6 TB a day, replicated and retained, is petabytes over a year. No disk holds that; no team should want to manage that on block storage. S3 turns “where do we keep a petabyte” into a line item that costs a couple of cents per GB-month and never fills up. The compute tier — distributors, queriers, even the stateful ingesters and store-gateways — stays comparatively small and elastic because the data doesn’t live on it. That separation of compute from storage is the single design decision that makes 6 TB a day affordable instead of ruinous. In the real build the object store is AWS S3, the hot tier holds roughly 540 TB at any time (6 TB a day across a 90-day window), and logs older than 90 days are tiered to S3 Glacier Deep Archive — which is where the storage bill drops by about 96%.
Retention is a component, not a setting
At small-team scale you set a retention period and forget it. At 6 TB a day, retention is an active, scheduled job that deletes data from object storage on a policy, and it is owned by a specific component: the compactor.
In Loki the compactor compacts the many per-ingester index files into a single file per day per tenant, and it owns retention. Retention only runs if you set retention_enabled: true; without it the compactor only compacts (Grafana Loki retention, accessed June 2026). It marks chunks for deletion asynchronously and waits retention_delete_delay before they actually vanish, so index gateways have time to refresh and queries don’t fail on already-deleted data (Grafana Loki retention, accessed June 2026). Two hard constraints worth knowing: retention “is only available if the index period is 24h,” and the compactor must run as a singleton — a single instance (Grafana Loki retention, accessed June 2026). The singleton constraint surprises people building for scale: every other component scales horizontally, the compactor does not.
The lever that makes retention a cost-control tool rather than just a cleanup job is per-stream and per-tenant retention. Loki supports a global retention_period in limits_config, overridden by retention_stream rules that target specific streams by label selector, and per-tenant overrides via runtime config (Grafana Loki retention, accessed June 2026). In the real build the retention policy is tiered by storage class rather than per-stream: logs sit hot in S3 for 90 days, then a lifecycle rule tiers them to Glacier Deep Archive, while metrics and traces stay hot in S3 for fast access. With 15 departments separated by X-Scope-OrgID tenant, retention is owned by the compactor and the lifecycle policy, not by hoping people clean up after themselves. In Tempo the compactor plays the same dual role, compressing and deduplicating stored data and expiring it after its retention period (Grafana Tempo architecture, accessed June 2026).
Cardinality is the cost lever nobody schedules
Here is the part that decides whether 6 TB a day costs what it should or three times that: cardinality. And it’s the part no Helm chart fixes for you.
In Loki, cardinality is the number of unique label/value combinations, which is the number of log streams. High cardinality is the silent killer — Grafana’s docs are blunt that “the fewer labels you use, the better,” because high cardinality makes Loki “build a huge index and flush thousands of tiny chunks to the object store,” which degrades query performance and can cause ingestion failures outright (Grafana Loki cardinality, accessed June 2026). The mechanism is worth internalising: a label with status codes (5 values) crossed with HTTP methods (5 values) is 25 streams; add an endpoint label with 100 values and it’s 2,500. One unbounded label — a user ID, a trace ID, an order ID — and you have effectively infinite streams (Grafana Loki cardinality, accessed June 2026).
Grafana puts an actual number on the ceiling: keep any single tenant under 100,000 active streams, and under a million streams in a 24-hour period — and those figures are “for HUGE tenants sending more than 10 TB a day” (Grafana Loki cardinality, accessed June 2026). The discipline is doctrine: use static labels that describe origin and context (application, namespace, environment); never put unbounded values (timestamp, trace ID, order ID) in a label; push the high-cardinality stuff into filter expressions at query time instead of into the index (Grafana Loki label best practices, accessed June 2026).
At 15 departments, the cardinality discipline cannot be advice — it has to be enforced limits. Per-tenant ingestion limits in the distributor, set deliberately per department, are what stop one team’s well-intentioned “let’s add a user_id label” from quietly tripling the index size and the bill for everyone. The real build enforces these limits per tenant in the Loki distributor rather than leaving them to good intentions. This is the one part of the scale build that is mostly cultural rather than architectural, and it’s the part that, left unmanaged, eats every cost saving the rest of the architecture won.
What actually changes from the small-team build
If you’ve read the small-team post, here’s the whole delta in one place. The small build is three single binaries; this build is roughly thirty distinct component types across three backends. The small build scales by running more identical copies; this one scales each component on its own signal — distributors on traffic, ingesters on retained memory, queriers on query burst, store-gateways on stored-block volume. The small build keeps S3 as a cheap convenience; this build leans on S3 as the only thing that makes a petabyte-a-year affordable. The small build sets retention as a line in a config; this build runs a compactor with per-department retention tiers as an active deletion job. And the small build can be lax about labels; this build enforces cardinality as a hard per-tenant limit because 15 teams will otherwise find every way to break it.
None of that means the scale build is “better.” It’s a different problem. If your volume is 20 GB a day, building this is the most expensive mistake you can make — you’ll run forty pods to do the work of three and call it engineering. The whole point of having both posts is that the right architecture is the one that matches your actual volume, and the gap between 20 GB and 6 TB is where most teams over-build or under-build. Match the architecture to the number. That’s the entire skill.
The one-paragraph version
At ~6 TB a day across 15 departments, you run Loki, Mimir, and Tempo in full microservices mode. Stateless distributors take the writes and shard them across a replicated ring of stateful ingesters. Query-frontend and query-scheduler split and fairly-queue reads across stateless queriers, with Mimir’s store-gateways shuffle-sharding blocks out of S3. One object store holds petabytes for cents per GB-month. A singleton compactor enforces per-department retention as scheduled deletion. And hard per-tenant stream limits keep cardinality — the real cost lever — from eating the savings. That’s the architecture. The small-team version is three binaries and a bucket; this is what those three binaries become when the number gets three hundred times bigger.