Thanos: Long-Term Prometheus Storage on Kubernetes (k3s Homelab Guide)


estimated read time: 5 minutes

The Problem with Prometheus Alone

Prometheus is excellent at what it does: scraping metrics, storing them locally, and making them queryable. But it has a practical limit — local disk. In my homelab cluster, keeping metrics for more than 30 days would exhaust the storage available on the Raspberry Pis. Older data would roll off the TSDB and be gone.

For most day-to-day monitoring this is fine. But there are questions you can only answer with long-term data: how has memory usage trended over the past year? When did that service first start degrading? What was the 99th percentile latency this time last quarter?

Thanos solves this without replacing Prometheus. It sits alongside it: Prometheus handles the first 30 days as normal, and Thanos takes care of everything beyond that, shipping blocks to object storage and making them queryable through a unified interface.

How the Architecture Works

There are three Thanos components running in my cluster:

Thanos Query is the unified query endpoint. It federates PromQL queries across both live Prometheus data and the long-term store, deduplicating replicas automatically. Grafana queries Thanos Query rather than Prometheus directly — the switch is transparent.

Thanos Store Gateway is the bridge to object storage. It reads metric blocks from a MinIO bucket on my TrueNAS NAS, serves them over gRPC to Thanos Query, and caches index lookups in memory to keep queries fast.

Thanos Ruler evaluates alerting and recording rules against the full dataset — both recent Prometheus data and historical store data — so your alerts can reference older windows.

Grafana
  └─▶ Thanos Query (port 9090)
        ├─▶ Prometheus (live, last 30 days)
        ├─▶ Thanos Store Gateway (historical, up to 2 years)
        └─▶ Thanos Ruler
                Store Gateway
                  └─▶ MinIO (TrueNAS NAS)

Thanos Query

Three replicas of Thanos Query run in the monitoring namespace. The key arguments that define how it works:

args:
  - query
  - --log.level=info
  - --grpc-address=0.0.0.0:10901
  - --http-address=0.0.0.0:9090
  - --query.replica-label=prometheus_replica
  - --query.replica-label=ruler_replica
  - --store=prometheus-operated.monitoring.svc.cluster.local:10901
  - --store=thanos-store-gateway.monitoring.svc.cluster.local:10901
  - --store=thanos-ruler-operated.monitoring.svc.cluster.local:10901
  - --query.auto-downsampling
  - --query.partial-response
  - --query.default-evaluation-interval=1m
  - --store.response-timeout=30s
  - --query.max-concurrent-select=4

A few things worth noting here:

Query cache config (ConfigMap mounted into the pod):

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-query-cache-config
  namespace: monitoring
data:
  query-cache.yml: |
    type: IN-MEMORY
    config:
      max_size: 64MB
      validity: 24h

Service definition — this is what Grafana points at:

apiVersion: v1
kind: Service
metadata:
  name: thanos-query
  namespace: monitoring
spec:
  selector:
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/component: query
  ports:
    - name: grpc
      port: 10901
      targetPort: grpc
    - name: http
      port: 9090
      targetPort: http
  type: ClusterIP

Thanos Store Gateway

The Store Gateway reads metric blocks from object storage and serves them over gRPC. My setup uses a MinIO instance running on TrueNAS as the S3-compatible backend.

Store container arguments:

args:
  - store
  - --data-dir=/var/thanos/store
  - --objstore.config-file=/etc/thanos/objstore.yml
  - --index-cache.config-file=/etc/cache/cache.yml
  - --log.level=info
  - --grpc-address=0.0.0.0:10901
  - --http-address=0.0.0.0:10902
  - --sync-block-duration=3m
  - --block-sync-concurrency=20
  - --store.grpc.series-max-concurrency=20
  - --store.grpc.series-sample-limit=0

Volumes:

volumes:
  - name: objstore-config
    secret:
      secretName: thanos-objstore-config # S3 credentials for MinIO
  - name: cache-config
    configMap:
      name: thanos-cache-config # 128MB in-memory index cache
  - name: data
    emptyDir: {} # Temporary working directory

The objstore-config secret contains the S3 connection details for MinIO — bucket name, endpoint, access key, and secret key. Thanos uses the same S3 API that AWS exposes, so any S3-compatible backend works here.

Index cache config:

type: IN-MEMORY
config:
  max_size: 128MB

The index cache holds recently accessed TSDB index data in memory. On a homelab NAS over the local network, this makes a significant difference to query latency on historical data.

Thanos Ruler

The Ruler evaluates alerting and recording rules against the full metric dataset. Without it, rules that reference historical data or long windows (e.g. rate(...[1h]) over a 24-hour period) could return incorrect results if run only against Prometheus’s local retention window.

The Ruler connects to both Prometheus and the Store Gateway, so it has the full two-year view when evaluating rules.

Pointing Grafana at Thanos

The only change needed in Grafana is to point your Prometheus data source at Thanos Query instead of Prometheus directly:

URL: http://thanos-query.monitoring.svc.cluster.local:9090

From a user perspective, nothing changes — it’s still PromQL, still the same dashboards. You just now have access to two years of data instead of 30 days.

The Result

ComponentRetention
Prometheus (local TSDB)30 days
Thanos Store Gateway (MinIO/TrueNAS)2 years
Combined query window via Thanos Query2 years

The full configuration repository is at dfoulkes/prometheus-setup.

If you are running Loki for logs alongside this, the Thanos Query endpoint in Grafana sits alongside your Loki datasource — together they give you correlated metrics and logs across the full retention window.