Setting Up Loki on K3s for Centralised Log Aggregation


estimated read time: 5 minutes

Setting Up Loki on K3s for Centralised Log Aggregation

Completing the Observability Stack

My homelab runs a full observability stack:

Loki fills the final gap. Without it, investigating an incident means SSH-ing into individual nodes and grepping logs in real time. With it, every log line from every pod in the cluster flows into a central store and is queryable in Grafana using LogQL — the same interface you use for PromQL dashboards.

Install Loki via Helm

Add the Grafana Helm repository if you have not already:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install Loki in microservices mode (separate components for ingester, distributor, querier, etc.) rather than the single-binary mode. This is the production-style deployment — it scales better and separates concerns cleanly:

helm install loki grafana/loki \
  -f microservices-values.yaml \
  -n monitoring

The microservices-values.yaml file is where you configure storage backends, retention, and resource limits to match your cluster. At minimum you will want to configure an object storage backend (S3, MinIO, or similar) rather than the default filesystem store, which does not work well in a multi-replica deployment. A minimal starting point:

loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: s3
    s3:
      endpoint: <your-minio-or-s3-endpoint>
      bucketnames: loki-chunks
      access_key_id: <access-key>
      secret_access_key: <secret-key>
      insecure: true # set false for real TLS endpoints
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

Verify all pods come up:

kubectl get pods -n monitoring

You should see pods for the distributor, ingester, querier, query-frontend, compactor, and gateway components. The gateway is the key endpoint — it is what Promtail (the log shipper) and Grafana both connect to.

The Gateway URL

Once deployed, the Loki gateway is available at:

http://loki-gateway.monitoring.svc.cluster.local

Note this down — you will need it in two places: the Grafana datasource configuration and your Promtail config.

Connect Loki to Grafana

Navigate to Connections → Data Sources → Add data source in Grafana and select Loki.

Set the URL:

http://loki-gateway.monitoring.svc.cluster.local

The X-Scope-OrgID Header

Here is the gotcha that catches almost everyone. Loki requires a tenant identifier header on every request. Without it, queries return no data and you get no useful error — just an empty result.

Under HTTP Headers, add:

HeaderValue
X-Scope-OrgIDdefault

This header tells Loki which tenant’s data you are querying. In a single-tenant homelab setup the value default is the right choice. If you were running a multi-tenant Loki for multiple teams, each team would have their own org ID.

Save and test the datasource. You should see a green “Data source connected” confirmation.

Note: The gateway URL in the Helm output may differ from the default shown above. Always check the output of helm install or run kubectl get svc -n monitoring to confirm the actual service name.

Ship Logs with Promtail

Loki itself does not scrape logs — it receives them from a shipper. Promtail is the standard choice for Kubernetes: it runs as a DaemonSet, tails container logs from every node, adds Kubernetes metadata labels (pod name, namespace, container), and forwards to the Loki gateway.

helm install promtail grafana/promtail \
  --set "config.lokiAddress=http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push" \
  -n monitoring

Once Promtail is running, logs will start appearing in Grafana within a minute or two.

Querying Logs in Grafana

Open Explore in Grafana, select the Loki datasource, and try a basic query:

{namespace="monitoring"}

This returns all logs from the monitoring namespace. You can filter further:

{namespace="monitoring", pod=~"loki.*"} |= "error"

Or parse structured JSON logs (useful if your applications log in JSON):

{namespace="default"} | json | level="error"

How This Connects to the Rest of the Stack

With Loki, Prometheus, and Jaeger all connected as datasources in Grafana, you can correlate across all three signals. A spike in error rate on a Prometheus dashboard links to log lines in Loki from that same time window, which links to traces in Jaeger for the specific requests that failed. This is the full observability loop.

The blog itself already emits structured JSON logs to stdout and records metrics to Prometheus — once Promtail is running in the cluster, those logs flow into Loki automatically without any application changes.