Ingest logs into the Observability Platform

How to make new logs available in the Observability Platform in self-service.

By default, Giant Swarm clusters starting from CAPA v29.2.0, and CAPZ v29.1.0 are equipped with Alloy, an Observability data collector. It’s configured to collect system logs from the cluster and forward them to a central Loki instance running on the management cluster. See Logging for more details.

The observability platform allows to ingest logs from your workloads in a self-service way using PodLogs to select which pods to ingest logs.

From release v29.5.0 in CAPA and CAPZ, and v29.2.0 in CAPV and CAPVCD, additional targets to collect logs from can be configured either with a pod label or a PodLogs resource.

Using pod labels

The Observability Platform automatically ingest all logs for pods that are labelled with the observability.giantswarm.io/tenant label via PodLogs resources.

Example:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        # This is where the tenant label must be configured
        observability.giantswarm.io/tenant: my-team
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
EOF

With this deployed manifest, Grafana Alloy will collect all logs for the nginx ingress pods and send the logs to the my-team tenant>.

Using PodLogs

PodLogs resources are more flexible than pod labels as they can be used to:

  • Filter pods targets using label selector
  • Filter pods’ namespaces using label selector
  • Modify labels using relabeling

PodLogs can be created via a Helm Chart or GitOps. Alternatively it can also be directly created in the cluster.

Example:

cat <<EOF | kubectl apply -f -
apiVersion: monitoring.grafana.com/v1alpha2
kind: PodLogs
metadata:
  name: example-podlogs
spec:
  selector:
    matchLabels:
      foo: bar
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: charlie
  relabelings:
  # This configures the tenant name under which the data will be stored in the observability platform
  - action: replace
    targetLabel: giantswarm_observability_tenant
    replacement: my-team
  - action: replace
    sourceLabels: [__meta_kubernetes_pod_node_name]
    targetLabel: node_name
EOF

This will select all pods with the foo: bar label in the namespace charlie and add the node_name label to the logs metadata. It will send all the logs extracted by this PodLog under the my-team tenant. More examples can be found here.

Logs lines can then be viewed in GrafanaUI on the Explore page; learn more about this in Exploring logs with LogQL.

Technical considerations

By default, log ingestion is enabled for the following namespaces:

  • kube-system
  • giantswarm

This is done through a set of predefined PodLogs resources created in the kube-system namespace and labelled with giantswarm.io/managed-by: alloy-logs. These PodLogs shouldn’t be overwritten or we can’t guarantee the operational safety of the clusters.

Performances considerations

Keep in mind that ingesting new logs into the observability platform comes with a cost. The resource consumption of the central Loki is related to the amount of logs it has to handle. This means ingesting more logs also leads to higher resource consumption of the observability platform overall. Also note that logs are tailed through Kubernetes API on the cluster, impacting network traffic and CPU consumption of the API server pods.

You can check the resources usage of Kubernetes API server pods in Kubernetes / Compute Resources / Pod dashboard in your installation’s Grafana.


This part of our documentation refers to our vintage product. The content may be not valid anymore for our current product. Please check our new documentation hub for the latest state of our docs.