Ingest logs into the Observability Platform

How to make new logs available in the Observability Platform in self-service.

By default, Giant Swarm clusters starting from CAPA v29.2.0, and CAPZ v29.1.0 are equipped with Alloy, an Observability data collector. It’s configured to collect system logs from the cluster and forward them to a central Loki instance running on the management cluster. See Logging for more details.

The observability platform allows to ingest logs from your workloads in a self-service way using PodLogs to select which pods to ingest logs.

Using PodLogs

From release v30.0.0 additional targets to collect logs from can be configured using PodLogs resource. Those resources can be created in any namespace. PodLogs resources can be used to:

Filter pods targets using label selector
Filter pods’ namespaces using label selector
Modify labels using relabeling

PodLogs can be created via a Helm Chart or GitOps. Alternatively it can also be directly created in the cluster.

Example:

cat <<EOF | kubectl apply -f -
apiVersion: monitoring.grafana.com/v1alpha2
kind: PodLogs
metadata:
  name: example-podlogs
spec:
  selector:
    matchLabels:
      foo: bar
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: charlie
  relabelings:
  - action: replace
    sourceLabels: [__meta_kubernetes_pod_node_name]
    targetLabel: node_name
EOF

This will select all pods with the foo: bar label in the namespace charlie and add the node_name label to the logs metadata. More examples can be found here.

Logs lines can then be viewed in GrafanaUI on the Explore page; learn more about this in Exploring logs with LogQL.

Technical considerations

Per default log ingestion is enabled for the following namespaces:

kube-system
giantswarm

This is done through a set of predefined PodLogs resources created in the kube-system namespace and labelled with giantswarm.io/managed-by: alloy-logs. These PodLogs shouldn’t be overwritten or we can’t guarantee the operational safety of the clusters.

Performances considerations

Keep in mind that ingesting new logs into the observability platform comes with a cost. The resource consumption of the central Loki is related to the amount of logs it has to handle. This means ingesting more logs also leads to higher resource consumption of the observability platform overall. Also note that logs are tailed through Kubernetes API on the cluster, impacting network traffic and CPU consumption of the API server pods.

You can check the resources usage of Kubernetes API server pods in Kubernetes / Compute Resources / Pod dashboard in your installation’s Grafana.