This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.

Documentation Grafana Loki Send data Kubernetes Monitoring Helm

Kubernetes Monitoring Helm tutorial

One of the primary use cases for Loki is to collect and store logs from your Kubernetes cluster. These logs fall into three categories:

1. Pod logs: Logs generated by pods running in your cluster.
2. Kubernetes Events: Logs generated by the Kubernetes API server.
3. Node logs: Logs generated by the nodes in your cluster.

In this tutorial, we will deploy Loki and the Kubernetes Monitoring Helm chart to collect two of these log types: Pod logs and Kubernetes events. We will also deploy Grafana to visualize these logs.

Things to know

Before you begin, here are some things you should know:

• Loki: Loki can run in a single binary mode or as a distributed system. In this tutorial, we will deploy Loki as a single binary, otherwise known as monolithic mode. Loki can be vertically scaled in this mode depending on the number of logs you are collecting. Grafana Labs recommends running Loki in a distributed/microservice mode for production use cases to monitor high volumes of logs.
• Deployment: You will deploy Loki, Grafana, and Alloy (as part of the Kubernetes Monitoring Helm chart) in the meta namespace of your Kubernetes cluster. Make sure you have the necessary permissions to create resources in this namespace. These pods will also require resources to run, so consider the amount of capacity your nodes have available. It is also possible to just deploy the Kubernetes monitoring Helm chart (since it has a minimal resource footprint) within your cluster and write logs to an external Loki instance or Grafana Cloud.
• Storage: In this tutorial, Loki will use the default object storage backend provided in the Loki Helm chart; MinIO. You should migrate to a more production-ready storage backend like S3, GCS, Azure Blob Storage or a MinIO Cluster for production use cases.

Prerequisites

Before you begin, you will need the following:

• A Kubernetes cluster running a currently supported version of Kubernetes. The Kubernetes Monitoring Helm chart aims to be compatible with all supported Kubernetes versions.
• kubectl installed on your local machine.
• Helm installed on your local machine.

Create the meta and prod namespaces

The K8s Monitoring Helm chart will monitor two namespaces: meta and prod:

• meta namespace: This namespace will be used to deploy Loki, Grafana, and Alloy.
• prod namespace: This namespace will be used to deploy the sample application that will generate logs.

Create the meta and prod namespaces by running the following command:

Bash Copy

kubectl create namespace meta && kubectl create namespace prod

Add the Grafana Helm repository

The Grafana and Kubernetes Monitoring Helm charts are available in the Grafana Helm repository. The Loki Helm chart is maintained separately in the Grafana Community Helm Charts repository. Add both Helm repositories by running the following commands:

Bash Copy

helm repo add grafana https://grafana.github.io/helm-charts && helm repo update
helm repo add grafana-community https://grafana-community.github.io/helm-charts && helm repo update

As well as adding the repos to your local helm list, you should also run helm repo update to ensure you have the latest version of the charts.

Clone the tutorial repository

Clone the tutorial repository by running the following command:

Bash Copy

git clone https://github.com/grafana/alloy-scenarios.git

Then change directories to the alloy-scenarios/k8s/logs directory:

Bash Copy

cd alloy-scenarios/k8s/logs

The rest of this tutorial assumes you are in the alloy-scenarios/k8s/logs directory.

Deploy Loki

Grafana Loki will be used to store our collected logs. In this tutorial we will deploy Loki with a minimal footprint and use the default storage backend provided by the Loki Helm chart, MinIO.

To deploy Loki run the following command:

Bash Copy

helm install --values loki-values.yml loki grafana-community/loki -n meta

This command will deploy Loki in the meta namespace. The command also includes a values file that specifies the configuration for Loki. For more details on how to configure the Loki Helm chart refer to the Loki Helm documentation.

Deploy Grafana

Next we will deploy Grafana to the meta namespace. You will use Grafana to visualize the logs stored in Loki. To deploy Grafana run the following command:

Bash Copy

helm install --values grafana-values.yml grafana grafana/grafana --namespace meta

As before, the command also includes a values file that specifies the configuration for Grafana. There are two important configuration attributes to take note of:

1. adminUser and adminPassword: These are the credentials you will use to log in to Grafana. The values are admin and adminadminadmin respectively. The recommended practice is to either use a Kubernetes secret or allow Grafana to generate a password for you. For more details on how to configure the Grafana Helm chart, refer to the Grafana Helm documentation.

2. datasources: This section of the configuration lets you define the data sources that Grafana should use. In this tutorial, you will define a Loki data source. The data source is defined as follows:

   YAML Copy

    datasources:
      datasources.yaml:
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki-gateway.meta.svc.cluster.local:80
          basicAuth: false
          isDefault: false
          version: 1
          editable: false

This configuration defines a data source named Loki that Grafana will use to query logs stored in Loki. The url attribute specifies the URL of the Loki gateway. The Loki gateway is a service that sits in front of the Loki API and provides a single endpoint for ingesting and querying logs. The URL is in the format http://loki-gateway.<NAMESPACE>.svc.cluster.local:80. The loki-gateway service is created by the Loki Helm chart and is used to query logs stored in Loki. If you choose to deploy Loki in a different namespace or with a different name, you will need to update the url attribute accordingly.

Deploy the Kubernetes Monitoring Helm chart

The Kubernetes Monitoring Helm chart is used for gathering, scraping, and forwarding Kubernetes telemetry data to a Grafana stack. This includes the ability to collect metrics, logs, traces, and continuous profiling data. The scope of this tutorial is to deploy the Kubernetes Monitoring Helm chart to collect Pod logs and Kubernetes events.

To deploy the Kubernetes Monitoring Helm chart run the following command:

Bash Copy

helm install --values ./k8s-monitoring-values.yml k8s grafana/k8s-monitoring -n meta --version "^4"

Pinning the chart to the 4.x line (--version "^4") keeps this tutorial working even after a future major release changes the configuration format again.

Within the configuration file k8s-monitoring-values.yml we have defined the following:

YAML Copy

---
cluster:
  name: meta-monitoring-tutorial

destinations:
  loki:
    type: loki
    url: http://loki-gateway.meta.svc.cluster.local/loki/api/v1/push

clusterEvents:
  enabled: true
  collector: alloy-singleton
  namespaces:
    - meta
    - prod

podLogsViaKubernetesApi:
  enabled: true
  collector: alloy-logs
  namespaces:
    - meta
    - prod
  structuredMetadata:
    pod: pod  # Set structured metadata "pod" from label "pod"

# Collectors
collectors:
  alloy-singleton:
    presets: [singleton]
  alloy-logs:
    presets: [clustered]

To break down the configuration file:

• Define the cluster name as meta-monitoring-tutorial. This is a static label that will be attached to all logs collected by the Kubernetes Monitoring Helm chart.

• Define a destination named loki that will be used to forward logs to Loki. The url attribute specifies the URL of the Loki gateway. If you choose to deploy Loki in a different namespace or in a different location entirely, you will need to update the url attribute accordingly.

• Enable the collection of cluster events and Pod logs. Pod logs are gathered via the Kubernetes API, using the podLogsViaKubernetesApi feature:

  • collector: specifies which collector to use to collect logs. The value must match a name defined under collectors: below. In this case, clusterEvents uses the alloy-singleton collector, and podLogsViaKubernetesApi uses the alloy-logs collector.
  • structuredMetadata: specifies the structured metadata to collect. In this case, we are setting the structured metadata pod so we can retain the pod name for querying, without promoting it to an indexed label (which would add high cardinality).
  • namespaces: specifies the namespaces to collect logs from. In this case, we are collecting logs from the meta and prod namespaces.

• Node log collection is not enabled in this tutorial. The nodeLogs feature defaults to disabled, and requires mounting /var/log/journal, which is out of scope here.

• Define the collectors: map. Each entry is a name you choose for an Alloy instance, configured with one or more presets: that determine how that instance is deployed:

  • alloy-singleton uses the singleton preset, which deploys a single Alloy replica. This is sufficient for gathering cluster-wide data such as Kubernetes events.
  • alloy-logs uses the clustered preset, which deploys Alloy as a coordinated group of replicas. This is required for collecting Pod logs through the Kubernetes API, since the replicas need to coordinate which Pods each one collects logs from.

  At least one collector must be defined under collectors:, and every collector: reference elsewhere in the file (such as clusterEvents.collector and podLogsViaKubernetesApi.collector) must match a key in this map, or the install fails with an error naming the missing collector.

Accessing Grafana

To access Grafana, you will need to port-forward the Grafana service to your local machine. To do this, run the following command:

Bash Copy

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}") && \
kubectl --namespace meta port-forward $POD_NAME 3000 --address 0.0.0.0

This command will port-forward the Grafana service to your local machine on port 3000.

You can now access Grafana by navigating to http://localhost:3000 in your browser. The default credentials are admin and adminadminadmin.

One of the first places you should visit is Logs Drilldown which lets you automatically visualize and explore your logs without having to write queries: http://localhost:3000/a/grafana-lokiexplore-app

(Optional) View the Alloy UI

The Kubernetes Monitoring Helm chart deploys Grafana Alloy to collect and forward telemetry data from the Kubernetes cluster. The Helm chart is designed to abstract you away from creating an Alloy configuration file. However if you would like to understand the pipeline you can view the Alloy UI. To access the Alloy UI, you will need to port-forward the Alloy service to your local machine. To do this, run the following command:

Bash Copy

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=alloy-logs,app.kubernetes.io/instance=k8s-alloy-logs" -o jsonpath="{.items[0].metadata.name}") && \
kubectl --namespace meta port-forward $POD_NAME 12345 --address 0.0.0.0

This command will port-forward the Alloy service to your local machine on port 12345. You can access the Alloy UI by navigating to http://localhost:12345 in your browser.

Adding a sample application to prod

Finally, lets deploy a sample application to the prod namespace that will generate some logs. To deploy the sample application run the following command:

Bash Copy

helm install tempo grafana/tempo-distributed -n prod

This will deploy a default version of Grafana Tempo to the prod namespace. Tempo is a distributed tracing backend that is used to store and query traces. Normally Tempo would sit alongside Loki and Grafana in the meta namespace, but for the purpose of this tutorial, we will pretend this is the primary application generating logs.

Once deployed lets expose Grafana once more:

Bash Copy

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}") && \
kubectl --namespace meta port-forward $POD_NAME 3000 --address 0.0.0.0

and navigate to http://localhost:3000/a/grafana-lokiexplore-app to view Grafana Tempo logs.

Conclusion

In this tutorial, you learned how to deploy Loki, Grafana, and the Kubernetes Monitoring Helm chart to collect and store logs from a Kubernetes cluster. We have deployed a minimal test version of each of these Helm charts to demonstrate how quickly you can get started with Loki. It is now worth exploring each of these Helm charts in more detail to understand how to scale them to meet your production needs:

• Loki Helm chart
• Grafana Helm chart
• Kubernetes Monitoring Helm chart