Steps to set up centralized logging with Grafana Loki and Grafana Agent for Kubernetes, VM Applications, and Azure services - Part 3

In Part 1, We have covered how to setup Grafana Loki and Grafana Agent to view Kubernetes pod logs

In Part 2, We have covered how to configure Grafana Agent on Windows VM and export application logs to Grafana Loki.

In this Part 3, We will see how to export Azure PAAS service logs to Grafana loki and view it from Grafana Dashboard.

Requirement: 

• Grafana loki
• Azure eventhub
• Azure AKS or any PAAS which is having option with "Diagnostics Settings"

Step 8: Create Azure Eventhub namespace,

Go to the Azure Portal and create the Event Hub namespace with one Event Hub. (Currently, we are going to use Azure AKS, so we will create one Event Hub named "aks" under the Event Hub namespace)

Step 9: Configure Azure AKS to send logs to Azure Eventhub,

Go to Azure AKS, in the side blade select "Diagnostic Settings", and choose "Add Diagnostic Setting".

Then, in the new page, select which logs need to be sent to the Event Hub and choose "Stream to an Event Hub". Here, provide the newly created Event Hub namespace and Event Hub.

Step 10: Configure Grafana Agent to scrap the messages from Azure eventhub,

Next, We need to pull the data from Azure eventhub and push it to Grafana loki,

In our existing grafana-agent-values.yaml add below lines to pull the messages from Azure eventhub and redeploy grafana agent in AKS.

Here is the reference github url and below is the yaml.

https://github.com/DevOpsArts/grafana_loki_agent/blob/main/grafana-agent-values-azure-aks.yaml

loki.source.azure_event_hubs "azure_aks" {

      fully_qualified_namespace = "==XXX Eventhub namespace hostname XX===:9093"

      event_hubs = ["aks"]

      forward_to = [loki.write.local.receiver]

      labels = {

        "job" = "azure_aks",

        }

      authentication {

        mechanism = "connection_string"

        connection_string = " ===XXX Eventhub connection String XX==="

      }

      }

Replace the correct value for the above RED color. We can add multiple Event hubs in the Grafana agent by providing different Job names for each Azure PAAS. 

Note : Make sure the communication is established between Azure AKS and Azure Eventhub to send the messages on port 9093.

Redeploy grafana agent in AKS using below command,

helm install --values grafana-agent-values-azure-aks.yaml grafana-agent grafana/grafana-agent -n observability

Check all the Grafana agent pods are up and running using below command,

kubectl get all -n observability

Now, the Grafana agent will pull the messages from Azure Event Hub and push them to Grafana Loki for Azure AKS, which is configured to send the logs in Diagnostic Settings.

We can verify the status of message processing from Azure Event Hub, including the status of incoming and outgoing messages.

Step 11: Access Azure AKS logs in Grafana dashboard,

Go to Grafan Dashboard, Home > Explore > Select Loki Datasource

In the filter section, select "Job" and value as the job name which is given in the grafana-agent-values-azure-aks.yaml. In our case the job name is "azure_aks"

Thats all, We have successfully deployed centralized logging with Grafana Loki, Grafana Agent for Kubernetes, VM application and Azure PAAS.