𝔻𝕖𝕧𝕆𝕡𝕤𝔸𝕣𝕥

The only Kubernetes log agent with intelligent error context capture, rule-based alerting, and 9 pluggable storage backends.

The Problem: Finding the Needle in the Log Haystack

Every SRE knows the pain: an alert fires at 3 AM, and you're digging through gigabytes of logs trying to understand what happened before the error. Traditional log solutions either capture everything (expensive) or miss crucial context (frustrating).

What if your log agent was smart enough to capture only what matters—the error AND the context around it—and alert you instantly?

Introducing Logsenta

Logsenta is an open-source Kubernetes log monitoring agent that solves this problem with intelligent error-aware context capture and rule-based alerting. Instead of blindly forwarding all logs, Logsenta:

• 🔍 Detects errors using configurable regex patterns
• ⏪ Captures context – logs BEFORE and AFTER the error
• 🚨 Alerts intelligently – route different errors to different teams
• 💾 Stores smartly – choose from 9 storage backends
• ⚡ Scales efficiently – handles 500+ pods with minimal resources

Key Features

🎯 Smart Error Detection

Logsenta uses regex and string-based pattern matching to detect errors across multiple languages and frameworks:

errorPatterns:
  - "ERROR"
  - "Exception"
  - "FATAL"
  - "panic:"
  - "Traceback"
  - "OOMKilled"
  - "CrashLoopBackOff"
  
  These patterns are "fully customizable"

🚨 Rule-Based Alerting NEW

Route different error patterns to different teams with customizable thresholds:

alerting:
  enabled: true
  rules:
    # Critical errors → On-call team immediately
    - name: "critical-errors"
      patterns: ["CRITICAL", "FATAL", "OOMKilled", "panic:"]
      threshold:
        count: 1          # Alert on FIRST occurrence
        windowSeconds: 60
      email:
        enabled: true
        toAddresses: ["oncall@company.com"]

    # Java exceptions → Backend team (after 2 occurrences)
    - name: "java-exceptions"  
      patterns: ["NullPointerException", "OutOfMemoryError"]
      threshold:
        count: 2          # Alert after 2 occurrences
        windowSeconds: 300
      email:
        enabled: true
        toAddresses: ["backend-team@company.com"]

    # Python errors → Data team
    - name: "python-errors"
      patterns: ["Traceback", "TypeError", "ValueError"]
      threshold:
        count: 1
      email:
        enabled: true
        toAddresses: ["data-team@company.com"]

Why rule-based alerting matters:

• 🎯 Reduce alert fatigue – only alert relevant teams
• ⏱️ Smart thresholds – distinguish between flaky tests and real outages
• 📧 Multiple channels – email and webhooks (Slack, PagerDuty, etc.)
• 🔄 Context included – alerts contain the actual error with surrounding logs

📦 9 Storage Backends

One agent, any storage destination:

🔄 Context Capture Window

Configure how much context to capture around errors:

captureWindow:
  bufferDurationMinutes: 2  # Lines BEFORE error
  captureAfterMinutes: 2    # Lines AFTER error

This means when an error occurs, you get the full story—not just the error line.

Quick Start

Deploy Logsenta in under 2 minutes:

# Clone the repository
git clone https://github.com/DevOpsArts/logsenta.git
cd logsenta

# Install with Helm (PostgreSQL backend + alerting)
helm install logsenta-engine ./charts/logsenta-engine \
  --namespace logsenta \
  --create-namespace \
  --set storage.type=postgresql \
  --set connections.postgresql.host=your-db-host \
  --set connections.postgresql.username=logsenta \
  --set connections.postgresql.password=YOUR_PASSWORD \
  --set alerting.enabled=true \
  --set alerting.email.smtpHost=smtp.company.com

Architecture

┌─────────────────────────────────────────────┐
│           Kubernetes Cluster                │
│                                             │
│  ┌─────┐  ┌─────┐  ┌─────┐                 │
│  │Pod A│  │Pod B│  │Pod C│  ← Monitored    │
│  └──┬──┘  └──┬──┘  └──┬──┘                 │
│     └────────┼────────┘                     │
│              ▼                              │
│     ┌────────────────────┐                  │
│     │   Logsenta-Engine  │                  │
│     │  • Error Detection │                  │
│     │  • Context Capture │                  │
│     │  • Rule-Based Alert│ ──► 📧 Email     │
│     │  • Rolling Buffer  │ ──► 🔗 Webhook   │
│     └─────────┬──────────┘                  │
└───────────────┼─────────────────────────────┘
                ▼
       ┌────────────────┐
       │ Storage Backend│
       │ (Your Choice)  │
       └────────────────┘
  

Production-Ready Features

• 🔒 Security: Non-root container, read-only filesystem, seccomp profiles
• 🔄 High Availability: Leader election for multi-replica deployments
• 📊 Scalability: ThreadPoolExecutor handles 500+ pods efficiently
• 🔐 SSL/TLS: Secure connections to all database backends
• 🛡️ Network Policies: Built-in network isolation templates
• 🚨 Smart Alerting: Rule-based routing with thresholds and cooldowns

Get Started Today

Logsenta is open-source and free to use. Check out the resources below:

• 📦 GitHub: github.com/DevOpsArts/logsenta
• 📖 Documentation: Wiki Documentation
• 🚨 Alerting Guide: Alerting Configuration
• 🐳 Docker Image: devopsart1/logsenta-engine

Have questions or feedback? Drop a comment below or open an issue on GitHub!

Tags: Kubernetes, DevOps, SRE, Logging, Monitoring, Alerting, Azure, AWS, GCP, Helm, Open Source

In Part 1, We covered how to setup Grafana Loki and Grafana Agent to view Kubernetes pod logs

Requirement:

• Grafana Loki
• Grafana agent
• Windows VM with one application
• Grafana Dashboard

Next double click the downloaded exe and install it, by default in windows the installer path is,

C:\Program Files\Grafana Agent

Once installation is completed, We need to update the configuration based on our needs like which application logs we need to send to Grafana loki.

In our case, we installed Grafana Dashboard in the windows VM and configured the Grafana dashboard logs in Grafana agent.

Similarly, we can add multiple application with different Job names.

Copy the grafana agent config file from below repo and update the required changes according on your needs.

We can start manually by below command in command prompt as well.

In command Prompt go to, C:\Program Files\Grafana Agent

Execute below command,

grafana-agent-windows-amd64.exe --config.file=agent-config.yaml

This will help to find any issue with the configuration.

Note : Here the Grafana loki distributed service endpoint(which is configured in the agent-config.yaml) should be accessible from the windows VM

Step 7 : Access VM application logs in Grafana Loki,

Go to Grafana Dashboard > Home > Explore > Select Loki Datasource

In the filter section, select "Job" and value as the job name which is given in the agent-config.yaml. In our case the job name is "devopsart-vm"

Now We are able to view the Grafana Dashboard logs in Grafana Loki. You can create the Dashboard from here based on your preference.

In Part 2, We covered how to export Windows VM application logs to Grafana Loki and how to view them from the Grafana Dashboard.

In Part 3, We will cover how to export Azure PAAS services logs to Grafana Loki

Dealing with multiple tools for capturing application logs from different sources can be a hassle for anyone. In this blog post, we'll dive into the steps required to establish centralized logging with Grafana Loki and Grafana Agent. This solution will allow us to unify the collection of logs from Kubernetes pods, VM services, and Azure PAAS services.

Grafana Loki : It is a highly scalable log aggregation system designed for cloud-native environments

Grafana Agent : It is an observability agent that collects metrics and logs from various application for visualization and analysis in Grafana

Requirement:

• Kuberentes Cluster (Latest version)
• Helm
• Azure PAAS(Eventhub and AKS, IOT, etc)
• VM with any application
• Azure storage account(Loki backend)
• Azure subscription with admin privileges

loki.write "local" {

        endpoint {

          url = "http://loki-loki-distributed-gateway.observability.svc.cluster.local/loki/api/v1/push"

        }

      }

In the grafana-agent-values.yaml, Currently it is added to export k8s pod logs, k8s events, etc.

Next deploy Grafana-Agent using below command,

helm install --values grafana-agent-values.yaml grafana-agent grafana/grafana-agent -n observability

Verify the pods are up and running by using below command,

kubectl get all -n observability

Now you can go to azure storage account which is configured in Grafana Loki and verify the logs are getting updated to the respective container.

Step 3: Deploy Grafana, to view the pod logs

Execute below command to install Grafana in K8s cluster,

helm install grafana grafana/grafana -n observability

Verify the pods are up and running by using below command,

kubectl get all -n observability

use below command to get the password for Grafana,

kubectl get secret --namespace observability grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Step 4:  Access Grafana and view the pod logs

Use port forward or use NodePort or Ingress to access the Grafana Dashboard.

- Configure loki as Datasource in Grafana,

Once login to the Grafana, Go to Home > Connections > Search for "Loki", then select it.

Next give connection url as, 

http://loki-loki-distributed-gateway.monitoring1.svc.cluster.local

then select save and test.

- Next import Loki dashboard,

Go to Grafana Home > Dashboard > select New and Import.

Use this below Grafana template ID "15141" , load it and save it.

https://grafana.com/grafana/dashboards/15141-kubernetes-service-logs/?source=post_page-----d2847d526f9e--------------------------------

Next click the dashboard, You can able to view all the pod logs now.

Step 5:  View the Kubernetes(K8s) Events in Grafana,

To view the Kubernetes events, Go to Grafana Home page > Explore > Select "Datasource name" > Select "Job" and value as "loki.source.kubernetes_events"

In this part, We have covered how to deploy Grafana Loki with Azure Storage as the backend, deployed Grafana Agent to view Kubernetes pod logs, and deployed Grafana dashboard to visualize the pod logs and Kubernetes events.

In the next Part 2, We will explain how to configure Grafana Agent for VM applications and send the logs to the same Grafana Loki.

Part 3 : We will cover how to export Azure PAAS services logs to Grafana Loki

In this blog, we will explore a new tool called 'Rover,' which helps to visualize the Terraform plan

Rover : This open-source tool is designed to visualize Terraform Plan output, offering insights into infrastructure and its dependencies.

We will use the "Rover" docker image, to do our setup and visualize the infra.

Requirements:

1.Linux/Windows VM

2. Docker

Steps 1 : Generate terraform plan output

I have a sample Azure terraform block in devopsart folder, will generate terraform plan output from there and store is locally.

cd devopsart

terraform plan -out tfplan.out

terraform show -json tfplan.out > tfplan.json

Now both the files are generated.

Step 2 : Run Rover tool locally,

Execute below docker command to run rover from the same step 1 path,

docker run --rm -it -p 9000:9000 -v $(pwd)/tfplan.json:/src/tfplan.json im2nguyen/rover:latest -planJSONPath=tfplan.json

Its run the webUI in port number 9000.

Step 3 : Accessing Rover WebUI,

Lets access the WebUI and check it,

Go to browser, and enter http://localhost:9000

In the UI, color codes on the left side provide assistance in understanding the actions that will take place for the resources when running terraform apply. 

When a specific resource is selected from the image, it will provide the name and parameter information.

Additionally, the image can be saved locally by clicking the 'Save' option

I hope this is helpful for someone who is genuinely confused by the Terraform plan output, especially when dealing with a large infrastructure.

Thanks for reading!! We have tried Rover tool and experimented with examples.

Reference:

https://github.com/im2nguyen/rover

Infracost :  It provides cloud cost projections from Terraform. It enables engineers to view a detailed cost breakdown and comprehend expenses before implementions.

Requirement :

1. One window/Linux VM

2.Terraform

3.Terraform examples

Step 1 : infracost installation,

For Mac, use below brew command to do the installation,

# brew install infracost

For other Operating systems, follow below link,

https://www.infracost.io/docs/#quick-start

Step 2 : Infracost configuration,

We need to set up the Infracost API key by signing up here,

https://dashboard.infracost.io

Once logged in, visit the following URL to obtain the API key,

https://dashboard.infracost.io/org/praboosingh/settings/general

Next, open the terminal and set the key as an environment variable using the following command,

# export INFRACOST_API_KEY=XXXXXXXXXXXXX

or You can log in to the Infracost UI and grant terminal access by using the following command,

# infracost auth login

Note : Infracost will not send any cloud information to their server.

Step 3 : Infracost validation

Next, We will do the validation. For validation purpose i have cloned below github repo which contains terraform examples.

# git clone https://github.com/alfonsof/terraform-azure-examples.git

# cd terraform-azure-examples/code/01-hello-world

try infracost by using below command to get the estimated cost for a month,

# infracost breakdown --path .

To save the report in json format and upload to infracost server, use below command,

# infracost breakdown --path . --format json --out-file infracost-demo.json

# infracost upload --path infracost-demo.json

In case we plan to upgrade the infrastructure and need to understand the new cost, execute the following command to compare it with the previously saved output from the Terraform code path.

# infracost diff --path . --compare-to infracost-demo.json

Thanks for reading!! We have installed infracost and experimented with examples.

References:

https://github.com/infracost/infracost

https://www.infracost.io/docs/#quick-start

In this blog, we will install and examine a new tool called Trivy, which helps identify vulnerabilities, misconfigurations, licenses, secrets, and software dependencies in the following,

1.Container image

2.Kubernetes Cluster

3.Virtual machine image

4.FileSystem

5.Git Repo

6.AWS

Requirements,

1.One Virtual Machine

2.Above mentioned tools anyone

Step 1 : Install Trivy

Exceute below command based on your OS,

For Mac : 

brew install trivy

In this blog post, We will explore a new tool called "KOR" (Kubernetes Orphaned Resources), which assists in identifying unused resources within a Kubernetes(K8S) cluster. This tool will be beneficial for those who are managing Kubernetes clusters.

Requirements:

1.One machine(Linux/Windows/Mac)

2.K8s cluster

Step 1 : Install kor in the machine.

Am using linux VM to do the experiment and for other flavours download the binaries from below link,

https://github.com/yonahd/kor/releases

Download the linux binary for linux VM,

wget https://github.com/yonahd/kor/releases/download/v0.1.8/kor_Linux_x86_64.tar.gz

tar -xvzf kor_Linux_x86_64.tar.gz

chmod 777 kor

cp -r kor /usr/bin

kor --help

Step 2 : Nginx Webserver deployment in K8s

I have a k8s cluster, We will deploy nginx webserver in K8s and try out "kor" tool

Create a namespace as "nginxweb"

kubectl create namespace nginxweb

Using helm, we will deploy nginx webserver by below command,

helm install nginx bitnami/nginx --namespace nginxweb 

kubectl get all -n nginxweb

Step 3 : Validate with kor tool

lets check the unused resources with kor tool in the nginx namespace,

Below command will list all the unused resources available in the given namespace,

Syntax : kor all -n namespace

kor all -n nginxweb

lets delete one service from the nginxweb namespace and try it.

kubectl delete deployments nginx -n nginxweb

Now check what are the resources are available in the namespace,

kubectl get all -n nginxweb

it gives the result of one k8s service is available under the nginxweb namespace

And now try out with kor tool using below command,

kor all -n nginxweb

it gives the same result, that the nginx service is not used anywhere in the namespace.

We can check only configmap/secret/services/serviceaccount/deployments/statefulsets/role/hpa by,

kor services -n nginxweb

kor serviceaccount -n nginxweb

kor secret -n nginxweb

That's all. We have installed the KOR tool and validated it by deleting one of the component in the Nginx web server deployment.

References:

https://github.com/yonahd/kor

The only Kubernetes log agent with intelligent error context capture, rule-based alerting, and 9 pluggable storage backends.

The Problem: Finding the Needle in the Log Haystack

Every SRE knows the pain: an alert fires at 3 AM, and you're digging through gigabytes of logs trying to understand what happened before the error. Traditional log solutions either capture everything (expensive) or miss crucial context (frustrating).

What if your log agent was smart enough to capture only what matters—the error AND the context around it—and alert you instantly?

Introducing Logsenta

Logsenta is an open-source Kubernetes log monitoring agent that solves this problem with intelligent error-aware context capture and rule-based alerting. Instead of blindly forwarding all logs, Logsenta:

• 🔍 Detects errors using configurable regex patterns
• ⏪ Captures context – logs BEFORE and AFTER the error
• 🚨 Alerts intelligently – route different errors to different teams
• 💾 Stores smartly – choose from 9 storage backends
• ⚡ Scales efficiently – handles 500+ pods with minimal resources

Key Features

🎯 Smart Error Detection

Logsenta uses regex and string-based pattern matching to detect errors across multiple languages and frameworks:

errorPatterns:
  - "ERROR"
  - "Exception"
  - "FATAL"
  - "panic:"
  - "Traceback"
  - "OOMKilled"
  - "CrashLoopBackOff"
  
  These patterns are "fully customizable"

🚨 Rule-Based Alerting NEW

Route different error patterns to different teams with customizable thresholds:

alerting:
  enabled: true
  rules:
    # Critical errors → On-call team immediately
    - name: "critical-errors"
      patterns: ["CRITICAL", "FATAL", "OOMKilled", "panic:"]
      threshold:
        count: 1          # Alert on FIRST occurrence
        windowSeconds: 60
      email:
        enabled: true
        toAddresses: ["oncall@company.com"]

    # Java exceptions → Backend team (after 2 occurrences)
    - name: "java-exceptions"  
      patterns: ["NullPointerException", "OutOfMemoryError"]
      threshold:
        count: 2          # Alert after 2 occurrences
        windowSeconds: 300
      email:
        enabled: true
        toAddresses: ["backend-team@company.com"]

    # Python errors → Data team
    - name: "python-errors"
      patterns: ["Traceback", "TypeError", "ValueError"]
      threshold:
        count: 1
      email:
        enabled: true
        toAddresses: ["data-team@company.com"]

Why rule-based alerting matters:

• 🎯 Reduce alert fatigue – only alert relevant teams
• ⏱️ Smart thresholds – distinguish between flaky tests and real outages
• 📧 Multiple channels – email and webhooks (Slack, PagerDuty, etc.)
• 🔄 Context included – alerts contain the actual error with surrounding logs

📦 9 Storage Backends

One agent, any storage destination:

🔄 Context Capture Window

Configure how much context to capture around errors:

captureWindow:
  bufferDurationMinutes: 2  # Lines BEFORE error
  captureAfterMinutes: 2    # Lines AFTER error

This means when an error occurs, you get the full story—not just the error line.

Quick Start

Deploy Logsenta in under 2 minutes:

# Clone the repository
git clone https://github.com/DevOpsArts/logsenta.git
cd logsenta

# Install with Helm (PostgreSQL backend + alerting)
helm install logsenta-engine ./charts/logsenta-engine \
  --namespace logsenta \
  --create-namespace \
  --set storage.type=postgresql \
  --set connections.postgresql.host=your-db-host \
  --set connections.postgresql.username=logsenta \
  --set connections.postgresql.password=YOUR_PASSWORD \
  --set alerting.enabled=true \
  --set alerting.email.smtpHost=smtp.company.com

Architecture

┌─────────────────────────────────────────────┐
│           Kubernetes Cluster                │
│                                             │
│  ┌─────┐  ┌─────┐  ┌─────┐                 │
│  │Pod A│  │Pod B│  │Pod C│  ← Monitored    │
│  └──┬──┘  └──┬──┘  └──┬──┘                 │
│     └────────┼────────┘                     │
│              ▼                              │
│     ┌────────────────────┐                  │
│     │   Logsenta-Engine  │                  │
│     │  • Error Detection │                  │
│     │  • Context Capture │                  │
│     │  • Rule-Based Alert│ ──► 📧 Email     │
│     │  • Rolling Buffer  │ ──► 🔗 Webhook   │
│     └─────────┬──────────┘                  │
└───────────────┼─────────────────────────────┘
                ▼
       ┌────────────────┐
       │ Storage Backend│
       │ (Your Choice)  │
       └────────────────┘
  

Production-Ready Features

• 🔒 Security: Non-root container, read-only filesystem, seccomp profiles
• 🔄 High Availability: Leader election for multi-replica deployments
• 📊 Scalability: ThreadPoolExecutor handles 500+ pods efficiently
• 🔐 SSL/TLS: Secure connections to all database backends
• 🛡️ Network Policies: Built-in network isolation templates
• 🚨 Smart Alerting: Rule-based routing with thresholds and cooldowns

Get Started Today

Logsenta is open-source and free to use. Check out the resources below:

• 📦 GitHub: github.com/DevOpsArts/logsenta
• 📖 Documentation: Wiki Documentation
• 🚨 Alerting Guide: Alerting Configuration
• 🐳 Docker Image: devopsart1/logsenta-engine

Have questions or feedback? Drop a comment below or open an issue on GitHub!

Tags: Kubernetes, DevOps, SRE, Logging, Monitoring, Alerting, Azure, AWS, GCP, Helm, Open Source

In Part 1, We covered how to setup Grafana Loki and Grafana Agent to view Kubernetes pod logs

Requirement:

• Grafana Loki
• Grafana agent
• Windows VM with one application
• Grafana Dashboard

Next double click the downloaded exe and install it, by default in windows the installer path is,

C:\Program Files\Grafana Agent

Once installation is completed, We need to update the configuration based on our needs like which application logs we need to send to Grafana loki.

In our case, we installed Grafana Dashboard in the windows VM and configured the Grafana dashboard logs in Grafana agent.

Similarly, we can add multiple application with different Job names.

Copy the grafana agent config file from below repo and update the required changes according on your needs.

We can start manually by below command in command prompt as well.

In command Prompt go to, C:\Program Files\Grafana Agent

Execute below command,

grafana-agent-windows-amd64.exe --config.file=agent-config.yaml

This will help to find any issue with the configuration.

Note : Here the Grafana loki distributed service endpoint(which is configured in the agent-config.yaml) should be accessible from the windows VM

Step 7 : Access VM application logs in Grafana Loki,

Go to Grafana Dashboard > Home > Explore > Select Loki Datasource

In the filter section, select "Job" and value as the job name which is given in the agent-config.yaml. In our case the job name is "devopsart-vm"

Now We are able to view the Grafana Dashboard logs in Grafana Loki. You can create the Dashboard from here based on your preference.

In Part 2, We covered how to export Windows VM application logs to Grafana Loki and how to view them from the Grafana Dashboard.

In Part 3, We will cover how to export Azure PAAS services logs to Grafana Loki

Dealing with multiple tools for capturing application logs from different sources can be a hassle for anyone. In this blog post, we'll dive into the steps required to establish centralized logging with Grafana Loki and Grafana Agent. This solution will allow us to unify the collection of logs from Kubernetes pods, VM services, and Azure PAAS services.

Grafana Loki : It is a highly scalable log aggregation system designed for cloud-native environments

Grafana Agent : It is an observability agent that collects metrics and logs from various application for visualization and analysis in Grafana

Requirement:

• Kuberentes Cluster (Latest version)
• Helm
• Azure PAAS(Eventhub and AKS, IOT, etc)
• VM with any application
• Azure storage account(Loki backend)
• Azure subscription with admin privileges

loki.write "local" {

        endpoint {

          url = "http://loki-loki-distributed-gateway.observability.svc.cluster.local/loki/api/v1/push"

        }

      }

In the grafana-agent-values.yaml, Currently it is added to export k8s pod logs, k8s events, etc.

Next deploy Grafana-Agent using below command,

helm install --values grafana-agent-values.yaml grafana-agent grafana/grafana-agent -n observability

Verify the pods are up and running by using below command,

kubectl get all -n observability

Now you can go to azure storage account which is configured in Grafana Loki and verify the logs are getting updated to the respective container.

Step 3: Deploy Grafana, to view the pod logs

Execute below command to install Grafana in K8s cluster,

helm install grafana grafana/grafana -n observability

Verify the pods are up and running by using below command,

kubectl get all -n observability

use below command to get the password for Grafana,

kubectl get secret --namespace observability grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Step 4:  Access Grafana and view the pod logs

Use port forward or use NodePort or Ingress to access the Grafana Dashboard.

- Configure loki as Datasource in Grafana,

Once login to the Grafana, Go to Home > Connections > Search for "Loki", then select it.

Next give connection url as, 

http://loki-loki-distributed-gateway.monitoring1.svc.cluster.local

then select save and test.

- Next import Loki dashboard,

Go to Grafana Home > Dashboard > select New and Import.

Use this below Grafana template ID "15141" , load it and save it.

https://grafana.com/grafana/dashboards/15141-kubernetes-service-logs/?source=post_page-----d2847d526f9e--------------------------------

Next click the dashboard, You can able to view all the pod logs now.

Step 5:  View the Kubernetes(K8s) Events in Grafana,

To view the Kubernetes events, Go to Grafana Home page > Explore > Select "Datasource name" > Select "Job" and value as "loki.source.kubernetes_events"

In this part, We have covered how to deploy Grafana Loki with Azure Storage as the backend, deployed Grafana Agent to view Kubernetes pod logs, and deployed Grafana dashboard to visualize the pod logs and Kubernetes events.

In the next Part 2, We will explain how to configure Grafana Agent for VM applications and send the logs to the same Grafana Loki.

Part 3 : We will cover how to export Azure PAAS services logs to Grafana Loki

In this blog, we will explore a new tool called 'Rover,' which helps to visualize the Terraform plan

Rover : This open-source tool is designed to visualize Terraform Plan output, offering insights into infrastructure and its dependencies.

We will use the "Rover" docker image, to do our setup and visualize the infra.

Requirements:

1.Linux/Windows VM

2. Docker

Steps 1 : Generate terraform plan output

I have a sample Azure terraform block in devopsart folder, will generate terraform plan output from there and store is locally.

cd devopsart

terraform plan -out tfplan.out

terraform show -json tfplan.out > tfplan.json

Now both the files are generated.

Step 2 : Run Rover tool locally,

Execute below docker command to run rover from the same step 1 path,

docker run --rm -it -p 9000:9000 -v $(pwd)/tfplan.json:/src/tfplan.json im2nguyen/rover:latest -planJSONPath=tfplan.json

Its run the webUI in port number 9000.

Step 3 : Accessing Rover WebUI,

Lets access the WebUI and check it,

Go to browser, and enter http://localhost:9000

In the UI, color codes on the left side provide assistance in understanding the actions that will take place for the resources when running terraform apply. 

When a specific resource is selected from the image, it will provide the name and parameter information.

Additionally, the image can be saved locally by clicking the 'Save' option

I hope this is helpful for someone who is genuinely confused by the Terraform plan output, especially when dealing with a large infrastructure.

Thanks for reading!! We have tried Rover tool and experimented with examples.

Reference:

https://github.com/im2nguyen/rover

Infracost :  It provides cloud cost projections from Terraform. It enables engineers to view a detailed cost breakdown and comprehend expenses before implementions.

Requirement :

1. One window/Linux VM

2.Terraform

3.Terraform examples

Step 1 : infracost installation,

For Mac, use below brew command to do the installation,

# brew install infracost

For other Operating systems, follow below link,

https://www.infracost.io/docs/#quick-start

Step 2 : Infracost configuration,

We need to set up the Infracost API key by signing up here,

https://dashboard.infracost.io

Once logged in, visit the following URL to obtain the API key,

https://dashboard.infracost.io/org/praboosingh/settings/general

Next, open the terminal and set the key as an environment variable using the following command,

# export INFRACOST_API_KEY=XXXXXXXXXXXXX

or You can log in to the Infracost UI and grant terminal access by using the following command,

# infracost auth login

Note : Infracost will not send any cloud information to their server.

Step 3 : Infracost validation

Next, We will do the validation. For validation purpose i have cloned below github repo which contains terraform examples.

# git clone https://github.com/alfonsof/terraform-azure-examples.git

# cd terraform-azure-examples/code/01-hello-world

try infracost by using below command to get the estimated cost for a month,

# infracost breakdown --path .

To save the report in json format and upload to infracost server, use below command,

# infracost breakdown --path . --format json --out-file infracost-demo.json

# infracost upload --path infracost-demo.json

In case we plan to upgrade the infrastructure and need to understand the new cost, execute the following command to compare it with the previously saved output from the Terraform code path.

# infracost diff --path . --compare-to infracost-demo.json

Thanks for reading!! We have installed infracost and experimented with examples.

References:

https://github.com/infracost/infracost

https://www.infracost.io/docs/#quick-start

In this blog, we will install and examine a new tool called Trivy, which helps identify vulnerabilities, misconfigurations, licenses, secrets, and software dependencies in the following,

1.Container image

2.Kubernetes Cluster

3.Virtual machine image

4.FileSystem

5.Git Repo

6.AWS

Requirements,

1.One Virtual Machine

2.Above mentioned tools anyone

Step 1 : Install Trivy

Exceute below command based on your OS,

For Mac : 

brew install trivy

In this blog post, We will explore a new tool called "KOR" (Kubernetes Orphaned Resources), which assists in identifying unused resources within a Kubernetes(K8S) cluster. This tool will be beneficial for those who are managing Kubernetes clusters.

Requirements:

1.One machine(Linux/Windows/Mac)

2.K8s cluster

Step 1 : Install kor in the machine.

Am using linux VM to do the experiment and for other flavours download the binaries from below link,

https://github.com/yonahd/kor/releases

Download the linux binary for linux VM,

wget https://github.com/yonahd/kor/releases/download/v0.1.8/kor_Linux_x86_64.tar.gz

tar -xvzf kor_Linux_x86_64.tar.gz

chmod 777 kor

cp -r kor /usr/bin

kor --help

Step 2 : Nginx Webserver deployment in K8s

I have a k8s cluster, We will deploy nginx webserver in K8s and try out "kor" tool

Create a namespace as "nginxweb"

kubectl create namespace nginxweb

Using helm, we will deploy nginx webserver by below command,

helm install nginx bitnami/nginx --namespace nginxweb 

kubectl get all -n nginxweb

Step 3 : Validate with kor tool

lets check the unused resources with kor tool in the nginx namespace,

Below command will list all the unused resources available in the given namespace,

Syntax : kor all -n namespace

kor all -n nginxweb

lets delete one service from the nginxweb namespace and try it.

kubectl delete deployments nginx -n nginxweb

Now check what are the resources are available in the namespace,

kubectl get all -n nginxweb

it gives the result of one k8s service is available under the nginxweb namespace

And now try out with kor tool using below command,

kor all -n nginxweb

it gives the same result, that the nginx service is not used anywhere in the namespace.

We can check only configmap/secret/services/serviceaccount/deployments/statefulsets/role/hpa by,

kor services -n nginxweb

kor serviceaccount -n nginxweb

kor secret -n nginxweb

That's all. We have installed the KOR tool and validated it by deleting one of the component in the Nginx web server deployment.

References:

https://github.com/yonahd/kor

The only Kubernetes log agent with intelligent error context capture, rule-based alerting, and 9 pluggable storage backends.

The Problem: Finding the Needle in the Log Haystack

Every SRE knows the pain: an alert fires at 3 AM, and you're digging through gigabytes of logs trying to understand what happened before the error. Traditional log solutions either capture everything (expensive) or miss crucial context (frustrating).

What if your log agent was smart enough to capture only what matters—the error AND the context around it—and alert you instantly?

Introducing Logsenta

Logsenta is an open-source Kubernetes log monitoring agent that solves this problem with intelligent error-aware context capture and rule-based alerting. Instead of blindly forwarding all logs, Logsenta:

• 🔍 Detects errors using configurable regex patterns
• ⏪ Captures context – logs BEFORE and AFTER the error
• 🚨 Alerts intelligently – route different errors to different teams
• 💾 Stores smartly – choose from 9 storage backends
• ⚡ Scales efficiently – handles 500+ pods with minimal resources

Key Features

🎯 Smart Error Detection

Logsenta uses regex and string-based pattern matching to detect errors across multiple languages and frameworks:

errorPatterns:
  - "ERROR"
  - "Exception"
  - "FATAL"
  - "panic:"
  - "Traceback"
  - "OOMKilled"
  - "CrashLoopBackOff"
  
  These patterns are "fully customizable"

🚨 Rule-Based Alerting NEW

Route different error patterns to different teams with customizable thresholds:

alerting:
  enabled: true
  rules:
    # Critical errors → On-call team immediately
    - name: "critical-errors"
      patterns: ["CRITICAL", "FATAL", "OOMKilled", "panic:"]
      threshold:
        count: 1          # Alert on FIRST occurrence
        windowSeconds: 60
      email:
        enabled: true
        toAddresses: ["oncall@company.com"]

    # Java exceptions → Backend team (after 2 occurrences)
    - name: "java-exceptions"  
      patterns: ["NullPointerException", "OutOfMemoryError"]
      threshold:
        count: 2          # Alert after 2 occurrences
        windowSeconds: 300
      email:
        enabled: true
        toAddresses: ["backend-team@company.com"]

    # Python errors → Data team
    - name: "python-errors"
      patterns: ["Traceback", "TypeError", "ValueError"]
      threshold:
        count: 1
      email:
        enabled: true
        toAddresses: ["data-team@company.com"]

Why rule-based alerting matters:

• 🎯 Reduce alert fatigue – only alert relevant teams
• ⏱️ Smart thresholds – distinguish between flaky tests and real outages
• 📧 Multiple channels – email and webhooks (Slack, PagerDuty, etc.)
• 🔄 Context included – alerts contain the actual error with surrounding logs

📦 9 Storage Backends

One agent, any storage destination:

🔄 Context Capture Window

Configure how much context to capture around errors:

captureWindow:
  bufferDurationMinutes: 2  # Lines BEFORE error
  captureAfterMinutes: 2    # Lines AFTER error

This means when an error occurs, you get the full story—not just the error line.

Quick Start

Deploy Logsenta in under 2 minutes:

# Clone the repository
git clone https://github.com/DevOpsArts/logsenta.git
cd logsenta

# Install with Helm (PostgreSQL backend + alerting)
helm install logsenta-engine ./charts/logsenta-engine \
  --namespace logsenta \
  --create-namespace \
  --set storage.type=postgresql \
  --set connections.postgresql.host=your-db-host \
  --set connections.postgresql.username=logsenta \
  --set connections.postgresql.password=YOUR_PASSWORD \
  --set alerting.enabled=true \
  --set alerting.email.smtpHost=smtp.company.com

Architecture

┌─────────────────────────────────────────────┐
│           Kubernetes Cluster                │
│                                             │
│  ┌─────┐  ┌─────┐  ┌─────┐                 │
│  │Pod A│  │Pod B│  │Pod C│  ← Monitored    │
│  └──┬──┘  └──┬──┘  └──┬──┘                 │
│     └────────┼────────┘                     │
│              ▼                              │
│     ┌────────────────────┐                  │
│     │   Logsenta-Engine  │                  │
│     │  • Error Detection │                  │
│     │  • Context Capture │                  │
│     │  • Rule-Based Alert│ ──► 📧 Email     │
│     │  • Rolling Buffer  │ ──► 🔗 Webhook   │
│     └─────────┬──────────┘                  │
└───────────────┼─────────────────────────────┘
                ▼
       ┌────────────────┐
       │ Storage Backend│
       │ (Your Choice)  │
       └────────────────┘
  

Production-Ready Features

• 🔒 Security: Non-root container, read-only filesystem, seccomp profiles
• 🔄 High Availability: Leader election for multi-replica deployments
• 📊 Scalability: ThreadPoolExecutor handles 500+ pods efficiently
• 🔐 SSL/TLS: Secure connections to all database backends
• 🛡️ Network Policies: Built-in network isolation templates
• 🚨 Smart Alerting: Rule-based routing with thresholds and cooldowns

Get Started Today

Logsenta is open-source and free to use. Check out the resources below:

• 📦 GitHub: github.com/DevOpsArts/logsenta
• 📖 Documentation: Wiki Documentation
• 🚨 Alerting Guide: Alerting Configuration
• 🐳 Docker Image: devopsart1/logsenta-engine

Have questions or feedback? Drop a comment below or open an issue on GitHub!

Tags: Kubernetes, DevOps, SRE, Logging, Monitoring, Alerting, Azure, AWS, GCP, Helm, Open Source

In Part 1, We covered how to setup Grafana Loki and Grafana Agent to view Kubernetes pod logs

Requirement:

• Grafana Loki
• Grafana agent
• Windows VM with one application
• Grafana Dashboard

Next double click the downloaded exe and install it, by default in windows the installer path is,

C:\Program Files\Grafana Agent

Once installation is completed, We need to update the configuration based on our needs like which application logs we need to send to Grafana loki.

In our case, we installed Grafana Dashboard in the windows VM and configured the Grafana dashboard logs in Grafana agent.

Similarly, we can add multiple application with different Job names.

Copy the grafana agent config file from below repo and update the required changes according on your needs.

We can start manually by below command in command prompt as well.

In command Prompt go to, C:\Program Files\Grafana Agent

Execute below command,

grafana-agent-windows-amd64.exe --config.file=agent-config.yaml

This will help to find any issue with the configuration.

Note : Here the Grafana loki distributed service endpoint(which is configured in the agent-config.yaml) should be accessible from the windows VM

Step 7 : Access VM application logs in Grafana Loki,

Go to Grafana Dashboard > Home > Explore > Select Loki Datasource

In the filter section, select "Job" and value as the job name which is given in the agent-config.yaml. In our case the job name is "devopsart-vm"

Now We are able to view the Grafana Dashboard logs in Grafana Loki. You can create the Dashboard from here based on your preference.

In Part 2, We covered how to export Windows VM application logs to Grafana Loki and how to view them from the Grafana Dashboard.

In Part 3, We will cover how to export Azure PAAS services logs to Grafana Loki

Dealing with multiple tools for capturing application logs from different sources can be a hassle for anyone. In this blog post, we'll dive into the steps required to establish centralized logging with Grafana Loki and Grafana Agent. This solution will allow us to unify the collection of logs from Kubernetes pods, VM services, and Azure PAAS services.

Grafana Loki : It is a highly scalable log aggregation system designed for cloud-native environments

Grafana Agent : It is an observability agent that collects metrics and logs from various application for visualization and analysis in Grafana

Requirement:

• Kuberentes Cluster (Latest version)
• Helm
• Azure PAAS(Eventhub and AKS, IOT, etc)
• VM with any application
• Azure storage account(Loki backend)
• Azure subscription with admin privileges

loki.write "local" {

        endpoint {

          url = "http://loki-loki-distributed-gateway.observability.svc.cluster.local/loki/api/v1/push"

        }

      }

In the grafana-agent-values.yaml, Currently it is added to export k8s pod logs, k8s events, etc.

Next deploy Grafana-Agent using below command,

helm install --values grafana-agent-values.yaml grafana-agent grafana/grafana-agent -n observability

Verify the pods are up and running by using below command,

kubectl get all -n observability

Now you can go to azure storage account which is configured in Grafana Loki and verify the logs are getting updated to the respective container.

Step 3: Deploy Grafana, to view the pod logs

Execute below command to install Grafana in K8s cluster,

helm install grafana grafana/grafana -n observability

Verify the pods are up and running by using below command,

kubectl get all -n observability

use below command to get the password for Grafana,

kubectl get secret --namespace observability grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Step 4:  Access Grafana and view the pod logs

Use port forward or use NodePort or Ingress to access the Grafana Dashboard.

- Configure loki as Datasource in Grafana,

Once login to the Grafana, Go to Home > Connections > Search for "Loki", then select it.

Next give connection url as, 

http://loki-loki-distributed-gateway.monitoring1.svc.cluster.local

then select save and test.

- Next import Loki dashboard,

Go to Grafana Home > Dashboard > select New and Import.

Use this below Grafana template ID "15141" , load it and save it.

https://grafana.com/grafana/dashboards/15141-kubernetes-service-logs/?source=post_page-----d2847d526f9e--------------------------------

Next click the dashboard, You can able to view all the pod logs now.

Step 5:  View the Kubernetes(K8s) Events in Grafana,

To view the Kubernetes events, Go to Grafana Home page > Explore > Select "Datasource name" > Select "Job" and value as "loki.source.kubernetes_events"

In this part, We have covered how to deploy Grafana Loki with Azure Storage as the backend, deployed Grafana Agent to view Kubernetes pod logs, and deployed Grafana dashboard to visualize the pod logs and Kubernetes events.

In the next Part 2, We will explain how to configure Grafana Agent for VM applications and send the logs to the same Grafana Loki.

Part 3 : We will cover how to export Azure PAAS services logs to Grafana Loki

In this blog, we will explore a new tool called 'Rover,' which helps to visualize the Terraform plan

Rover : This open-source tool is designed to visualize Terraform Plan output, offering insights into infrastructure and its dependencies.

We will use the "Rover" docker image, to do our setup and visualize the infra.

Requirements:

1.Linux/Windows VM

2. Docker

Steps 1 : Generate terraform plan output

I have a sample Azure terraform block in devopsart folder, will generate terraform plan output from there and store is locally.

cd devopsart

terraform plan -out tfplan.out

terraform show -json tfplan.out > tfplan.json

Now both the files are generated.

Step 2 : Run Rover tool locally,

Execute below docker command to run rover from the same step 1 path,

docker run --rm -it -p 9000:9000 -v $(pwd)/tfplan.json:/src/tfplan.json im2nguyen/rover:latest -planJSONPath=tfplan.json

Its run the webUI in port number 9000.

Step 3 : Accessing Rover WebUI,

Lets access the WebUI and check it,

Go to browser, and enter http://localhost:9000

In the UI, color codes on the left side provide assistance in understanding the actions that will take place for the resources when running terraform apply. 

When a specific resource is selected from the image, it will provide the name and parameter information.

Additionally, the image can be saved locally by clicking the 'Save' option

I hope this is helpful for someone who is genuinely confused by the Terraform plan output, especially when dealing with a large infrastructure.

Thanks for reading!! We have tried Rover tool and experimented with examples.

Reference:

https://github.com/im2nguyen/rover

Infracost :  It provides cloud cost projections from Terraform. It enables engineers to view a detailed cost breakdown and comprehend expenses before implementions.

Requirement :

1. One window/Linux VM

2.Terraform

3.Terraform examples

Step 1 : infracost installation,

For Mac, use below brew command to do the installation,

# brew install infracost

For other Operating systems, follow below link,

https://www.infracost.io/docs/#quick-start

Step 2 : Infracost configuration,

We need to set up the Infracost API key by signing up here,

https://dashboard.infracost.io

Once logged in, visit the following URL to obtain the API key,

https://dashboard.infracost.io/org/praboosingh/settings/general

Next, open the terminal and set the key as an environment variable using the following command,

# export INFRACOST_API_KEY=XXXXXXXXXXXXX

or You can log in to the Infracost UI and grant terminal access by using the following command,

# infracost auth login

Note : Infracost will not send any cloud information to their server.

Step 3 : Infracost validation

Next, We will do the validation. For validation purpose i have cloned below github repo which contains terraform examples.

# git clone https://github.com/alfonsof/terraform-azure-examples.git

# cd terraform-azure-examples/code/01-hello-world

try infracost by using below command to get the estimated cost for a month,

# infracost breakdown --path .

To save the report in json format and upload to infracost server, use below command,

# infracost breakdown --path . --format json --out-file infracost-demo.json

# infracost upload --path infracost-demo.json

In case we plan to upgrade the infrastructure and need to understand the new cost, execute the following command to compare it with the previously saved output from the Terraform code path.

# infracost diff --path . --compare-to infracost-demo.json

Thanks for reading!! We have installed infracost and experimented with examples.

References:

https://github.com/infracost/infracost

https://www.infracost.io/docs/#quick-start

In this blog, we will install and examine a new tool called Trivy, which helps identify vulnerabilities, misconfigurations, licenses, secrets, and software dependencies in the following,

1.Container image

2.Kubernetes Cluster

3.Virtual machine image

4.FileSystem

5.Git Repo

6.AWS

Requirements,

1.One Virtual Machine

2.Above mentioned tools anyone

Step 1 : Install Trivy

Exceute below command based on your OS,

For Mac : 

brew install trivy

In this blog post, We will explore a new tool called "KOR" (Kubernetes Orphaned Resources), which assists in identifying unused resources within a Kubernetes(K8S) cluster. This tool will be beneficial for those who are managing Kubernetes clusters.

Requirements:

1.One machine(Linux/Windows/Mac)

2.K8s cluster

Step 1 : Install kor in the machine.

Am using linux VM to do the experiment and for other flavours download the binaries from below link,

https://github.com/yonahd/kor/releases

Download the linux binary for linux VM,

wget https://github.com/yonahd/kor/releases/download/v0.1.8/kor_Linux_x86_64.tar.gz

tar -xvzf kor_Linux_x86_64.tar.gz

chmod 777 kor

cp -r kor /usr/bin

kor --help

Step 2 : Nginx Webserver deployment in K8s

I have a k8s cluster, We will deploy nginx webserver in K8s and try out "kor" tool

Create a namespace as "nginxweb"

kubectl create namespace nginxweb

Using helm, we will deploy nginx webserver by below command,

helm install nginx bitnami/nginx --namespace nginxweb 

kubectl get all -n nginxweb

Step 3 : Validate with kor tool

lets check the unused resources with kor tool in the nginx namespace,

Below command will list all the unused resources available in the given namespace,

Syntax : kor all -n namespace

kor all -n nginxweb

lets delete one service from the nginxweb namespace and try it.

kubectl delete deployments nginx -n nginxweb

Now check what are the resources are available in the namespace,

kubectl get all -n nginxweb

it gives the result of one k8s service is available under the nginxweb namespace

And now try out with kor tool using below command,

kor all -n nginxweb

it gives the same result, that the nginx service is not used anywhere in the namespace.

We can check only configmap/secret/services/serviceaccount/deployments/statefulsets/role/hpa by,

kor services -n nginxweb

kor serviceaccount -n nginxweb

kor secret -n nginxweb

That's all. We have installed the KOR tool and validated it by deleting one of the component in the Nginx web server deployment.

References:

https://github.com/yonahd/kor