Mastering Cost-Effective Log Management: A Guide to Adaptive Logs Drop Rules

Overview

Platform and observability teams constantly battle noisy log lines—those throwaway health checks, forgotten debug statements, or verbose info logs from rarely-used services. These not only clutter dashboards but also inflate costs. Until recently, eliminating them required cumbersome infrastructure changes. Now, with the Adaptive Logs drop rules feature in Grafana Cloud (in public preview), you can define custom rules to drop low-value logs before they are ingested, reducing noise and saving money immediately.

Mastering Cost-Effective Log Management: A Guide to Adaptive Logs Drop Rules

Drop rules complement the intelligent optimization recommendations already available in Adaptive Metrics and Adaptive Traces. You can create logic using any combination of log labels, detected log levels, or line content to drop logs before they are written to Cloud logs. This guide walks you through everything you need to start using drop rules effectively.

Prerequisites

An active Grafana Cloud account with logs enabled.
Logs being shipped to Grafana Cloud via Promtail, Grafana Agent, or other supported pipelines.
Basic understanding of log labels (e.g., service, namespace) and log levels (DEBUG, INFO, WARN, ERROR).
Access to the Adaptive Logs configuration interface (under 'Logs' in the Grafana Cloud Portal).

Step-by-Step Instructions

Understanding the Pipeline Order

Before creating drop rules, know the evaluation order when a log line arrives:

Exemptions: Logs matching exemption rules pass through untouched.
Drop rules (your custom rules): Evaluated in priority order. The first matching rule applies its drop rate.
Patterns: Optimization recommendations (intelligent sampling) apply to remaining logs not exempted or dropped.

This hierarchy ensures known noise is removed first, then intelligent sampling handles the rest.

Creating a Basic Drop Rule

Navigate to the Adaptive Logs section in Grafana Cloud. Click "Add drop rule". You will need to define:

Name: Descriptive identifier (e.g., "Drop all DEBUG logs").
Label selector: Optional, to target specific services or namespaces.
Log level filter: Choose from DEBUG, INFO, WARN, ERROR.
Line content matching: Regex or substring to match specific messages (e.g., healthcheck).
Drop percentage: 0% to 100%. 100% means drop all matching logs entirely.

Example: Drop all DEBUG logs from any service. Set label selector to {} (match all), log level to DEBUG, drop percentage 100%.

Example 1: Drop Noisy DEBUG Logs

DEBUG logs often consume the logging budget without providing value. To drop them:

Rule: "Drop All DEBUG"
Label selector: {} (all)
Log level: DEBUG
Drop %: 100

This immediately prevents any DEBUG log from being stored.

Example 2: Sampling Chatty, Repetitive Logs

If you suspect a verbose service generates many identical INFO lines, you can sample them by using a partial drop percentage:

Rule: "Sample HTTP request logs"
Label selector: {service="api-server"}
Line content: "GET /status"
Drop %: 90

This keeps 10% of those logs (representative sample) while discarding 90%.

Example 3: Targeting a Specific Noisy Producer

Suppose a batch processing job starts emitting high-volume, low-value logs. Target it with a label selector:

Rule: "Sample batch-job logs"
Label selector: {namespace="batch", service="data-processor"}
Drop %: 95

Combine with log level or line content for more precision.

Setting Priority Among Multiple Rules

Drop rules are evaluated in priority order (ascending number). Lower numbers have higher priority. For example:

Priority 1: Drop all DEBUG logs (100%).
Priority 2: Sample api-server GET requests (90%).

The first matching rule applies; if a log matches both, only the highest priority rule is used.

Confirming Drop Effectiveness

After saving a rule, monitor the Log Volume dashboard to see a reduction in ingested bytes. Adaptive Logs provides metrics on how many logs were dropped per rule.

Common Mistakes

Overly broad rules: Using {} with a high drop percentage on INFO can accidentally discard important logs from other services. Always start with a specific label selector.
Ignoring exemptions: If you have logs that must be kept (e.g., security audits), add them as exemptions first. Otherwise, a drop rule might catch them.
Wrong drop percentage: Sampling at 100% when you only intended to sample means you permanently lose all matching logs. Double-check percentages.
Not ordering rules correctly: A broad rule at high priority can overshadow narrower rules placed below it. Order from most specific to least specific.
Forgetting to test: Before applying to production, use Grafana Cloud's preview feature (if available) to estimate the impact.

Summary

Adaptive Logs drop rules empower you to eliminate known noisy log lines without touching application code or infrastructure. By combining label selectors, log levels, and line content matching with a drop percentage, you can precisely control which logs are stored. Used alongside exemptions and intelligent patterns, drop rules form a complete system for log cost management. Start with a small scope, monitor the effect, and iterate—your logging bill will thank you.

Tags: