Best practices for sizing Azure Monitor pipeline

The Azure Monitor pipeline is CPU-bound and scales linearly with available cores. This article provides throughput baselines, scaling strategies, and capacity planning examples to help you size a pipeline deployment for your environment.

Use these baselines alongside the Azure Monitor pipeline overview and Configure Azure Monitor pipeline to plan your deployment.

Pipeline types tested

Three pipeline configurations were tested, each representing a different level of processing:

• Syslog Basic. Minimal processing. The raw syslog message is passed through to a custom Log Analytics table with just three columns: TimeGenerated, Body (the full syslog message as-is), and SeverityText. No parsing of the syslog header or payload.

• Syslog Fully Formed. The pipeline parses the RFC 5424 syslog header and maps it into the standard Syslog table schema with structured columns: EventTime, Facility, SeverityLevel, Computer, HostName, HostIP, ProcessName, ProcessID, SyslogMessage, etc.

• CEF Fully Formed. The pipeline parses both the syslog header and the CEF payload inside it, mapping into the standard CommonSecurityLog table schema with 90+ columns: DeviceVendor, DeviceProduct, Activity, SourceIP, DestinationIP, DeviceAction, Protocol, RequestURL, and many more.

Measured throughput

All numbers are from a single replica under sustained full load for 5 minutes, averaged across 3 consecutive runs. Each message has randomized fields (IPs, ports, session IDs, etc.) to reflect realistic payloads.

4-core node (Standard_D4as_v6):

The pipeline automatically uses all available CPU cores on the node. No configuration changes are needed when scaling.

Capacity planning

Since throughput scales linearly with CPU, the per-vCPU rate is the simplest way to estimate capacity for any node size:

Multiply by cores per node to get per-replica throughput. Add 20–30% buffer for production workloads. To scale beyond a single node, add more replicas. Each replica handles traffic independently.

Per-vCPU throughput baselines

The pipeline scales linearly with CPU cores. Per-vCPU throughput is consistent across node sizes, which makes it the best starting point for capacity planning.

The difference between pipeline types reflects the parsing cost. Syslog Basic passes data through with minimal processing, while Syslog Fully Formed and CEF Fully Formed parse and map fields to their respective standard table schemas.

Scale vertically, horizontally, or both

To handle more load, use nodes with more CPU cores (vertical scaling), add more pipeline replicas (horizontal scaling), or combine both approaches. All three approaches yield linear throughput increases.

The pipeline automatically uses all available CPU cores per node. You don't need to make any configuration changes when you change node sizes.

Per-replica throughput = per-vCPU throughput x cores per node
Total throughput = per-replica throughput x replica count

Plan capacity for a target throughput

Calculate the resources you need for a target throughput by using these three steps:

1. Per-replica throughput = per-vCPU throughput × cores available per node.
2. Replicas needed = target throughput / per-replica throughput (round up).
3. Memory per replica = memory per vCPU × cores available per node.

Add 20-30% buffer to your resource allocation for production workloads. Each replica handles traffic independently, so running more replicas also improves fault tolerance.

Example: Syslog Basic at 500,000 logs/sec

Per-vCPU throughput for Syslog Basic: ~32,000 logs/sec.

Example: Syslog Fully Formed at 100,000 logs/sec

Per-vCPU throughput for Syslog Fully Formed: ~8,000 logs/sec.

Example: CEF Fully Formed at 100,000 logs/sec

Per-vCPU throughput for CEF Fully Formed: about 6,500 logs/sec.

Factors that might affect throughput

The baselines in this article are measured under controlled conditions. The following factors might reduce throughput in your environment and aren't yet reflected in the baselines:

• External ingress (gateway): Sending traffic through a gateway like Traefik adds network hops and potential TLS termination overhead compared to in-cluster delivery.
• TLS enabled: Enabling TLS or mTLS adds encryption overhead per connection.
• Variable payloads: The baselines use identical ~1.2 KB messages. Real-world traffic with variable message sizes and formats might affect parsing throughput.
• Durable buffering: Enabling disk-backed buffers adds I/O overhead but improves reliability during connectivity gaps.
• Transformations: Complex KQL transformations applied to data before export add processing overhead per message.

Test setup

• Node tested: Standard_D4as_v6 (4 vCPU, 16 GB)
• Transport: TCP
• Payload: RFC 5424 formatted syslog messages, ~1.2 KB each, with randomized fields (IPs, ports, session IDs, counters) per message. Syslog Basic and Syslog Fully Formed use a generic (non-CEF) message body. CEF Fully Formed uses a CEF-formatted message body.
• The load generator runs in the same cluster but uses minimal resources (< 2 cores, < 2 GB memory), so the cluster capacity is effectively dedicated to pipeline pods.
• Each test run scrapes the pipeline's internal Prometheus metrics to measure exact receive and export counts, and queries Log Analytics to verify end-to-end delivery. Pipeline received = exported = LA received (99.6–100%).
• Memory figures are working set as reported by kubectl top pods.

<13>1 2026-04-03T15:30:00.000000Z loadgen-host ThreatDetection 1234 ID47 - NOTCEF: This is a generic syslog test message for performance benchmarking. source=loadgen action=test severity=info category=benchmark host=testhost process=loadgen pid=5678 user=testuser session=abc123 status=success duration=0 bytes_in=0 bytes_out=0 packets=0 protocol=tcp src_ip=192.168.1.100 dst_ip=10.0.0.1 src_port=44840 dst_port=514 message=Performance test log entry for throughput measurement XXXXXXXXXXXXXX...

<13>1 2026-04-03T15:30:00.000000Z loadgen-host ThreatDetection 1234 ID47 - CEF:0|PaloAltoNetworks|PAN-OS|9.1.8|SSH2 Login Attempt(31914)|SSH2 Login Attempt(31914)|1|act=alert actionflags=0x2000000000000000 app=ssh cat=any cn1=67640598 cn2=1207111110 cnt=1 cs1=THREAT cs2=vulnerability cs3=Tap_Allow dst=172.21.166.15 dpt=22 src=172.21.76.92 spt=44840 proto=tcp dvchost=PA-820 ...

Related articles

• What is Azure Monitor pipeline?
• Configure Azure Monitor pipeline
• Azure Monitor pipeline extension versions
• Pod placement