OpenTelemetry Collector Deployment Patterns: Agent, Gateway and Sidecar Trade-offs

TL;DR: There is no single "right" OpenTelemetry Collector topology. Start with the simplest thing that works (apps exporting OTLP straight to a backend, or one agent per node), add a gateway when you need central policy, credentials or tail sampling, and use sidecars only where per-workload isolation justifies the extra pods. The decisions that hurt later are not "agent or gateway" but processor ordering, memory limits, and how spans for one trace reach the same sampler.

Why deployment topology matters more than the config file

Most teams adopt the OpenTelemetry Collector by copying a working YAML file and pointing it at a backend. That works until the second cluster, the first backend outage, or the first invoice for trace storage. At that point the questions change: where does credential handling live, who owns sampling policy, what happens to telemetry when a node runs out of memory, and how many places do you have to patch when a processor needs changing?

The OpenTelemetry project describes several ways to run the Collector: as an agent, as a gateway, and combinations of the two, plus the option of not running a Collector at all (Deploy the Collector). In Kubernetes you will also meet the sidecar variant, which the project's scaling guidance treats as a way to place a Collector container inside the workload pod (Scaling the Collector).

This article compares those patterns for platform and DevOps teams running Kubernetes-based estates, with a view to the concerns we most often hear from engineering buyers in the US, UK and Singapore: operating cost, blast radius, data-handling controls, and how much the platform team has to own. Everything factual below is taken from the official OpenTelemetry documentation and component READMEs linked at the end; where we give opinion, we say so.

The four patterns at a glance

flowchart LR
  subgraph A[No Collector]
    A1[App + SDK] -->|OTLP| A2[(Backend)]
  end
  subgraph B[Agent - DaemonSet]
    B1[Pods] -->|OTLP| B2[Node agent] --> B3[(Backend)]
  end
  subgraph C[Agent to Gateway]
    C1[Pods] --> C2[Node agent] -->|OTLP| C3[Gateway pool] --> C4[(Backend)]
  end
  subgraph D[Sidecar]
    D1[App container] --> D2[Collector container in same pod] --> D3[(Backend or gateway)]
  end

Pattern 0: no Collector

The agent-to-gateway documentation is explicit that simpler patterns can be the better choice: applications can send OTLP directly to backends, and you may not need host metrics, log collection or tail sampling, or your deployment may simply be small (Agent-to-gateway deployment pattern). This is a legitimate production choice. Its cost is that retry, batching, credentials and backend choice live in every application's configuration, so changing vendors means redeploying services.

Pattern 1: agent (typically a DaemonSet)

In the agent pattern, signals go to a Collector that runs alongside the application or on the same host, "such as a sidecar or DaemonSet" (Agent deployment pattern). The documented upsides are that it is straightforward to start with and gives a clear one-to-one mapping between application and Collector. The documented downsides are limited scalability for teams and infrastructure resources, and inflexibility for complex or evolving deployments.

On Kubernetes, a DaemonSet agent gives every node a local OTLP endpoint, which makes it the natural place to enrich telemetry with Kubernetes metadata and to collect node-level signals. The agent-to-gateway page notes that agents "typically don't require horizontal scaling because they run on each host" (source), which is a real operational advantage: capacity grows with your node count.

Pattern 2: gateway

A gateway is a Collector deployment that receives telemetry from many sources and sits in front of the backend. The gateway documentation says the endpoint is typically provided per cluster, per data center or per region, and lists the trade-offs plainly: separation of concerns (for example centrally managed credentials) and centralised policy (filtering, sampling) on the plus side; one more thing to maintain and that can fail, added latency in cascaded setups, and higher overall resource usage on the minus side (Gateway deployment pattern).

Pattern 3: agent to gateway

Combining the two is the most common shape in multi-team platforms. Agents do the cheap, local work (enrichment, a first memory guard, small batches). The gateway does the work that needs a global view: tail sampling, filtering, transformation, and holding backend credentials. The documentation lists centralised processing, network isolation and cost optimisation at scale as the reasons to choose it.

Pattern 4: sidecar

The scaling guide describes the sidecar pattern as adding a container into the workload pod, and notes that the OpenTelemetry Operator can inject it automatically. It also gives a rule of thumb: when the number of nodes is low and the number of pods is high, sidecars might make more sense, and they avoid the situation where one failing DaemonSet pod takes out telemetry for every pod on its node (Scaling the Collector). The flip side, which follows directly from the mechanism, is one extra container per pod to schedule, size and upgrade.

Trade-off table

Treat this as a decision aid rather than a ranking. A small team with a single cluster is often best served by the first or second column; a platform serving many teams usually ends up in the fourth.

Decision flow

flowchart TD
  S[Start] --> Q1{Need host/node signals,<br/>logs, or K8s enrichment?}
  Q1 -- No --> Q2{Need central sampling,<br/>filtering or credential control?}
  Q2 -- No --> N[Export OTLP directly<br/>no Collector]
  Q2 -- Yes --> G[Gateway]
  Q1 -- Yes --> Q3{Need central policy<br/>or tail sampling?}
  Q3 -- No --> AG[DaemonSet agent only]
  Q3 -- Yes --> AGW[Agent + Gateway]
  AG --> Q4{Few nodes, many pods,<br/>or per-workload isolation?}
  AGW --> Q4
  Q4 -- Yes --> SC[Consider sidecars for those workloads]
  Q4 -- No --> Done[Keep DaemonSet]

Building the agent tier

The following agent configuration is a deliberately small sketch of the agent-to-gateway shape. Treat the values as starting points to be sized against your own load, not as recommendations from us or the project.

# Agent (DaemonSet) - collector config sketch
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  k8sattributes:
    filter:
      node_from_env_var: KUBE_NODE_NAME
  batch: {}

exporters:
  otlp:
    endpoint: otel-gateway:4317
    tls:
      insecure: true   # in-cluster example only; use TLS in production

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]

A few things in this sketch are directly grounded in the documentation:

• memory_limiter is first. The memory limiter README says the best practice is to add it as the first processor in a pipeline, and the agent-to-gateway guide repeats the advice (memory limiter README).
• k8sattributes appears before batch. The k8sattributes README states that when using connection-based pod association it must appear before any batching or tail sampling, because those remove the connection information (k8sattributes README).
• The KUBE_NODE_NAME filter follows the README's DaemonSet example, where the environment variable is populated from the pod's spec.nodeName field via the downward API. Restricting the processor to its own node keeps its API watch small.
• The agent exports OTLP to a service name, matching the guide's recommendation to use OTLP between agents and gateways.

The same guidance says agents should use smaller batch sizes and shorter timeouts than gateways, which should use larger batches and longer timeouts. Tune batch accordingly once you have measured.

Sizing the memory limiter

The memory limiter has a few defaults worth knowing. check_interval defaults to 0s with a recommended value of 1 second, you must set either limit_mib or limit_percentage, and the spike limit defaults to 20% of the limit. The README recommends limit_percentage in containers and limit_mib on bare metal or VMs, suggests setting GOMEMLIMIT to 80% of the Collector's hard memory limit, and is blunt that the processor is not a replacement for properly sizing the Collector. It also warns that data is permanently lost if the component before the limiter does not retry correctly, so check that your SDKs and upstream senders retry on refusal. The scaling guide suggests watching otelcol_processor_refused_spans and scaling up if data is refused too often.

Building the gateway tier

A gateway without tail sampling is the easy case. The agent-to-gateway page says such gateways can sit behind any load balancer or Kubernetes service with round-robin distribution. Two cautions from the scaling guide apply even then:

1. For gRPC and OTLP, use an L7 load balancer. Standard L4 balancers might establish a persistent connection to a single backend, leaving the rest of the pool idle.
2. Size queues and retries for outages. The documentation notes gateways often need larger queues and retry policies to handle backend outages, and the scaling guide suggests considering a scale-up when a queue reaches 60 to 70% of capacity, and keeping a small number of replicas (it gives three as an example) for resilience.

For metrics there is a correctness rule rather than a performance one: all metric streams within OTLP must have a single writer. With several gateway replicas, each stream has to be routed to one of them and carry a globally unique identity. Cumulative-to-delta conversion needs data-aware load balancing for the same reason. The gateway page also recommends the k8sattributes and resourcedetection processors as best practice.

Tail sampling: where topology gets hard

Tail sampling is the main reason teams introduce a gateway, and it is where most designs go wrong. The processor README is unambiguous: all spans for a given trace must be received by the same Collector instance for effective sampling decisions. Its documented defaults include a decision_wait of 30s and num_traces of 50000, and it buffers spans in memory while it waits, which is why it is stateful.

If you scale the sampler horizontally with an ordinary load balancer, different replicas receive spans of the same trace and each makes a decision on a partial view. The documented answer is a two-tier gateway: a first layer running the load-balancing exporter, which routes by trace ID so that the same trace always lands on the same instance of the second layer, which runs the tail sampling processor. The gateway page says to use this two-tiered setup when telemetry must be processed in a specific Collector.

# Tier 1 - routing layer (exporter section)
exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true   # example only
    resolver:
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317

# Tier 2 - sampling layer (processor section)
processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: sample-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Details from the load-balancing exporter README that matter in production:

• routing_key options include service, traceID, metric, resource, streamID and attributes. The default is traceID for traces and service for logs and metrics. traceID is invalid for metrics.
• Resolvers are static, dns, k8s and aws_cloud_map. The DNS resolver re-queries on an interval (5s default), and the scaling guide warns that each Collector may run the DNS query at different times, so routing can briefly disagree between tier-1 instances during scale events.
• The k8s resolver needs RBAC: the service account must be able to get, list and watch EndpointSlice objects in the target namespace, otherwise the exporter's cache stays empty and it logs a missing-exporter error.
• When the backend list changes, around R/N of routes are rerouted differently, so scaling tier 2 in or out will split some in-flight traces. Plan sampling changes and scale events accordingly.
• Queue, retry and timeout settings are disabled by default in the load-balancing section, and by default the exporter does not re-route to a healthy endpoint on delivery failure. You must configure resilience explicitly.

The agent-to-gateway page adds a candid caution: running tail sampling across multiple gateways is an advanced setup with practical caveats, and unless you have a robust sticky-routing strategy you should prefer a single well-resourced tail-sampling gateway. Our view, as an opinion rather than a documented fact, is that many teams should start there and add the routing tier only when one instance genuinely cannot hold the working set.

Also mind the ordering rule from the tail-sampling README: it must come after any processor that relies on context, such as k8sattributes. If you sample probabilistically across several Collectors, make sure they share the same hash seed.

Running sidecars with the Operator

The OpenTelemetry Operator can inject a Collector sidecar. The scaling guide shows a resource with mode: sidecar, and a pod annotation to request injection. The apiVersion shown there at the time of writing is v1alpha1, so check the current CRD version before copying.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: sidecar
spec:
  mode: sidecar
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 127.0.0.1:4317
    exporters:
      otlp:
        endpoint: otel-gateway:4317
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]
---
# In the workload's pod template metadata:
#   annotations:
#     sidecar.opentelemetry.io/inject: "true"

Binding to localhost is the safer default when every client is in the same pod; the official examples use 0.0.0.0 as a convenience and the agent-to-gateway page notes localhost is generally preferable when all clients are local.

When do sidecars earn their keep? Typical reasons include per-tenant or per-workload configuration, strict isolation between workloads sharing nodes, and clusters with few large nodes but many pods (the scaling guide's own rule of thumb). When sidecars are used, they usually forward to a gateway rather than directly to a backend, so credentials still stay central.

Operational concerns buyers should ask about

Data handling and residency. A gateway gives you one place to filter, redact or drop attributes before data leaves a region, and one place to hold backend credentials. If you operate under UK GDPR, the EU's rules, or Singapore's PDPA, that central choke point is often easier to explain to an auditor than policy scattered across hundreds of application configs. This is a design observation, not legal advice; check the actual requirements with your compliance team.

Multi-region estates. The gateway documentation positions a gateway per cluster, data center or region. For a team serving US, UK and Singapore users, that usually means one gateway pool per region with local agents sending to it, rather than a single global pool with long-haul links.

Upgrades and supply chain. The project's own anti-patterns post advises against using a single Collector to collect all telemetry for all infrastructure and applications, and recommends building your own distribution with the OpenTelemetry Collector Builder rather than shipping the Core or Contrib distributions as-is in production. That post dates from March 2024, so confirm its current wording, but the underlying argument (ship only the components you use) still reduces attack surface and upgrade friction.

Cost. The gateway documentation lists higher overall resource usage as a cost of the pattern, and sidecars multiply Collector instances by pod count. Neither is a reason to avoid them; both are reasons to measure before and after rather than assume.

Common mistakes

1. Putting batch before k8sattributes or tail_sampling. Connection-based association stops working, and enrichment silently goes missing.
2. Leaving memory_limiter off, or not first. An overloaded Collector then falls over instead of refusing data and applying back-pressure.
3. Using an L4 load balancer in front of OTLP/gRPC. One long-lived connection pins traffic to a single replica.
4. Scaling a tail-sampling gateway with a plain Deployment and Service. Traces split across replicas and sampling decisions are made on partial data.
5. Sending metrics through round-robin gateways without a single-writer plan. Duplicate or conflicting series follow.
6. Treating memory_limiter as a substitute for sizing. The README says it is not.
7. Forgetting RBAC for the k8s resolver or k8sattributes. Both fail quietly or with confusing errors when the service account cannot list the resources they watch.
8. Using insecure TLS outside examples. The snippets above use it for brevity only.
9. One giant Collector for everything. The anti-patterns post advises against it; split by signal or responsibility.

Pre-production checklist

• [ ] Decided and documented why each tier exists (what does it do that the previous one cannot?)
• [ ] memory_limiter first in every pipeline, check_interval set, GOMEMLIMIT aligned with the container limit
• [ ] k8sattributes before batch and before tail_sampling; RBAC reviewed and, where possible, namespace-scoped
• [ ] OTLP between tiers, over TLS in production
• [ ] L7 load balancing (or a headless service with client-side balancing) for gRPC traffic
• [ ] Queue and retry settings reviewed on every exporter, including the load-balancing exporter where they are off by default
• [ ] Tail sampling routed by trace ID, with a documented plan for scale-in and scale-out events
• [ ] Metrics have a single-writer strategy and unique resource identity
• [ ] Alerts on refused spans and on exporter queue utilisation
• [ ] Custom Collector build (or a documented component allow-list) instead of shipping everything
• [ ] Rollback plan for Collector upgrades, tested in a non-production cluster first

How Syslabs can help

Syslabs works with engineering teams on cloud infrastructure, CI/CD and delivery pipelines, and migrations onto Kubernetes-based platforms. For observability this typically means reviewing an existing Collector estate against the checklist above, designing the agent and gateway tiers around your regions and compliance constraints, and building the rollout and upgrade pipeline so changes to telemetry configuration are reviewed and tested like any other code. We keep our recommendations tied to your measured load rather than a template.

If you are weighing one of these topologies, a short architecture review is usually the cheapest way to avoid a rebuild.

Book a technical conversation: https://www.syslabs.in/book/meeting

References

• OpenTelemetry. Deploy the Collector
• OpenTelemetry. Agent deployment pattern
• OpenTelemetry. Gateway deployment pattern
• OpenTelemetry. Agent-to-gateway deployment pattern
• OpenTelemetry. Scaling the Collector
• OpenTelemetry. OpenTelemetry Collector Antipatterns
• OpenTelemetry Collector. memorylimiterprocessor README
• OpenTelemetry Collector Contrib. loadbalancingexporter README
• OpenTelemetry Collector Contrib. tailsamplingprocessor README
• OpenTelemetry Collector Contrib. k8sattributesprocessor README

Component defaults and API versions change between releases; confirm against the version you deploy.