TL;DR: A bulk endpoint is not "the same endpoint but with an array in the body." It needs its own limits, its own idempotency contract, its own partial-failure semantics, and — past a certain payload size — its own execution model (synchronous vs. queued). Skip any one of these and the endpoint that was supposed to reduce load on your system becomes the thing that takes it down.

Every API eventually gets a bulk endpoint. A customer wants to import 50,000 contacts instead of making 50,000 POST /contacts calls. An internal service wants to reconcile a nightly batch of 200,000 inventory updates against a set of supplier APIs in one shot instead of hammering your API with individual requests. This is exactly the kind of problem teams run into when doing custom software development around fragmented supplier or partner integrations, where the bulk endpoint is the seam between "works in the demo" and "works at scale." The instinct is reasonable — round-trips are expensive, and doing work in bigger chunks amortizes fixed costs like auth, TLS handshakes, and connection setup.

The problem is that most bulk endpoints are built by taking the existing single-item handler, wrapping it in a loop, and accepting an array instead of an object. That works fine in a demo with ten items. It falls over in production with ten thousand, and it falls over in a way that's much harder to debug than a single failed request, because now you have partial state, ambiguous retries, and a client that has no idea what actually happened.

This piece covers how to design bulk API endpoints that hold up under real load: sizing and pagination, partial failure semantics, idempotency across a batch, the sync-vs-async decision, and the database-side failure modes that bulk endpoints trigger disproportionately often.

Why "just loop over the array" breaks

A naive bulk handler does something like this:

python
@app.post("/contacts/bulk")
def bulk_create_contacts(items: list[ContactIn]):
    results = []
    for item in items:
        contact = db.session.add(Contact(**item.dict()))
        db.session.commit()  # one commit per item, or one big commit at the end
        results.append(contact)
    return results

Two failure modes show up almost immediately once this hits real traffic:

Unbounded work per request. If there's no cap on array length, a client (or an attacker) can send a 500,000-item payload in a single call. That single HTTP request now holds a worker thread, a database connection, and potentially a long-running transaction for as long as it takes to process — which for anything doing real validation or side effects (webhooks, search indexing, email) can be minutes. One request can starve your entire connection pool.

All-or-nothing semantics nobody asked for. If you commit once at the end and item #4,998 fails a foreign key constraint, either the whole batch rolls back (the client redoes 5,000 items to fix one bad row) or you've silently created 4,997 rows and have no clean way to tell the client which ones. Neither is what anyone actually wants.

DynamoDB's BatchWriteItem is instructive here: AWS caps it at 25 items and 16MB per call, and the operation will reject the entire batch if you exceed either limit. That's not an arbitrary restriction — it's an explicit statement that "bulk" doesn't mean "unlimited," and that the failure boundary needs to be predictable and enforced at the API layer, not discovered by whichever client sends the biggest payload first.

Set a real cap, and make pagination the client's problem, not yours

The fix is to decide, explicitly, what the maximum batch size is — and reject anything larger with a clear 4xx before you touch the database.

A reasonable approach:

  • Cap batch size at something your system can process comfortably within a normal request timeout (commonly 100–1,000 items depending on per-item cost — DynamoDB uses 25, most REST APIs land between 100 and 500).
  • Return 413 Payload Too Large or 400 Bad Request with an explicit message ("max 500 items per request, received 12,000") rather than silently truncating the array or timing out.
  • Document the cap in the API reference, not just in an error message nobody sees until production.

This pushes chunking responsibility onto the client, which is where it belongs — the client knows how many items it has and can loop client-side, with backoff, far more gracefully than your server can absorb an arbitrary payload.

Partial failure: use 207 Multi-Status, and mean it

Once you accept that a 500-item batch can have some items succeed and some fail, you need a response shape that says so per item. The relevant convention is HTTP 207 Multi-Status (borrowed from WebDAV, now used broadly for batch APIs):

json
{
  "status": "partial_success",
  "succeeded": 497,
  "failed": 3,
  "results": [
    { "index": 0, "status": "created", "id": "ct_8a1f2e" },
    { "index": 1, "status": "created", "id": "ct_8a1f2f" },
    { "index": 2, "status": "error", "code": "validation_error", "message": "email is required" }
  ]
}

A few rules that keep this honest:

  • Every input item maps to exactly one result item, in the same order, even the failures. A client that submitted 500 items should never have to guess which ones landed.
  • Item-level failures should not be caused by other items in the batch. If item #3's failure rolls back item #1 and #2 because they're all in one transaction, you don't have partial success — you have a lie. Either commit per-item (with the throughput cost that implies) or commit in small sub-batches (e.g., groups of 20–50) so a single bad row only takes out its own group.
  • Distinguish retryable from non-retryable failures explicitly in the error code (validation_error vs. rate_limited vs. internal_error) so the client knows whether resubmitting that one item is useful.

Idempotency across a batch, not just a single call

Single-request idempotency (an Idempotency-Key header, per Stripe's well-documented pattern) is well understood: the client generates a key, the server stores the eventual response keyed by that ID, and a retry with the same key returns the stored response instead of re-executing.

Batch endpoints complicate this in a way that's easy to get wrong: a single Idempotency-Key for the whole batch tells you the batch wasn't replayed, but says nothing about whether an individual item inside it was already processed by an earlier, partially-failed attempt.

The pattern that actually holds up:

  1. The client sends one top-level Idempotency-Key for the batch call itself, so a full retry of the same request doesn't get charged or processed twice.
  2. Each item in the batch also carries a caller-supplied idempotency key or natural unique key (e.g., an external order_id), and the server upserts on that key rather than blind-inserting.
  3. On retry after a partial failure, the server re-evaluates every item, but items that already succeeded resolve to "already exists, no-op" rather than a duplicate or an error — the retry is safe to run against the entire original batch, not just the failed subset, which is what most real client retry logic actually does.

This is the detail that separates "we support idempotency" from "we support idempotency for exactly the case we tested." Design for the client that doesn't track which three items failed and just resubmits the whole 500-item array.

When to go async: the 202 + polling / webhook pattern

Past a certain size or per-item processing cost, synchronous request/response stops being viable — not because of the HTTP timeout alone, but because holding a connection, a worker, and (often) a database transaction open for tens of seconds at high concurrency is how you exhaust your connection pool during exactly the traffic spike you most need to survive.

The standard shape for this is:

  1. Client POSTs the batch job. Server validates structure and size, persists the job (status: queued), and returns 202 Accepted immediately with a job ID and a Location header pointing at a status endpoint.
  2. A worker (separate from the request-handling process) picks up the job and processes it, updating per-item status as it goes.
  3. Client either polls GET /batch-jobs/{id} for status, or receives a signed webhook on completion — ideally both, since webhooks can be missed and polling is the reliable fallback.
text
POST /contacts/bulk-import
→ 202 Accepted
  Location: /batch-jobs/job_7f3a
  { "job_id": "job_7f3a", "status": "queued", "item_count": 12000 }

GET /batch-jobs/job_7f3a
→ 200 OK
  { "status": "processing", "succeeded": 8210, "failed": 4, "pending": 3786 }

A rough rule of thumb: if the batch can realistically complete inside 2–5 seconds under normal load, synchronous is simpler and fine. If per-item cost involves external calls (webhooks, third-party API calls, email sends, search re-indexing), or the batch size is client-controlled and can reach the thousands, go async from the start — retrofitting it later means a breaking API change.

The database side: where bulk endpoints actually die

Most bulk-endpoint outages aren't API design bugs — they're database exhaustion, and the API design decisions above are largely about preventing exactly this:

  • Connection pool exhaustion. A handful of concurrent bulk requests, each holding a connection for the full duration of a large synchronous batch, can exhaust a pool sized for many short-lived requests. If your normal request holds a connection for 20ms and your bulk request holds one for 20 seconds, three concurrent bulk callers can lock out everyone else. Keep bulk-request database work in short transactions, and consider a separate, smaller connection pool for bulk/batch traffic so it can't starve normal API traffic.
  • Lock contention from long transactions. A single transaction wrapping 10,000 inserts holds row and index locks for the entire duration, which serializes against any other writer touching the same rows or index ranges. Sub-batching commits (e.g., every 100–500 items) trades a little throughput for dramatically shorter lock hold times.
  • Downstream rate limits you don't control. If bulk processing fans out to third-party APIs (payment processors, CRMs, shipping providers), your batch is now bounded by their rate limit, not yours — Shopify's Admin API, for instance, enforces roughly 2 requests/second sustained with a small burst allowance on standard plans. Queue-based processing with backoff handles this naturally; synchronous bulk endpoints that fan out to rate-limited third parties will time out unpredictably.

A decision framework

Batch size / per-item costRecommended pattern
Small (<100 items), cheap per-item, no external callsSynchronous, single transaction acceptable
Medium (100–1,000 items), moderate costSynchronous, sub-batched commits, 207 partial response
Large (1,000+ items) or external calls per itemAsync: 202 + job ID, poll/webhook, queue-backed worker
Client-controlled, potentially unbounded sizeAsync only, with a hard cap on enqueued job size and client-side chunking required above it

Designing bulk endpoints that hold up under production traffic is one of the more common pieces of architecture work we do when building or hardening API integrations for clients — it tends to be a small surface area with outsized failure consequences if it's skipped.

Sources