Loading a million records without breaking the org

Under a thousand records, use whatever you already have. The synchronous REST or SOAP API is simpler, the feedback is immediate, and the volume does not matter.

Above roughly ten thousand, the synchronous path is the wrong tool. It consumes daily API allocation at one request per batch of 200, it holds the caller waiting, and any interruption leaves you guessing which records made it.

Between those two numbers, it depends on how often you do it. A one-off of 5,000 records is fine synchronously. The same 5,000 every hour is not.

Salesforce's introduction to Bulk API 2.0 and Bulk API is the reference, and the practical framing is simple: the asynchronous path trades immediate feedback for throughput and for a per-record result file you can act on.

Salesforce ships two generations under the same name, which causes a lot of confused documentation searches.

Bulk API 2.0 is the current one. You create a job, upload the data, close the job, and the platform handles batching for you. Fewer moving parts, fewer things to get wrong.

The original Bulk API makes batching your responsibility: you create the job, add batches yourself, and monitor each batch. That is more work, and it is occasionally the right answer because it gives you control over batch boundaries, which matters when records within a batch interact.

Salesforce documents the comparison directly in what is the difference between Bulk API 2.0 and Bulk API. Default to 2.0 unless you have a specific reason, and write the reason down when you deviate.

Bulk loads are CSV, and CSV is where a surprising share of failed loads originate. The format is specified in RFC 4180, and the specification exists because the naive reading of "comma separated" is wrong in several ways that matter.

Fields containing a comma must be quoted. Fields containing a quote must escape it by doubling. Fields containing a line break must be quoted, and a line break inside a quoted field does not end the record. Get any of these wrong and the row count in your file does not match the row count Salesforce parsed.

Company names are the usual culprit. "Acme, Inc." unquoted turns one record into two malformed ones, and the failure appears as a field-level error on a record that looks fine when you open the file in a spreadsheet.

The pre-flight check is mechanical. Parse your own file with a real CSV parser before you upload it, assert the row count and the column count per row, and reject the file if either is off. That check takes minutes to write and it catches a category of failure that is otherwise diagnosed one confusing error message at a time.

Character encoding deserves the same treatment. Declare UTF-8, verify UTF-8, and check the file for a byte order mark, because a stray BOM turns your first column header into something Salesforce does not recognise.

This is the mindset difference between bulk loading and everything else. A bulk job does not succeed or fail. It processes, and some rows succeed while others do not.

Salesforce returns results per record, and the documentation on handling failed records in batches describes the mechanism. The failures are yours to reconcile, and if nobody built a process for that, they are silently discarded.

The pattern I insist on is a load ledger. Before the job runs, write one row per input record with its source identifier and a pending status. After the job completes, download the results and update each row to success or failure with the platform's error text. The ledger is now the answer to every question anybody will ask.

Without the ledger, a job that loads 940,000 of a million records looks like a success in the job status and leaves 60,000 records missing with no list of which ones. Recovering from that means a full re-extract and a comparison, which costs more than building the ledger ever would.

Use external identifiers and upsert rather than insert wherever the source system has a stable key. Upsert makes a re-run idempotent, which converts "we do not know what got in" into "run it again".

Six checks, in order. Each one has cost me a Saturday at some point.

Check the allocation. Bulk API has its own limits and allocations covering daily records processed and concurrent jobs. Confirm your load fits before you start rather than after it stops.

Check what will fire. Triggers, flows, validation rules, workflow and roll-up summaries all run during a bulk load unless deliberately bypassed, and each one multiplies the work per record. A trigger that is fine at 200 records behaves differently across a million, which is the same class of problem as the governor limits that bite in production.

Check the permissions of the loading user. A bulk load runs as somebody, and that somebody needs object access, field access and the API permission. Salesforce's user permissions reference is the place to confirm exactly which ones, and a load that silently drops a field because the user cannot write it is a genuinely miserable thing to diagnose.

Check ownership and sharing recalculation. Loading records owned by many users triggers sharing recalculation, which can run for hours after the load appears finished. The visibility layer this touches is covered in who can see which record, and why.

Check the data model assumptions with a sample. Run 500 records first, inspect the created records by hand, and only then run the million. Salesforce's data model documentation will tell you what a field expects; a sample run will tell you what your file actually contains.

Check the clock. Salesforce's guidance on planning bulk data loads covers sequencing and timing, and the practical rule is to avoid business hours and avoid the same window as your other scheduled jobs.

Reading a million records out has its own failure mode, and it is the one people underestimate because reads feel safe.

A large query can exceed the platform's query cursor limits or time out mid-retrieval, and a retry starts from the beginning. The established mitigation is chunking by record identifier, which Salesforce documents in its PK chunking walkthrough.

The design principle is the same as for writes. Make each chunk independently retryable, record which chunks completed, and never treat the whole extract as one atomic operation. Microsoft's retry pattern is the standard formulation, and it applies directly.

Where this is a recurring extract feeding analytics rather than a one-off migration, the right destination is usually not another operational system. Landing Salesforce data beside ad platform and finance data so the joins can be defined once is the whole argument for a marketing data warehouse.

A large data load has a rollback question, a validation question and an owner question, and none of them are technical.

The rollback question: if this goes wrong, what exactly do we do? "Delete the records" is only an answer if you can identify them, which means every loaded record needs a marker field or a batch identifier written at load time. Add it before the load, not after.

The validation question: how will we know it worked? Row counts are necessary and insufficient. Pick three business assertions, for example total pipeline amount within a tolerance, count of records per owner, and count of records with a populated required field, and check all three against the source.

The owner question: who signs off? A data load changes what every report returns, so somebody who reads those reports has to accept the result. Naming that person before the load is how you avoid discovering the objection a week later.

Bounding all of this in writing, with the assumptions stated, is the same discipline as scoping any fixed-fee technical project. A load quoted as a day of work with no ledger, no markers and no validation assertions is a number invented to sound cheap, and it will be renegotiated the first time a batch fails.

A transit technology enterprise ran the same shape of problem at platform scale, moving 154 data sources with zero unplanned production outages against a hard vendor deadline. The thing that made it survivable was not the loading technique. It was that the inventory, the validation standard and the owner per asset all existed before anybody started.