OpenAI's Dota agents beat the world champions after training the equivalent of tens of thousands of years of game time. ANYmal's quadruped walked over rubble through forests and stairs. Google's data-center cooling RL posted a 40% efficiency gain that has held for years in production. Sim-to-real demonstrably works.
And yet most industry RL projects fail at exactly this step. The sim agent performs well, the deployment goes live, and within hours the policy behaves in ways that match no trajectory the lab has seen. Engineers blame the simulator and add more domain randomization. The policy grows slightly more robust in test, slightly less reliable in production, and the project stalls.
The difficulty is procedural. Two decades of research have produced techniques that work in combination, but practitioners arrive expecting a single switch: turn on domain randomization, deploy, done. The real workflow is a checklist where skipping one step invalidates the rest.
Five principles for closing the gap in industrial RL deployments, with examples from the three domains where we do most of our work: biotech (lab-to-clinical and bioprocess control), robotics (contact-rich manipulation and locomotion), and energy (grid simulators to SCADA-real control). The order matters, and most projects skip at least two.
1. Treat the "reality gap" as a taxonomy, not a single problem
When a deployed policy underperforms its sim benchmarks, the temptation is to blame the simulator and add randomization. That rarely works, because there are at least five distinct gaps:
- Visual gap: rendered appearance differs from camera imagery (lighting, texture, motion blur, sensor noise, lens distortion).
- Dynamics gap: the simulator's transition function doesn't match reality (contact physics, fluid flow, friction models, hysteresis).
- Sensor-noise gap: real sensors have biases, drift, and stochasticity that the simulator doesn't replicate.
- Latency gap: actuation delays and observation-to-action timing differ from what the policy was trained against.
- Contact / actuator gap: the simulator treats actuator outputs as commands; reality treats them as suggestions that motors, hydraulics, valves, and grippers interpret differently.
Each closes by a different technique, so treating them as one problem means fixing one and ignoring the rest. Three examples:
Your cell-culture simulator models growth kinetics with a doubling-time parameter, but the actual bioreactor has gas transfer rates that drift by 5–15% across runs, foam dynamics the simulator ignores entirely, and pH sensor drift of 0.02 units per day. A policy trained in sim optimizes for glucose feed rates that work in the simulated curve but overshoot the real one within the first batch.
MuJoCo's contact model treats objects as rigid bodies with simple Coulomb friction. Real deformable objects — a bag of frozen vegetables, a soft package, a wet piece of fruit — have viscoelastic contact dynamics that the simulator doesn't see. A sim-trained grasp policy that succeeds 95% of the time in the lab fails 40%+ of the time when deployed on actual produce.
Your grid simulator steps at 1-second resolution, but SCADA real-time control runs on a 100-ms clock with jitter. A policy trained in sim operates on the assumption that observations are temporally consistent — they're not. The deployed policy makes decisions based on a state estimate that's already 60 ms out of date.
Name the gap you are trying to close. If you cannot say which of the five your project has, you are guessing.
2. Domain randomization is necessary but rarely sufficient
Domain randomization (DR), training across a distribution of simulator configurations, is the best-known tool for closing the gap. It works, and it fails in a characteristic way.
Exposing the policy to many simulator variations (textures, lighting, mass, friction, sensor noise, latency) should produce a policy that generalizes to the real system, which is one more variation. The intuition holds; teams apply it too narrowly.
The OpenAI Rubik's-cube hand (Andrychowicz et al., 2020) is the canonical example: trained in sim with extensive DR, then transferred to a real robotic hand. Most summaries omit the mechanism that made it work, an asymmetric actor-critic setup with residual adaptation in the critic. The critic stayed in sim while the actor ran in reality, fine-tuned online. The paper is explicit that DR alone would not have produced the result.
A practical lab-analog example: in a recent robotic insertion task (electrical connector mating, 6 DOF), DR over pose, friction, and lighting raised sim success from 88% to 96%, but real-world success was still 71%. Adding a small real-world fine-tune phase — 200 trajectories on the actual hardware, used to update only the last layer of the policy network — closed the remaining gap to 92%. DR did the coarse work; the residual model made the policy deployable.
You've budgeted for at least two transfer mechanisms: DR as the coarse adapter, plus residual models (Principle 3), learned world models (Principle 4), or limited real-world fine-tuning. They cover different failure modes.
Your plan calls for DR as the only transfer mechanism, and "fine-tuning on the real system" isn't in scope because of safety, cost, or sample-budget constraints. You'll likely get a policy that's robust to surface variation but brittle to dynamics the simulator doesn't model.
3. Calibrate on the actual instrument you intend to deploy on
The most common pattern in failed transfers is calibration on the wrong machine. It happens constantly.
Your team trains a policy in a simulator tuned to nominal hardware: a specific robotic arm model, bioreactor, or power-grid model. The simulator parameters (link lengths, joint friction, mass distribution, sensor noise floor) come from a manufacturer's spec sheet, a paper on a similar instrument, or a one-time measurement six months ago.
On the actual hardware you discover that the spec quotes nominal values, the paper covered a slightly different model, and the six-month-old measurement predates a sensor recalibration. The resulting calibration error looks small: five millimeters of arm deflection, two parts per million of glucose sensor offset, a 30-ms latency the controller does not have. It compounds into a deployed policy that misses its targets.
The fix is to system-identify the actual deployment target, the specific unit you will deploy on, and then learn either (a) a dynamics correction on the simulator that brings it into alignment with the real system, or (b) a residual model that the policy queries for "what the simulator says vs. what reality does" and uses to adjust its actions.
The biotech version: calibrate your bioprocess simulator on the actual bioreactor you'll deploy on, with the actual probe lot, the actual media batch, and at least three runs to capture run-to-run variability. The robotics version: characterize your actual robot arm, in its actual cell, using its actual gripper, with the actual cables attached (cable routing affects dynamics more than most teams realize). The energy version: validate your grid simulator against your actual SCADA feed on at least two weeks of representative operating conditions, including the failure cases.
Your project plan includes system identification of the actual deployment hardware, with multiple runs to capture variability, and the simulator parameters are updated from those measurements rather than from spec sheets.
Your simulator parameters come from the manufacturer's datasheet, a literature reference, or a single earlier calibration session — and the deployment hardware is the same model but a different physical unit.
4. Consider learned simulators (world models) when the hand-engineered one falls short
For decades, sim-to-real meant: build or buy a simulator, train the policy in it, transfer to reality. DreamerV3 and its predecessors added another option: learn a simulator from data, then train the policy inside the learned model. This "world model" approach now outperforms hand-engineered simulators in several Atari benchmarks and in Minecraft, and the technique has matured enough to be relevant for industrial work.
Learned simulators beat hand-engineered ones when (a) the real system is too complex to model from first principles — protein folding, multi-body contact in unstructured environments, the full national grid with weather and demand variability — and (b) you have enough real-system data to train a faithful enough dynamics model. They lose to hand-engineered simulators when (a) data is scarce, (b) the system has rare but catastrophic failure modes the world model hasn't seen, or (c) you're operating in a safety-critical domain where unmodelled extrapolation is unacceptable.
In practice, world models are most useful as a second simulator alongside the main one: train on the hand-engineered sim for the bulk of policy learning, then fine-tune or consistency-check against the world model before deployment. Pure world-model training remains high-risk for industrial deployments: the failure mode is silent incoherence rather than obvious divergence, which is harder to catch before production.
Your system has clean, dense data from real operations (process logs, sensor streams, trajectory archives), and you've at least considered whether a learned world model would close dynamics gaps the hand-engineered simulator can't. You're using the world model as a consistency check, not as a sole training environment.
Data is sparse, intermittent, or only available from a different system than the one you'll deploy on. The world model can only be trusted, not validated, which is a known failure pattern.
5. Wrap the deployed policy in safety bounds — always
Even after you've closed four of the five gap types, the deployed policy will face conditions it didn't encounter in training. Production is non-stationary in ways simulation is not: distribution shift, novel environments, instrument wear, mode changes.
The fix is to wrap the policy rather than perfect it. Three wraps matter:
- Bounded action projection: clip every action the policy produces to a safe envelope, defined by the dynamics of the real system. The wrapper enforces what the policy is allowed to do.
- Confidence-aware fallback: when the policy's uncertainty exceeds a threshold (e.g., the observation is far from any training state, or an ensemble disagrees by more than X%), fall back to a hand-coded safe controller — a PID loop, a rule-based policy, a stop-and-alarm condition.
- Monitor and rollback: monitor the deployed policy's key signals in real time. If the policy drifts off-distribution in a way that affects outcomes, automatically roll back to the previous (known-good) policy version. The rollback decision should be automatable, not require an engineer to wake up at 3am.
The energy case shows why: a grid-control policy that's slightly miscalibrated after a sensor drift can issue setpoints that exceed thermal limits on transmission lines. The safety wrapper is the part of the system that says "the policy wants to push 2.4 GW through this line; the limit is 2.2 GW; clip to 2.2 and alarm." Without it, a policy rewarded for throughput in simulation trades grid stability for a few percent of efficiency. With it, the worst case is bounded.
Your deployment plan includes action bounds, a fallback policy that's been validated independently of the learned one, and monitoring + rollback hooks — and these have been tested with deliberate fault injection before the policy went live.
You have some safety wrapper but it's been defined by the training team rather than by the operations team, hasn't been exercised under failure conditions, or only monitors the policy's outputs without rollback capability.
The reality-gap diagnostic
Before you commit to a sim-to-real engagement, run through this checklist. Each row corresponds to one of the five gap types. Not every green flag is required. Amber flags need a concrete mitigation in the plan; red flags mean you are not ready to deploy.
| Gap | Green (closing it) | Amber (mitigation planned) | Red (blocker) |
|---|---|---|---|
| Visual | DR covers relevant variation | DR for some axes only | Sim visuals don't match camera |
| Dynamics | System ID of actual hardware done | Tuned from spec sheet | First-principles model only |
| Sensor / noise | Real noise profile injected in sim | Noise modeled partially | Sensors assumed ideal |
| Contact / actuator | Hardware-in-the-loop validated | Bounded by actuator models | Treats commands as effects |
| Latency | Realistic timing in sim + wrapper | Approximate timing only | Sim assumes zero latency |
| Safety wrapper | Bounds, fallback, monitor + rollback tested | Partial wrapper, not fault-tested | Raw policy with monitoring only |
You cannot skip the real-system calibration step. There is no published technique that makes a simulator faithful enough to a specific physical system without measurements from that system. Teams that train purely in sim, because the real system is too expensive, too risky, or too slow to touch, are not doing transfer at all. The techniques below help when access is constrained, but they do not replace measurement.
What to do when you can't deploy to the real system for training
Many industrial systems are too expensive, too risky, or too slow to expose a learning agent to directly. A reinforcement learning policy can't grind through ten thousand episodes on a clinical infusion pump, a high-voltage transformer, or a five-axis CNC mill the way it can on a robot arm. The answer is a structured path from sim to reality that does not depend on open-ended real-world exploration.
Instrumented physical proxies
For robotics, this is the most common answer: build or buy a lab-grade version of the production hardware, run the policy there for hundreds to thousands of episodes, then transfer to the production system with a residual adaptation phase. The lab surrogate need not be identical, only share the dominant dynamics modes.
Hardware-in-the-loop simulation
For grid-scale energy systems and bioprocess control, hardware-in-the-loop (HIL) is the standard answer: the actual control hardware (PLCs, SCADA interfaces, sensor stacks) runs against a real-time simulator of the physical plant. The control software sees realistic timing, sensor noise, and protocol behavior while the policy's actions affect a virtual plant. This closes the latency and actuator gaps without touching the real system during training.
Human-in-the-loop trajectory collection
When the real system is too risky for autonomous exploration but too important to skip entirely, domain experts operating the system can produce demonstration trajectories that get the policy to a "safe enough" region of state space from which it can be deployed with bounded autonomy. This is imitation learning as a warm start, avoiding the early exploration phase on the real system.
Low-stakes shadow-mode runs
The most under-used technique. Deploy the policy in shadow mode, observing and logging without acting, for an initial period in which its recommendations are evaluated against a known-good baseline. The system remains under human or hand-coded control. The policy accumulates a real distribution of observations without any action risk. After validation, the policy is promoted to a monitored advisory role, then to limited autonomy, then to full deployment.
What the sim-to-real playbook actually looks like
Pulling the five principles together, an industrial sim-to-real engagement has a recognizable shape:
- Gap audit at week 2: identify which of the five gaps matter for your domain; rank them by likely impact on policy behavior; pick the two or three that get explicit mitigation resources.
- System identification on the actual deployment hardware: calibrate the simulator (or train a residual model) against measurements from the unit you'll deploy on. If a proxy is the only option, document the gap between proxy and target.
- DR plus at least one second transfer mechanism: residual models, real-system fine-tuning, or a learned world model serving as a consistency check. DR alone is rarely sufficient for industrial workloads.
- Hardware-in-the-loop validation where applicable: run the policy against the actual control stack, real timing, real protocols, before it ever sees the real plant.
- Safety wrapper designed by the operations team, not the training team: action bounds, fallback policy, monitor + rollback. Tested with deliberate fault injection. Deployed before the policy goes live, not after.
- Shadow-mode deployment period: weeks to months depending on stakes, with the policy providing advisory recommendations while human or hand-coded control stays in charge.
- Phased rollout to limited autonomy, then full autonomy: with monitoring gates at each step. Rollback at any time, by policy version, not by engineer availability.
None of this is a research program. For most engagements the gap audit and system ID take 3–6 weeks, the safety wrapper is built in parallel with the policy work, and the shadow-mode period runs 4–12 weeks depending on domain. From sim-only prototype to monitored production autonomy is usually 4–9 months when the engagement is scoped correctly at the start.
DataWorks runs targeted sim-to-real engagements: gap audit, system identification on your actual hardware, transfer-mechanism selection, and safety-wrapper design. The deliverable is a deployable policy.
- Reality-gap audit across visual, dynamics, sensor, latency, and contact axes
- System ID on the actual deployment hardware (not a spec sheet)
- Domain randomization + residual models / world models selection
- Hardware-in-the-loop validation against your real control stack
- Safety wrapper: action bounds, fallback policy, monitor + rollback
- Phased rollout engineering with shadow-mode gate