Reward Shaping Without Reward Hacking: A Practitioner's Framework for Non-ML Operators

I have worked on reinforcement learning for over 25 years. Across those decades one failure mode has cost industrial projects more time than any other: a team writes a reward that is easy to compute, trains a policy on it, watches the metric go up, and gets a system that does something nobody wanted.

The industrial version looks different from the canonical research examples. Rather than a robot falling forward to maximize forward velocity, you get a bioreactor control loop held at the wrong pH setpoint for six weeks, because the reward discounted terminal yield and never noticed a 0.3-unit pH offset. Or a grasping policy that rejects 40% of the SKUs it was meant to pick, because "successful grasp" was defined by lift height and the policy learned to lift only what the simulator modelled. Or a grid-flexibility agent that pushes load into the most expensive window of the day, because the reward left peak-shaving violations unpenalized.

The cause is rarely a shortage of ML skill. It is that the reward gets designed as a thing to optimize rather than as a formal statement of the business objective, the framing I use in "Why Your RL Project Fails". It ranks third in the causal stack of industrial RL failures, behind sim-to-real gaps and distribution shift, which the prior two posts covered.

Six principles for designing reward functions that direct policy search without handing the agent a way to game the metric, with examples from biotech (bioprocess control and lab-to-clinical transfer), robotics (contact-rich manipulation and locomotion), and energy (grid flexibility and cooling control). They build on each other, and the post closes with a checklist to run before training.

1. Treat the reward function as a formal statement of business objective, not a proxy to optimize

The most important design move is to ask, before any training begins: under what policy would an agent maximize this reward while producing behavior we don't want? I call this a static analysis of the reward function. In consulting work across biotech, robotics, and energy it has caught more downstream problems than any algorithmic technique.

The framing matters because the easy-to-measure reward is almost never the one you want. Reward a robot for forward velocity and it learns to fall forward; reward a recommender for clicks and it surfaces outrageous content. A bioreactor policy rewarded for titer will spike glucose above the toxic threshold in the last 24 hours of a 14-day run, because the marginal titer in that window exceeds the long-run toxicity penalty. A grid-flexibility agent rewarded for renewable utilization will push load into the next-day peak window, where the marginal carbon cost of deferrable demand exceeds that of curtailed supply.

The static analysis is a discipline rather than a single technique. For each candidate reward, write down (a) the optimal policy you hope for, (b) the worst-case policy you can imagine, and (c) whether the reward discriminates between them. If an unwanted behavior scores as well as the behavior you want, the reward is the wrong shape.

2. Audit the reward for the Goodhart patterns: terminal Goodhart, gaming, and tight proxy

Goodhart's Law, "when a measure becomes a target, it ceases to be a good measure", was stated by Charles Goodhart in 1975 for monetary policy, and it sits at the heart of reward design. Later work in ML safety refined it into a vocabulary of three failure modes that map cleanly onto industrial reward design:

• Terminal Goodhart: optimizing the metric so tightly it loses its connection to the original objective (e.g., training on glucose feed-rate to maximize titer, ending up with a policy that meets the metric but produces cell lines that fail in scale-up).
• Gaming Goodhart: reaching states where the metric is high but in a way that doesn't actually achieve the goal (e.g., a grasping policy that maximizes "grasp successful" events by retracting before contact registration, scoring lift without lifting).
• Tight-proxy Goodhart: optimizing a proxy reward so hard that the gap between proxy and true objective becomes the failure mode (e.g., a cooling RL agent optimizing for kWh that ends up routing load onto less-efficient chillers to meet a courtyard temperature constraint).

All three surface again in sim-to-real transfer and distribution shift. The static analysis of Principle 1 is the diagnostic that catches them; this principle is about naming the pattern once you have.

3. Use potential-based reward shaping (Ng et al., 1999) for the additive case, not ad-hoc bonuses

If you have a base reward and want to encourage the policy to reach a goal state faster (a "shaped" reward), there's a celebrated 1999 result by Andrew Ng, Daishi Harada, and Stuart Russell that tells you how to add a shaping term without changing the optimal policy: choose the shaping term in the form γΦ(s′) − Φ(s), where Φ is any function from states to reals, γ is the discount factor, and s′ is the resulting state. This is called potential-based reward shaping, and it's one of the few "free lunch" results in RL.

The proof is short: potential-based shaping preserves the policy optimum. Non-potential shaping, such as a constant bonus, a step bonus firing at goal states, or a sparse reward shaped by a hand-coded heuristic, can introduce an optimum the base reward would have rejected. Occasionally that is deliberate, to bias the agent toward short trajectories. Usually it is an accident.

In practice, when you add a sub-reward for partial credit (a small reward for each timestep the agent keeps a bioreactor in spec, or each timestep a flex-contract window stays below the peak threshold), derive it from a potential Φ mapping the state to its closeness to goal rather than from a hand-tuned constant. The math is no harder, the simulation is easier to interpret, and the resulting policies are easier to audit when an operator asks why the policy did something.

QV-learning (the algorithm my research group helped develop, which biological brains use to navigate partially observable environments) provides one useful construction for the potential: at each state, value the prospective cumulative reward against a known terminal state — such as a safe-stop or a near-goal state. The bipartite-QV decomposition produces a potential function naturally.

4. Recompose the reward from sub-signals, not unit-weighted sums

A common error is to weight sub-objectives by their unit magnitudes. Combining −0.5×titer-penalty with −1.0×yield-penalty across different unit scales lets one term dominate, and the policy exploits whichever carries the largest coefficient. Recompose the reward from sub-signals on a single scale, typically a dimensionless 0–1 score, and do that recomposition only after deciding what the operator actually wants.

In biotech this means: instead of rewarding yield in grams-per-liter-times-reaction-time, score each batch as the projected margin against a static reference and penalize as the deviation from a target titer distribution. In robotics it means: instead of weighting contact-distance and lift-height by their natural units, map each to a "0 if out of spec, 1 if in spec" and sum. In energy, a multi-objective reward that includes curtailment cost, ancillary services revenue, and capacity-factor floor should be written as a Pareto composite after the cost is computed, not as a weighted sum of the underlying quantities.

The symptom is that a small step in one sub-signal moves the reward far more than the same step in another, and the policy trades between them discontinuously, sometimes accepting a 10% degradation in metric A for a 1% improvement in metric B. The cure is to pick a formulation with unit-consistent sub-signals, or pass the per-component rewards to a multi-objective optimizer and record the Pareto frontier.

5. Stress-test the reward on hand-crafted counterfactual trajectories before training the policy

The static analysis of the reward function from Principle 1 is best done as a checklist exercise before any training begins. But the same conceptual move can be performed empirically by writing counterfactual trajectories: a series of state-action sequences that step through plausible policy behaviors, with the reward evaluated for each. Badly-shaped rewards reveal themselves long before the RL training loop does, and at a fraction of the engineering cost.

The most productive counterfactuals are adversarial: they describe the worst-case policy a learning algorithm could find that still maximizes the reward. In biotech, a policy that delivers glucose boluses at end-of-batch so titer spikes just before the reward is computed. In robotics, one that lifts the easiest 30% of the SKUs and stops. In energy, one that exploits a regulatory threshold by routing load to a less-efficient but legally distinct asset. Each answers the same question: given this reward and the state space the model allows, what would a smart adversary build?

Once you have that picture, you can either (a) reshape the reward to remove the new optimum, (b) add a constraint that bounds the trajectory shape — using the safety wrapper from Principle 6 — or (c) document the failure mode and let it be a known attack surface.

6. Pair the reward with a safety envelope that doesn't depend on the reward

The final principle connects this reward-shaping discussion back to the prior two posts in the series: sim-to-real transfer and distribution shift in deployment. The most robust industrial RL pipelines I have seen wrap the policy in a safety envelope that constrains actions by the dynamics of the system rather than by the reward signal.

A reward-shaped envelope ("penalize this for being out of bounds") is still a reward, and a sufficiently clever policy can trade against it. A dynamics-shaped one ("never exceed 2.2 GW on this line") is a hard bound the reward cannot argue with. The same move appears in sim-to-real transfer as the action-projection wrapper for contact-rich grasping, and in distribution shift as the rollback-to-known-good safeguard. The reward drives learning; the dynamics drive what the policy is allowed to do.

For each industrial domain, the dynamics-side constraint is:

• Biotech: pH / dissolved-oxygen / temperature bounds that are hard floors/ceilings the reward cannot override. The cell line will not survive even a "near-optimal" reward-driving violation of the bioprocess envelope.
• Robotics: contact-force and joint-angle bounds that the policy's outputs are clipped against, with a PID fallback when the policy's value estimate disagrees with the actual state by more than a configured threshold.
• Energy: thermal-line, transformer, and frequency-response bounds the policy cannot exceed — even if the reward signals the policy to. The reward can be loose (it sets the optimization direction); the envelope is independent and tight.

The reward plus the envelope is the minimum viable system to train. With less, a failure the reward cannot capture will surface during sim-to-real transfer or after deployment.

The reward-audit matrix

Before training a policy, run this diagnostic to surface where the reward is exposed. Each row corresponds to one of the six principles. Not every green is required. Every amber needs a documented mitigation, and a red usually means the reward needs another design pass.

What "production-ready" reward design looks like

Pulling the six principles together, a production-grade reward design is recognizable in five ways:

• The reward is a formal statement of the business objective rather than an aggregation of measurable proxies. The team can articulate the worst-case policy before training begins and verify that the reward discriminates against it.
• The shaping is potential-based where additive terms are needed. The reason for any "near-optimality violation" is documented in writing, not by accident.
• The sub-signals are recomposed to a dimensionless scale before summing. The optimum is robust to units of measurement, and a domain expert can read the reward and trace it back to the business.
• The counterfactual adversarial review is run on hand-crafted trajectories and the worst-case policies are either blocked or documented as known attack surfaces.
• The safety envelope is dynamics-side. The reward sets the optimization direction; the envelope bounds the trajectory shape, and it is tested with deliberate fault injection before the policy is deployed.

This is a 6–10 week effort for most industrial engagements, depending on how mature the underlying problem formalization is. Across the biotech, robotics, and energy projects we work on, the reward audit is the throughput-limiting step, and where most of the delay between "we have an RL project" and "it is live" accumulates.