Simulation RL Environments

Simulation RL environments built from real records: AI agents plan a port, dispatch trains and coordinate airline operations through storm days.

Affiliation

Hugging Face

Published

October 6, 2026

PDF

Work in progress

This is an early, experimental series. The environments, numbers and conclusions will change as we go.

Update, 22 October 2026

PortSimEnv now has v2–v5, built by Edgar Arakelyan (Scale AI): a live port where the week unfolds watch by watch, with pilots and tugs, real weather and shipping lines that answer back. See the live port. Bhavish Pahwa built a second job, RailSimEnv, in What the community built. Now Edgar and @agutmanstein, a licensed pilot, have taken the same idea to airline operations with AirOpsEnv: three agents at the ops, crew and customer desks coordinate through a real storm day. Built at Scale AI using AgentEnv Framework, in collaboration with Hugging Face and FineEnvs.

Simulation RL environments

Most RL environments for language models are puzzles, code, maths or web tasks. A lot of valuable work looks different. Someone keeps a schedule running under real rules while things keep going wrong. I want agents that can do that work, which raises a general question. How do we build simulation environments, digital twins of real work, that are realistic enough to learn from?

Each part of this series takes one real job, rebuilds it from the records it leaves behind, and grades the agent against a proven optimum where tractable, or a disclosed best-known reference and lower bound when it is not.

PortSimEnv v1

PortSimEnv is a deterministic RL environment for berth planning at the Port of Barcelona. It is built from the 1,784 container-ship calls the port recorded in 2024 at its two container terminals, with the real ships, their sizes, berthing times and quay sections. The agent gets one quay, the ships that really called there in a 2024 week, and a week that has just gone wrong. It re-plans the week, deciding for every ship when it docks, where along the quay and with how many cranes.

A recorded eval run, replayed on the 3D twin: GLM-5.3 on week 6 at APM Terminals (17 ships, 9 cranes, two cranes down and two quay closures). It checks three drafts and submits a plan that matches the proven optimum, reward 1.0. The cursor only shows its tool calls. The agent itself sends structured plans.

Every plan plays out on a 3D twin of the real port, built from open data. The agent’s berthing hours, quay sections and crane counts turn into ships, tugs and cranes moving through the week.

The disruptions come from what happens in real ports. Red Sea diversions send ships in late and in bunches, gales stop ships of 300 m or more above 25 knots and every ship above 30, a crane goes down for two days, or an emergency needs a berth now.

Try it

This is the environment’s OpenEnv Space, embedded live. Pick a split and a task index, load the episode and plan it with the same three tools the agent gets. The Playground tab is OpenEnv’s MCP playground. The code is in 07-simulation-environments/portsim-v1 on GitHub.

The PortSimEnv environment, live
It opens on week 7 at APM Terminals (17 ships). Drag ships on the dock chart or type their hour, section and cranes, and the 3D quay follows. Check plan reports rule breaks and cost, 10 times per episode. Submit plan ends the episode with a single grade.

Why it matters

A ship’s containers only move once it is alongside under a crane, so the berth plan decides how long ships wait, how much fuel they burn at anchor and whether a whole service stays on schedule. Schedules rarely hold, so the plan keeps changing.

Barcelona alone handled 3.9 million TEU from 2,163 container-ship calls in 2024, and in 2025 it changed its rules to give berth priority to regular services during congestion. Rotterdam reports that its port-call app cut waiting by 20%.

The eval

Six models each ran the 50 held-out weeks once, with the same three tools, 12 turns and 32k output tokens per turn.

The eval at a glance
Each bar is one model's 50 eval weeks by outcome; the number is its mean reward. Hover for counts.

Watch a rollout

All 300 eval rollouts replay on the 3D quay in the eval Space, FineEnvs/PortSimEnv-Eval, with the plan at each tool call, the dock chart, the grade and the full transcript.

Rollouts on the 3D quay: five eval weeks, six models
Pick a week and a model. The line under the replay explains the reward.

How the reward works

The reward is deterministic, so the same plan always gets the same score. It is computed from the plan alone, without an LLM judge or sampling, once per episode when the agent calls submit_plan.

Reward against cost, with each model's plan
1.0 at the proven optimum, falling towards 0.2 as the plan gets costlier; plans that break a rule sit in the shaded band. Measuring the gap against the avoidable cost means a week full of unavoidable delay is not punished.

An independent checker re-verifies every plan. Across the 1,100 tasks the naive re-plan never scores above 0.44 and a greedy heuristic never above 0.82, because tasks they nearly solve were dropped when the dataset was built.

In the film’s run, GLM-5.3 checks three drafts that come back with 9, 5 and then 0 rule breaks. Its last edits bring NAGOYA EXPRESS down from eight cranes to its six-crane limit and move CMA CGM MONTREAL from hour 131 to 133. The plan it submits costs 87: 72 in weighted delay plus three berth changes at 5 each. That equals the proven optimum, so the gap is zero and the reward is 0.2 + 0.8 × e⁰ = 1.00.

Built on real Port of Barcelona data

Each task keeps a real week at one quay and the ships that really called, adds disruptions modelled on real events, and gets an answer key from CP-SAT. I drop the tasks a simple heuristic nearly solves, which leaves 1,050 training and 50 eval tasks. No week appears in both.

What is real and what is simulated. The records give us the berth plan the port actually ran, with the ships, their sizes, times and quay sections. They don’t include crane assignments, container manifests or time spent waiting at anchor. Workloads, disruptions and the physical handling are simulated, and the film replays an agent’s plan on that simulation.

From seed data to tasks

  1. Seed: the real calls of one quay for one to three weeks, with the berth plan the port actually executed.
  2. Fill the gaps: crane fleets, wind and traffic rules, and a crane rate of 28 moves per hour turn each call’s real time alongside into a workload.
  3. Disturb: inject the disruptions below, scaled by tier (standard, busy, storm, extreme).
  4. Solve: CP-SAT proves the optimal re-plan, the answer key; the naive and greedy re-plans set the floor.
  5. Filter: drop tasks a heuristic nearly solves and keep whole weeks out of training for the eval.
DisruptionWhat happensReal-world modelShare of the 1,100 tasks
Late shipsships arrive hours or days lateRed Sea diversions, half of ships late100%
Closuresquay sections closed for repairs or dredgingcrane-rail repair, fender repair, dredging100%
Crane outagescranes out of service for hours or daysbreakdowns, maintenance89%
Priority cargosome ships count ×3transshipment connections86%
Emergenciesa ship must dock by a deadlinea reefer ship losing power76%
Extra callsunscheduled ships need a berthad-hoc calls63%
Galesno ship moves; ≥300 m stop above 25 kn, all above 30the port’s traffic ordinance62%
Bunchingseveral ships arrive togetherdelayed services catching up46%
Diverted trafficships moved from the other terminalterminal outages15%

Data sources

DataSourceLicence
2024 ship calls: 1,784 at BEST and APM, with times, sections, portsPort of Barcelona open data via berth-allocation-problemsCC BY-SA 4.0
Crane fleets: 13 at BEST, 9 at APMBEST, APM Terminalspublished
Wind, traffic and berth rulesBOE-A-2023-6719, BOE-A-2025-5157, port-call procedurepublic
3D twin: coast, buildings, yards, cranesOpenStreetMapODbL 1.0
TerrainTerrain Tiles on AWSopen
Anchorage and routes (checked)Copernicus Sentinel-2open
Environment: the OpenEnv Space, with the editor and 3D view and an MCP playgroundFineEnvs/PortSimEnv
Eval: all 300 rollouts, replayed in 3DFineEnvs/PortSimEnv-Eval
Dataset: tasks (train, eval), the source calls, the eval rolloutsdatasets/FineEnvs/PortSimEnv
Bucket: 3D twin data and raw eval rolloutsbuckets/FineEnvs/PortSimEnv
CollectionSimulation RL Envs
Codegithub.com/adithya-s-k/FineEnvs: 07-simulation-environments/portsim-v1
DiscussionGitHub discussion #36

Contains data from the Port de Barcelona open data portal. The extracted data and the task pack built from it keep the source’s CC BY-SA 4.0 licence.

The live port: PortSimEnv v2–v5

Edgar Arakelyan at Scale AI took PortSimEnv and made the week happen in order. In v1 the agent sees the whole broken week up front and submits one plan. In his versions it starts on Monday at 00:00 knowing only what the port knows then, and the rest of the week reaches it the way it would reach a real planner. He built four versions on AgentEnv, Scale’s framework for agent environments, each adding another part of the real port. All four keep v1’s quay, weeks, rules checker and reward, so a plan means the same thing in every version. The code is in 07-simulation-environments/portsim-live.

A week in watches

At hour 0 the agent knows the published arrivals and the planned closures. Every other disruption reaches it as a message from whoever would send it: the ship, the harbour master, terminal ops or the line desk. Each message arrives 24 hours before the hour it is about, or 12 hours for an emergency. Messages come in 6-hour bulletins, and each bulletin with news is a watch, between 2 and 9 in a week.

One live week, watch by watch
Week 7 at APM Terminals, the week the v1 Space opens on. Step through the watches or press Play. Filled dots are news the agent has, hollow ones are still to come, and the shaded band is the next six hours, which are frozen.

At each watch the agent has four tools:

toolwhat it does
get_situation()the watch, its time, the week as known now, the confirmed windows and the ships without one
check_plan(plan)rule breaks and cost of a draft, merged over the confirmed windows; confirms nothing
confirm_berths(plan)sets or changes windows; each refusal gives the harbour master’s reason
advance()ends the watch; after the last one the rest of the week runs and is graded

v3: pilots and tugs

Every berthing and departure needs a pilot, and bigger ships need tugs. In v3 both come from port-wide pools that the quay shares with the rest of the port’s real 2024 traffic: ferries, tankers, car carriers, cruise ships and container ships at other quays. Edgar set the pools at 7 pilots and 8 tugs, calibrated against that traffic, and the tug company and the pilot station announce cuts.

Pilots and tugs over the same week
Hour by hour over 11 days: what the rest of the port takes, what is left for the quay's own ships, and the two announced cuts. Hover for any hour.

A plan that worked in v2 can now break. Two ships moving in the same hour may need more tugs than are free, especially inside a wind window, where every move takes one more tug.

v4: real wind, real forecasts

v1 and v2 simulate their gales. v4 uses real weather instead. Each week is moved into a real Barcelona weather week from 2023 to 2025, picked for its storms or for forecast gales that never came. At every watch Port Control sends the latest ECMWF forecast as it was actually issued, adjusted to the port’s anemometer. The agent plans on the forecast, and the grade uses the wind that really blew.

How a wind week grades one ship
Three cases for the same ship. What counts is the forecast in force at its last chance to change, and the wind that then blew.

The agent can only know the forecast, so a wind week scores 1.0 at its anchor: the cheaper of the plan in hindsight and a re-planner that follows each forecast as it comes. The 15 weeks are picked so that this re-planner reaches the anchor, while one that ignores the wind scores 0.9 or less. Reading the forecast is what scores.

v5: the shipping lines answer back

In v5 the schedule is only an estimate. Each ship has its agent, the firm the port’s 2024 records name for that call, running as an AI agent of its own. It reports an ETA 48 and 24 hours out and calls an hour before the ship arrives, and its ETAs are optimistic. The planner gets a fifth tool, message_agent, to ask a ship for its latest ETA or agree an arrival hour. Ships with an agreed arrival berth first come, first served. The port berths a late ship wherever there’s room, and every hour at anchor costs as much as an hour late.

One ship, three ways to plan it
Booking on the ETAs leaves the ship late, booking with a margin leaves it waiting at anchor, and agreeing an hour with its agent gets it in on time.

The ships’ agents have no model of their own. They answer from the environment, so the same request always gets the same answer and a week replays exactly.

The eval

Twelve models played 15 weeks of each version, 720 runs in all, graded by the same code.

PortSimEnv v2–v5: twelve models, 15 weeks per version
Pick a version. Bars are mean reward with the 95% interval, and vertical rules are the eval card's baselines. Open 'How this eval works' to see how the weeks were picked and what 1.0 means.

Watch a run

Every run replays in Edgar’s Space on the same 3D quay as v1. Here is week 7 at APM Terminals in each version.

Week 7 at APM Terminals, in all four versions
Pick a version and a model. Each replay is the published run: step through its watches or play the week, with the dock chart, the grade and the transcript below.

Every run is in the dataset earakely-scale/PortSimEnv-AgentEnv, and the agentenv-portsim plugin has the command line, the sweeps and the Hub tooling.

AirOpsEnv: three desks, one storm day

A flight can have an aircraft and still be unable to leave. Its crew might run out of legal duty time. Waiting a few minutes could save a passenger’s last connection, but delay everyone already on board and the aircraft’s next flight. Then another weather advisory changes the plan.

AirOpsEnv takes FineEnvs from ports to airline operations, in collaboration with Scale AI (@scale_ai) and Hugging Face. It was developed by Edgar Arakelyan (@earakely) with @agutmanstein, a licensed pilot, using Scale AI’s AgentEnv Framework.

The simulation is grounded in real flight records for New York JFK, Boston Logan and the wider JetBlue network, using US Bureau of Transportation Statistics data, FAA traffic-management advisories and NWS weather reports. In v3, three AI agents run the operations, crew and customer desks. They have different tools and must ask each other for changes. The environment executes the decisions and grades the resulting day.

AirOpsEnv v3: a storm day in one minute

Try the interactive replay · AgentEnv Framework · Environment code in FineEnvs · Source and reproducibility instructions · Pinned data and recorded runs

What the agents actually do

The ops controller changes holds, cancellations and arrival-slot choices. The crew scheduler assigns legal crews, calls reserves and moves crew members between stations. The customer desk identifies endangered connections and asks operations to hold or protect flights. A customer request does not itself change a departure: operations has to accept it. Passenger rebooking is automatic in the simulator; the agents do not directly rebook individual travellers.

Each watch reveals the information available at that point. The desks inspect the situation, check proposed changes, commit decisions and hand over to the next desk. New FAA advisories arrive as the virtual clock advances. Past events cannot be rewritten, and a run must pass the environment’s audit before it receives a score.

The film also follows a crew-agent recovery from the 08:00 watch. The agent confirms that a standby pilot will travel on B6 488 from Orlando to Worcester, then cover B6 487 back to Orlando. It messages ops through request R3 to preserve the positioning flights. That is why the desks need to communicate: cancelling a flight can also remove the crew needed for a later departure. This action comes from the recorded run; the crew roster itself is generated.

Three minutes, three connections

The film follows a recorded run with 637 flights and 81,279 simulated passenger trips. At the 08:00 watch, the customer desk projects that three passengers connecting from B6 126 would miss B6 2902, JFK–Buffalo, by three minutes. It asks ops to hold departure from 10:25 to 10:28. That costs roughly 381 passenger-minutes for the 127 passengers booked on the departing flight, so the answer depends on what happens to the rest of the plan.

Follow one recorded decision
Request R4, GPT-6.1 Sol, b6-20250708-0400-joint. Text is condensed from the recorded exchange. Times are Eastern Daylight Time.

Ops checks the downstream effects and accepts. In the executed simulation B6 2902 leaves at 10:28, carrying all 127 booked passengers, with no missed connections recorded on that departure. This is a useful example because the agent’s work is visible: identify a problem, ask another desk, check the trade-off, change the plan, then see the consequences.

Across the whole day the agents make 24 crew moves, hold 31 departures and cancel one flight. The run scores 0.795, with a total cost of 4.24 million passenger-minute equivalents versus 6.01 million for the standard crew baseline: 29.4% lower cost on this particular day. It still ends with delays and stranded passengers. A good score is not a claim that the day was painless.

What is real, and what is simulated?

Part of the environmentWhere it comes from
Flight schedules, tails and recorded delaysUS Bureau of Transportation Statistics on-time records
Aircraft characteristicsFAA registry and BTS Form 41 B-43
Ground stops and ground-delay programsFAA Command Center advisories, revealed as issued
WeatherNWS METARs, forecasts and discussions via Iowa Environmental Mesonet
Passenger trips and connectionsGenerated using BTS T-100 load data and DB1B/DB1C itinerary surveys; domestic itineraries
Crews, pairings and reservesGenerated from the schedule under the model’s duty and rest constraints
Airport geometry and imageryOpenStreetMap and USGS; gates, taxi routes, walking paths and the control room are display reconstructions

These are simulated airline operations grounded in public records. The passenger trips are not private airline itineraries, and the generated crew roster is not JetBlue’s actual roster. The 3D movement explains the result; the simulator’s audited flight, crew and passenger outcomes determine the score. The film’s network board follows recorded departure and arrival times along reconstructed paths; it is not an ADS-B track. Its opening globe uses NASA Blue Marble imagery, a 2004 satellite composite used as geographic context.

From one desk to a passenger-aware network

AirOpsEnv v1 asks one hub desk to recover a short storm window and compares its plan with a proven hindsight optimum. V2 expands to the whole network with operations and crew agents. V3 adds the customer desk and changes the objective to account for passengers reaching their final destinations, including rebooking, nights away and refund exposure, alongside aircraft and crew costs.

The v3 pack has 44 training dates, 24 evaluation dates and two calm controls. The evaluation dates cover separate summer and winter storm events at JFK and Boston. The published board runs six models with three desks on all 24 dates, plus a one-agent comparison for GPT-6.1 Sol: 168 scored runs in total.

AirOpsEnv v1–v3: every published result
Switch versions and metrics, inspect all model configurations and baselines, or explore individual runs. Includes 221 scored runs and eight failed or superseded attempts. Model means and spend were recomputed from the pinned records and match the published cards.

The explorer includes all three published evaluations:

All 221 scored runs are available in the run explorer and downloads. The complete attempt ledger also retains seven failed Terra attempts and the superseded v2 attempt; these are excluded from the published means. The registered v2 retry replaces the interrupted run on 9 July 2026. Each row links to the original record and a transcript. These versions change the task and objective, so their rewards should not be read as a progression on one unchanged benchmark.

The leading v3 averages are close: GPT-6 Astra scores 0.742 and GPT-6.1 Sol 0.721, with overlapping confidence intervals. Sol’s three-desk result is 0.044 above its one-agent result, but it uses about twice the compute. This comparison does not isolate the benefit of communication.

A reference is not always an optimum

For v2 and v3 the reference is the best plan found in hindsight, accompanied by a lower bound where available. Reward is 1.0 at or below that best-found plan; it falls towards 0.2 as cost rises. In v3, 24 of 168 runs beat the published reference and are flagged accordingly. A separate lower-bound reward is retained. The board is a benchmark of this simulator, not a measured improvement over an operating airline.

The recorded requests, tool calls and outcomes are published with the data, so the demo can be replayed and checked without making new model calls. That is the point of this series: real operational constraints, explicit decisions and an outcome that can be inspected.

Film music: “Felicity” by Scott Buckley, licensed under CC BY 4.0. Edited and mixed with original aviation sound effects.

What the community built

Within a week of part 1, the community was already building on it. Edgar Arakelyan built the live port, and Bhavish Pahwa took the same recipe to train dispatching. Edgar later developed AirOpsEnv with @agutmanstein, a licensed pilot, at Scale AI. Their environments are in the FineEnvs catalog, credited by name, each with its own eval.

RailSimEnv, by Bhavish Pahwa

In RailSimEnv the agent dispatches Arriva’s trains on the single-track Groningen–Leeuwarden line in the Netherlands after a real 2025 breakdown, collision, or signal or points failure. On a single track, trains in opposite directions can only pass where the line doubles, so one closed section backs up the whole line. The agent holds, short-turns and cancels trains so they cross safely, and every plan is graded against a CP-SAT-proven optimum. It is built from Rijden de Treinen’s open train archive and OpenStreetMap, and it runs on OpenEnv with a 3D replay.

On 45 hard eval tasks, as reported by the author:

ModelRewardLegal plansAt the optimum
Qwen3.8-2.4T-A95B0.55639 of 4512
DeepSeek V4.1 Flash0.53433 of 4512
Claude Sonnet 5.50.46745 of 454
GLM-5.3-Flash0.36024 of 456

An open model leads here. The one closed model keeps every plan legal but rarely finds the best one. The code is in bp-high/RLSimEnvs.

Add yours

The catalog takes environments whose code lives in FineEnvs and ones that stay in their authors’ own repos. Every entry has a manifest that credits its authors, a short card, and an eval card in a shared format. That shared format is what lets this article draw each environment’s eval the same way, like the PortSimEnv v2–v5 chart above. Here is how to add one.

Ongoing work

Most of the first roadmap is built now, and most of it by the community. A live port with news over time, pilots and tugs, weather history and ship-to-port talk all exist as PortSimEnv v2–v5. RailSimEnv brought train dispatching, and AirOpsEnv now adds airline operations with three desks, real storm records and simulated passenger connections. What is still open: post-training on these environments, more data (AIS tracks, terminal logs, other ports) and more jobs worth simulating from public records.

The question behind the series is still the same. We can build realistic environments from seed data, but how do we get them as close to the real world as possible? If you have an idea or an environment, add it to the catalog or start a thread in the GitHub discussion.