This piece draws on work completed as part of an MSc dissertation in partnership with Kainos, building a cross-modal disruption prediction pipeline using UK open transport feeds. The pipeline is publicly available at github.com/nehavadapally/master-dissertation-project.
The gap between data availability and usability
In March 2026, the Department for Transport published its Transport Data Action Plan, committing to resolve identifier mismatches that make transport data "laboursome to join up". I spent a few months attempting exactly that join, using the feeds that currently exist. What follows is an account of where it works, where it breaks and what closing that gap actually requires.
The feeds, taken at face value
Five datasets cover this problem and no single one solves it on its own.
National Highways publishes DATEX II, giving road closures as coordinates. GB Stations and CORPUS resolve the station coordinates against CRS, STANOX and TIPLOC into one station reference. That reference is the only table that can be joined to DATEX II through a haversine distance calculation. The join then splits again: Network Rail's TRUST builds a retrospective dataset through STANOX, while the Rail Delivery Group's Darwin builds a forward one through TIPLOC. The goal is to predict rail delay near unplanned closures, and chaining these five datasets together was the prerequisite for any signal at all.
The identifier problem
CORPUS is the critical constraint. It bridges the identifier systems that DATEX II, TRUST and Darwin each use independently, but its coverage is not complete and the gaps are not random. Major interchanges and principal routes are well represented, while smaller stations on freight-only and rural lines drop out. The result is a non-random subset of the national network before any modelling decision has been made, weighted towards high-volume stations in ways that cannot be corrected downstream. The Action Plan's Action 3 commits to reviewing naming conventions to address this. That work has not yet happened.
The granularity problem
Identifier alignment is the first obstacle. Granularity mismatch is the second, and it only surfaces once you start building the join.
The intuitive approach is to match each TRUST movement to the road closures active nearby at the same time. A single closure can overlap with dozens of passing services simultaneously, so a direct service-level join produces one row per movement per nearby closure: one movement with ten active closures nearby becomes ten records, each carrying the same delay value under slightly different closure attributes. Any classifier trained on this learns from duplicated observations presented as independent evidence. The two datasets diverge sharply as a result: the retrospective training set reached 93,749 rows while the equivalent Darwin timetable dataset for forward prediction reached 2,863,334 rows at the same granularity, a ratio of 1:30.5.
The fix is to reverse the join order, collapsing train movements to station-day level: one row per station per day summarising mean delay and volume. This choice settles more than the row count. It decides what the classifier is asked to predict — station-level risk on a given day rather than delay on any single service. Road closures are aggregated independently to the same unit, each closure exploded across every calendar day it was active before grouping by station and date. The two tables then join on a three-column key: station name, STANOX and planned date.
The reframing brings the retrospective dataset to 33,941 rows and the timetable dataset to 51,437, a ratio of 1:1.5. Equally important, 59% of retrospective rows carry at least one nearby closure and 41% do not, giving a classifier balanced exposure to both conditions. Within-day temporal resolution is lost in the process, but that is a tractable feature-signal problem, not a structural dead end.
The geometry problem
DATEX II encodes closure locations across three formats depending on event type, each requiring separate parsing logic, and none of them exposes a ready-made centroid. For a twelve-kilometre motorway closure, the mean of its polyline coordinate pairs can sit several kilometres from the nearest affected junction. That positional error flows directly into the distance features carrying most of the model's predictive weight, so what it learns is a spatial relationship distorted by the error rather than the one closures and stations actually have on the ground. The format was built for traffic management display; spatial analysis was never the intent.
What is revealed?
These are not gaps waiting for better governance. DATEX II was built for traffic management. TRUST was built for operational performance monitoring. CORPUS is a patch over identifier misalignment that accumulated as the network evolved. None were designed to speak to each other, and the 1:30.5 granularity ratio is not a data quality failure — it is a structural property of two systems measuring the same railway at different resolutions for different operational reasons.
Between a dataset being published and a dataset being usable sits a set of tasks.
- Build the identifier chain across systems that were never meant to share one.
- Choose a unit of analysis before the join happens rather than after it.
- Parse whatever location format each source uses.
- Document where coverage drops before a model has the chance to hide it.
None of these four tasks show up in a schema or a data catalogue. They only surface once someone tries to build the join, which is exactly why plans that stop at publication leave them out.
So, if the data is already there, what is the missing piece?
The missing piece is the engineering that turns available data into usable data — the kind a model, or a person, can rely on. Road and rail data happened to be the case here. The same four tasks wait behind most public datasets that look finished the moment they are published.