Cloud based ITS

As volumes of traffic data grow exponentially, so the challenge of storing it in an organised and accessible way mount. In the UK, Transport for West Midlands has built a world-class data capability in-house, using smart funding methods to create a long-lasting legacy of usability.

Seven years ago, Transport for West Midlands (TfWM) in the UK had a problem familiar to many public sector organisations: lots of data, little control, and no real idea what good looked like. Multiple versions of the same map stored across various network drives. No governance. No catalogue. Just a small team doing their best in an unmanaged environment. 
Today, the organisation ingests 10 million records daily, operates sophisticated real-time dashboards refreshing every two minutes, and does it all for around €800 per month in cloud costs. It’s a transformation story that challenges assumptions about public sector digital capability – and it was built almost entirely in-house.

“We didn’t really know what we had, and we also weren’t really good at knowing what good looks like in terms of data management,” says Anna Watt, transport research lead at Transport for West Midlands, describing the pre-2018 situation. “The same data was stored multiple times, with a slightly different version of a map that might have a little tweak here, and might have been updated – that meant there was confusion between us.”
The transformation began with necessity. The 2022 Commonwealth Games created an imperative for sophisticated, real-time data management that the existing infrastructure simply couldn’t support.

“There was a big set of data that we needed to be able to ingest, manage and pipeline out for real-time decision making,” says Watt, who spoke at a presentation at Highways UK in Birmingham in October 2025. “There was a lot of learning that we went through in that process.”

Direct management

Rather than outsourcing the challenge, Transport for West Midlands partnered with AWS and Esri to build internal capability. The decision proved transformative – not just for the Commonwealth Games, but for the organisation’s entire approach to data. Now, the data pipeline requires multiple approvals before anything reaches everyday use. “It’s a controlled process, so nothing enters our environment unless we’ve properly controlled it,” says Watt.

The process begins when a team member identifies a data need. This is when data engineers assess the use case: How big is the dataset? How often will it be retrieved? What aggregations are needed? Based on these requirements, engineers select appropriate storage solutions from the organisation’s diverse tech stack and load a sample into the UAT (user acceptance testing) environment. Only after approval from both the user and data management does data move into production.

Cost-effective cloud architecture

The technical evolution has been deliberate and cost-conscious. From network drives, the organisation moved to on-premises Postgres databases, then into the cloud. In 2022, Transport for West Midlands adopted a serverless cloud architecture, selecting different storage methods optimised for different data types based on retrieval speed, volume and frequency of access.
The organisation is now exploring zero ETL (extract, transform, load) approaches, eliminating the need to copy large datasets repeatedly for processing, which saves time. They’re also implementing AWS SageMaker Unified Studio, an AI tool that simplifies code creation and provides federated data access across different storage solutions.

“It doesn’t matter which of those storage solutions your data is in – you can use one platform to access it. All you need to know is the schema, you don’t need anything else,” says Watt.

Behind the technical infrastructure stands a dramatically expanded team. From seven people in 2017 doing “various bits of analytics,” the data-insight team has grown to 26 specialists spanning data management, data engineering, product management, research, performance modelling and mapping.

TfWM decided to develop much of this new data-management capability in-house, which meant it was able to take various government grants that might have in the past have gone to external consultancies, and get greater value from them, creating lasting organisational capacity rather than temporary project deliverables.

“We’ve treated code as a capitalisable asset, and therefore we were able to use funding to grow the team,” says Watt. “Instead of going to a consultancy and saying, ‘Can you deliver this?’ and letting them walk away with all that knowledge, we got some help from AWS and Esri grew our internal capability.”

Managing output

The product management team plays a crucial role in ensuring the data capability delivers genuine value rather than just technical outputs.

 “Our product management team are really good at engaging with our users, trying to understand really what they’re trying to achieve with that data,” says Watt.

 “Not just blindly putting a whole load of data out there. Sometimes we’ll get requests to say, ‘I need this,’ and when you unpick it, they don’t really need that. What they really need is a tweak to an existing product or a copy of something else. We have to ask what question it is they’re trying to answer, and what decisions they are going to try to make from whatever solution we provide.”

For TfWM, data can assist in decisions in a wide range of different activities – from operational performance to strategic insights and monitoring. At the operational end sits the Regional Transport Coordination Centre (RTCC)’s Operations Dashboard for the West Midlands.
At the time of the opening of the RTCC, ahead of the Commonwealth Games, Cllr Ian Ward, leader of Birmingham City Council said: “By pooling our expertise and resources with partners we are now able to manage our roads and public transport on a much larger scale than ever before to make a real difference for travellers.”

The Operations Dashboard draws in major-roads data from National Highways’ Single View of The Network (SVN) (see page 44) and while it also runs off Esri’s ArcGIS, it differs from SVN by also including data from local roads, buses, railways and trams, making it the RTCC the UK’s first traffic control centre to bring together such complete multimodal transport information.
“Although we obviously use our model for a lot of business cases, what people really want to know is: what is happening on the network now, and how is our network performing now?” says Watt. “That’s one of the key use cases.”

Strategic shift

When it comes to strategic planning, the newly ordered data is hugely valuable in analysing and improving cycling infrastructure, public transport and the understanding of travel trends – only now the team is transitioning from data analytics towards data science capability.

“We’re moving away from spreadsheets into SQL (structure query language). We’re moving into starting to play with machine learning and AI so that we can upskill our team and produce more insight more quickly and more robustly,” says Watt.

The zero ETL and federated data approaches lower barriers for this transition.
 

 “You don’t need to know where everything’s stored,” says Watt. “

It makes it easier for people to transition from ‘I used to do everything in a spreadsheet, now I’m learning a bit of SQL, but it’s really difficult and I don’t really understand how this is structured,’ because you don’t need to know that–you can use SageMaker Studio to enable you to learn more quickly with fewer barriers.”

The organisation anticipates reduced reliance on traditional dashboards as large language models enable natural language queries. 
 

“Some groups might find that particularly useful,” says Watt. “

I’d imagine our policy partners might find that an easier way to engage and get insight out of our data rather than ‘I need to remember to go to that dashboard and click this and put that filter on.’
 

“But as people move into data science, they need to be a bit more literate around the cost of the queries that they run. There’s a little bit of nervousness about letting everybody loose and being able to run whatever queries they want without understanding the computational costs.”
The journey from network drive chaos to sophisticated cloud infrastructure took seven years, cost less per month than many organisations spend on coffee, and happened largely without the army of consultants that conventional wisdom suggests would be necessary. TfWM didn’t wait for perfect conditions or unlimited budgets – they started with the Commonwealth Games deadline, partnered strategically and built incrementally.

The implementation for the Games was so successful that Anne Shaw, executive eirector of TfWM, was been honoured with an OBE (Order of the British Empire) in recognition of her role in spearheading the successful transport planning and operations. “It’s a great honour to be recognised for our contribution to the success of Birmingham 2022,” she said. “This achievement is not just for me but also for the fantastic team of talented individuals here at TfWM, as well as the wider transport community.”

What makes the transformation particularly significant isn’t the technology stack or the cost efficiency, though both are impressive. It’s the organisational model: treating code as a capitalisable asset, using grant funding to build permanent capability rather than temporary deliverables, and developing in-house expertise that can evolve with changing needs; transforming data management from an IT project into genuine organisational intelligence.
“Everybody’s excited about the journey that we’re going on,” Watt concludes –¬a sentiment that applies as much to what’s been achieved as it does to what’s still to come.
[main copy ends]

 “We’ve treated code as a capitalisable asset, and therefore we were able to use funding to grow the team” Anna Watt, transport research lead, Transport for West Midlands"

“By pooling our expertise and resources with partners we are now able to manage our roads and public transport on a much larger scale than ever before to make a real difference for travellers” Cllr Ian Ward, leader, Birmingham City Council"

Lessons for the sector
The Transport for West Midlands story offers several lessons for other transport organisations contemplating similar transformations:
■ Build internal capability rather than relying entirely on consultancies. The knowledge stays when projects end, and the team can continuously evolve solutions.
■ Use grant funding strategically to capitalise digital assets and grow teams, rather than only funding external delivery.
■ Implement rigorous governance from the start – the discipline prevents future chaos and builds confidence in data quality.
■ Cost-engineer cloud infrastructure thoughtfully. Public sector budgets don’t permit waste, but properly structured cloud services can deliver remarkable value.
■ Focus product management on user needs and decisions, not just technical delivery.
■ Create clear pathways from operational through strategic data use, ensuring investments serve multiple purposes.

This article was first published in the 2026 edition of Intertraffic World Magazine.