Building an AI Race Engineer

On April 13, 2026, I walked into an in-person fireside chat at Cisco's Santana Row office in San Jose. McLaren Racing CEO Zak Brown was sitting down with Cisco EVP and Chief People, Policy & Purpose Officer Francine Katsoudas to talk about life in the fast lane.

I went in as a big McLaren fan.

I walked out with an engineering problem I could not stop thinking about.

One conversation, two very memorable moments, and the start of the AI Race Engineer idea.

Zak was explaining what happens around team radio during a Formula 1 race. There is a constant stream of communication, but not every message can be taken at face value. Some messages are purely operational. Some reveal intent. Others may be deliberately framed to influence what rival teams think is about to happen.

For the team hearing it, one message can open an entire decision tree:

• Is it real?
• Why was it said now?
• How will the other teams interpret it?
• What might they do next?
• How should we respond to each possibility?

All of that has to happen before the moment is gone.

That was the spark.

The hard problem was not simply the amount of data. It was decision latency: how quickly can you turn many imperfect signals into useful, explainable guidance?

The Idea I Could Not Leave Alone

A modern race team has telemetry, timing, weather, radio, video and human observations arriving at once. When another stream needs attention, the natural answer is often to add another specialist. But human attention does not scale forever. Every additional person also adds cost and another handoff in a sport where budgets and milliseconds both matter.

So I started asking a different question:

What if an AI agent could follow those streams together, investigate what changed and give the team a recommendation while there was still time to act?

I was not interested in replacing the race engineer. I wanted to give the engineer another set of eyes: one that could keep up with the data, attach evidence to its recommendation and leave the final call with the human.

That question became our AI Race Engineer MVP.

A Game Became Our Proving Ground

We built the MVP on EA SPORTS F1 25. The game gave us a safe, controllable environment, real drivers behind the wheel and a live UDP telemetry feed. We could create race conditions without waiting for an actual race weekend or touching real team data.

We did not try to solve audio, video, weather and telemetry all at once. We started with telemetry because it was structured, fast and measurable. If we could prove the data-to-decision loop there, we could add the other signals later.

The pitch sounded simple: collect the data, give it to an agent and return useful coaching in real time.

Then we started building.

The first problem we had to solve was not AI. It was plumbing.

We needed to get every useful signal out of the game at maximum speed. UDP is fast, but it does not guarantee that every packet will arrive. We wrote a tiny collector that decoded the game's telemetry packets, transformed them into JSON and sent them over TCP so the downstream pipeline received a reliable, usable stream.

That little collector was not the glamorous part of the demo. It was one of the most important parts. If the data arrives late, out of order or not at all, the smartest model in the world cannot help you.

We also kept it deliberately small. In a real-time system, the collector cannot become the slowest car on the grid.

If You Cannot Replay It, You Cannot Improve It

The second challenge was making the data reusable.

We wanted many people to drive under their own names. We also needed to record sessions, normalize and relabel the data, and replay the same corner, incident or lap whenever we wanted.

That replay layer changed the project. We could send an identical event through the pipeline again, compare two prompts or model configurations and see whether a change actually improved the answer. We could build examples from different driving styles instead of tuning the experience around one driver and one clean lap.

This was the point where the MVP stopped being a fun one-off demo and started becoming an AI engineering system.

Live data made it exciting. Repeatable data made it useful.

Putting the AI Close to the Data

Challenge three was the AI stack. This is also where the MVP became a real One Cisco team effort.

Andrey Tsediakov brought the Cisco AI POD expertise. Michael Szebenyi and Bartosz Jedrecki brought the Splunk engineering depth. Together, we worked through the model, infrastructure and integration choices.

Splunk became the data and context layer. The MCP Server for Splunk gave the agent a controlled interface to the telemetry and tools it needed. The Splunk AI Toolkit helped us work with data and models inside the Splunk platform.

Cisco AI PODs gave us the enterprise AI infrastructure: Cisco compute and networking, accelerated by NVIDIA GPUs and built to support training, fine-tuning and high-throughput inference. We deployed NVIDIA Nemotron Nano 12B v2 locally for the agent's reasoning and recommendations.

The flow looked roughly like this:

Race telemetry > Splunk > MCP tools and context > NVIDIA Nemotron Nano 12B v2 > recommendation

^{One continuous loop: The system collects, contextualizes and analyzes live race telemetry, returns an evidence-backed recommendation for human review, and uses Splunk Observability and Splunk Agent Observability across the complete path.}

The model mattered, of course. But the full loop mattered more. A brilliant answer delivered ten seconds too late is still the wrong answer.

Our target was a useful response within two to three seconds of a corner or race event.

Observing the Observer

The fourth challenge was one every AI team eventually meets: how do you know the agent itself is healthy?

Asking only whether the model returned an answer is like monitoring a race by asking whether the car crossed the finish line. It misses everything that explains the result.

We wanted to see the full path: end-to-end latency, where the time was spent, GPU utilization, queue depth, errors, tool calls, token consumption and whether the answer was grounded in the race data.

This is where Splunk felt completely at home.

Splunk Observability and our dashboards connected the experience in the simulator to the application and infrastructure underneath it. Following Cisco's acquisition of Galileo, Splunk Agent Observability gave us the large language model (LLM) observability and tokenomics insights we needed.

We could trace the agent from request to response, see latency and errors beside model quality, follow tool calls and watch token consumption by interaction and workflow. That mattered because an agent can be technically available while still giving a slow, expensive or poorly grounded answer.

The language model stopped being an exciting black box. It became something we could operate: visible, measurable and improvable.

In other words, the AI became another production system with telemetry. That is exactly what it should be.

Black Hat Was the Reality Check

For the real-world test, we chose Black Hat USA 2026 in Las Vegas.

A conference floor is where beautiful architecture diagrams go to meet reality. The network is unpredictable. The environment is noisy. People do things you never tested. And the demo still needs to work, over and over again.

^{The race in one view: The Splunk dashboard brings the leaderboard, live car telemetry, lap timing, track position, weather and data-pipeline health together in real time.}

More than 100 people per day got behind the wheel. The system had to collect the live telemetry, understand the event and respond within two to three seconds of a corner or incident.

It delivered.

The audience reaction is the part I will remember. People wanted to know how we built it. Then came the question that changed the conversation:

“Wait, this can run on premises?”

Cloud, On premises, Air gapped or Hybrid. The architecture supports all four deployment models. It can run fully on premises, with no public cloud service required. For organizations with strict data-sovereignty or connectivity requirements, it can also operate in an air-gapped environment. The game, collector, Splunk platform, MCP connection, model, agent observability and GPU infrastructure can all stay inside the organization's boundary.

For the on-premises or air-gapped options, that gives the organization direct control over its data, model, performance and operating environment. It also means that unreliable conference internet does not sit in the critical path between a race event and the agent's response.

At Black Hat, that was not an architecture footnote. It was part of why the demo worked.

A Blueprint That Goes Beyond Racing

The AI Race Engineer is a racing experience, but the engineering pattern applies anywhere that an AI agent must make a useful recommendation from fast-moving operational data.

• The reusable blueprint is straightforward:
• Define the decision and its time budget before choosing the model.
• Build a small, reliable path from the source data into a format the rest of the system can use.
• Record and replay real scenarios so that prompts, models and infrastructure can be compared against identical inputs.
• Give the agent controlled access to trusted context and tools.
• Place the model close to the data when latency, privacy or resilience demands it.
• Observe the complete agent workflow, including quality, infrastructure, latency and token economics.
• Keep a human responsible for the final decision.

The pit wall does not need more AI theater. It needs an answer that arrives in time, shows its work and earns trust.

The Next Lap

Now we are thinking about the signals a simulator cannot fully capture.

Biometrics such as heart rate could show when a driver is under stress or cognitive load. Video could add racing line, traffic and visual context. Input from real racers could help us understand the gap between what the telemetry reports and what the driver actually feels.

The car can tell us what happened. The driver may be the only one who can tell us why.

That is why the goal is still not to replace human expertise. It is to give experts better context, faster, with evidence they can inspect and a system they can trust.

And who knows? Maybe one day we can contribute something useful back to our incredible McLaren Racing partner, all because its CEO answered one small question from a very enthusiastic fan during a fireside chat.

I love that this project started that way.

Sometimes innovation starts with a strategy deck. Sometimes it starts with a sentence that stays in your head until you build something.

For me, this one became an AI Race Engineer.

The lesson goes well beyond racing: AI becomes useful when the data arrives in time, the context can be trusted, the system can be observed and a human can act on the result.

See it. Measure it. Replay it. Improve it.

That is a very Splunky way to build.