The Curb Cut for AI Agents: The Metadata Layer Accessibility Is Missing

The ramp you build for one user is the ramp the next one rolls up.

Around 2001 I helped a neighbor named Oscar set up his first computer. He lived a few houses up the hill from us, and he'd been blind since childhood. He knew me through my father — also an Oscar — the two of them were shortwave radio aficionados, which was a kind of internet before the internet. He'd heard I was good with computers, and he asked for a hand.

The machine came with JAWS, a screen reader, on a stack of floppies he'd gotten through an association he worked with. Oscar already knew how to type — he'd learned on a typewriter — so once I showed him how to turn the computer on and start the screen reader, he was off, moving through the system with Tab and key combinations. I went back most days that week. By the end he was opening folders, writing notes, finding his way around on his own.

It was slow, and the reason it was slow has stuck with me for twenty years. A screen reader reads the whole screen. To get anywhere you listen, hold it all in your head, and assemble the path to the one thing you actually want — every single time. The computer was describing itself to Oscar in full, and telling him almost nothing about what mattered or how to reach it. That work was left entirely to him.

Then one day he asked if we could get the thing onto the internet. We found a modem, dialed up, and he read the news. He told me he hadn't read the news himself in thirty or forty years — this was the first time he'd done it on his own. It was a genuinely moving thing to be in the room for. I visited again a few years later and he was still at it: news, email, the lot. I've never forgotten it, and it's why the mechanics of accessibility have fascinated me ever since.

Here's what I didn't appreciate at the time. The interface Oscar was navigating with Tab and patience is the same interface I now point AI agents at.

There's a term from urban design worth stealing: the curb-cut effect. Cities cut ramps into sidewalk corners for wheelchair users, and then discovered everyone with wheels used them — strollers, delivery carts, luggage, cyclists. A feature built for one constituency quietly became infrastructure for many. The accessibility tree is a curb cut. It was built so a screen reader could describe an application to a blind user — every button, its label, its state, read aloud on demand. Nobody designing it was thinking about AI agents. And yet, when you want an agent to operate software reliably — not by staring at a screenshot and guessing where the button probably is, but by knowing — that same tree is the cleanest source of truth on the machine. The ramp built for Oscar is exactly the ramp the agent rolls up.

I've spent a while now building on that idea, most recently in a little project called Telekinesis that drives desktop apps through the accessibility API instead of the mouse. It works. It's fast, it's pixel-perfect where the tree is honest, and the pointer never moves. But living with it surfaced a wall — the same wall a blind user hits — and I think the fix is small, cheap, and good for both of us.

The tree is legible, and thin

Walk an app's accessibility tree and it hands you real facts:

role = grid,      name = "Invoices"
role = combobox,  name = "Status"

That is genuinely useful. It's the difference between "there's probably something clickable around x=420" and "this is a grid, and it is called Invoices." But look at what it doesn't say. It doesn't tell you the grid can be filtered. It doesn't tell you where the filter surface is. It doesn't tell you that the Status control is the way to narrow the grid to unpaid invoices, or how to operate it without the dropdown collapsing before your selection registers.

An agent standing in front of that tree still has to infer four things: which vendor's control library built the component, which actions it actually supports, where the surface for each action lives, and how to execute it correctly. That's a lot of guessing on top of a system that was supposed to end the guessing.

Here's the part that matters: a blind user hits the same wall. A screen reader that reads "grid, Invoices; combobox, Status" has told the person what's on screen, not what they can do with it or how to do it. The metadata is descriptive, not operational. This is the wall Oscar climbed every time he listened to a full screen and assembled the path in his head — the machine described itself completely and told him nothing about what to reach for. The gap is identical for both constituencies, which is the first clue that one fix serves both.

A thin manifest, not a rewrite

The proposal is not "rebuild your UI for agents." It's to add a small machine-readable layer on top of the accessibility tree — a manifest — that vendors or app developers publish. A DevExpress grid might expose:

{
  "id": "invoice_grid",
  "vendor": "devexpress",
  "component": "dxDataGrid",
  "semantic": "invoice_list",
  "actions": ["filter_column", "sort_column", "select_row", "open_row"]
}

And for each action, where it lives and — this is the part people skip — how to work it:

{
  "action": "filter_column",
  "target": "invoice_grid",
  "column": "Status",
  "surface": "filter_row",
  "selector": "[data-agent-action='invoice_grid.filter.status']",
  "accessibleName": "Filter Status column",
  "bounds": { "x": 420, "y": 310, "width": 140, "height": 32 },
  "executor": {
    "strategy": "focus_keyboard_select",
    "steps": ["click selector", "ArrowDown to option", "Enter"]
  }
}

Now the tree isn't just legible. It's operational — for whoever is reading it.

The model picks the intent; something dumb does the clicking

The instinct is to let the model output coordinates or keystrokes. Don't. The model's job is to choose the semantic intent, and nothing else:

{ "action": "filter_column", "target": "invoice_grid", "column": "Status", "value": "Pending" }

That's the whole output. A deterministic executor then translates it into real UI operations against whichever platform the app lives on — Playwright for web, Microsoft UI Automation on Windows, the Apple Accessibility API on macOS, AT-SPI on Linux. The model never touches a pixel.

This split buys three things at once. It's faster, because there's far less visual reasoning per step. It's safer, because a validator sits between intent and execution and rejects anything that isn't a declared capability — the model literally cannot invent an action the control never offered. And it means a small local model can do the job, because the metadata has removed the ambiguity that used to require a frontier model's world knowledge. The pipeline is boring on purpose: read the tree, read the manifest, normalize into semantic controls, let a small model choose one strict JSON action, validate it, execute it, observe the result, log the trace. Boring is what reliable looks like.

Control packs

Bundle those manifests per UI library and you get a control pack: a portable description of how to recognize a vendor's controls, what semantic type each represents, what actions they support, where the surfaces are, how to execute them, and how to confirm success. Write the DevExpress pack once and every DevExpress app in the enterprise becomes agent-operable. Then Telerik/Kendo, Syncfusion, SAP GUI, WinForms and WPF, Electron — and, crucially, the internal proprietary apps that RPA teams have been holding together with screen-scraping and prayer. A control pack is a wrapper that makes an opaque interface legible without touching its source.

The DevExtreme experiment

I wanted to know if this was real or just a nice diagram, so I built a small MVP on the web with DevExtreme React components — a DataGrid, a SelectBox, a DateBox, and a TabPanel. On each control I added two things: ordinary aria-labels for the browser accessibility tree, and data-agent-* attributes for the agent metadata.

data-agent-id="invoice_grid"
data-agent-vendor="devexpress"
data-agent-component="dxDataGrid"
data-agent-semantic="invoice_list"
data-agent-actions="filter_column,sort_column,select_row,open_row"
aria-label="Invoices grid. Filterable and sortable invoice list."

Then I gave a deterministic executor one instruction: filter to Pending invoices. It read the declared action location for the Status filter, drove it through the keyboard behavior the manifest specified, and the grid narrowed to Pending. First try. The whole thing — the controls, the metadata, the action manifest, a replay evaluator, and a trace view of what the agent did — is on GitHub if you want to poke at it.

The lesson came from the failures before that. It is not enough to know where an action is; the manifest has to describe how to perform it. DevExtreme dropdowns render their options in a hidden overlay that isn't a child of the control you're looking at — so the naive "find the text and click it" fails, because the text isn't where the tree says the control is. A manifest that only carries a bounding box is a manifest that breaks. A reliable one carries the whole recipe: the semantic action, the target control, the action location, an accessible name or selector, a fallback bounding box, an executor strategy, and a validation rule. Location plus method plus proof.

Why this is the accessibility story, not a detour

Look again at that aria-label: "Invoices grid. Filterable and sortable invoice list." I wrote it for the agent. But it's also a strictly better thing for a screen reader to announce than "grid, Invoices" — it tells the person not just what the control is but what they can do with it. That's a label that would have saved Oscar a few of the paths he had to hold in his head. The discipline of describing a control well enough for an agent to operate it is the same discipline that makes it legible to a human who can't see it. Most enterprise apps ship thin, perfunctory accessibility metadata — the legally-required minimum and not a word more. Building for agents gives teams a concrete, testable, self-interested reason to finally fill it in. That's the curb-cut effect running in reverse: a ramp poured for the newcomer turns out to widen the one the original user was already on.

The opportunity

There's a product shaped like this, and I've been calling it Telekinesis Control Studio — a tool where a developer or an enterprise automation team can inspect a running app's accessibility tree, add or verify the agent metadata, label controls, record workflows, generate a control pack, run evaluations against it, and eventually distill a small local UI-policy model that operates that app without a cloud round-trip. Vendors ship control packs for their own libraries; enterprises author packs for the proprietary software no vendor will ever touch.

None of this requires a breakthrough. It's a thin metadata layer, a deterministic executor per platform, a validator, and the honesty to describe not just where a control is but how it works. It sits exactly in the seam between three worlds that have been circling each other for years — accessibility, RPA, and local AI agents — and it lets a small, cheap model do reliably what we keep asking an expensive one to do by guesswork. The accessibility tree already cut the curb. This is the paving that lets everyone use it.