LLM function-calling as a structured-extraction tool

Most of the interesting LLM work I've done isn't chat. It's using a model as a parser: point it at unstructured noise, hand it a schema, and get back rows. The least glamorous framing of a language model—a very expensive, very flexible str → dict—is also the most useful one I've found. Here's the case that taught me that.

The setup: thousands of failures, no two alike

We had a Cypress end-to-end suite for a LIMS platform—thousands of browser tests. During a stretch where we were migrating the front end—Vue 2 to Vue 3, Cypress 9 to 10, a grid component upgraded underneath—a single CI run could throw a hundred or two failures. And the thing about end-to-end failures is that they're all phrased differently, because they fail for different reasons at different layers: an assertion timed out waiting for a selector, a component threw during render, a network stub didn't match, the test was just flaky.

The question that mattered wasn't "did it fail." The dashboard already told us that. The question was "why, and is this the migration or a real regression?"—and answering it meant a human reading stack traces for the better part of a day, every bad run. Eight to sixteen hours of senior-engineer attention spent squinting at tracebacks. That's the cost I was trying to kill.

What I tried first (the graveyard)

The instinct is to parse. A stack trace has structure—file paths, line numbers, error types—so you write regexes. I did. There's a file in that repo, mostly commented out now, that's a monument to it: patterns to pull the spec file and line out of a webpack-mangled URL, logic to grep the source for grid-component imports and guess whether a failure was migration-related. It worked for the traces that matched the patterns and fell apart on the ones that didn't, which is the eternal story of parsing adversarial text with regexes.

The second attempt was smarter and still wrong. I tokenized the error text—lemmatized it, dropped stopwords, counted terms—and fed the top tokens to a model, asking for a plain-English summary of what was going wrong across a spec file. That produced readable paragraphs. The trouble with readable paragraphs is that you can't query them. "How many failures this run were timing issues versus real regressions?" is not a question you can ask a pile of prose summaries. I'd moved the unstructured noise from one shape into another.

What I actually wanted was structured data—the same fields for every failure, so I could count, group, and filter. That's a different use of the model entirely.

Function-calling as a contract

The reframe: don't ask the model to describe the failure. Make it fill out a form.

Function-calling—tool use, structured outputs; the vocabulary has churned, the idea hasn't—lets you hand the model a schema and require that its response be a call to a function whose arguments conform to that schema. You define the shape you want; the model's only job is to populate it. The schema stops being documentation and becomes the actual contract at the API boundary. In a real sense the schema is the prompt—most of the instruction is carried by the field names and types, not by the prose around them.

I defined the failure schema with Pydantic and generated the function signature straight off the model classes. The core of it:

• type—the error class: AssertionError, TypeError, CypressError.
• cause—an enum, not free text: timing, selectors, environment, application. This is the field I actually wanted to group by.
• refs—the file / line / column / method frames pulled out of the trace.
• targets—the CSS selectors implicated in the failure (.selected-options .option-text and the like), which turned out to be gold for spotting which UI components the migration had broken.

Then I forced the call: hand the model the raw Cypress output for one failure, give it a single function—"report a test from this output"—whose parameters were that schema, and set the API to require that function. No "please respond in JSON." No parsing prose back into fields and praying. The structured object is what comes back, type-checked at the door.

Confidence as a number, not a boolean

The migration questions—is this an AG-Grid failure, a Vue 3 failure, a Cypress 10 failure—were the whole reason for the project, and my first instinct was to make them boolean flags. That was wrong, and catching why is the design decision I'm proudest of on this one.

A model asked "is this an AG-Grid issue?" and forced to answer true or false will answer even when the trace is genuinely ambiguous—and you've now baked a coin-flip into your data as if it were certain. So instead of a boolean I used a signed integer from −10 to +10: −10 is "confidently not," +10 is "confidently yes," 0 is "the trace doesn't say." It's a more honest representation of what the model actually knows, and—the practical payoff—it let me filter by confidence. Show me the failures the model is sure are AG-Grid. Show me the ambiguous ones for a human to eyeball. A binary flag throws that gradient away; keeping it is what made the output trustworthy enough to act on.

Treat the model like the flaky API it is

Function-calling guarantees the shape of the response. It does not guarantee the response arrives—you're still talking to a rate-limited network service that fails transiently. So the runner around it treated the model exactly like any other unreliable dependency: a parallel async processor with a token-aware rate limiter (staying under the requests-per-minute and tokens-per-minute ceilings), a retry queue with a bounded number of attempts, and per-request cost accounting. Failures went back on the queue; successes were appended to a JSONL file, one line per failure, each line carrying the original request, the structured classification, and the metadata.

That last detail matters more than it looks: I kept the raw trace right next to the structured extraction. When you're trusting a model to classify, you want to be able to pull up the input that produced any given row and check it. Structured output you can't trace back to its source is structured output you can't trust.

The economics were the quiet punchline. At the model I was using, classification ran to something like a fifth of a cent per failure—a full bad run of a couple hundred failures cost pennies and finished in minutes, against eight to sixteen hours of a person. The migration breakdown that used to be a day of reading became a GROUP BY.

Why this generalizes

None of this is really about tests. The pattern is: unstructured input on one side, a consumer that needs structured data on the other, and a translation in the middle that used to require either brittle parsing or a human. That shape is everywhere. I've since used the identical move to map thousands of free-text requirements onto a test-traceability matrix—an LLM parsing each requirement into structured form and judging every test-to-requirement pairing, with a human validating the calls. Different problem, same three moves—define the schema you wish the data already had, force the model to emit exactly that, and validate at the boundary.

The mental shift is to stop thinking of the model as something you converse with and start thinking of it as a typed adapter you drop between two systems that already know what they want. Chat is the demo. Extraction is the job. The most valuable thing a language model does, most days, is turn text you can't query into rows you can—and function-calling is how you make it do that reliably instead of hopefully.

I built this in 2023, on gpt-3.5-turbo-16k, before "structured outputs" was a checkbox in the API and before this was the standard advice it's since become. The technique has only gotten easier. The lesson hasn't changed: when you catch yourself writing a parser for the output of an intelligent system, consider handing the system the schema and letting it do the parsing—into a shape you defined, checked at the door.