It Was Never the Pipeline

I have built a speech pipeline five times in two years. Audio comes in, words come out, somebody scores them. Each time I was sure the hard part would be the model. Each time it was something else, and each time a different something else. Putting the five next to each other is how I arrived at what I am building now.

One: the model is an ingredient

The first was for a call center, and it started on a speech-to-text API. Halfway through, transcription became a prompt to a general model, then a newer one, then a newer one again. Nothing downstream cared. The scoring did not change, the storage did not change, the people reading the results never noticed. I had built the whole thing around the one component that turned out to be easiest to swap.

That was the first conclusion, and I resisted it for a while. The model is an ingredient. You do not build a kitchen around one ingredient.

Two: silence is money

The second was only the front of the pipeline, the part that touches audio before anyone pays for a token. Cut silences longer than three seconds, at a threshold conservative enough to keep every real pause. Detect hold music, and only call it hold music after thirty seconds, because a false positive deletes a customer's words and a false negative costs cents. Downsample, mono, lossless.

Two things came out of that. The thresholds were the product, and they were sitting as constants in a file only I could touch. And the person who should tune them was not an engineer. A quality analyst knows what a long pause means on a collections call. I do not. The parameters needed an owner who was not me, and safe ranges so that owner could not break anything.

Three: nobody asked for a product

The third was for a health network: three jobs and almost nothing else. Fifty workers over the audio, fifty over the text, one process to consolidate. It was the smallest of the five and the one that shipped fastest, because I had stopped building a product and built only the recipe. Everything the first two had that this one lacked was surface. Screens, endpoints, auth. The recipe underneath was the same.

Four: the answer is the product

The fourth was field audio from a utility. Transcribe, summarise, extract fifteen attributes. What mattered was not the attributes. It was that every one came back with evidence and a confidence, so a supervisor could check the one they doubted instead of trusting or distrusting all fifteen at once. The shape of the answer decided whether anyone used the system. I had been treating output as the last step. It is the first decision.

By then I could also see the cost. More than one version had grown its own cost calculator, always late, always bolted on. Cost per minute was never a report. It was a constraint the recipe should have carried from the start.

Five: you cannot freeze a conversation

The fifth is realtime, a voice on the phone with a person. Every rule above breaks. You cannot cut silence, silence is how you know it is your turn. You cannot pick the slow accurate model, latency wins. You cannot retry, someone is waiting. And the first serious bug was not about the model at all. The voice lost its accent after an interruption, because the accent was defined in three places and none of them was in charge.

What you can freeze, before the call, is who owns what. Who owns the audio path and who owns the effects. Once that boundary was a contract instead of an assumption, the accent bug and its whole family went away.

What that adds up to

Put the five conclusions in a row and they describe one thing.

An application should not say "use this model, this endpoint, this prompt". It should say "I need this result, under these constraints": this budget, this latency, this privacy, this quality. Something turns that request into an exact recipe and freezes it. A specialised kitchen executes the recipe, and only that recipe. It hands back the result with a receipt: what was done, by which steps, at what cost, with what confidence.

The model is one ingredient the recipe names. The thresholds are parameters the recipe carries, with an owner who is not an engineer. Cost is a budget the recipe cannot exceed. The shape of the output is declared before anything runs. And each step says whether it may run twice.

I did not design that. I kept building the same thing until the parts that never changed had names. That is what I am building now.