Keep the cases
Whoever writes the test owns the agent
Labeling money is leaving the labs. Write ten cases in-house before a vendor scores your work.
The model you rent is not the asset. The asset is the short list of what good looks like when a real ticket, quote, or exception hits your floor. Whoever writes that list owns the agent. If a labeling vendor writes it, you are renting your own judgment back to yourself.
That is not a research-lab problem. A team of twenty already pays someone to decide whether a reply was allowed to send. This week the work is to put that decision in a doc you keep, before a vendor files it in a queue you cannot take with you.
The renewal sitting in finance
Open the contracts that come due in the next ninety days. Anything that labels data, scores outputs, or promises to improve a bot's quality is in scope. You do not need a giant legal bench to ask a single ownership question: when this work is done, who holds the training data, the labels, the evals, and any model tuned on your examples.
Alex Lieberman (@businessbarista), who built Morning Brew and now sits with operators buying this kind of work, talked to one of the large labeling shops. They told him their revenue will tilt toward big enterprises, not labs, in a few years. They also said every company will want to own its intelligence, and that owning it does not require an open source model.
Read that as a sales map, not a compliment. They are walking off campus and into companies that look more like a shop than like a research group. What they will offer is help writing and running the tests. What you must not hand over is the tests.
@businessbaristaA company's evals will become their main proprietary IP given the improvement in agent performance after properly setting up & running internal eval environments
Alex Lieberman (@businessbarista)
Lieberman is passing along the vendor's own claim. Use it as a prompt for your renewal, not as a number you have to believe.
Coding agents were the easy part
If your engineers already let an assistant draft a patch, that was the cheap graduation. The expensive one is the work that is not code. A refund that cannot auto-approve. A quote that cannot ship the wrong SKU. A ticket that still needs a named closer.
The same labeling contact told Lieberman that most enterprises have not moved past coding agents. The reason they gave was not the model. It was the missing eval setup that would make a non-engineering agent hold a standard. You already know that gap. The demo looks sharp. The live queue still needs a person to catch the miss.
You do not close that gap by buying a thicker labeling package. You close it by scoring last month's real work, in-house, with the person who already knows the miss when they see it.
Ten cases from last month
The ops lead names one live workflow by Monday. Ticket replies, quote drafts, or invoice exceptions. Sit with the person who already closes that work. Do not invent a future process.
- Pull ten real examples from the last thirty days
- For each, write the input, what send looks like, and what fail looks like in one sentence
- Put a named closer on the doc, the same person who would have caught the miss last month
- Score all ten before Wednesday. Pass or fail. No maybe.
The domain expert writes the cases. The closer scores them. The owner does not write the cases. The owner decides whether any customer text in those cases is allowed to leave the building. That last call stays human even when a vendor offers to host the environment for you.
If you cannot score the ten without calling the vendor, you do not own the intelligence. You are renting a grade.
When cases fail, do not send the misses to a labeling queue. The closer sits with the expert and rewrites the fail line until a new hire could apply it. That rewrite is the IP. It is also the only way a non-engineering agent gets better without you paying someone else to remember how your shop works.
Who holds the pen
The handoff is people, not a platform. Ops names the work. The expert writes. The closer scores. The owner draws the data line. Finance waits on names.
Figure 1
Finance does not write cases. Finance holds the invoice until the one-pager has names in the ownership cells. Ops does not sign. The owner does.
What you can actually count
Fill this on Friday. Empty cells today are honest. Invented percentages are not.
| Measure | Count |
|---|---|
| Eval cases a named closer can score without a vendor | |
| Open vendor contracts that name who owns labels and evals |
Zero on the first row means you still do not have a score. That is a finding. Zero on the second row means the next invoice waits. Ops and finance can fill both cells in a fifteen minute sit-down. If the closer cannot name the number of cases they scored, the cell stays empty. Do not round it into a story.
Write the ownership page before Friday
One page. Same folder as the ten cases. Not a deck.
- Name the workflow and the closer
- State who owns training data, labels, evals, and any model tuned on your examples
- State what customer data may not leave your boundary
- Send it to finance before the next labeling or quality invoice is paid
Put the page next to the cases so a new hire could score next month without a vendor login. You do not need a new stack to do this. You need ten examples and four ownership lines.
You can run this on a shared doc you already use.
Sources
If one workflow is eating your week, tell me which one. I'll tell you if it's worth deploying — and if it isn't.
Message Mikko on LinkedIn