Most agent pilots go live the same way. Someone builds a demo, it looks good on five hand-picked examples, and the team flips it on for real work. Two weeks later there is a cleanup project.
There is a cheaper first step. Run the agent in shadow mode.
What shadow mode is
The agent gets the same inputs the person doing the job gets today: the same inbox, the same tickets, the same spreadsheet. It does the work and writes its answer down. But the answer goes to a log. Nobody acts on it. The customer never sees it, the ledger never changes, and the CRM stays exactly as the person left it.
Then, every day, someone compares the two. What the person did, next to what the agent would have done.
That is the whole idea. It is not new. Teams have run new systems beside old ones for a long time before switching over. It just gets skipped with AI because the demo looked finished.
What the comparison tells you
A side-by-side log on your real work answers questions a demo cannot.
- Where it agrees with your best person. These cases are candidates for handoff. You can stop worrying about them.
- Where it disagrees. Sometimes the agent is wrong. Sometimes it caught something the person missed. Either way you learn something, and nobody got hurt.
- Where it should have stopped. The cases where the right move was to skip, ask a human, or refuse, and the agent pushed ahead anyway. These are the ones that would have turned into the cleanup project.
Write down what “agree” means before you start. For some jobs it is an exact match. For others it is “a reviewer would accept this without edits.” If you cannot define it, you are not ready to automate the job yet, and that is useful to know too.
Cut over one slice, not the workflow
When the disagreements get boring, do not flip the whole thing on. Pick one slice.
One case type. One queue. One customer segment. One time of day. Hand that slice to the agent, keep a person approving anything that cannot be undone, and keep shadowing everything else.
Then do it again with the next slice. Each cutover is small enough that a bad week is a bad week, not a crisis. And you always have a fresh comparison running on the parts you have not handed over yet.
What it costs
Shadow mode is not free. Someone has to read the comparison, and that takes real time in the first week. But it is far cheaper than the alternatives: discovering problems in front of customers, or never trusting the system enough to use it.
It also gives you something most pilots lack: a record. When someone asks “is it working?”, you can point at the log instead of a feeling.
What we watch for
A few things make shadow mode go wrong.
- Comparing on easy cases only. If the log only covers the clean inputs, it will look great and tell you nothing. Shadow the messy ones too.
- Nobody owns the review. A log nobody reads is not shadow mode. Name the person, and give them a short daily slot.
- Shadowing forever. At some point the diff is boring and you need to hand over a slice. Set a rough date for the first cutover decision up front.
How we use this at Viking Labs
We start agent builds and productized automations this way, and custom software when a template will not fit. Before anything acts on its own, it runs beside the person who does the job today, and we compare. Then we hand over one slice at a time, with a human still approving the irreversible steps.
ResearcherFlow, the product we run ourselves, follows the same pattern. It proposes, a researcher compares and approves, and only then does anything become the record.
The impressive part of an AI system is not the demo. It is the boring log that shows it did the job your way, on your work, before anyone let it act alone.
If you have one workflow you would like to try this on, map the first build: tell us the workflow
Jon Marrs · About Viking Labs
Leave a Reply