Todd here.
TL;DR
- Anthropic is placing evaluators from another company inside its own, with access comparable to an employee's.
- Its framing: this doesn't reduce accountability, it makes accountability more verifiable.
- And it names the awkward part: it pays the evaluator itself, while saying that long-term the money should come from somewhere else.
- A lab that size deciding it needs outsiders looking from inside answers the question for your business.
- What you leave with: three questions for whoever runs your marketing, and which answers stop being acceptable.
On September 18 Anthropic published that Accenture will evaluate its models from the inside, with access comparable to an employee's, each side expecting to invest at least a billion dollars over five years. That figure is an investment both say they expect to make, not money spent. The number isn't the useful part. The structure is: whoever does the work isn't the one who says it went well, and the reviewer needs real access, not a summary.
You already run that arrangement. You just haven't called it that.
Whoever built your site sends the report on how the site is doing. Whoever runs your ads explains why the month was soft. Nobody's being dishonest: they're simply the only person with the numbers open, and you're asking them about their own work.
What was announced, in plain terms
Accenture staff will work inside Anthropic, evaluating and deliberately attacking its models and testing its safeguards. Three lines from the announcement matter to a service business:
On access. Unlike today's external evaluators, these will work inside, "with access comparable to an employee's": enough to watch models take shape during training, follow the decisions behind them, and talk to staff directly.
On what it's for. Independent embedded evaluators don't reduce their accountability; they help make it more verifiable. The responsibility stays theirs.
And the part nobody made them publish. They acknowledge that no standards exist yet for what information those evaluators should have access to or how they should report findings, and no settled system for funding them. Given the urgency, Anthropic will fund Accenture's work directly, while stating that long-term the money should come from pooled or government sources.
That's the most honest of the three, and it's what makes the example usable. The evaluated pays the evaluator, in those words.
And one nuance worth not skipping, because it keeps this from reading as black and white: that isn't their only arrangement. The same announcement says they're in dialogue with METR and other nonprofit evaluators to pilot this using the evaluators' own funding, and that the Accenture partnership is non-exclusive. So there are degrees. Someone paid by you and someone paying their own way don't look the same, and they're testing both.
Why this lands on a ten-person business
Because the shape doesn't change with size. Your vendor does the work and is also your only source of information about the work. Paid to do it, and author of the document saying whether it was worth it. Neither half is suspicious alone; together they leave a gap.
Worth knowing where each thing lives, which is what you rent the tool, you own the asset covers: if the dashboard belongs to the vendor, so does the key.
The gap doesn't close with suspicion. It closes with access.
The three questions, and they fit in one call
You don't need a new vendor or an audit. Three questions, and what matters isn't the answer but whether the answer can be checked somewhere else.
1. "Where does this number come from?"
Not whether the number is good. Which screen it came off, and who else can open that screen.
A useful answer names a tool and tells you how to get in. An unusable one is "from our system." If the only place that figure exists is a document your vendor sends you, that isn't a measurement. It's an account.
2. "Who else sees this without going through you?"
The independence question, and almost nobody asks it.
"Nobody" tells you where you stand. Not that you're being misled: that nobody unpaid on the outcome has looked at the outcome, which is exactly what Anthropic just decided wasn't good enough for itself.
3. "If this stops working, how do I find out, and when?"
The bad-news question. Is there an agreed moment when someone tells you something is underperforming, or does that depend on somebody feeling like it?
Useful answers carry a threshold: "below X, I write to you." "Don't worry, we'll let you know" is an intention, not a commitment.
Where this doesn't apply, so it doesn't become dogma
Full access since day one, and you can log in whenever? Solved. Whether you look is a separate matter, and it's yours.
Pure execution, where you set the goal, measure the result and delegate only the doing? No conflict. The trouble starts when one person defines, executes and grades.
And this isn't a pitch to hire an auditor. For most small businesses a second vendor watching the first is spending that doesn't pay for itself. Having the tool logins in your name rather than your vendor's does.
Naty and Todd's corner
Todd: What stayed with me is where the pressure came from. Not a regulator, not an angry customer. It came from inside, out of an essay by the company's own head. That's the hardest part to copy: almost nobody puts a reviewer above themselves when nobody's asking them to.
Naty: What I see is the other side, the owner receiving the report. Almost nobody asks where a number came from, because asking feels like accusing. It isn't. When I had the spa I didn't ask either, and it wasn't trust, it was not wanting to be the difficult client.
Todd: That's the part that isn't in the article. The question doesn't get blocked by distrust, it gets blocked by embarrassment. Which is why it's worth asking at the start of a relationship, while it's still a procedure and not a suspicion. Nobody minds being asked that in a first meeting.
Naty: And if you're two years in and never asked, there's a way to open it that doesn't sound like an audit: say you want to learn to read it yourself. Which is also true. Anyone doing the work properly will be glad to teach you, because it's in their interest that you can read what they send.
The only thing to do this week
Open the last report your vendor sent. Site, ads, social, any of them. Pick one number, the one you care about most, and answer two things in writing:
Which other screen can I see this number on without asking anyone? Do I have the login for that screen?
Two yeses and you're ahead of most people, and this took four minutes.
Any no and you've got the subject line for your next email, and it isn't a complaint. It's asking for a key.
Frequently asked questions
Does this mean my agency is lying to me? No. It means the people doing the work and grading it are the same, which is structural, not a character question. Anthropic didn't swap teams out of distrust.
Do I have to hire someone to review my vendor? For almost no small business, no. What you need is the logins in your own name. A second vendor only pays off when monthly spend is large enough that a mistake costs more than the review.
My vendor says the dashboard is too technical for me. Is that true? Usually not. Most tools have a simple view, and whoever runs your work can bookmark the three figures you care about. If the answer to "I'd like access" is "it's complicated," that's information, even when it's genuinely complicated.
What if I'm the one doing the work, with no vendor? Then you're both parties at once, the hardest case. Bring in someone with nothing invested in the answer being good: another owner, someone on your team who didn't build it. They don't need to know the subject. They need the result not to sting.
What if my vendor takes offense? Anyone doing the work properly rarely does, because these questions make their life easier: a client who can read their own numbers argues less and asks better. If the question creates friction, that's data, and it's data you wanted sooner rather than later.
To close
A large company decided its accountability for its own work should be verifiable by an outsider, and the way to get there wasn't promising harder. It was letting someone in.
Your version costs nothing: ask for the login to the screen where the number lives.
Open the last report, pick one number, find out whether you can see it somewhere else. It costs no money and little time. What it costs is asking a question that feels like an accusation and isn't.
Most of the time the answer is that you can see it, and never logged in. The work was already done there. All that was missing was a password.
Keep reading
- SEO vs. GEO vs. AEO - what you're actually asking whoever runs your marketing to do
- You rent the tool. You own the asset - where your data lives and what happens when a vendor stops answering
- Partnering with Accenture on embedded evaluation - the original announcement
