Wizard of Oz Testing SOP

Wizard of Oz testing lets you find out if people want an automated product before you build the automation, and one round takes three to five weeks. Users see a form, an app or a chat window that looks finished. Behind it, a person (the "wizard") does the work by hand. You learn two things at once: whether users keep coming back for the result, and exactly which tasks a machine would need to handle.

What is Wizard of Oz testing and when should you use it?

In a Wizard of Oz test, the front end is real and the back end is a human. The user submits a request, and a team member reads it, produces the answer and sends it back through the same channel a finished product would use. The name comes from the film character who turns out to be a man pulling levers behind a curtain. Psychologist J. F. Kelley named the method in the early 1980s while studying natural language computer systems, and IBM researchers Gould, Conti and Hovanyecz used a hidden typist to simulate a "listening typewriter" years before speech recognition worked.

Use this SOP:

• When the hard part is the technology and you don't want to build it until you know the output is wanted.
• When you sell an AI or matching product and need real examples of what users ask for, in their own words.
• When you must estimate automation effort. Logging every manual request shows which ones follow a pattern and which ones need judgment.

The difference from the concierge testing SOP is visibility. In a concierge test, customers know you serve them personally. In a Wizard of Oz test, they interact with the product as if it were automatic, so you see how they behave with the real thing.

What do you need before you start?

• One automated promise to fake, in a sentence: "Send a voice note, get a week of social posts back within 2 hours."
• A front end built with no-code tools: a Tally or Typeform form, a Glide or Softr app, an Intercom chat widget or a dedicated email address.
• One or two wizards with time blocked to answer requests within the promised response time.
• 10 to 30 users from your target segment who did not help design the product.
• Honest terms and a privacy notice that say requests are processed with human review during the beta.
• A wizard log in Google Sheets, ready before the first request.

How do you run it, step by step?

1. Define the promise and the response time

Write down what the "system" delivers, in what format and how fast. The response time matters because it is part of the promise. If the future product answers in seconds, your wizards need to answer in minutes, or you test a slower product than the one you plan to sell.

2. Build the front end and nothing else

Build only the screens the user touches: input, waiting state, result. Connect the input to a Slack channel or a shared inbox with Zapier so wizards see each request at once. Resist building any real logic. Every hour spent on code is an hour of learning delayed.

3. Write the wizard playbook

List the rules wizards follow: output format, tone, templates for common requests, what to do with odd requests, and what never to do (medical, legal or financial advice, for one). Two wizards answering in different styles muddy the results.

4. Recruit users and set expectations

Invite 10 to 30 people who match your segment. Tell them it is a beta, that the service is free during the test and that their data is reviewed by the team to improve quality. That sentence keeps you honest while the experience stays product-like.

5. Run it live and log every request

For two to four weeks, wizards handle every request and log it: timestamp, request type, minutes spent, template used or not, and any edge case. The log is your future product spec. Resist the urge to over-deliver, because users will expect that quality from the automated version.

6. Measure behaviour, then ask about money

Track who comes back without a reminder, who completes the full flow and who asks about price. In week three, show a price and a "keep using it" button. Behaviour beats any satisfaction score here.

7. Sort the log into automate, assist and drop

Group requests by type. Mark each group as fully templatable (automate), needs a human check (assist) or rare and costly (drop). This tells your developer what the first build must do, and a technical feasibility assessment can then check if it can be built.

8. Debrief every user

Tell users how the beta worked and interview five of them. Ask what they would miss if it stopped tomorrow.

Template you can copy

WIZARD PLAYBOOK (one page)
Promise: [input] -> [output] within [response time]
Output format: [structure, length, tone]
Templates: T1 [common request], T2 [...], T3 [...]
Never do: [advice types, data uses, promises]
Odd request: reply "We're working on this one" and flag in #wizard-edge

WIZARD LOG (one row per request)
Date/time in | User ID | Request type | Template used (T1/T2/none) |
Minutes spent | Time to reply | Edge case Y/N | User came back within 7 days Y/N |
Notes (exact user words)

WEEKLY SUMMARY
Active users | Requests | Median minutes per request |
% handled by a template | Users who returned unprompted | Price questions

Worked example: "automatic" social posts for cafés

This case is illustrative. Two founders want to build an AI tool that turns a café owner's 2-minute voice note into a week of Instagram posts. They build a WhatsApp number and a Glide app, then answer every voice note by hand for three weeks.

• 22 café owners joined. They sent 141 voice notes in total.
• Median wizard time was 18 minutes per request, with replies inside 2 hours.
• 15 of 22 (68%) sent a voice note in week three without a reminder.
• 99 of 141 requests (70%) fit one of three templates. The other 30% needed knowledge of local events.
• 11 owners asked what it would cost before the price screen appeared.

The founders decided to automate the three templates first and keep a human editor for local-event posts. The 18-minute median also gave them a ceiling for what the human-assisted part could cost per customer.

What should you measure, and when do you stop?

Stop after three weeks of stable numbers, or earlier if usage collapses in week one. If repeat use is strong but template coverage is low, the product works as a service and not yet as software. In that case, run a proof of concept on the hardest request type before you plan any build.

Which mistakes should you avoid?

• Over-delivering as the wizard. If you write each answer like a senior expert, users fall for quality the software will never match.
• Hiding human review in the privacy terms. People share data with a "machine" they would not share with a stranger. Disclose human review up front.
• Skipping the log. Without minutes and request types, you finish with happy users and no build plan.
• Running too long. Past four weeks, wizard fatigue sets in and answer quality drifts.
• Testing with friends. They forgive slow replies. Real users don't, and that is part of the signal.

How can AI speed this up?

You can let an AI model draft each answer and have the wizard review and send it. This tests a human-in-the-loop version of your product and shows how far current models get on their own: log how many drafts needed no edits. AI also helps at the end, by clustering a few hundred logged requests into types. The human check is the playbook: a person decides what goes out, and a person handles anything involving health, money or safety.

If AI sits at the heart of your product, read the free AI for startups module next, and the Fe/male Switch startup game lets you practise validation steps with real deadlines.

Where does this SOP fit in your validation plan?

Wizard of Oz testing comes after you have shown interest and before you write production code.

1. Before: prove interest with a landing page test, and if the service itself is unclear, start with a concierge test.
2. Alongside: run prototype testing on the screens to fix usability problems separately from demand questions.
3. After: check buildability with a technical feasibility assessment, then watch real retention with customer retention analysis.

Browse all Solution Validation SOPs.

FAQ

Is Wizard of Oz testing ethical?

Yes, when you disclose human review in your terms, avoid sensitive advice and debrief users afterwards. The deception covers how the output is produced, never what users get or how their data is handled.

What is the difference between Wizard of Oz and concierge testing?

In concierge testing, the customer knows a person serves them. In Wizard of Oz testing, the customer uses an interface that looks automated. Use concierge to design the service and Wizard of Oz to test the product experience.

How many users do I need for a Wizard of Oz test?

10 to 30 active users is enough to see repeat behaviour and request patterns. More than that, and your wizards can't keep up with the response time.

Can I use an AI model as the wizard?

Partly. Let the model draft and a person approve. That tells you how much human time the real product still needs.