Validation SOPs › Technical Validation
SOP 69 of 100 · Technical Validation
Device Compatibility Testing SOP
- Difficulty
- Intermediate
- Time
- 1 day to build the matrix, 1-2 weeks of testing and fixes
- Cost
- $0-500
- Best for
- Consumer apps, websites, e-commerce and any product whose buyers use phones, tablets and mixed browsers
- Tools
- BrowserStack, LambdaTest, Playwright, Firebase Test Lab, StatCounter Global Stats
- Measure
- Audience covered by the matrix, Blockers per device, Top-task completion per device, Load time on a mid-range phone, Accessibility checks passed
A device compatibility testing SOP checks that the people you want as customers can complete your product's main tasks on the phones, tablets, laptops and browsers they really own, and it takes one to two weeks. You build a test matrix from visitor data, run the same short task list on each device and fix whatever stops a task. A checkout button hidden behind the iPhone address bar loses sales as surely as a bad price, and no analytics dashboard will tell you why. The result is a matrix, a ranked bug list and the proof that your top tasks work where your buyers are.
What is the device compatibility testing SOP and when should you use it?
Device compatibility testing checks that your product looks right and works across the combinations of device, operating system, browser, screen size and user settings your audience uses. It covers layout, input (touch, keyboard, mouse), browser features, speed on cheaper hardware and settings such as large text and dark mode.
It sits next to two other SOPs. Prototype testing asks whether people understand and want the product. Technical performance testing measures server speed under load. This SOP asks whether the same product works on a five-year-old Android phone and a new iPad alike.
Use it:
- Before launch, once the top tasks are built and before you pay for traffic.
- Before a landing page test or ad campaign, so a broken mobile layout doesn't fake a low conversion rate.
- When analytics show a gap, such as mobile visitors converting at a third of the desktop rate.
- After a redesign or a framework upgrade.
What do you need before you start?
- Visitor data: device, OS and browser breakdown from your analytics, landing page or waitlist. With no traffic yet, use StatCounter Global Stats for your target country.
- Your top three tasks, written as steps: sign up, complete the main action, pay.
- Access to devices: your own phones, friends' old phones and a cloud device service (BrowserStack, LambdaTest and Sauce Labs offer real devices; Firebase Test Lab covers Android and iOS apps).
- Feature support references: Can I use for browser support of each web feature, and Google's Baseline labels for features that work across major browsers.
- A bug tracker with a severity field.
How do you run it, step by step?
1. Build the matrix from real sessions
List every device, OS and browser combination with its share of your sessions. Sort by share and draw a line where the cumulative total passes 90%: everything above the line is tier 1. Add one low-end Android phone to tier 1 even if its share is small, since it shows speed problems first.
2. Add the settings people forget
For each tier-1 device, add three variants: largest system text size, dark mode and a screen reader (VoiceOver on Apple devices, TalkBack on Android). These catch broken layouts and unlabelled buttons that standard tests miss.
3. Write a short task script
Each tester runs the same top three tasks with the same test account and records pass, fail or workaround, with a screenshot of every problem. Keep the script under 15 minutes per device so you can cover the matrix in a few days.
4. Run the matrix on real devices first
Start with real phones for tier 1: touch, keyboards, address bars and notches behave differently on emulators. On iPhone, Apple's App Store rules long required every browser to use the WebKit engine (the EU now allows exceptions), so Safari testing covers most iOS browsing.
5. Automate the repeat checks
Write the top tasks as Playwright tests, which run on Chromium, Firefox and WebKit, and run them on every code change. Use the cloud device service for tier-2 combinations you can't own.
6. Rate every bug by severity
Use four levels: blocker (task impossible), major (task possible with a workaround), minor (looks wrong) and cosmetic. Fix every blocker on tier 1 before launch. Check tap target sizes against WCAG 2.2 success criterion 2.5.8, which sets a minimum of 24 by 24 CSS pixels; Apple's Human Interface Guidelines recommend 44 by 44 points.
7. Retest and publish a support list
Rerun the full script on every device where you fixed something. Then write the list of supported browsers and devices on your help page, so support has an answer when someone on an old browser writes in.
Template you can copy
| Device | OS version | Browser | Share of sessions | Tier | Task 1: Sign up | Task 2: Main action | Task 3: Pay | Large text | Dark mode | Screen reader | Blockers | Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| iPhone (recent) | iOS current | Safari | 1 | Pass/Fail | ||||||||
| iPhone (3+ years old) | iOS current minus 2 | Safari | 1 | |||||||||
| Samsung Galaxy A series | Android current minus 1 | Chrome | 1 | |||||||||
| Low-end Android | Android oldest supported | Chrome | 1 | |||||||||
| Windows laptop | Windows 11 | Chrome / Edge | 1 | |||||||||
| MacBook | macOS current | Safari | 1 | |||||||||
| iPad | iPadOS current | Safari | 2 |
Bug log columns: ID, device row, task, step, what happened, screenshot, severity (blocker / major / minor / cosmetic), owner, fixed Y/N, retested Y/N.
Worked example: a meal-planning web app for parents
This is an illustrative case. A founder runs a waitlist page for a meal-planning app and looks at 2,400 visits: 73% from phones, mostly iPhone Safari and Android Chrome, with a small share from Samsung Internet. Seven combinations cover 92% of sessions, so they become tier 1, plus one low-cost Android phone.
Two testers run the task script on real devices in three days. They log 31 bugs, four of them blockers. On iPhone, the "Save plan" button sits behind the browser toolbar, because the layout uses a fixed 100vh height; switching to the newer dvh unit fixes it. On the largest text size, the shopping list overlaps the price column and hides the "Pay" button. The low-end Android phone needs 6.8 seconds to show the first meal card, mostly because of uncompressed food photos. The screen reader announces the add-to-list button as "button" with no label.
The fixes take four developer days. Compressed images bring the low-end phone's Largest Contentful Paint down to 2.3 seconds, and all three tasks pass on every tier-1 device.
What should you measure, and when do you stop?
| Metric | Pass signal | Warning signal |
|---|---|---|
| Audience covered by the matrix | Tier 1 covers 90%+ of sessions | Matrix built from guesses, below 80% |
| Blockers per device | 0 on every tier-1 device | Any blocker on a device with 5%+ share |
| Top-task completion per device | 100% on tier 1 | Workarounds needed on mobile |
| Load time on a mid-range phone | Largest Contentful Paint under 2.5 s (Google's published "good" threshold) | Over 4 s |
| Accessibility checks passed | Large text, dark mode and screen reader pass on tier 1 | Unlabelled buttons, clipped text |
Stop when tier 1 passes every row and two consecutive automated runs are green. Rebuild the matrix every quarter from fresh visitor data, since your audience's devices change as you reach new segments.
Which mistakes should you avoid?
- Testing only on the founders' new phones. Your buyers own older, cheaper and smaller devices.
- Trusting browser resize tools alone. Shrinking a desktop window doesn't reproduce touch, mobile toolbars or slow processors.
- Picking devices from global charts when your visitors come from one country. Use your own data first.
- Skipping large text and screen readers. These settings break layouts, and in many countries accessibility is also a legal duty.
- Fixing cosmetic bugs before blockers. Rank by severity and share of sessions.
How can AI speed this up?
Paste your analytics export into an AI assistant and ask it to build the tier-1 and tier-2 matrix with cumulative shares. Ask it to write Playwright tests from your task script, and to check your CSS for known mobile traps such as fixed viewport heights.
The human check: AI can read code and tables, but it can't feel a thumb missing a tiny button or a screen reader reading nonsense. Real hands on real devices still run the final pass. For the page that brings visitors in, the F/MS landing page module covers what to test before you send traffic.
Where does this SOP fit in your validation plan?
Device testing closes the build phase of the Technical Validation SOPs:
- Before it: confirm people want the product with prototype testing and get server speed right with technical performance testing.
- Alongside it: compare how competitors behave on the same phones with the competitor user experience analysis SOP.
- After it: run your landing page test knowing every device converts fairly.
Google indexes the mobile version of your pages first, so mobile bugs also cost search traffic; the F/MS digital marketing and SEO module explains the rest.
FAQ
How many devices should a startup test on?
Enough to cover 90% of your sessions, which for most consumer products means 6 to 10 combinations. Add one low-end Android phone and test settings such as large text on each.
Is an emulator good enough for mobile testing?
Emulators catch layout problems fast and cost nothing. Use real devices for the final pass, because touch, toolbars, keyboards and slow processors behave differently.
What is the best free tool for cross-browser testing?
Playwright is free and runs automated tests on Chromium, Firefox and WebKit. Pair it with a few real phones and a trial of a cloud device service for devices you don't own.
Do I need to support old browsers?
Support what your visitors use. If an old browser has under 1% of sessions, show a clear message that asks people to update, and spend the time on tier 1.