LEADERSHIP / PLAYBOOK 04 / QUALITY SYSTEM · DESIGN SYSTEMS · ACCESSIBILITY · MEASUREMENT
Making product quality repeatable
A playbook for connecting direction, research, design systems, accessibility, content,
delivery, and measurement. I make product quality repeatable by treating it as a system
rather than a final review. Research creates reusable knowledge. UX architecture and
design systems turn that knowledge into shared patterns. Accessibility and content
standards make those patterns inclusive and understandable. Delivery reviews protect their
integrity. Product measurement reveals whether the experience works in the market.
USE WHEN
Quality depends on a final review, teams keep solving the same problems, or polish
hides failures in usefulness, accessibility or outcomes
HORIZON
A 90-day implementation, then continuous assurance and quarterly review of the quality
scorecard
WHO IT IS FOR
Design, product, engineering and QA leaders responsible for the quality of a
multi-team product
CORE ARTIFACTS
Experience Quality Plan · Quality profile · Operating-condition matrix · Pattern
contribution record · Exception log · Quality scorecard
THE QUALITY FLYWHEEL
Quality is a system, not a gate.
The flywheel allows teams to deliver coherent experiences without repeatedly solving the
same problems. Each turn produces evidence that improves the shared standards the next
team starts from, so quality compounds instead of being re-inspected at the end of every
release.
01
Evidence
02
Shared standards
03
Product execution
04
Validation
05
Measured outcomes
06
Improved standards
01 / FOUR STREAMS
Define quality across four streams.
Quality means something different at each stage of product development. These streams
should overlap: direction continues to evolve during discovery, designers remain
involved during development, and market learning informs the next planning cycle.
Quality is not transferred from one function to another through handoffs.
STREAM
PRIMARY QUESTION
DEFINITION OF QUALITY
Product direction
Are we solving the right problem?
The initiative supports strategy, addresses an important customer need, and has a
credible outcome hypothesis
Feature discovery
Are we designing the right solution?
The proposed experience is useful, understandable, appropriately scoped, coherent,
accessible, and feasible
Engineering delivery
Did we build it right?
The implemented product preserves the intended experience and performs reliably under
realistic conditions
Customer-value release
Did it deliver value?
Customers discover, adopt, and successfully use the capability, producing the intended
customer and business outcomes
02 / SHARED DEFINITION
Establish a shared definition of product quality.
Every initiative should be evaluated across eight dimensions. A product should not be
called high quality because it looks polished while failing one or more of them.
01
Outcome alignment
Does the work address an important customer problem and contribute to a defined
business result?
02
Usefulness
Does the capability meaningfully help the client or end user complete an important
job?
03
Usability
Can intended users understand and successfully operate the experience without
unnecessary effort?
04
Coherence
Does the solution fit the broader product, UX architecture, brand, navigation model,
workflows, and terminology?
05
Accessibility and inclusion
Can people with different abilities, technologies, languages, contexts, and levels
of expertise use the experience?
06
Content quality
Is information clear, accurate, actionable, consistent, appropriately timed, and
suitable for its audience?
07
Technical and operational resilience
Does the solution perform reliably across expected devices, data conditions,
permissions, integrations, and failure scenarios?
08
Measurable effectiveness
Can the organization determine whether customers discovered the capability, received
value, and changed their behaviour?
03 / PROPORTIONATE CONTROL
Determine how much quality control is needed.
Not every initiative requires the same review process. The level of control should
reflect risk, novelty, reach, and reversibility. Baseline security, privacy,
accessibility, and domain-correctness requirements cannot be waived simply because an
initiative is small.
AT THE START, CREATE A QUALITY PROFILE BASED ON
Consequence of failure for the customer.
Financial, legal, security, or regulatory exposure.
Number and diversity of customers affected.
Difficulty of reversing the decision after release.
Novelty of the interaction or technology.
Number of teams, systems, and workflows involved.
Visibility and effect on the brand.
Accessibility and inclusion implications.
Amount of customer behaviour change required.
Strength of the available evidence.
REVIEW LEVEL
Standard
Used for low-risk work based on established patterns.
Quality control can rely on team review, automated testing, design-system
conformance, and normal release monitoring.
REVIEW LEVEL
Substantial
Used for important workflows, new patterns, cross-team dependencies, or meaningful
customer change.
Requires cross-functional review, representative customer validation,
accessibility assessment, and explicit success measures.
REVIEW LEVEL
Critical
Used for consequential decisions, regulated activities, major platform changes,
sensitive data, high-volume journeys, or new AI-enabled behaviour.
Requires specialist review, broader customer evidence, formal risk acceptance,
progressive rollout, and stronger post-release monitoring.
04 / EXPERIENCE QUALITY PLAN
Create an Experience Quality Plan.
Each initiative should begin with a short Experience Quality Plan. It determines what
needs to be reviewed, by whom, and at what point. It should be proportionate to the
initiative and remain a living part of the product work.
Experience Quality Plan
ONE PER INITIATIVE · PROPORTIONATE · LIVING
01
The customer and business outcomes.
02
The most important workflow or decision.
03
Applicable quality dimensions.
04
Initiative risk level.
05
Customer segments and operating conditions.
06
Design-system and UX architecture dependencies.
07
Domain, content, accessibility, technical, and control specialists required.
08
Validation moments.
09
Evidence and acceptance thresholds.
10
Required analytics and feedback mechanisms.
11
Release approach.
12
Accountable decision-makers.
13
Known exceptions and quality debt.
STREAM 1 / PRODUCT DIRECTION
Confirm the organization is addressing the right problem.
This stream confirms that the organization is investing in an experience that supports
product strategy. It protects teams from producing an elegant solution to a low-value or
misunderstood problem.
WHAT SHOULD BE REVIEWED
Business goal and strategic alignment.
Target clients, users, and market segments.
Customer jobs, needs, concerns, and current alternatives.
Evidence that the problem is important.
Product positioning and competitive context.
Intended customer behaviour.
Expected business result.
Proposed success measures.
UX architecture and platform implications.
Functional and non-functional requirements.
Key assumptions and risks.
Scope, exclusions, and opportunity cost.
Relationship to other roadmap initiatives.
EVIDENCE EXPECTED
Customer, market, and operational evidence.
A clear problem statement.
A testable outcome hypothesis.
An assumption and risk map.
Alternative strategic or experience directions.
An initial measurement plan.
A Visual Roadmap or experience narrative showing how the initiative contributes to
the future product.
WHO SHOULD BE INVOLVED
Executive or business sponsor.
Product, design, and engineering leaders.
Customer research or insights.
Data and analytics.
Client-facing teams.
Relevant domain subject-matter experts.
Operations, compliance, legal, security, or risk partners when applicable.
Representatives of affected product areas.
Selected clients or end users.
QUALITY DECISION
The accountable product or business leader decides whether to proceed, revise, defer,
or stop. Design is accountable for making the customer problem and experience
implications clear. Engineering identifies feasibility and platform consequences.
EXIT CONDITIONS
Do not move into detailed design until:
The priority problem is understood.
The target audience is defined.
The intended result is measurable.
Important assumptions are visible.
The proposed scope is consistent with the evidence.
Leadership understands the investment and opportunity cost.
STREAM 2 / FEATURE DISCOVERY AND DESIGN
Identify the smallest coherent solution that resolves the priority problem.
This is where teams explore alternatives, validate uncertain assumptions, and establish
the intended experience before major delivery commitments. Domain experts verify that
the design correctly represents business rules, terminology, workflows, risks, and
operating realities. They extend the team's domain knowledge, but they do not replace
customer research: an expert can confirm a workflow is correct; only representative
users can show it is understandable and useful in practice.
WHAT SHOULD BE REVIEWED
Completeness of the problem framing.
Alternative solution concepts.
End-to-end workflow and journey.
Alignment with the UX architecture.
Use of established components and patterns.
Navigation and information architecture.
Role, permission, and access behaviour.
Content hierarchy, terminology, and instructions.
Accessibility and inclusive interaction.
Empty, loading, error, success, and recovery states.
Responsive and cross-channel behaviour.
Feasibility and operational implications.
Analytics and feedback requirements.
Whether the solution does too much or too little.
EVIDENCE EXPECTED
Relevant research findings.
Alternative concepts and trade-offs.
Workflow models or journey maps.
Testable prototypes.
Usability and comprehension evidence.
Accessibility review.
Technical feasibility assessment.
Content review.
A design-system contribution or exception proposal.
Defined acceptance criteria and analytics events.
WHO SHOULD BE INVOLVED
Product owner.
Product designer and researcher.
Engineering lead or architect.
Design-system or UX architecture owner.
Content designer.
Accessibility specialist.
Data or analytics partner.
Domain subject-matter experts.
Security, privacy, legal, or compliance partners where necessary.
Representative clients and end users.
QUALITY DECISION
Design owns the coherence and usability of the proposed experience. Product owns
priority and scope. Engineering owns technical viability. Material trade-offs should
be resolved by the product-design-engineering triad and recorded.
EXIT CONDITIONS
Ready for committed delivery when:
The primary workflow solves the intended problem.
Consequential assumptions have been evaluated.
Representative users can understand and operate it.
Accessibility and content requirements are addressed.
The solution fits the broader product.
Engineering understands how it can be delivered.
The scope is sufficient without unnecessary complexity.
Success can be measured after release.
STREAM 3 / ENGINEERING DELIVERY AND DESIGN EXECUTION
Ensure the working product preserves the intended experience.
The product must remain effective under realistic operating conditions. Design quality
cannot be determined solely from design files. It must be reviewed in the product
customers will use. A smaller solution must still be complete for the conditions it
claims to support: reducing scope does not justify ignoring relevant failure states.
WHAT SHOULD BE REVIEWED
Fidelity to the intended workflow and interaction.
Use of approved components, tokens, and patterns.
Content accuracy and hierarchy.
Keyboard and assistive-technology behaviour.
Focus order and accessible names.
Responsive layouts and supported browsers.
Roles, permissions, and restricted states.
Performance and perceived responsiveness.
Data integrity and formatting.
Empty, dense, delayed, stale, or conflicting data.
Errors, interruptions, timeouts, and recovery.
Localization and long-text behaviour.
Integration with upstream and downstream systems.
Analytics implementation.
Security and privacy behaviour.
Alignment with non-functional requirements.
EVIDENCE EXPECTED
Working software reviewed in its actual environment.
Coverage of the operating-condition matrix.
Accessibility and keyboard test results.
Design-system conformance report.
Analytics validation.
Documented exceptions with owners and dates.
WHO SHOULD BE INVOLVED
Product designer.
Engineering lead and implementing engineers.
Quality assurance.
Product owner.
Content and accessibility specialists.
Design-system maintainers.
Security, privacy, performance, and operational specialists as required.
Domain experts for complex rules.
Customer support or service teams for high-impact workflows.
REVIEW THE REAL OPERATING CONDITIONS · TESTING SHOULD COVER MORE THAN THE HAPPY PATH
Customer conditions
New and experienced users.
Different responsibilities and permission levels.
Different languages and levels of domain expertise.
Users relying on assistive technologies.
Customers working under time pressure or interruption.
Device and environment
Supported screen sizes and browsers.
Keyboard, touch, and alternative input.
Slow or unstable network conditions.
Limited viewport or high zoom.
Changes in orientation or window size.
Data conditions
Empty, sparse, typical, and dense datasets.
Long names and translated content.
Invalid, incomplete, stale, or duplicated information.
Partial access to related information.
System conditions
Service latency or dependency failure.
Interrupted submission.
Concurrent edits.
Partial success.
Retry, recovery, and rollback.
Unavailable functionality.
QUALITY DECISION
The product owner remains accountable for release readiness. Design accepts the
experience quality, while engineering accepts technical quality. Material exceptions
should have a description of the gap, the customer and business consequence, an
accountable owner, a remediation date, a monitoring plan, and explicit acceptance by
the appropriate decision-maker.
EXIT CONDITIONS
Ready for release when:
Critical customer journeys work end to end.
Blocking usability and accessibility problems are resolved.
The implementation is coherent with the product and design system.
Important operating conditions have been tested.
Required instrumentation is functioning.
Remaining quality debt is understood and accepted.
Support and recovery mechanisms are ready.
STREAM 4 / CUSTOMER-VALUE RELEASE
Determine whether the solution is understood, adopted, and effective.
Delivery is complete only when the organization has learned whether the product produced
the intended customer and business results. Release should increase exposure as evidence
and confidence grow, and each stage should have entry criteria, success thresholds,
monitoring, and an explicit decision to expand, revise, pause, or withdraw.
WHAT SHOULD BE REVIEWED
Product positioning and value proposition.
In-product discovery.
Onboarding and activation.
Marketing and release communication.
Sales, service, and support readiness.
Training and documentation.
Customer eligibility and rollout sequencing.
Completion of critical workflows.
Errors, abandonment, and support demand.
Adoption across relevant customer segments.
Engagement depth and frequency.
Customer satisfaction and trust.
Intended business outcomes.
Unexpected consequences or unmet needs.
EVIDENCE EXPECTED
Internal acceptance and operational testing.
Alpha or pilot feedback.
Beta telemetry from a representative cohort.
Usability and workflow-performance measures.
Support and service signals.
Qualitative customer feedback.
Adoption, engagement, and retention measures.
Comparison with the original outcome hypothesis.
A documented investment recommendation.
WHO SHOULD BE INVOLVED
Product, design, and engineering.
Data and analytics.
Marketing and product marketing.
Sales, customer success, service, and support.
Operations.
Domain experts.
Risk and control partners where applicable.
Selected early adopters.
Representative customers from the broader market.
Executive or business sponsor for material investment decisions.
PROGRESSIVE EXPOSURE · INCREASE EXPOSURE AS EVIDENCE AND CONFIDENCE GROW
01
Internal validation
02
Selected clients
03
Representative cohort
04
Staged rollout
05
General availability
QUALITY DECISION
The executive or business sponsor makes material investment decisions on the evidence.
Product, design and engineering jointly compare the released result with the original
outcome hypothesis and recommend whether to expand, revise, pause, or withdraw.
EXIT CONDITIONS
The initiative is successful when:
Customers understand the capability.
Eligible users can discover and activate it.
Important workflows succeed under real conditions.
Adoption is sustained beyond initial promotion.
The intended customer behaviour changes.
The business outcome is progressing.
Adverse effects remain within agreed thresholds.
The organization has made an evidence-based next investment decision.
05 / RIGHT-SIZING
Right-size the solution.
Elegant design is not the addition of every potentially valuable capability. It is the
simplest coherent solution that addresses the priority problem, works under relevant
conditions, and creates a foundation for learning. This prevents under-designing
necessary states while avoiding speculative features.
EVALUATE SCOPE THROUGH FOUR TESTS
01
Priority
Does each part of the solution directly support the intended outcome or a necessary
operating requirement?
02
Sufficiency
Does the solution fully resolve the core job for the intended audience?
03
Simplicity
Can anything be removed without weakening comprehension, trust, accessibility, or
successful task completion?
04
Extensibility
Does the solution use patterns and architecture that allow it to evolve without
premature complexity?
CLASSIFY SCOPE INTO
Essential
Required to deliver the customer outcome.
Assurance
Required for accessibility, security, reliability, compliance, measurement, or
recovery.
Enabling
Shared capability that prevents repeated work or supports near-term evolution.
Enhancement
Valuable but not required to validate the central hypothesis.
Future
Plausible work that should not complicate the current solution.
06 / PARTICIPATION
Involve the right people at the right moment.
“Stakeholder review” is often too vague. Different participants serve different
purposes. Each quality review should name the purpose of participation. Inviting
everyone to every review creates slow decisions without improving accountability.
Decide
The accountable owner makes the decision and accepts the resulting trade-offs.
Align
Adjacent departments help ensure that the solution fits their strategies, systems,
commitments, and customer interactions.
Advise
Subject-matter experts provide specialized knowledge about the domain, technology,
content, risk, or operating environment.
Validate
Representative clients and users provide evidence about usefulness, comprehension,
usability, and behaviour.
Assure
Specialists confirm that required quality, accessibility, security, privacy, legal,
and operational standards have been met.
Inform
Affected groups receive enough context to prepare for the change without being
required to approve it.
07 / SHARED CAPABILITIES
Turn repeated solutions into shared capabilities.
When teams repeatedly encounter the same problem, the result should become reusable
organizational knowledge. A design system is therefore more than a component library. It
is a governed record of recurring experience decisions.
THE PATH FROM RECURRING PROBLEM TO SHARED CAPABILITY
01
Research identifies a recurring need.
02
UX architecture establishes the structural solution.
03
Design defines the interaction and content pattern.
04
Accessibility requirements are embedded.
05
Engineering creates the reusable implementation.
06
Guidance documents when and how to use it.
07
Automated tests protect expected behaviour.
08
Product teams adopt it.
09
Measurement reveals whether the pattern remains effective.
10
New evidence improves the shared solution.
REUSABLE CAPABILITIES CAN INCLUDE
Navigation and workspace patterns
Search and filtering
Data tables and dense information displays
Multi-step workflows
Forms and validation
Notifications and status
Permissions and restricted access
Empty, error, and recovery states
Onboarding and feature activation
Content and terminology patterns
Analytics specifications
Accessibility behaviours
08 / GOVERNANCE
Establish lightweight quality governance.
Governance should create a dependable route to quality without turning a central team
into an approval bottleneck. Standard-pattern work should move quickly through
documented guardrails. Novel or high-risk work should receive deeper review.
USEFUL FORUMS
Product-direction review
Strategic fit, customer problem, and intended outcome.
Experience review
Workflow, usability, content, accessibility, and coherence.
UX architecture review
Cross-product structures and dependencies.
Design-system contribution review
New patterns, extensions, and exceptions.
Implementation review
Working-product quality during development.
Release review
Experience, technical, operational, and communication readiness.
Outcome review
Adoption, customer results, business results, and next investment.
MANAGE EXCEPTIONS EXPLICITLY
When a team cannot meet a standard, record:
The standard being bypassed.
Why the exception is necessary.
Customer and organizational consequences.
The accountable owner.
A remediation or expiry date.
Whether the exception should become a supported pattern.
Undocumented exceptions eventually become accidental architecture.
09 / CONTINUOUS ASSURANCE
Use automation and AI for continuous assurance.
Automation and AI can move quality detection closer to the moment work is created.
Automated evaluation should handle repeatable checks. People should remain accountable
for strategic alignment, domain correctness, usefulness, elegance, ethical consequences,
and final quality decisions. Synthetic users may help expose questions, but they should
not replace validation with representative customers.
THEY CAN HELP
01
Compare implementations with design-system standards.
02
Detect visual regressions.
03
Identify missing states and edge cases.
04
Scan for accessibility issues.
05
Check terminology, reading level, and content consistency.
06
Generate test scenarios from requirements.
07
Examine designs under different data and localization conditions.
08
Summarize research and quality findings.
09
Find recurring defects across product areas.
10
Detect changes in adoption, errors, or support demand.
11
Maintain traceability between requirements, designs, tests, and metrics.
10 / MEASUREMENT
Measure whether the quality system works.
No single quality score can represent product quality. Use a balanced set of measures.
Quality metrics should lead to a decision: continue, correct, invest, standardize,
simplify, or retire.
DIRECTION QUALITY
Initiatives tied to a defined value driver
Initiatives with a measurable outcome hypothesis
Time from opportunity identification to meaningful evidence
Initiatives revised or stopped before major investment
DISCOVERY QUALITY
Critical assumptions evaluated before delivery
Usability and comprehension results
Accessibility problems found before development
Reuse of established patterns
Amount of late scope change
DELIVERY QUALITY
Design and accessibility defects found before release
Escaped defects after release
Rework caused by missing requirements or design states
Performance and reliability of critical workflows
Design-system conformance
Time required to resolve quality debt
MARKET EFFECTIVENESS
Awareness and activation
Task success and time to value
Adoption breadth and engagement depth
Abandonment and support demand
Retention and progression to higher-value capabilities
Customer and business outcome movement
SYSTEM LEVERAGE
Shared-pattern adoption
Reduction in duplicated solutions
Teams enabled by common capabilities
Time saved in discovery, design, and development
Consistency across product areas
Improvements made to the system from product evidence
IMPLEMENTATION
A 90-day implementation plan.
Define the quality dimensions and baseline; pilot the model on one initiative that
crosses all four streams; then systematize what works and expand to additional value
streams.
DAYS 1–30
Define and baseline
Agree on the eight quality dimensions.
Map existing reviews to the four streams.
Inventory recurring defects and duplicated solutions.
Establish initiative risk levels.
Identify required subject-matter experts and control partners.
Baseline customer, delivery, accessibility, and system-quality measures.
DAYS 31–60
Pilot
Select one initiative crossing all four streams.
Create its Experience Quality Plan.
Establish validation and exit criteria.
Test the operating-condition matrix.
Conduct design reviews in the working product.
Record pattern contributions and exceptions.
Instrument the release for adoption and outcome measurement.
DAYS 61–90
Systematize
Convert repeated solutions into shared patterns.
Integrate automated quality checks into delivery.
Establish lightweight review forums.
Publish decision rights and escalation paths.
Create a quality scorecard.
Review the pilot and simplify unnecessary steps.
Expand the model to additional value streams.
WHAT SUCCESS LOOKS LIKE
Product quality is repeatable when:
In this model, quality is not a gate at the end of delivery. It is a connected set of
decisions, standards, evidence, and feedback loops that helps every team produce a
coherent experience, and makes the next team faster and more effective.
Teams agree on what quality means before delivery begins.
Review effort is proportional to risk.
The right specialists participate at the right time.
Clients validate value while experts validate domain correctness.
Designers remain involved through implementation and release.
Products work across realistic roles, data, devices, and failure conditions.
Small solutions are complete without becoming overbuilt.
Recurring decisions become reusable patterns.
Accessibility and content are built into the system.
Quality debt is explicit and managed.
Released experiences are measured for adoption and value.
Evidence continuously improves the product's shared foundations.
NEXT / PLAYBOOK 05
Lead change through experience direction
Continue reading
Discuss making product quality repeatable
Start a conversation