DESIGN LEADERSHIP
Establishment of UX Research Competency
Company
Caseware
Date Completed
July 2023
Leadership Team
Christopher Moorehead Head of User Experience
Louise Bonnamy UX Research Manager
Overview
Background
For most of its history, UX research did not exist at Caseware. Overall product strategy and individual product decisions were based almost entirely on the opinions of in-house industry experts (generally accountants with previous experience in auditing) and Caseware’s worldwide network of distributors. User testing was rarely performed, and when it was, it tended to involve a “friendly audience” of representatives from major clients being asked leading questions in an informal, “focus group” environment.
Problems with Operating Model
Industry subject-matter experts tended to make decisions regarding customer requirements entirely through intuition, usually based on their own experiences as auditors (sometimes from decades earlier). They consulted with distributors (who were often focused on unvalidated feature requests from their most vocal customers), but rarely if ever talked to actual end users of Caseware’s products. Critical product decisions were made without considering the potential for cognitive bias on the part of the decision-makers.
..........
If you skip research, you risk creating a superficially good user interface for a product or solution that really doesn’t provide value to the end user.
Paul Eisen
The result of this flawed process was almost always adding features to the product, rather than refining and simplifying existing functionality or fixing long-standing problems. We therefore made the decision to build a UX Research competency to provide evidence-based product decisions.
Building UX Research at Caseware
The first course of action in building a UX Research competency was to hire an experienced UX Research Manager to lead it. To this end, we were extremely fortunate in that we found an ideal candidate, Louise Bonnamy, very early in our search. Recognizing her value to Caseware, we brought her on board almost immediately.
With our research leader in place, our next priority was to develop our long-term UX Research strategy and a roadmap for its implementation, including the expansion of the UX Research team to include additional staff. The expanded team executed on the key elements of the roadmap, including developing a robust user testing program with users of Caseware products, as well as with users of competitor products. At the same time (working with Product Design and Design Operations), the UX Research team identified key customer and operational performance metrics, and developed a system for their measurement, tracking, and analysis.
As part of the overall UX Research strategy, we modified Caseware’s processes to insert continuing UX Research into every stage of the product life cycle. This was done in parallel with our Integration of UX into the Software Development Life Cycle initiative, and followed the same cross-functional collaboration model. The three phases of continuous research paralleled the phases of the software development life cycle’s cross-functional approach.
Phase 1: Define the Problem
An accurate definition of the problem to be solved requires extensive user research. It is therefore essential that the UX Research team be involved at this early stage. Ideally, Product’s identification of and alignment with the company’s business needs is informed by this early user research work.
Phase 2: Discover & Envision
In this stage, the UX Research team works closely with Product Design to map user journeys, create service blueprints, and develop relevant user stories. Researchers and designers test prototypes, obtain feedback from users, and refine the initial designs.
This was the stage in which user research was seen as “normal” by Product and Development (although the reality is that any research started at this stage was likely to be too late to be truly effective). Since this stage is led by UX, inserting UX Research required the least amount of change management.
Phase 3: Build & Reflect
UX Research continues to play an important role even after the design has been handed off to Development. Usability testing is essential to identifying remaining issues, allowing them to be fixed with minimal impact to the release schedule. Even after a product has launched, the UX Research team will review and analyze survey results and feedback as a means of understanding evolving customer needs and expectations. There is truly no research “stage” — it continues throughout the entire product life cycle.
Eliminating Cognitive Bias
The existing method of gathering information from customers was particularly susceptible to being influenced by the cognitive biases of industry experts, product managers, and product owners.
Some of the cognitive biases that adversely impact effective user research.
The UX Research team created a series of training sessions on identifying the various forms of cognitive bias, and how to minimize their effects through asking non-leading questions. These sessions were held regularly for the entire Product team, and the recordings of the sessions were made a required part of the Product onboarding process.
Detail from one of the UX Research team’s cognitive bias training presentations (created by Louise Bonnamy).
Outcome
Our combination of process changes and training resulted in a new appreciation of the value of user research throughout the Product organization, and continuing UX research was successfully inserted into every stage of the product life cycle. To measure success, we established metrics that were more user-focused than those used in the past. For example, although we continued to collect NPS data for marketing purposes, we replaced it with UMUX-Lite as our primary usability metric. Our research had identified UMUX-Lite as a highly reliable source of actionable usability feedback similar to SUS, but was much shorter and fit easily into surveys.
..........
Moving fast is risky. Research de-risks the situation by providing evidence to support (or refute) the direction we’re taking.
Louise Bonnamy
Research allowed us to identify and focus on business and user problems that were truly important to our customers. Effective, timely research kept us focused on solving the user’s actual problems, rather than simply adding features and complexity to the product. Strategic research provided a holistic view of customer needs, allowing us to address the root causes of customer and user problems. Finally, research ensured that we were not basing our product decisions entirely on our own opinions, which were subject to cognitive bias and would result in significant time and effort being wasted if we were wrong.
..........
When you talk directly to users, you hear the truth.
Louise Bonnamy
-
Share