No. 4: Measuring Audience Experience Success

This is the fourth installment of our series on the 12 Competencies of UX Design.

In the late eighties, psychological researcher Dr. John Gottman set out to determine if he could discriminate happy from unhappy couples by analyzing their behavioral patterns.

In a study that would eventually be dubbed the Love Lab, he invited couples to discuss a particularly contentious topic on camera. From there, he analyzed the recording and codified behaviors like eye rolls, sighs, and tone of voice. Then he combined this with quantitative data like heart rate, skin conductivity, and blood pressure to map stress levels.

Gottman and his team analyzed hundreds of couples over three years before finally releasing their research. By the time they completed their study, Gottman was able to predict which relationships would remain intact with 83% accuracy.

Gottman might not be in the design field, but his methodology isn’t all that different from how savvy UX professionals approach user research.

Like Gottman, UX designers have to analyze not just what someone is saying and doing, but what they are thinking and feeling as well. This is what we at Drawbackwards call “Measuring Audience Experience Success”.

Three Elements for the Right Measurement Mix

So how do product leaders and designers go about measuring audience experience success? It depends. There are a number of research methods UX designers use to understand user experience before, during, and after the experience, and not all make sense for every situation.

More often than not, successful audience experience measurement comes down to the right combination of a few of these methods, and not all have to happen at the same time with the same audience. This is especially true if you’re dealing with a diverse audience with complex needs.

No matter which methods you use, your strategy should be guided by three critical factors:

#1. Timing and Key Interactions

Getting quality data from a specific interaction in audience research often comes down to when you ask. Ask too early, and users are more likely to shrug their shoulders than provide any meaningful feedback. Too late, and they won’t remember enough to provide any value. Perhaps worst of all is asking in the middle of an interaction, something that often undermines the success of the experience you’re trying to measure.

Generally speaking, the sweet spot is right after a key interaction. This way you don’t interrupt the experience, but a user’s thoughts and feelings about a particular interaction are still fresh in their minds.

#2. Context and Users

Just like understanding a user’s state of mind and current situation can help guide initial design, it can also influence how a particular experience is measured. New users and established users, for example, won’t provide the same type of feedback. Rather than throwing an audience into a single testing bucket, designers need to identify what kind of insights they can get from different types of users, and how those users change over time.

Consider grouping an audience into buckets based on things like experience level, professional role, buying phase, or average web usage. This will remove tricky outliers that can throw off your findings, and help you hone in on patterns that really matter.

#3. Adaptivity and Fatigue

Nobody likes getting asked the same questions over and over again. Of course, sometimes getting quality data requires asking the same users for feedback multiple times over the course of a given period or time or within a specific workflow. Simply changing the way questions are phrased can be a great way to fight research fatigue. The same goes for developing new questions based on users’ new needs.

Some companies do a great job of balancing timing, context, and adaptivity in their audience measurement strategy. Cloud-based productivity tool Asana, for example, has a robust user research strategy that accounts for all three of these factors in equal measure.

Before Asana users are included in their research, they must first fill out an extensive form that covers everything from how much they use Asana to which industry they work in (context). From there, users are contacted to participate in either an in-person test or one that takes place over the phone, depending on their preference (adaptivity). Users that do participate in these tests take part in a carefully moderated study that gauges their experience in real-time (timing).

Not every company has the resources to conduct such involved audience research, but even very simple methods can be enormously successful when they account for these three factors.

The Importance of Audience Experience

When Gottman created his relationship study, his goal was to determine how current behavior patterns would dictate future decisions. This is no different than the goal of measuring audience research. For Gottman, the goal was determining whether or not a couple would split — but for designers, it’s whether a user will continue to use a product or service.

Just like some couples Gottman studied, digital audiences today have plenty of options — and are constantly evaluating these options to determine which they prefer. Since digital audiences sometimes have thousands of options to choose from, the quality of their experience is a major deciding factor in which option they select.

Consequently, product leaders and designers need a way to determine how their experience rates against competing experiences and where they need to improve in order to win over their audience — and that’s not always easy when you’re dealing with complex human emotions.

To do this, the team at Drawbackwards uses the Experience Success Ladder, which codifies levels of user experience into five distinct “rungs”, each building on the one below it. The higher an experience goes on the ladder, the more likely users are to develop loyalty and engagement with it.

Functional

Functional experiences fulfill technical specifications but don’t include much in the way of clear labels, logical order or placement of buttons or user-friendliness. Settling for an experience that is merely “functional” is a great way to alienate the majority of human audiences looking to move through a workflow and achieve something easily. It’s also where human error is a frequent result.

Usable

Usable experiences are different from functional experiences because a user can typically complete a task correctly with some work. “Usable” and “usability” is too low of a bar in the UX discipline today, yet many teams stop here. Merely usable experiences still make people think before acting and often leave a person searching for the right way to complete a task. This level still results in wasted interaction time, especially for new users.

Comfortable

When an experience moves from usable to comfortable, that means it is inherently more intuitive for a user. They won’t need to dig around to find what they need, and are able to move through a workflow with less friction — but experiences that are merely comfortable may not leave a lasting impression.

Delightful

Move one rung up to delightful, and you are beginning to leave users feeling noticeably happy rather than just satisfied. Surprises appear that elevate and compliment the core experience in unexpected and fulfilling ways. Users will enjoy the experience and might prefer to use it unless something truly meaningful comes around.

Meaningful

Meaningful experiences are those that users are likely to say they “can’t live without”. These are the product moments that lead people to tell their friends about what they are missing out on. Savvy product managers and UX designers aspire to create meaningful experiences by understanding their audience’s deeper emotions, needs, wants. They also know how to use time, space and context to deliver information, tools, and insights in a way that alters how people think and act in their world.

Codifying Qualitative Data

Meaningful experiences don’t happen on accident. They require a deep understanding of users’ emotions, needs, and wants — all of which can’t be captured with quantitative data alone. The trick is understanding how you can create meaning from qualitative data, and that comes down to repetition. If you analyze enough of qualitative data, patterns will start to emerge, and these patterns can tell you a lot about how and why users are behaving (or not behaving) in a particular way.

As Drawbackwards, we created a tool called UX Rings, part of an overall assessment process,  that uses patterns and impressions in user feedback to “grade” particular experiences based on the five rungs of the Experience Success Ladder. This helps designers make sense of complex qualitative data in a format they can actually understand and use. Plus, it helps them understand what changes they need to make in order to move from merely comfortable to meaningful.

Slack’s rating based on UX Rings’ report.

Creating Meaningful Experiences

Creating meaningful experiences comes down to understanding and designing for human emotions, wants, and needs. Doing so means taking enough time, looking at the right metrics and using the right tools to understand exactly what an audience is telling you.

Though the process takes considerable time and energy, neglecting to measure audience experience success is a one-way ticket to mediocrity or failure.

Smart UX designers will always recognize the importance of knowing what their audience thinks and feels, and will allow those insights to guide their entire design approach.