hypothesis product design

• Product Management

How to Generate and Validate Product Hypotheses

What is a product hypothesis.

A hypothesis is a testable statement that predicts the relationship between two or more variables. In product development, we generate hypotheses to validate assumptions about customer behavior, market needs, or the potential impact of product changes. These experimental efforts help us refine the user experience and get closer to finding a product-market fit.

Product hypotheses are a key element of data-driven product development and decision-making. Testing them enables us to solve problems more efficiently and remove our own biases from the solutions we put forward.

Here’s an example: ‘If we improve the page load speed on our website (variable 1), then we will increase the number of signups by 15% (variable 2).’ So if we improve the page load speed, and the number of signups increases, then our hypothesis has been proven. If the number did not increase significantly (or not at all), then our hypothesis has been disproven.

In general, product managers are constantly creating and testing hypotheses. But in the context of new product development , hypothesis generation/testing occurs during the validation stage, right after idea screening .

Now before we go any further, let’s get one thing straight: What’s the difference between an idea and a hypothesis?

Idea vs hypothesis

Innovation expert Michael Schrage makes this distinction between hypotheses and ideas – unlike an idea, a hypothesis comes with built-in accountability. “But what’s the accountability for a good idea?” Schrage asks. “The fact that a lot of people think it’s a good idea? That’s a popularity contest.” So, not only should a hypothesis be tested, but by its very nature, it can be tested.

At Railsware, we’ve built our product development services on the careful selection, prioritization, and validation of ideas. Here’s how we distinguish between ideas and hypotheses:

Idea: A creative suggestion about how we might exploit a gap in the market, add value to an existing product, or bring attention to our product. Crucially, an idea is just a thought. It can form the basis of a hypothesis but it is not necessarily expected to be proven or disproven.

• We should get an interview with the CEO of our company published on TechCrunch.
• Why don’t we redesign our website?
• The Coupler.io team should create video tutorials on how to export data from different apps, and publish them on YouTube.
• Why not add a new ‘email templates’ feature to our Mailtrap product?

Hypothesis: A way of framing an idea or assumption so that it is testable, specific, and aligns with our wider product/team/organizational goals.

Examples: 

• If we add a new ‘email templates’ feature to Mailtrap, we’ll see an increase in active usage of our email-sending API.
• Creating relevant video tutorials and uploading them to YouTube will lead to an increase in Coupler.io signups.
• If we publish an interview with our CEO on TechCrunch, 500 people will visit our website and 10 of them will install our product.

Now, it’s worth mentioning that not all hypotheses require testing . Sometimes, the process of creating hypotheses is just an exercise in critical thinking. And the simple act of analyzing your statement tells whether you should run an experiment or not. Remember: testing isn’t mandatory, but your hypotheses should always be inherently testable.

Let’s consider the TechCrunch article example again. In that hypothesis, we expect 500 readers to visit our product website, and a 2% conversion rate of those unique visitors to product users i.e. 10 people. But is that marginal increase worth all the effort? Conducting an interview with our CEO, creating the content, and collaborating with the TechCrunch content team – all of these tasks take time (and money) to execute. And by formulating that hypothesis, we can clearly see that in this case, the drawbacks (efforts) outweigh the benefits. So, no need to test it.

In a similar vein, a hypothesis statement can be a tool to prioritize your activities based on impact. We typically use the following criteria:

• The quality of impact
• The size of the impact
• The probability of impact

This lets us organize our efforts according to their potential outcomes – not the coolness of the idea, its popularity among the team, etc.

Now that we’ve established what a product hypothesis is, let’s discuss how to create one.

Start with a problem statement

Before you jump into product hypothesis generation, we highly recommend formulating a problem statement. This is a short, concise description of the issue you are trying to solve. It helps teams stay on track as they formalize the hypothesis and design the product experiments. It can also be shared with stakeholders to ensure that everyone is on the same page.

The statement can be worded however you like, as long as it’s actionable, specific, and based on data-driven insights or research. It should clearly outline the problem or opportunity you want to address.

Here’s an example: Our bounce rate is high (more than 90%) and we are struggling to convert website visitors into actual users. How might we improve site performance to boost our conversion rate?

How to generate product hypotheses

Now let’s explore some common, everyday scenarios that lead to product hypothesis generation. For our teams here at Railsware, it’s when:

• There’s a problem with an unclear root cause e.g. a sudden drop in one part of the onboarding funnel. We identify these issues by checking our product metrics or reviewing customer complaints.
• We are running ideation sessions on how to reach our goals (increase MRR, increase the number of users invited to an account, etc.)
• We are exploring growth opportunities e.g. changing a pricing plan, making product improvements , breaking into a new market.
• We receive customer feedback. For example, some users have complained about difficulties setting up a workspace within the product. So, we build a hypothesis on how to help them with the setup.

BRIDGES framework for ideation

When we are tackling a complex problem or looking for ways to grow the product, our teams use BRIDGeS – a robust decision-making and ideation framework. BRIDGeS makes our product discovery sessions more efficient. It lets us dive deep into the context of our problem so that we can develop targeted solutions worthy of testing.

Between 2-8 stakeholders take part in a BRIDGeS session. The ideation sessions are usually led by a product manager and can include other subject matter experts such as developers, designers, data analysts, or marketing specialists. You can use a virtual whiteboard such as Figjam or Miro (see our Figma template ) to record each colored note.

In the first half of a BRIDGeS session, participants examine the Benefits, Risks, Issues, and Goals of their subject in the ‘Problem Space.’ A subject is anything that is being described or dealt with; for instance, Coupler.io’s growth opportunities. Benefits are the value that a future solution can bring, Risks are potential issues they might face, Issues are their existing problems, and Goals are what the subject hopes to gain from the future solution. Each descriptor should have a designated color.

After we have broken down the problem using each of these descriptors, we move into the Solution Space. This is where we develop solution variations based on all of the benefits/risks/issues identified in the Problem Space (see the Uber case study for an in-depth example).

In the Solution Space, we start prioritizing those solutions and deciding which ones are worthy of further exploration outside of the framework – via product hypothesis formulation and testing, for example. At the very least, after the session, we will have a list of epics and nested tasks ready to add to our product roadmap.

How to write a product hypothesis statement

Across organizations, product hypothesis statements might vary in their subject, tone, and precise wording. But some elements never change. As we mentioned earlier, a hypothesis statement must always have two or more variables and a connecting factor.

1. Identify variables

Since these components form the bulk of a hypothesis statement, let’s start with a brief definition.

First of all, variables in a hypothesis statement can be split into two camps: dependent and independent. Without getting too theoretical, we can describe the independent variable as the cause, and the dependent variable as the effect . So in the Mailtrap example we mentioned earlier, the ‘add email templates feature’ is the cause i.e. the element we want to manipulate. Meanwhile, ‘increased usage of email sending API’ is the effect i.e the element we will observe.

Independent variables can be any change you plan to make to your product. For example, tweaking some landing page copy, adding a chatbot to the homepage, or enhancing the search bar filter functionality.

Dependent variables are usually metrics. Here are a few that we often test in product development:

• Number of sign-ups
• Number of purchases
• Activation rate (activation signals differ from product to product)
• Number of specific plans purchased
• Feature usage (API activation, for example)
• Number of active users

Bear in mind that your concept or desired change can be measured with different metrics. Make sure that your variables are well-defined, and be deliberate in how you measure your concepts so that there’s no room for misinterpretation or ambiguity.

For example, in the hypothesis ‘Users drop off because they find it hard to set up a project’ variables are poorly defined. Phrases like ‘drop off’ and ‘hard to set up’ are too vague. A much better way of saying it would be: If project automation rules are pre-defined (email sequence to responsible, scheduled tickets creation), we’ll see a decrease in churn. In this example, it’s clear which dependent variable has been chosen and why.

And remember, when product managers focus on delighting users and building something of value, it’s easier to market and monetize it. That’s why at Railsware, our product hypotheses often focus on how to increase the usage of a feature or product. If users love our product(s) and know how to leverage its benefits, we can spend less time worrying about how to improve conversion rates or actively grow our revenue, and more time enhancing the user experience and nurturing our audience.

2. Make the connection

The relationship between variables should be clear and logical. If it’s not, then it doesn’t matter how well-chosen your variables are – your test results won’t be reliable.

To demonstrate this point, let’s explore a previous example again: page load speed and signups.

Through prior research, you might already know that conversion rates are 3x higher for sites that load in 1 second compared to sites that take 5 seconds to load. Since there appears to be a strong connection between load speed and signups in general, you might want to see if this is also true for your product.

Here are some common pitfalls to avoid when defining the relationship between two or more variables:

Relationship is weak. Let’s say you hypothesize that an increase in website traffic will lead to an increase in sign-ups. This is a weak connection since website visitors aren’t necessarily motivated to use your product; there are more steps involved. A better example is ‘If we change the CTA on the pricing page, then the number of signups will increase.’ This connection is much stronger and more direct.

Relationship is far-fetched. This often happens when one of the variables is founded on a vanity metric. For example, increasing the number of social media subscribers will lead to an increase in sign-ups. However, there’s no particular reason why a social media follower would be interested in using your product. Oftentimes, it’s simply your social media content that appeals to them (and your audience isn’t interested in a product).

Variables are co-dependent. Variables should always be isolated from one another. Let’s say we removed the option “Register with Google” from our app. In this case, we can expect fewer users with Google workspace accounts to register. Obviously, it’s because there’s a direct dependency between variables (no registration with Google→no users with Google workspace accounts).

3. Set validation criteria

First, build some confirmation criteria into your statement . Think in terms of percentages (e.g. increase/decrease by 5%) and choose a relevant product metric to track e.g. activation rate if your hypothesis relates to onboarding. Consider that you don’t always have to hit the bullseye for your hypothesis to be considered valid. Perhaps a 3% increase is just as acceptable as a 5% one. And it still proves that a connection between your variables exists.

Secondly, you should also make sure that your hypothesis statement is realistic . Let’s say you have a hypothesis that ‘If we show users a banner with our new feature, then feature usage will increase by 10%.’ A few questions to ask yourself are: Is 10% a reasonable increase, based on your current feature usage data? Do you have the resources to create the tests (experimenting with multiple variations, distributing on different channels: in-app, emails, blog posts)?

Null hypothesis and alternative hypothesis

In statistical research, there are two ways of stating a hypothesis: null or alternative. But this scientific method has its place in hypothesis-driven development too…

Alternative hypothesis: A statement that you intend to prove as being true by running an experiment and analyzing the results. Hint: it’s the same as the other hypothesis examples we’ve described so far.

Example: If we change the landing page copy, then the number of signups will increase.

Null hypothesis: A statement you want to disprove by running an experiment and analyzing the results. It predicts that your new feature or change to the user experience will not have the desired effect.

Example: The number of signups will not increase if we make a change to the landing page copy.

What’s the point? Well, let’s consider the phrase ‘innocent until proven guilty’ as a version of a null hypothesis. We don’t assume that there is any relationship between the ‘defendant’ and the ‘crime’ until we have proof. So, we run a test, gather data, and analyze our findings — which gives us enough proof to reject the null hypothesis and validate the alternative. All of this helps us to have more confidence in our results.

Now that you have generated your hypotheses, and created statements, it’s time to prepare your list for testing.

Prioritizing hypotheses for testing

Not all hypotheses are created equal. Some will be essential to your immediate goal of growing the product e.g. adding a new data destination for Coupler.io. Others will be based on nice-to-haves or small fixes e.g. updating graphics on the website homepage.

Prioritization helps us focus on the most impactful solutions as we are building a product roadmap or narrowing down the backlog . To determine which hypotheses are the most critical, we use the MoSCoW framework. It allows us to assign a level of urgency and importance to each product hypothesis so we can filter the best 3-5 for testing.

MoSCoW is an acronym for Must-have, Should-have, Could-have, and Won’t-have. Here’s a breakdown:

• Must-have – hypotheses that must be tested, because they are strongly linked to our immediate project goals.
• Should-have – hypotheses that are closely related to our immediate project goals, but aren’t the top priority.
• Could-have – hypotheses of nice-to-haves that can wait until later for testing. 
• Won’t-have – low-priority hypotheses that we may or may not test later on when we have more time.

How to test product hypotheses

Once you have selected a hypothesis, it’s time to test it. This will involve running one or more product experiments in order to check the validity of your claim.

The tricky part is deciding what type of experiment to run, and how many. Ultimately, this all depends on the subject of your hypothesis – whether it’s a simple copy change or a whole new feature. For instance, it’s not necessary to create a clickable prototype for a landing page redesign. In that case, a user-wide update would do.

On that note, here are some of the approaches we take to hypothesis testing at Railsware:

A/B testing

A/B or split testing involves creating two or more different versions of a webpage/feature/functionality and collecting information about how users respond to them.

Let’s say you wanted to validate a hypothesis about the placement of a search bar on your application homepage. You could design an A/B test that shows two different versions of that search bar’s placement to your users (who have been split equally into two camps: a control group and a variant group). Then, you would choose the best option based on user data. A/B tests are suitable for testing responses to user experience changes, especially if you have more than one solution to test.

Prototyping

When it comes to testing a new product design, prototyping is the method of choice for many Lean startups and organizations. It’s a cost-effective way of collecting feedback from users, fast, and it’s possible to create prototypes of individual features too. You may take this approach to hypothesis testing if you are working on rolling out a significant new change e.g adding a brand-new feature, redesigning some aspect of the user flow, etc. To control costs at this point in the new product development process , choose the right tools — think Figma for clickable walkthroughs or no-code platforms like Bubble.

Deliveroo feature prototype example

Let’s look at how feature prototyping worked for the food delivery app, Deliveroo, when their product team wanted to ‘explore personalized recommendations, better filtering and improved search’ in 2018. To begin, they created a prototype of the customer discovery feature using web design application, Framer.

One of the most important aspects of this feature prototype was that it contained live data — real restaurants, real locations. For test users, this made the hypothetical feature feel more authentic. They were seeing listings and recommendations for real restaurants in their area, which helped immerse them in the user experience, and generate more honest and specific feedback. Deliveroo was then able to implement this feedback in subsequent iterations.

Asking your users

Interviewing customers is an excellent way to validate product hypotheses. It’s a form of qualitative testing that, in our experience, produces better insights than user surveys or general user research. Sessions are typically run by product managers and involve asking  in-depth interview questions  to one customer at a time. They can be conducted in person or online (through a virtual call center , for instance) and last anywhere between 30 minutes to 1 hour.

Although CustDev interviews may require more effort to execute than other tests (the process of finding participants, devising questions, organizing interviews, and honing interview skills can be time-consuming), it’s still a highly rewarding approach. You can quickly validate assumptions by asking customers about their pain points, concerns, habits, processes they follow, and analyzing how your solution fits into all of that.

Wizard of Oz

The Wizard of Oz approach is suitable for gauging user interest in new features or functionalities. It’s done by creating a prototype of a fake or future feature and monitoring how your customers or test users interact with it.

For example, you might have a hypothesis that your number of active users will increase by 15% if you introduce a new feature. So, you design a new bare-bones page or simple button that invites users to access it. But when they click on the button, a pop-up appears with a message such as ‘coming soon.’

By measuring the frequency of those clicks, you could learn a lot about the demand for this new feature/functionality. However, while these tests can deliver fast results, they carry the risk of backfiring. Some customers may find fake features misleading, making them less likely to engage with your product in the future.

User-wide updates

One of the speediest ways to test your hypothesis is by rolling out an update for all users. It can take less time and effort to set up than other tests (depending on how big of an update it is). But due to the risk involved, you should stick to only performing these kinds of tests on small-scale hypotheses. Our teams only take this approach when we are almost certain that our hypothesis is valid.

For example, we once had an assumption that the name of one of Mailtrap ’s entities was the root cause of a low activation rate. Being an active Mailtrap customer meant that you were regularly sending test emails to a place called ‘Demo Inbox.’ We hypothesized that the name was confusing (the word ‘demo’ implied it was not the main inbox) and this was preventing new users from engaging with their accounts. So, we updated the page, changed the name to ‘My Inbox’ and added some ‘to-do’ steps for new users. We saw an increase in our activation rate almost immediately, validating our hypothesis.

Feature flags

Creating feature flags involves only releasing a new feature to a particular subset or small percentage of users. These features come with a built-in kill switch; a piece of code that can be executed or skipped, depending on who’s interacting with your product.

Since you are only showing this new feature to a selected group, feature flags are an especially low-risk method of testing your product hypothesis (compared to Wizard of Oz, for example, where you have much less control). However, they are also a little bit more complex to execute than the others — you will need to have an actual coded product for starters, as well as some technical knowledge, in order to add the modifiers ( only when… ) to your new coded feature.

Let’s revisit the landing page copy example again, this time in the context of testing.

So, for the hypothesis ‘If we change the landing page copy, then the number of signups will increase,’ there are several options for experimentation. We could share the copy with a small sample of our users, or even release a user-wide update. But A/B testing is probably the best fit for this task. Depending on our budget and goal, we could test several different pieces of copy, such as:

• The current landing page copy
• Copy that we paid a marketing agency 10 grand for
• Generic copy we wrote ourselves, or removing most of the original copy – just to see how making even a small change might affect our numbers.

Remember, every hypothesis test must have a reasonable endpoint. The exact length of the test will depend on the type of feature/functionality you are testing, the size of your user base, and how much data you need to gather. Just make sure that the experiment running time matches the hypothesis scope. For instance, there is no need to spend 8 weeks experimenting with a piece of landing page copy. That timeline is more appropriate for say, a Wizard of Oz feature.

Recording hypotheses statements and test results

Finally, it’s time to talk about where you will write down and keep track of your hypotheses. Creating a single source of truth will enable you to track all aspects of hypothesis generation and testing with ease.

At Railsware, our product managers create a document for each individual hypothesis, using tools such as Coda or Google Sheets. In that document, we record the hypothesis statement, as well as our plans, process, results, screenshots, product metrics, and assumptions.

We share this document with our team and stakeholders, to ensure transparency and invite feedback. It’s also a resource we can refer back to when we are discussing a new hypothesis — a place where we can quickly access information relating to a previous test.

Understanding test results and taking action

The other half of validating product hypotheses involves evaluating data and drawing reasonable conclusions based on what you find. We do so by analyzing our chosen product metric(s) and deciding whether there is enough data available to make a solid decision. If not, we may extend the test’s duration or run another one. Otherwise, we move forward. An experimental feature becomes a real feature, a chatbot gets implemented on the customer support page, and so on.

Something to keep in mind: the integrity of your data is tied to how well the test was executed, so here are a few points to consider when you are testing and analyzing results:

Gather and analyze data carefully. Ensure that your data is clean and up-to-date when running quantitative tests and tracking responses via analytics dashboards. If you are doing customer interviews, make sure to record the meetings (with consent) so that your notes will be as accurate as possible.

Conduct the right amount of product experiments. It can take more than one test to determine whether your hypothesis is valid or invalid. However, don’t waste too much time experimenting in the hopes of getting the result you want. Know when to accept the evidence and move on.

Choose the right audience segment. Don’t cast your net too wide. Be specific about who you want to collect data from prior to running the test. Otherwise, your test results will be misleading and you won’t learn anything new.

Watch out for bias. Avoid confirmation bias at all costs. Don’t make the mistake of including irrelevant data just because it bolsters your results. For example, if you are gathering data about how users are interacting with your product Monday-Friday, don’t include weekend data just because doing so would alter the data and ‘validate’ your hypothesis.

• Not all failed hypotheses should be treated as losses. Even if you didn’t get the outcome you were hoping for, you may still have improved your product. Let’s say you implemented SSO authentication for premium users, but unfortunately, your free users didn’t end up switching to premium plans. In this case, you still added value to the product by streamlining the login process for paying users.
• Yes, taking a hypothesis-driven approach to product development is important. But remember, you don’t have to test everything . Use common sense first. For example, if your website copy is confusing and doesn’t portray the value of the product, then you should still strive to replace it with better copy – regardless of how this affects your numbers in the short term.

Wrapping Up

The process of generating and validating product hypotheses is actually pretty straightforward once you’ve got the hang of it. All you need is a valid question or problem, a testable statement, and a method of validation. Sure, hypothesis-driven development requires more of a time commitment than just ‘giving it a go.’ But ultimately, it will help you tune the product to the wants and needs of your customers.

If you share our data-driven approach to product development and engineering, check out our services page to learn more about how we work with our clients!

Hypothesis-driven product management

Saikiran Chandha

Saikiran Chandha is the CEO and founder of SciSpace — the only integrated research platform to discover, write, publish, and disseminate your research paper. He holds notable experience in research, development, and applications. Forbes, Fortune, and NASDAQ recently captured his entrepreneurial journey.

Join the community

Sign up for free to share your thoughts

Going global: How to implement a product localization strategy

How to conduct productive sprint planning meetings for product managers

Advice from product leaders: Five ways product and design can collaborate effectively

A case study: how to speed up your bug reporting workflow

How to Generate and Validate Product Hypotheses

Every product owner knows that it takes effort to build something that'll cater to user needs. You'll have to make many tough calls if you wish to grow the company and evolve the product so it delivers more value. But how do you decide what to change in the product, your marketing strategy, or the overall direction to succeed? And how do you make a product that truly resonates with your target audience?

There are many unknowns in business, so many fundamental decisions start from a simple "what if?". But they can't be based on guesses, as you need some proof to fill in the blanks reasonably.

Because there's no universal recipe for successfully building a product, teams collect data, do research, study the dynamics, and generate hypotheses according to the given facts. They then take corresponding actions to find out whether they were right or wrong, make conclusions, and most likely restart the process again.

On this page, we thoroughly inspect product hypotheses. We'll go over what they are, how to create hypothesis statements and validate them, and what goes after this step.

What Is a Hypothesis in Product Management?

A hypothesis in product development and product management is a statement or assumption about the product, planned feature, market, or customer (e.g., their needs, behavior, or expectations) that you can put to the test, evaluate, and base your further decisions on . This may, for instance, regard the upcoming product changes as well as the impact they can result in.

A hypothesis implies that there is limited knowledge. Hence, the teams need to undergo testing activities to validate their ideas and confirm whether they are true or false.

Hypotheses guide the product development process and may point at important findings to help build a better product that'll serve user needs. In essence, teams create hypothesis statements in an attempt to improve the offering, boost engagement, increase revenue, find product-market fit quicker, or for other business-related reasons.

It's sort of like an experiment with trial and error, yet, it is data-driven and should be unbiased . This means that teams don't make assumptions out of the blue. Instead, they turn to the collected data, conducted market research , and factual information, which helps avoid completely missing the mark. The obtained results are then carefully analyzed and may influence decision-making.

Such experiments backed by data and analysis are an integral aspect of successful product development and allow startups or businesses to dodge costly startup mistakes .

‍ When do teams create hypothesis statements and validate them? To some extent, hypothesis testing is an ongoing process to work on constantly. It may occur during various product development life cycle stages, from early phases like initiation to late ones like scaling.

In any event, the key here is learning how to generate hypothesis statements and validate them effectively. We'll go over this in more detail later on.

Idea vs. Hypothesis Compared

You might be wondering whether ideas and hypotheses are the same thing. Well, there are a few distinctions.

An idea is simply a suggested proposal. Say, a teammate comes up with something you can bring to life during a brainstorming session or pitches in a suggestion like "How about we shorten the checkout process?". You can jot down such ideas and then consider working on them if they'll truly make a difference and improve the product, strategy, or result in other business benefits. Ideas may thus be used as the hypothesis foundation when you decide to prove a concept.

A hypothesis is the next step, when an idea gets wrapped with specifics to become an assumption that may be tested. As such, you can refine the idea by adding details to it. The previously mentioned idea can be worded into a product hypothesis statement like: "The cart abandonment rate is high, and many users flee at checkout. But if we shorten the checkout process by cutting down the number of steps to only two and get rid of four excessive fields, we'll simplify the user journey, boost satisfaction, and may get up to 15% more completed orders".

A hypothesis is something you can test in an attempt to reach a certain goal. Testing isn't obligatory in this scenario, of course, but the idea may be tested if you weigh the pros and cons and decide that the required effort is worth a try. We'll explain how to create hypothesis statements next.

How to Generate a Hypothesis for a Product

The last thing those developing a product want is to invest time and effort into something that won't bring any visible results, fall short of customer expectations, or won't live up to their needs. Therefore, to increase the chances of achieving a successful outcome and product-led growth , teams may need to revisit their product development approach by optimizing one of the starting points of the process: learning to make reasonable product hypotheses.

If the entire procedure is structured, this may assist you during such stages as the discovery phase and raise the odds of reaching your product goals and setting your business up for success. Yet, what's the entire process like?

• It all starts with identifying an existing problem . Is there a product area that's experiencing a downfall, a visible trend, or a market gap? Are users often complaining about something in their feedback? Or is there something you're willing to change (say, if you aim to get more profit, increase engagement, optimize a process, expand to a new market, or reach your OKRs and KPIs faster)?
• Teams then need to work on formulating a hypothesis . They put the statement into concise and short wording that describes what is expected to achieve. Importantly, it has to be relevant, actionable, backed by data, and without generalizations.
• Next, they have to test the hypothesis by running experiments to validate it (for instance, via A/B or multivariate testing, prototyping, feedback collection, or other ways).
• Then, the obtained results of the test must be analyzed . Did one element or page version outperform the other? Depending on what you're testing, you can look into various merits or product performance metrics (such as the click rate, bounce rate, or the number of sign-ups) to assess whether your prediction was correct.
• Finally, the teams can make conclusions that could lead to data-driven decisions. For example, they can make corresponding changes or roll back a step.

How Else Can You Generate Product Hypotheses?

Such processes imply sharing ideas when a problem is spotted by digging deep into facts and studying the possible risks, goals, benefits, and outcomes. You may apply various MVP tools like (FigJam, Notion, or Miro) that were designed to simplify brainstorming sessions, systemize pitched suggestions, and keep everyone organized without losing any ideas.

Predictive product analysis can also be integrated into this process, leveraging data and insights to anticipate market trends and consumer preferences, thus enhancing decision-making and product development strategies. This approach fosters a more proactive and informed approach to innovation, ensuring products are not only relevant but also resonate with the target audience, ultimately increasing their chances of success in the market.

Besides, you can settle on one of the many frameworks that facilitate decision-making processes , ideation phases, or feature prioritization . Such frameworks are best applicable if you need to test your assumptions and structure the validation process. These are a few common ones if you're looking toward a systematic approach:

• Business Model Canvas (used to establish the foundation of the business model and helps find answers to vitals like your value proposition, finding the right customer segment, or the ways to make revenue);
• Lean Startup framework (the lean startup framework uses a diagram-like format for capturing major processes and can be handy for testing various hypotheses like how much value a product brings or assumptions on personas, the problem, growth, etc.);
• Design Thinking Process (is all about interactive learning and involves getting an in-depth understanding of the customer needs and pain points, which can be formulated into hypotheses followed by simple prototypes and tests).

Need a hand with product development?

Upsilon's team of pros is ready to share our expertise in building tech products.

How to Make a Hypothesis Statement for a Product

Once you've indicated the addressable problem or opportunity and broken down the issue in focus, you need to work on formulating the hypotheses and associated tasks. By the way, it works the same way if you want to prove that something will be false (a.k.a null hypothesis).

If you're unsure how to write a hypothesis statement, let's explore the essential steps that'll set you on the right track.

Step 1: Allocate the Variable Components

Product hypotheses are generally different for each case, so begin by pinpointing the major variables, i.e., the cause and effect . You'll need to outline what you think is supposed to happen if a change or action gets implemented.

Put simply, the "cause" is what you're planning to change, and the "effect" is what will indicate whether the change is bringing in the expected results. Falling back on the example we brought up earlier, the ineffective checkout process can be the cause, while the increased percentage of completed orders is the metric that'll show the effect.

Make sure to also note such vital points as:

• what the problem and solution are;
• what are the benefits or the expected impact/successful outcome;
• which user group is affected;
• what are the risks;
• what kind of experiments can help test the hypothesis;
• what can measure whether you were right or wrong.

Step 2: Ensure the Connection Is Specific and Logical

Mind that generic connections that lack specifics will get you nowhere. So if you're thinking about how to word a hypothesis statement, make sure that the cause and effect include clear reasons and a logical dependency .

Think about what can be the precise and link showing why A affects B. In our checkout example, it could be: fewer steps in the checkout and the removed excessive fields will speed up the process, help avoid confusion, irritate users less, and lead to more completed orders. That's much more explicit than just stating the fact that the checkout needs to be changed to get more completed orders.

Step 3: Decide on the Data You'll Collect

Certainly, multiple things can be used to measure the effect. Therefore, you need to choose the optimal metrics and validation criteria that'll best envision if you're moving in the right direction.

If you need a tip on how to create hypothesis statements that won't result in a waste of time, try to avoid vagueness and be as specific as you can when selecting what can best measure and assess the results of your hypothesis test. The criteria must be measurable and tied to the hypotheses . This can be a realistic percentage or number (say, you expect a 15% increase in completed orders or 2x fewer cart abandonment cases during the checkout phase).

Once again, if you're not realistic, then you might end up misinterpreting the results. Remember that sometimes an increase that's even as little as 2% can make a huge difference, so why make 50% the merit if it's not achievable in the first place?

Step 4: Settle on the Sequence

It's quite common that you'll end up with multiple product hypotheses. Some are more important than others, of course, and some will require more effort and input.

Therefore, just as with the features on your product development roadmap , prioritize your hypotheses according to their impact and importance. Then, group and order them, especially if the results of some hypotheses influence others on your list.

Product Hypothesis Examples

To demonstrate how to formulate your assumptions clearly, here are several more apart from the example of a hypothesis statement given above:

• Adding a wishlist feature to the cart with the possibility to send a gift hint to friends via email will increase the likelihood of making a sale and bring in additional sign-ups.
• Placing a limited-time promo code banner stripe on the home page will increase the number of sales in March.
• Moving up the call to action element on the landing page and changing the button text will increase the click-through rate twice.
• By highlighting a new way to use the product, we'll target a niche customer segment (i.e., single parents under 30) and acquire 5% more leads. 

How to Validate Hypothesis Statements: The Process Explained

There are multiple options when it comes to validating hypothesis statements. To get appropriate results, you have to come up with the right experiment that'll help you test the hypothesis. You'll need a control group or people who represent your target audience segments or groups to participate (otherwise, your results might not be accurate).

‍ What can serve as the experiment you may run? Experiments may take tons of different forms, and you'll need to choose the one that clicks best with your hypothesis goals (and your available resources, of course). The same goes for how long you'll have to carry out the test (say, a time period of two months or as little as two weeks). Here are several to get you started.

Feedback and User Testing

Talking to users, potential customers, or members of your own online startup community can be another way to test your hypotheses. You may use surveys, questionnaires, or opt for more extensive interviews to validate hypothesis statements and find out what people think. This assumption validation approach involves your existing or potential users and might require some additional time, but can bring you many insights.

Conduct A/B or Multivariate Tests

One of the experiments you may develop involves making more than one version of an element or page to see which option resonates with the users more. As such, you can have a call to action block with different wording or play around with the colors, imagery, visuals, and other things.

To run such split experiments, you can apply tools like VWO that allows to easily construct alternative designs and split what your users see (e.g., one half of the users will see version one, while the other half will see version two). You can track various metrics and apply heatmaps, click maps, and screen recordings to learn more about user response and behavior. Mind, though, that the key to such tests is to get as many users as you can give the tests time. Don't jump to conclusions too soon or if very few people participated in your experiment.

Build Prototypes and Fake Doors

Demos and clickable prototypes can be a great way to save time and money on costly feature or product development. A prototype also allows you to refine the design. However, they can also serve as experiments for validating hypotheses, collecting data, and getting feedback.

For instance, if you have a new feature in mind and want to ensure there is interest, you can utilize such MVP types as fake doors . Make a short demo recording of the feature and place it on your landing page to track interest or test how many people sign up.

Usability Testing

Similarly, you can run experiments to observe how users interact with the feature, page, product, etc. Usually, such experiments are held on prototype testing platforms with a focus group representing your target visitors. By showing a prototype or early version of the design to users, you can view how people use the solution, where they face problems, or what they don't understand. This may be very helpful if you have hypotheses regarding redesigns and user experience improvements before you move on from prototype to MVP development.

You can even take it a few steps further and build a barebone feature version that people can really interact with, yet you'll be the one behind the curtain to make it happen. There were many MVP examples when companies applied Wizard of Oz or concierge MVPs to validate their hypotheses.

Or you can actually develop some functionality but release it for only a limited number of people to see. This is referred to as a feature flag , which can show really specific results but is effort-intensive. 

What Comes After Hypothesis Validation?

Analysis is what you move on to once you've run the experiment. This is the time to review the collected data, metrics, and feedback to validate (or invalidate) the hypothesis.

You have to evaluate the experiment's results to determine whether your product hypotheses were valid or not. For example, if you were testing two versions of an element design, color scheme, or copy, look into which one performed best.

It is crucial to be certain that you have enough data to draw conclusions, though, and that it's accurate and unbiased . Because if you don't, this may be a sign that your experiment needs to be run for some additional time, be altered, or held once again. You won't want to make a solid decision based on uncertain or misleading results, right?

• If the hypothesis was supported , proceed to making corresponding changes (such as implementing a new feature, changing the design, rephrasing your copy, etc.). Remember that your aim was to learn and iterate to improve.
• If your hypothesis was proven false , think of it as a valuable learning experience. The main goal is to learn from the results and be able to adjust your processes accordingly. Dig deep to find out what went wrong, look for patterns and things that may have skewed the results. But if all signs show that you were wrong with your hypothesis, accept this outcome as a fact, and move on. This can help you make conclusions on how to better formulate your product hypotheses next time. Don't be too judgemental, though, as a failed experiment might only mean that you need to improve the current hypothesis, revise it, or create a new one based on the results of this experiment, and run the process once more.

On another note, make sure to record your hypotheses and experiment results . Some companies use CRMs to jot down the key findings, while others use something as simple as Google Docs. Either way, this can be your single source of truth that can help you avoid running the same experiments or allow you to compare results over time.

Have doubts about how to bring your product to life?

Upsilon's team of pros can help you build a product most optimally.

Final Thoughts on Product Hypotheses

The hypothesis-driven approach in product development is a great way to avoid uncalled-for risks and pricey mistakes. You can back up your assumptions with facts, observe your target audience's reactions, and be more certain that this move will deliver value.

However, this only makes sense if the validation of hypothesis statements is backed by relevant data that'll allow you to determine whether the hypothesis is valid or not. By doing so, you can be certain that you're developing and testing hypotheses to accelerate your product management and avoiding decisions based on guesswork.

Certainly, a failed experiment may bring you just as much knowledge and findings as one that succeeds. Teams have to learn from their mistakes, boost their hypothesis generation and testing knowledge , and make improvements according to the results of their experiments. This is an ongoing process, of course, as no product can grow if it isn't iterated and improved.

If you're only planning to or are currently building a product, Upsilon can lend you a helping hand. Our team has years of experience providing product development services for growth-stage startups and building MVPs for early-stage businesses , so you can use our expertise and knowledge to dodge many mistakes. Don't be shy to contact us to discuss your needs! 

How to Make an MVP Roadmap

How Much Does It Cost to Build an AI Solution in 2024?

Top 30 Financial Terms for Startups

Never miss an update.

Ask a question

Start a discussion.

• Atlassian logo Jira Product Discovery
• Jira Service Desk Jira Service Management
• Confluence Confluence
• Trello Trello
• Atlassian logo Atlassian Guard

Community resources

• Announcements
• Documentation and support

Atlassian Community Events

• Atlassian University
• groups-icon Welcome Center
• groups-icon Featured Groups
• groups-icon Product Groups
• groups-icon Regional Groups
• groups-icon Industry Groups
• groups-icon Community Groups
• Learning Paths
• Certifications
• Courses by Product
• Live learning
• Local meet ups
• Community led conferences

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

• Featured Groups
• App Central

How to create and validate hypotheses for product improvement

Was this helpful?

Halyna Kudlak _SaaSJet_

About this author

Marketing Team Lead

2 accepted answers

46 total posts

• +15 more...
• atlassian-marketplace
• Community Guidelines
• Privacy policy
• Notice at Collection
• Terms of use
• © 2024 Atlassian

InVisionApp, Inc.

Inside Design

5 steps to a hypothesis-driven design process

  •   mar 22, 2018.

S ay you’re starting a greenfield project, or you’re redesigning a legacy app. The product owner gives you some high-level goals. Lots of ideas and questions are in your mind, and you’re not sure where to start.

Hypothesis-driven design will help you navigate through a unknown space so you can come out at the end of the process with actionable next steps.

Ready? Let’s dive in.

Step 1: Start with questions and assumptions

On the first day of the project, you’re curious about all the different aspects of your product. “How could we increase the engagement on the homepage? ” “ What features are important for our users? ”

Related: 6 ways to speed up and improve your product design process

To reduce risk, I like to take some time to write down all the unanswered questions and assumptions. So grab some sticky notes and write all your questions down on the notes (one question per note).

I recommend that you use the How Might We technique from IDEO to phrase the questions and turn your assumptions into questions. It’ll help you frame the questions in a more open-ended way to avoid building the solution into the statement prematurely. For example, you have an idea that you want to make riders feel more comfortable by showing them how many rides the driver has completed. You can rephrase the question to “ How might we ensure rider feel comfortable when taking ride, ” and leave the solution part out to the later step.

“It’s easy to come up with design ideas, but it’s hard to solve the right problem.”

It’s even more valuable to have your team members participate in the question brainstorming session. Having diverse disciplines in the room always brings fresh perspectives and leads to a more productive conversation.

Step 2: Prioritize the questions and assumptions

Now that you have all the questions on sticky notes, organize them into groups to make it easier to review them. It’s especially helpful if you can do the activity with your team so you can have more input from everybody.

When it comes to choosing which question to tackle first, think about what would impact your product the most or what would bring the most value to your users.

If you have a big group, you can Dot Vote to prioritize the questions. Here’s how it works: Everyone has three dots, and each person gets to vote on what they think is the most important question to answer in order to build a successful product. It’s a common prioritization technique that’s also used in the Sprint book by Jake Knapp —he writes, “ The prioritization process isn’t perfect, but it leads to pretty good decisions and it happens fast. ”

Related: Go inside design at Google Ventures

Step 3: Turn them into hypotheses

After the prioritization, you now have a clear question in mind. It’s time to turn the question into a hypothesis. Think about how you would answer the question.

Let’s continue the previous ride-hailing service example. The question you have is “ How might we make people feel safe and comfortable when using the service? ”

Based on this question, the solutions can be:

• Sharing the rider’s location with friends and family automatically
• Displaying more information about the driver
• Showing feedback from previous riders

Now you can combine the solution and question, and turn it into a hypothesis. Hypothesis is a framework that can help you clearly define the question and solution, and eliminate assumption.

From Lean UX

We believe that [ sharing more information about the driver’s experience and stories ] For [ the riders ] Will [ make riders feel more comfortable and connected throughout the ride ]

4. Develop an experiment and testing the hypothesis

Develop an experiment so you can test your hypothesis. Our test will follow the scientific methods, so it’s subject to collecting empirical and measurable evidence in order to obtain new knowledge. In other words, it’s crucial to have a measurable outcome for the hypothesis so we can determine whether it has succeeded or failed.

There are different ways you can create an experiment, such as interview, survey , landing page validation, usability testing, etc. It could also be something that’s built into the software to get quantitative data from users. Write down what the experiment will be, and define the outcomes that determine whether the hypothesis is valids. A well-defined experiment can validate/invalidate the hypothesis.

In our example, we could define the experiment as “ We will run X studies to show more information about a driver (number of ride, years of experience), and ask follow-up questions to identify the rider’s emotion associated with this ride (safe, fun, interesting, etc.). We will know the hypothesis is valid when we get more than 70% identify the ride as safe or comfortable. ”

After defining the experiment, it’s time to get the design done. You don’t need to have every design detail thought through. You can focus on designing what is needed to be tested.

When the design is ready, you’re ready to run the test. Recruit the users you want to target , have a time frame, and put the design in front of the users.

5. Learn and build

You just learned that the result was positive and you’re excited to roll out the feature. That’s great! If the hypothesis failed, don’t worry—you’ll be able to gain some insights from that experiment. Now you have some new evidence that you can use to run your next experiment. In each experiment, you’ll learn something new about your product and your customers.

“Design is a never-ending process.”

What other information can you show to make riders feel safe and comfortable? That can be your next hypothesis. You now have a feature that’s ready to be built, and a new hypothesis to be tested.

Principles from from The Lean Startup

We often assume that we understand our users and know what they want. It’s important to slow down and take a moment to understand the questions and assumptions we have about our product.

After testing each hypothesis, you’ll get a clearer path of what’s most important to the users and where you need to dig deeper. You’ll have a clear direction for what to do next.

by Sylvia Lai

Sylvia Lai helps startup and enterprise solve complex problems through design thinking and user-centered design methodologies at Pivotal Labs . She is the biggest advocate for the users, making sure their voices are heard is her number one priority. Outside of work, she loves mentoring other designers through one-on-one conversation. Connect with her through LinkedIn or Twitter .

Collaborate in real time on a digital whiteboard Try Freehand

Get awesome design content in your inbox each week, give it a try—it only takes a click to unsubscribe., thanks for signing up, you should have a thank you gift in your inbox now-and you’ll hear from us again soon, get started designing better. faster. together. and free forever., give it a try. nothing’s holding you back..

4.7 STARS ON G2

Analyze your mobile app for free. No credit card required. 100k sessions.

SHARE THIS POST

Product best practices

Product hypothesis - a guide to create meaningful hypotheses.

13 December, 2023

Growth Manager

Data-driven development is no different than a scientific experiment. You repeatedly form hypotheses, test them, and either implement (or reject) them based on the results. It’s a proven system that leads to better apps and happier users.

Let’s get started.

What is a product hypothesis?

A product hypothesis is an educated guess about how a change to a product will impact important metrics like revenue or user engagement. It's a testable statement that needs to be validated to determine its accuracy.

The most common format for product hypotheses is “If… than…”:

“If we increase the font size on our homepage, then more customers will convert.”

“If we reduce form fields from 5 to 3, then more users will complete the signup process.”

At UXCam, we believe in a data-driven approach to developing product features. Hypotheses provide an effective way to structure development and measure results so you can make informed decisions about how your product evolves over time.

Take PlaceMakers , for example.

PlaceMakers faced challenges with their app during the COVID-19 pandemic. Due to supply chain shortages, stock levels were not being updated in real-time, causing customers to add unavailable products to their baskets. The team added a “Constrained Product” label, but this caused sales to plummet.

The team then turned to UXCam’s session replays and heatmaps to investigate, and hypothesized that their messaging for constrained products was too strong. The team redesigned the messaging with a more positive approach, and sales didn’t just recover—they doubled.

Types of product hypothesis

1. counter-hypothesis.

A counter-hypothesis is an alternative proposition that challenges the initial hypothesis. It’s used to test the robustness of the original hypothesis and make sure that the product development process considers all possible scenarios. 

For instance, if the original hypothesis is “Reducing the sign-up steps from 3 to 1 will increase sign-ups by 25% for new visitors after 1,000 visits to the sign-up page,” a counter-hypothesis could be “Reducing the sign-up steps will not significantly affect the sign-up rate.

2. Alternative hypothesis

An alternative hypothesis predicts an effect in the population. It’s the opposite of the null hypothesis, which states there’s no effect. 

For example, if the null hypothesis is “improving the page load speed on our mobile app will not affect the number of sign-ups,” the alternative hypothesis could be “improving the page load speed on our mobile app will increase the number of sign-ups by 15%.”

3. Second-order hypothesis

Second-order hypotheses are derived from the initial hypothesis and provide more specific predictions. 

For instance, “if the initial hypothesis is Improving the page load speed on our mobile app will increase the number of sign-ups,” a second-order hypothesis could be “Improving the page load speed on our mobile app will increase the number of sign-ups.”

Why is a product hypothesis important?

Guided product development.

A product hypothesis serves as a guiding light in the product development process. In the case of PlaceMakers, the product owner’s hypothesis that users would benefit from knowing the availability of items upfront before adding them to the basket helped their team focus on the most critical aspects of the product. It ensured that their efforts were directed towards features and improvements that have the potential to deliver the most value. 

Improved efficiency

Product hypotheses enable teams to solve problems more efficiently and remove biases from the solutions they put forward. By testing the hypothesis, PlaceMakers aimed to improve efficiency by addressing the issue of stock levels not being updated in real-time and customers adding unavailable products to their baskets.

Risk mitigation

By validating assumptions before building the product, teams can significantly reduce the risk of failure. This is particularly important in today’s fast-paced, highly competitive business environment, where the cost of failure can be high.

Validating assumptions through the hypothesis helped mitigate the risk of failure for PlaceMakers, as they were able to identify and solve the issue within a three-day period.

Data-driven decision-making

Product hypotheses are a key element of data-driven product development and decision-making. They provide a solid foundation for making informed, data-driven decisions, which can lead to more effective and successful product development strategies. 

The use of UXCam's Session Replay and Heatmaps features provided valuable data for data-driven decision-making, allowing PlaceMakers to quickly identify the problem and revise their messaging approach, leading to a doubling of sales.

How to create a great product hypothesis

Map important user flows

Identify any bottlenecks

Look for interesting behavior patterns

Turn patterns into hypotheses

Step 1 - Map important user flows

A good product hypothesis starts with an understanding of how users more around your product—what paths they take, what features they use, how often they return, etc. Before you can begin hypothesizing, it’s important to map out key user flows and journey maps that will help inform your hypothesis.

To do that, you’ll need to use a monitoring tool like UXCam .

UXCam integrates with your app through a lightweight SDK and automatically tracks every user interaction using tagless autocapture. That leads to tons of data on user behavior that you can use to form hypotheses.

At this stage, there are two specific visualizations that are especially helpful:

Funnels : Funnels are great for identifying drop off points and understanding which steps in a process, transition or journey lead to success.

In other words, you’re using these two tools to define key in-app flows and to measure the effectiveness of these flows (in that order).

Average time to conversion in highlights bar.

Step 2 - Identify any bottlenecks

Once you’ve set up monitoring and have started collecting data, you’ll start looking for bottlenecks—points along a key app flow that are tripping users up. At every stage in a funnel, there’s going to be dropoffs, but too many dropoffs can be a sign of a problem.

UXCam makes it easy to spot dropoffs by displaying them visually in every funnel. While there’s no benchmark for when you should be concerned, anything above a 10% dropoff could mean that further investigation is needed.

How do you investigate? By zooming in.

Step 3 - Look for interesting behavior patterns

At this stage, you’ve noticed a concerning trend and are zooming in on individual user experiences to humanize the trend and add important context.

The best way to do this is with session replay tools and event analytics. With a tool like UXCam, you can segment app data to isolate sessions that fit the trend. You can then investigate real user sessions by watching videos of their experience or by looking into their event logs. This helps you see exactly what caused the behavior you’re investigating.

For example, let’s say you notice that 20% of users who add an item to their cart leave the app about 5 minutes later. You can use session replay to look for the behavioral patterns that lead up to users leaving—such as how long they linger on a certain page or if they get stuck in the checkout process.

Step 4 - Turn patterns into hypotheses

Once you’ve checked out a number of user sessions, you can start to craft a product hypothesis.

This usually takes the form of an “If… then…” statement, like:

“If we optimize the checkout process for mobile users, then more customers will complete their purchase.”

These hypotheses can be tested using A/B testing and other user research tools to help you understand if your changes are having an impact on user behavior.

Product hypothesis emphasizes the importance of formulating clear and testable hypotheses when developing a product. It highlights that a well-defined hypothesis can guide the product development process, align stakeholders, and minimize uncertainty.

UXCam arms product teams with all the tools they need to form meaningful hypotheses that drive development in a positive direction. Put your app’s data to work and start optimizing today— sign up for a free account .

You might also be interested in these;

Product experimentation framework for mobile product teams

7 Best AB testing tools for mobile apps

A practical guide to product experimentation

5 Best product experimentation tools & software

How to use data to challenge the HiPPO

Ardent technophile exploring the world of mobile app product management at UXCam.

Get the latest from UXCam

Stay up-to-date with UXCam's latest features, insights, and industry news for an exceptional user experience.

Related articles

5 best user behavioral analytics tools for mobile app teams

Behavioral analytics offers deep insights into user interaction. Learn about the best behavioral analytics tools to optimize the user experience on your mobile...

Jonas Kurzweg

Growth Lead

How UXCam compares to Amplitude

Amplitude is a data analytics platform providing insights about user behavior, UXCam is a dedicated mobile analytics solution, what's the best decision for...

Hannah Squire

Content Manager

How to Find Active Users of an App

Discover proven strategies to identify and engage active users for your app, boost retention, drive growth, and maximize your app's...

From Theory to Practice: The Role of Hypotheses in Product Development

This article explores why working with hypotheses is not just a quirky aspect of product management but an essential practice in the field.

Let's dive into what a hypothesis actually is when it comes to crafting a standout product. Think of a hypothesis as your project's leading detective, uncovering the mysteries of user behavior, pinpointing problem sources, and suggesting solutions to not just improve your product, but to make it a market sensation.

Consider a straightforward example. Imagine you have a pizza delivery app. You hypothesize that enlarging the "Order" button will lead to more orders. This is your hypothesis! You're assuming that a change in X (the button size) will result in outcome Y (increased orders).

Or, suppose you plan to refine the product filtering on your e-commerce site, enabling users to find what they need faster. Your hypothesis might be, "Implementing a new filtering system by price and brand will boost purchase conversions."

In product development, a hypothesis isn't just a guess or an idea; it's a data-driven assumption about how certain changes can achieve desired outcomes. It serves as a map, guiding you through the ocean of user needs and transforming your product into a true gem.

So, don't hesitate to formulate hypotheses, test them through experiments and data analysis—you'll surely navigate your product towards success!

Hypothesis vs. Simple Statement: Understanding the Nuance

Let's clear up the difference between a hypothesis and a simple statement, in a way that both you and your grandmother can grasp.

A simple statement is like saying, "My cat loves milk." It seems like an obvious fact. But a hypothesis is more like a weather forecast: "If today is sunny, my cat will be happier." Here, there's an assumption and a link between two phenomena.

For example, a statement might be, "My grandmother enjoys knitting sweaters." This is a fact of life.

However, a hypothesis could be, "If I help my grandmother with household chores every day, she will be happier." Here, there's a presumption that active participation will lead to my grandmother's happiness.

See the difference? A hypothesis attempts to predict and explain the relationship between phenomena, while a statement just provides information about something. Remember, to develop your product like a boss, you need to craft compelling hypotheses and test them in reality!

Why Formulate Hypotheses in Product Development

Imagine you have an idea for an app that makes it faster and more convenient for people to catch up on news. You could formulate a hypothesis that adding a feature to alert users about significant events will increase app usage. This is your working hypothesis!

Here's why it's so crucial. Hypotheses help us understand which product modifications can make it even better. They allow us to test our assumptions and adapt our product development strategy on the fly.

Moreover, by testing hypotheses in the early stages of development, we can save time and money by identifying potential problems and fixing them before the product hits the market.

Formulating Hypotheses: A Step-by-Step Approach

Step 1: identifying key problems or opportunities for verification.

The first step is akin to treasure hunting in the business realm. You need to unearth the primary issues or potential opportunities that will form the basis of your hypotheses.

For instance, imagine you're developing a fitness app and users are reporting that the interface is too complex. The problem is already highlighted, and your task is to formulate and test a hypothesis!

The best way to proceed is by gathering data. Embrace your inner detective and delve into user data, analytics, reviews, and more. Remember, everything must be fact-based to ensure your hypothesis isn't mere speculation.

Once you've pinpointed your targets and problems, you're ready to craft a hypothesis. It should be specific, measurable, and include an anticipated outcome, such as "Simplifying the app's interface will increase user satisfaction and the time spent using it."

Step 2: Crafting Your Hypothesis: How to Structure It

Clarity comes first - start with a clear formulation of your hypothesis. If you're developing a financial management app, your hypothesis might be, "Introducing a feature for upcoming payment alerts will enhance user engagement and reduce the number of late payments."

Measurability is key - decide how you will measure the success of your hypothesis. For example, you could track an increase in user activity post-notification implementation.

Hypothesis vs. Goal - understand that a hypothesis is not a goal! It's an assumption about the outcomes of a change that can be tested, whereas a goal is the ultimate outcome you aim to achieve.

Consider alternatives and limitations - don’t forget to account for alternative scenarios and potential limitations, such as other factors that could impact your success metrics.

Testing is where the fun begins - after formulating your hypothesis, launch an experiment, collect data, and analyze the outcomes. If the hypothesis is disproven, it’s still valuable insight for future research.

Step 3: Defining Key Metrics and Experiments to Test Your Hypothesis

Before diving into the verification of product development hypotheses, let’s talk about how to define key metrics and design experiments for their testing. Imagine standing before a door of opportunities, behind which lie the answers to making your product even better. Ready for the challenge?

First, determine how to measure the success of your product change. These key metrics should be specific, measurable, and tied to your product's goals. For example, if your hypothesis is about improving user attraction, a key metric might be the conversion rate from the homepage to the sign-up page.

Hypothesis example: Adding video reviews of products will increase the conversion rate on the product page.

Key metric: Increased time spent on the product page.

With your key metrics in hand, it's time to unleash your creativity and devise experiments to test your hypothesis. Experiments should be structured, controlled, and capable of providing a definitive result on whether the hypothesis holds.

Experiment example:

Hypothesis: Simplifying the checkout process will increase purchase conversions.

Experiment: Split users into two groups—one with a simplified checkout process and the other with the standard process. Measure the purchase conversion rate in each group.

Typical Mistakes in Working with Hypotheses

The importance of specificity in hypotheses.

Let's discuss why specificity is crucial in the world of product development hypothesis formulation. Imagine trying to solve a puzzle, but instead of clear instructions, you're overwhelmed with numerous ambiguous paths. Intriguing, yes, but where to go and what to do? Similarly, vague hypotheses create confusion and can lead us nowhere.

Formulating a vague hypothesis is like playing the lottery with your product. You're giving it a chance to succeed, but without a clear plan, it's more luck than strategy. Knowing your direction ensures you move forward confidently rather than wandering in the dark.

For example:

Vague Hypothesis: "Improving the interface will increase user satisfaction."

This hypothesis leaves too many questions unanswered: What exactly should be improved in the interface? Which specific changes will lead to increased satisfaction?

To make a hypothesis clear and specific, ask yourself several questions. What do we want to change? How will this change affect users? How will we measure the effect? Let's be careful architects building dreams from the bricks under our feet, not explorers without a map in a land of unknown opportunities.

For instance:

Specific Hypothesis: "Increasing the size and contrast of the 'Order' button on the product page will increase conversion by 20% within a month."

This hypothesis is precise, measurable, and clearly defines the goal.

Avoiding Ill-Conceived Experiments: How to Save Resources

Let's talk about how we can avoid the pitfalls of ill-conceived experiments that can lead to wasted time, money, and effort. Imagine embarking on a journey without knowing your destination or how to get there—a purposeless wandering in a sea of opportunities. Let's be more goal-oriented!

Neglecting careful planning of experiments risks wasting resources. Ill-conceived experiments often end up as a drain on the evergreen garden of new ideas, potentially leading to a situation where effort is high but results don't meet expectations.

Ill-Conceived Experiment: Changing the "Buy" button to a random shade of the rainbow without data analysis.

Result: No change in conversion or, worse, a decrease.

How to Avoid Wasting Resources?

To dodge this trap, meticulously plan each experiment before launch. Set clear objectives, define expected outcomes, and identify key metrics to measure success. Be like detectives with a detailed plan of action before starting an investigation.

Well-Planned Experiment: Changing the text on the "Try for Free" button to "Start Free and Access All Features for 7 Days."

Result: An increase in users registering for the trial period.

Ignoring Data: The Importance of Basing Hypotheses on Facts

Imagine building a ship without considering a sea map—you might get lost in the ocean of possibilities. Let's dive into the world of data and discover why it's our invaluable treasure!

Why Base Hypotheses on Facts?

Ignoring data risks creating hypotheses based on assumptions and intuition, which could be far from reality. Data are our reliable compasses in the world of change. They help us understand where to go, which paths to take, and how to avoid pitfalls.

Data-Based Hypothesis: "Increasing the number of product recommendations based on user preferences will increase the average order value by 15%."

This hypothesis is grounded in real shopping preferences, making it more likely to succeed.

To successfully work with hypotheses, carefully analyze data. Use information about user behavior, feedback, and results from past experiments. Be like archaeologists sifting through traces of the past to formulate fact-based hypotheses, not guesses.

Data-Based Hypothesis: "Reducing the number of steps to checkout based on analysis of customer behavior will increase conversion at the checkout stage."

This hypothesis stems from specific data on user difficulties during the purchase phase.

Tools for Working with Hypotheses

Popular online tools and platforms for formulating and testing hypotheses.

Let's explore a few popular online tools that will become your faithful allies in innovating and enhancing user experience. Ready for the adventure? Let's dive in!

Optimizely is a convenient tool for A/B testing and personalization, enabling you to test different page versions, design elements, and product functionalities.

Usage example: Suppose you hypothesize that changing the "Buy" button color will increase conversion rates. With Optimizely, you can easily set up an A/B test and compare which variant truly attracts more customers.

Google Optimize is a free tool from Google for A/B testing, helping you conduct experiments with web pages and analyze their effectiveness.

Usage example: If you want to test the hypothesis that altering the homepage headline will improve user retention, Google Optimize allows you to set up the test and monitor changes in user behavior.

Hotjar offers tools for analyzing user behavior on your site, including heatmaps, session recordings, and surveys.

Usage example: Imagine you hypothesize that users can't find the "Call Us" button due to its invisibility on the page. Hotjar enables you to analyze user behavior and either confirm or refute your hypothesis.

Recommendations for Choosing Tools Based on Team Needs

Choosing tools is like picking out a suit—it needs to fit both your size and style. Let's figure out how to determine which tool is right for your team!

For teams passionate about analytics and experiments:

Recommendation: A/B testing tools like Optimizely or Google Optimize are suitable for those eager to put every hypothesis to the test and extract valuable data from each experiment.

Usage example: Your e-commerce team suspects that changing the order of product display on the homepage will increase conversion rates. Using Optimizely, you conduct an A/B test to find the optimal arrangement.

For teams focused on user experience:

Recommendation: Behavior analysis tools like Hotjar will help you understand how users interact with your product and where issues arise.

Usage example: Through Hotjar, your team discovers that most users don't scroll to the end of the service description page. This insight becomes the basis for a hypothesis about the need for brevity and clarity in the text.

For teams emphasizing design and visual experience:

Recommendation: Prototyping and design tools like Figma or Adobe XD can be an excellent choice for teams working on improving user interfaces.

Usage example: After receiving feedback that site navigation is cumbersome, your team uses Figma to create a new prototype with an improved structure and navigation.

Wrapping It Up

So, why is effective hypothesis management the key to success in the product world? Hypotheses are not just assumptions; they are a powerful tool that helps teams align, move forward, and achieve success. Properly managing hypotheses reduces risks, speeds up product development, and leads to more targeted outcomes. Hypotheses are your guide in the world of endless possibilities for development and improvement. Remember, diligent work, patience, and data analysis will help you unlock new horizons and bring the most ambitious ideas to life. Let your product development journey be paved with valuable hypotheses and successful solutions!

If you need assistance with setting up analytics or developing a data collection flow from various analytical tools, don't hesitate to book a free call with our CTO or leave your contact details on our website, and we will surely help you address your concerns!

Last updated 5 months ago

Integrations

What's new?

In-Product Prompts

Participant Management

Interview Studies

Prototype Testing

Card Sorting

Tree Testing

Live Website Testing

Automated Reports

Templates Gallery

Choose from our library of pre-built mazes to copy, customize, and share with your own users

Browse all templates

Financial Services

Tech & Software

Product Designers

Product Managers

User Researchers

By use case

Concept & Idea Validation

Wireframe & Usability Test

Content & Copy Testing

Feedback & Satisfaction

Content Hub

Educational resources for product, research and design teams

Explore all resources

Question Bank

Maze Research Success Hub

Guides & Reports

Help Center

Future of User Research Report

The Optimal Path Podcast

User Research

Mar 21, 2024

Creating a research hypothesis: How to formulate and test UX expectations

A research hypothesis helps guide your UX research with focused predictions you can test and learn from. Here’s how to formulate your own hypotheses.

Armin Tanovic

All great products were once just thoughts—the spark of an idea waiting to be turned into something tangible.

A research hypothesis in UX is very similar. It’s the starting point for your user research; the jumping off point for your product development initiatives.

Formulating a UX research hypothesis helps guide your UX research project in the right direction, collect insights, and evaluate not only whether an idea is worth pursuing, but how to go after it.

In this article, we’ll cover what a research hypothesis is, how it's relevant to UX research, and the best formula to create your own hypothesis and put it to the test.

Test your hypothesis with Maze

Maze lets you validate your design and test research hypotheses to move forward with authentic user insights.

What defines a research hypothesis?

A research hypothesis is a statement or prediction that needs testing to be proven or disproven.

Let’s say you’ve got an inkling that making a change to a feature icon will increase the number of users that engage with it—with some minor adjustments, this theory becomes a research hypothesis: “ Adjusting Feature X’s icon will increase daily average users by 20% ”.

A research hypothesis is the starting point that guides user research . It takes your thought and turns it into something you can quantify and evaluate. In this case, you could conduct usability tests and user surveys, and run A/B tests to see if you’re right—or, just as importantly, wrong .

A good research hypothesis has three main features:

• Specificity: A hypothesis should clearly define what variables you’re studying and what you expect an outcome to be, without ambiguity in its wording
• Relevance: A research hypothesis should have significance for your research project by addressing a potential opportunity for improvement
• Testability: Your research hypothesis must be able to be tested in some way such as empirical observation or data collection

What is the difference between a research hypothesis and a research question?

Research questions and research hypotheses are often treated as one and the same, but they’re not quite identical.

A research hypothesis acts as a prediction or educated guess of outcomes , while a research question poses a query on the subject you’re investigating. Put simply, a research hypothesis is a statement, whereas a research question is (you guessed it) a question.

For example, here’s a research hypothesis: “ Implementing a navigation bar on our dashboard will improve customer satisfaction scores by 10%. ”

This statement acts as a testable prediction. It doesn’t pose a question, it’s a prediction. Here’s what the same hypothesis would look like as a research question: “ Will integrating a navigation bar on our dashboard improve customer satisfaction scores? ”

The distinction is minor, and both are focused on uncovering the truth behind the topic, but they’re not quite the same.

Why do you use a research hypothesis in UX?

Research hypotheses in UX are used to establish the direction of a particular study, research project, or test. Formulating a hypothesis and testing it ensures the UX research you conduct is methodical, focused, and actionable. It aids every phase of your research process , acting as a north star that guides your efforts toward successful product development .

Typically, UX researchers will formulate a testable hypothesis to help them fulfill a broader objective, such as improving customer experience or product usability. They’ll then conduct user research to gain insights into their prediction and confirm or reject the hypothesis.

A proven or disproven hypothesis will tell if your prediction is right, and whether you should move forward with your proposed design—or if it's back to the drawing board.

Formulating a hypothesis can be helpful in anything from prototype testing to idea validation, and design iteration. Put simply, it’s one of the first steps in conducting user research.

Whether you’re in the initial stages of product discovery for a new product, a single feature, or conducting ongoing research, a strong hypothesis presents a clear purpose and angle for your research It also helps understand which user research methodology to use to get your answers.

What are the types of research hypotheses?

Not all hypotheses are built the same—there are different types with different objectives. Understanding the different types enables you to formulate a research hypothesis that outlines the angle you need to take to prove or disprove your predictions.

Here are some of the different types of hypotheses to keep in mind.

Null and alternative hypotheses

While a normal research hypothesis predicts that a specific outcome will occur based upon a certain change of variables, a null hypothesis predicts that no difference will occur when you introduce a new condition.

By that reasoning, a null hypothesis would be:

• Adding a new CTA button to the top of our homepage will make no difference in conversions

Null hypotheses are useful because they help outline what your test or research study is trying to dis prove, rather than prove, through a research hypothesis.

An alternative hypothesis states the exact opposite of a null hypothesis. It proposes that a certain change will occur when you introduce a new condition or variable. For example:

• Adding a CTA button to the top of our homepage will cause a difference in conversion rates

Simple hypotheses and complex hypotheses

A simple hypothesis is a prediction that includes only two variables in a cause-and-effect sequence, with one variable dependent on the other. It predicts that you'll achieve a particular outcome based on a certain condition. The outcome is known as the dependent variable and the change causing it is the independent variable .

For example, this is a simple hypothesis:

• Including the search function on our mobile app will increase user retention

The expected outcome of increasing user retention is based on the condition of including a new search function. But, what happens when there are more than two factors at play?

We get what’s called a complex hypothesis. Instead of a simple condition and outcome, complex hypotheses include multiple results. This makes them a perfect research hypothesis type for framing complex studies or tracking multiple KPIs based on a single action.

Building upon our previous example, a complex research hypothesis could be:

• Including the search function on our mobile app will increase user retention and boost conversions

Directional and non-directional hypotheses

Research hypotheses can also differ in the specificity of outcomes. Put simply, any hypothesis that has a specific outcome or direction based on the relationship of its variables is a directional hypothesis . That means that our previous example of a simple hypothesis is also a directional hypothesis.

Non-directional hypotheses don’t specify the outcome or difference the variables will see. They just state that a difference exists. Following our example above, here’s what a non-directional hypothesis would look like:

• Including the search function on our mobile app will make a difference in user retention

In this non-directional hypothesis, the direction of difference (increase/decrease) hasn’t been specified, we’ve just noted that there will be a difference.

The type of hypothesis you write helps guide your research—let’s get into it.

How to write and test your UX research hypothesis

Now we’ve covered the types of research hypothesis examples, it’s time to get practical.

Creating your research hypothesis is the first step in conducting successful user research.

Here are the four steps for writing and testing a UX research hypothesis to help you make informed, data-backed decisions for product design and development.

1. Formulate your hypothesis

Start by writing out your hypothesis in a way that’s specific and relevant to a distinct aspect of your user or product experience. Meaning: your prediction should include a design choice followed by the outcome you’d expect—this is what you’re looking to validate or reject.

Your proposed research hypothesis should also be testable through user research data analysis. There’s little point in a hypothesis you can’t test!

Let’s say your focus is your product’s user interface—and how you can improve it to better meet customer needs. A research hypothesis in this instance might be:

• Adding a settings tab to the navigation bar will improve usability

By writing out a research hypothesis in this way, you’re able to conduct relevant user research to prove or disprove your hypothesis. You can then use the results of your research—and the validation or rejection of your hypothesis—to decide whether or not you need to make changes to your product’s interface.

2. Identify variables and choose your research method

Once you’ve got your hypothesis, you need to map out how exactly you’ll test it. Consider what variables relate to your hypothesis. In our case, the main variable of our outcome is adding a settings tab to the navigation bar.

Once you’ve defined the relevant variables, you’re in a better position to decide on the best UX research method for the job. If you’re after metrics that signal improvement, you’ll want to select a method yielding quantifiable results—like usability testing . If your outcome is geared toward what users feel, then research methods for qualitative user insights, like user interviews , are the way to go.

3. Carry out your study

It’s go time. Now you’ve got your hypothesis, identified the relevant variables, and outlined your method for testing them, you’re ready to run your study. This step involves recruiting participants for your study and reaching out to them through relevant channels like email, live website testing , or social media.

Given our hypothesis, our best bet is to conduct A/B and usability tests with a prototype that includes the additional UI elements, then compare the usability metrics to see whether users find navigation easier with or without the settings button.

We can also follow up with UX surveys to get qualitative insights and ask users how they found the task, what they preferred about each design, and to see what additional customer insights we uncover.

💡 Want more insights from your usability tests? Maze Clips enables you to gather real-time recordings and reactions of users participating in usability tests .

4. Analyze your results and compare them to your hypothesis

By this point, you’ve neatly outlined a hypothesis, chosen a research method, and carried out your study. It’s now time to analyze your findings and evaluate whether they support or reject your hypothesis.

Look at the data you’ve collected and what it means. Given that we conducted usability testing, we’ll want to look to some key usability metrics for an indication of whether the additional settings button improves usability.

For example, with the usability task of ‘ In account settings, find your profile and change your username ’, we can conduct task analysis to compare the times spent on task and misclick rates of the new design, with those same metrics from the old design.

If you also conduct follow-up surveys or interviews, you can ask users directly about their experience and analyze their answers to gather additional qualitative data . Maze AI can handle the analysis automatically, but you can also manually read through responses to get an idea of what users think about the change.

By comparing the findings to your research hypothesis, you can identify whether your research accepts or rejects your hypothesis. If the majority of users struggle with finding the settings page within usability tests, but had a higher success rate with your new prototype, you’ve proved the hypothesis.

However, it's also crucial to acknowledge if the findings refute your hypothesis rather than prove it as true. Ruling something out is just as valuable as confirming a suspicion.

In either case, make sure to draw conclusions based on the relationship between the variables and store findings in your UX research repository . You can conduct deeper analysis with techniques like thematic analysis or affinity mapping .

UX research hypotheses: four best practices to guide your research

Knowing the big steps for formulating and testing a research hypothesis ensures that your next UX research project gives you focused, impactful results and insights. But, that’s only the tip of the research hypothesis iceberg. There are some best practices you’ll want to consider when using a hypothesis to test your UX design ideas.

Here are four research hypothesis best practices to help guide testing and make your UX research systematic and actionable.

Align your hypothesis to broader business and UX goals

Before you begin to formulate your hypothesis, be sure to pause and think about how it connects to broader goals in your UX strategy . This ensures that your efforts and predictions align with your overarching design and development goals.

For example, implementing a brand new navigation menu for current account holders might work for usability, but if the wider team is focused on boosting conversion rates for first-time site viewers, there might be a different research project to prioritize.

Create clear and actionable reports for stakeholders

Once you’ve conducted your testing and proved or disproved your hypothesis, UX reporting and analysis is the next step. You’ll need to present your findings to stakeholders in a way that's clear, concise, and actionable. If your hypothesis insights come in the form of metrics and statistics, then quantitative data visualization tools and reports will help stakeholders understand the significance of your study, while setting the stage for design changes and solutions.

If you went with a research method like user interviews, a narrative UX research report including key themes and findings, proposed solutions, and your original hypothesis will help inform your stakeholders on the best course of action.

Consider different user segments

While getting enough responses is crucial for proving or disproving your hypothesis, you’ll want to consider which users will give you the highest quality and most relevant responses. Remember to consider user personas —e.g. If you’re only introducing a change for premium users, exclude testing with users who are on a free trial of your product.

You can recruit and target specific user demographics with the Maze Panel —which enables you to search for and filter participants that meet your requirements. Doing so allows you to better understand how different users will respond to your hypothesis testing. It also helps you uncover specific needs or issues different users may have.

Involve stakeholders from the start

Before testing or even formulating a research hypothesis by yourself, ensure all your stakeholders are on board. Informing everyone of your plan to formulate and test your hypothesis does three things:

Firstly, it keeps your team in the loop . They’ll be able to inform you of any relevant insights, special considerations, or existing data they already have about your particular design change idea, or KPIs to consider that would benefit the wider team.

Secondly, informing stakeholders ensures seamless collaboration across multiple departments . Together, you’ll be able to fit your testing results into your overall CX strategy , ensuring alignment with business goals and broader objectives.

Finally, getting everyone involved enables them to contribute potential hypotheses to test . You’re not the only one with ideas about what changes could positively impact the user experience, and keeping everyone in the loop brings fresh ideas and perspectives to the table.

Test your UX research hypotheses with Maze

Formulating and testing out a research hypothesis is a great way to define the scope of your UX research project clearly. It helps keep research on track by providing a single statement to come back to and anchor your research in.

Whether you run usability tests or user interviews to assess your hypothesis—Maze's suite of advanced research methods enables you to get the in-depth user and customer insights you need.

Frequently asked questions about research hypothesis

What is the difference between a hypothesis and a problem statement in UX?

A research hypothesis describes the prediction or method of solving that problem. A problem statement, on the other hand, identifies a specific issue in your design that you intend to solve. A problem statement will typically include a user persona, an issue they have, and a desired outcome they need.

How many hypotheses should a UX research problem have?

Technically, there are no limits to the amount of hypotheses you can have for a certain problem or study. However, you should limit it to one hypothesis per specific issue in UX research. This ensures that you can conduct focused testing and reach clear, actionable results.

• Data Product Managers
• Product Managers
• Technical Product Managers
• App Product Managers
• Product Strategy Consultants
• Digital Marketing Product Managers
• Business Analysts
• Digital Product Managers

Shipping Your Product in Iterations: A Guide to Hypothesis Testing

Glancing at the App Store on any phone will reveal that most installed apps have had updates released within the last week. Software products today are shipped in iterations to validate assumptions and hypotheses about what makes the product experience better for users.

By Kumara Raghavendra

Kumara has successfully delivered high-impact products in various industries ranging from eCommerce, healthcare, travel, and ride-hailing.

PREVIOUSLY AT

A look at the App Store on any phone will reveal that most installed apps have had updates released within the last week. A website visit after a few weeks might show some changes in the layout, user experience, or copy.

Today, software is shipped in iterations to validate assumptions and the product hypothesis about what makes a better user experience. At any given time, companies like booking.com (where I worked before) run hundreds of A/B tests on their sites for this very purpose.

For applications delivered over the internet, there is no need to decide on the look of a product 12-18 months in advance, and then build and eventually ship it. Instead, it is perfectly practical to release small changes that deliver value to users as they are being implemented, removing the need to make assumptions about user preferences and ideal solutions—for every assumption and hypothesis can be validated by designing a test to isolate the effect of each change.

In addition to delivering continuous value through improvements, this approach allows a product team to gather continuous feedback from users and then course-correct as needed. Creating and testing hypotheses every couple of weeks is a cheaper and easier way to build a course-correcting and iterative approach to creating product value .

What Is Hypothesis Testing in Product Management?

While shipping a feature to users, it is imperative to validate assumptions about design and features in order to understand their impact in the real world.

This validation is traditionally done through product hypothesis testing , during which the experimenter outlines a hypothesis for a change and then defines success. For instance, if a data product manager at Amazon has a hypothesis that showing bigger product images will raise conversion rates, then success is defined by higher conversion rates.

One of the key aspects of hypothesis testing is the isolation of different variables in the product experience in order to be able to attribute success (or failure) to the changes made. So, if our Amazon product manager had a further hypothesis that showing customer reviews right next to product images would improve conversion, it would not be possible to test both hypotheses at the same time. Doing so would result in failure to properly attribute causes and effects; therefore, the two changes must be isolated and tested individually.

Thus, product decisions on features should be backed by hypothesis testing to validate the performance of features.

Different Types of Hypothesis Testing

A/b testing.

One of the most common use cases to achieve hypothesis validation is randomized A/B testing, in which a change or feature is released at random to one-half of users (A) and withheld from the other half (B). Returning to the hypothesis of bigger product images improving conversion on Amazon, one-half of users will be shown the change, while the other half will see the website as it was before. The conversion will then be measured for each group (A and B) and compared. In case of a significant uplift in conversion for the group shown bigger product images, the conclusion would be that the original hypothesis was correct, and the change can be rolled out to all users.

Multivariate Testing

Ideally, each variable should be isolated and tested separately so as to conclusively attribute changes. However, such a sequential approach to testing can be very slow, especially when there are several versions to test. To continue with the example, in the hypothesis that bigger product images lead to higher conversion rates on Amazon, “bigger” is subjective, and several versions of “bigger” (e.g., 1.1x, 1.3x, and 1.5x) might need to be tested.

Instead of testing such cases sequentially, a multivariate test can be adopted, in which users are not split in half but into multiple variants. For instance, four groups (A, B, C, D) are made up of 25% of users each, where A-group users will not see any change, whereas those in variants B, C, and D will see images bigger by 1.1x, 1.3x, and 1.5x, respectively. In this test, multiple variants are simultaneously tested against the current version of the product in order to identify the best variant.

Before/After Testing

Sometimes, it is not possible to split the users in half (or into multiple variants) as there might be network effects in place. For example, if the test involves determining whether one logic for formulating surge prices on Uber is better than another, the drivers cannot be divided into different variants, as the logic takes into account the demand and supply mismatch of the entire city. In such cases, a test will have to compare the effects before the change and after the change in order to arrive at a conclusion.

However, the constraint here is the inability to isolate the effects of seasonality and externality that can differently affect the test and control periods. Suppose a change to the logic that determines surge pricing on Uber is made at time t , such that logic A is used before and logic B is used after. While the effects before and after time t can be compared, there is no guarantee that the effects are solely due to the change in logic. There could have been a difference in demand or other factors between the two time periods that resulted in a difference between the two.

Time-based On/Off Testing

The downsides of before/after testing can be overcome to a large extent by deploying time-based on/off testing, in which the change is introduced to all users for a certain period of time, turned off for an equal period of time, and then repeated for a longer duration.

For example, in the Uber use case, the change can be shown to drivers on Monday, withdrawn on Tuesday, shown again on Wednesday, and so on.

While this method doesn’t fully remove the effects of seasonality and externality, it does reduce them significantly, making such tests more robust.

Test Design

Choosing the right test for the use case at hand is an essential step in validating a hypothesis in the quickest and most robust way. Once the choice is made, the details of the test design can be outlined.

The test design is simply a coherent outline of:

• The hypothesis to be tested: Showing users bigger product images will lead them to purchase more products.
• Success metrics for the test: Customer conversion
• Decision-making criteria for the test: The test validates the hypothesis that users in the variant show a higher conversion rate than those in the control group.
• Metrics that need to be instrumented to learn from the test: Customer conversion, clicks on product images

In the case of the product hypothesis example that bigger product images will lead to improved conversion on Amazon, the success metric is conversion and the decision criteria is an improvement in conversion.

After the right test is chosen and designed, and the success criteria and metrics are identified, the results must be analyzed. To do that, some statistical concepts are necessary.

When running tests, it is important to ensure that the two variants picked for the test (A and B) do not have a bias with respect to the success metric. For instance, if the variant that sees the bigger images already has a higher conversion than the variant that doesn’t see the change, then the test is biased and can lead to wrong conclusions.

In order to ensure no bias in sampling, one can observe the mean and variance for the success metric before the change is introduced.

Significance and Power

Once a difference between the two variants is observed, it is important to conclude that the change observed is an actual effect and not a random one. This can be done by computing the significance of the change in the success metric.

In layman’s terms, significance measures the frequency with which the test shows that bigger images lead to higher conversion when they actually don’t. Power measures the frequency with which the test tells us that bigger images lead to higher conversion when they actually do.

So, tests need to have a high value of power and a low value of significance for more accurate results.

While an in-depth exploration of the statistical concepts involved in product management hypothesis testing is out of scope here, the following actions are recommended to enhance knowledge on this front:

• Data analysts and data engineers are usually adept at identifying the right test designs and can guide product managers, so make sure to utilize their expertise early in the process.
• There are numerous online courses on hypothesis testing, A/B testing, and related statistical concepts, such as Udemy , Udacity , and Coursera .
• Using tools such as Google’s Firebase and Optimizely can make the process easier thanks to a large amount of out-of-the-box capabilities for running the right tests.

Using Hypothesis Testing for Successful Product Management

In order to continuously deliver value to users, it is imperative to test various hypotheses, for the purpose of which several types of product hypothesis testing can be employed. Each hypothesis needs to have an accompanying test design, as described above, in order to conclusively validate or invalidate it.

This approach helps to quantify the value delivered by new changes and features, bring focus to the most valuable features, and deliver incremental iterations.

• How to Conduct Remote User Interviews [Infographic]
• A/B Testing UX for Component-based Frameworks
• Building an AI Product? Maximize Value With an Implementation Framework

Further Reading on the Toptal Blog:

• Evolving UX: Experimental Product Design with a CXO
• How to Conduct Usability Testing in Six Steps
• 3 Product-led Growth Frameworks to Build Your Business
• A Product Designer’s Guide to Competitive Analysis

Understanding the basics

What is a product hypothesis.

A product hypothesis is an assumption that some improvement in the product will bring an increase in important metrics like revenue or product usage statistics.

What are the three required parts of a hypothesis?

The three required parts of a hypothesis are the assumption, the condition, and the prediction.

Why do we do A/B testing?

We do A/B testing to make sure that any improvement in the product increases our tracked metrics.

What is A/B testing used for?

A/B testing is used to check if our product improvements create the desired change in metrics.

What is A/B testing and multivariate testing?

A/B testing and multivariate testing are types of hypothesis testing. A/B testing checks how important metrics change with and without a single change in the product. Multivariate testing can track multiple variations of the same product improvement.

Kumara Raghavendra

Dubai, United Arab Emirates

Member since August 6, 2019

About the author

World-class articles, delivered weekly.

By entering your email, you are agreeing to our privacy policy .

Toptal Product Managers

• Artificial Intelligence Product Managers
• Blockchain Product Managers
• Business Systems Analysts
• Cloud Product Managers
• Data Science Product Managers
• Directors of Product
• E-commerce Product Managers
• Enterprise Product Managers
• Enterprise Resource Planning Product Managers
• Interim CPOs
• Jira Product Managers
• Kanban Product Managers
• Lean Product Managers
• Mobile Product Managers
• Product Consultants
• Product Development Managers
• Product Owners
• Product Portfolio Managers
• Product Tour Consultants
• Robotic Process Automation Product Managers
• Robotics Product Managers
• SaaS Product Managers
• Salesforce Product Managers
• Scrum Product Owner Contractors
• Web Product Managers
• View More Freelance Product Managers

Join the Toptal ® community.

Advisory boards aren’t only for executives. Join the LogRocket Content Advisory Board today →

• Product Management
• Solve User-Reported Issues
• Find Issues Faster
• Optimize Conversion and Adoption

4 types of product assumptions and how to test them

Understanding, identifying, and testing product assumptions is a cornerstone of product development.

To some extent, it’s the primary responsibility of a product manager to handle assumptions well to drive product outcomes.

Let’s dive deep into what assumptions are, why they are critical, the common types of assumptions, and, most importantly, how to test them.

What are product assumptions?

Product assumptions are preconceived beliefs or hypotheses that product managers establish during the product development cycle, providing an initial framework for decision-making. These assumptions, which can involve features, user behaviors, market trends, or technical feasibility, are integral to the iterative process of product creation and validation.

Assumptions guide the prototyping, testing, and adjustment stages, allowing the team to refine and improve the product in response to real-world feedback.

Leveraging product assumptions effectively is a cornerstone of risk management in product development because it aids in reducing uncertainty, saving resources, and accelerating time to market. Remember, a key part of a product manager’s role is to continuously challenge and validate product assumptions to ensure the product remains aligned with consumer needs and market dynamics.

Whatever you do, you don’t do it without a reason. For example, if you are building a retention-focused feature to drive revenue, you automatically assume that the feature will improve your revenue metrics and that it’ll deliver enough value for users that they’ll retain better.

In short, assumptions are all the beliefs you have when pursuing a particular idea, whether validated or not.

Why are assumptions important for product managers?

You can’t overemphasize the importance of assumptions in product management. For PMs, they are the building block of everything we do.

Ultimately, our job is to drive product outcomes by pursuing various initiatives we believe will contribute to the outcome. We decide which initiatives to pursue based on the beliefs we hold:

If our assumptions are correct, the initiative is a success, and there should be a tangible impact on the outcome. If they turn out wrong, we might fail to drive the impact we hope to see. We may even do more harm than good.

Because one initiative is often based on numerous assumptions, and various solutions can share the same assumptions, testing individual hypotheses is faster and cheaper than testing whole initiatives:

Moreover, testing an initiative with multiple unvalidated assumptions makes it hard to distinguish which hypotheses contributed to its success and which didn’t. Testing shared assumptions can help us raise confidence in multiple solutions simultaneously.

Over 200k developers and product managers use LogRocket to create better digital experiences

In most cases, you’re better off focusing on testing individual assumptions first than jumping straight into solution development.

4 types of product assumptions

There are various types of assumptions. However, as a product manager, there are four important assumptions that you must understand and learn how to test:

• Desirability assumptions
• Viability assumptions
• Feasibility assumptions
• Usability assumptions

1. Desirability assumptions

When you assume solution desirability, you are trying to answer the question, “Do our users want this solution?”

After all, in the vast majority of cases, there’s no reason to pursue an initiative that isn’t interesting for your end-users.

Desirability assumptions include questions such as:

• Does this problem solve a painful enough problem?
• Is the problem we are solving relevant to enough users?
• Is our proposed way of solving the problem optimal?
• Will users understand the value they can get from this solution?

2. Viability assumptions

Viability determines whether the initiative makes sense from a business perspective.

Delivering value for users is great, but to be truly successful, an initiative must also deliver enough ROI for the business to grow and prosper. Of course, you might work for an NGO that doesn’t care about the revenue.

Viability assumptions include questions such as:

• Will we see a positive impact on business metrics?
• Does this initiative fit our current business model?
• Does the solution align with our long-term product strategy?
• Can we expect a satisfactory return on investment?

3. Feasibility assumptions

Even the most desirable and viable solutions are only relevant if they are possible to build, implement, and maintain.

Before committing to any direction, ensure you can deliver the initiative within your current constraints.

You can assess feasibility by answering questions such as:

• Does our current technology stack allow such an implementation?
• Do we have the resources and skillset to proceed with this initiative?
• Do we have means of maintaining the initiative?
• Can we handle the technical complexity of this solution?

4. Usability assumptions

Even after you implement a desirable, viable, and feasible solution, it won’t drive the expected results if users don’t understand how to use it.

The more usable the solution is, the more optimal outcomes it’ll yield.

Focus on answering questions such as:

• Are our users aware that the new solution exists?
• Do they understand what value they can get from it?
• Is it clear how to find and use the solution?
• Is there friction or needless complexity that might prevent users from adopting the solution?

How to use an assumption map

An assumption map is a powerful technique that can help you identify, organize, and prioritize assumptions you make with your initiatives.

Check out our assumption mapping article for more details if that sounds valuable.

For the purpose of this article, I’ll assume you’ve already identified and prioritized your assumptions.

Testing product assumptions

Now let’s take a look at some ways you can test your assumptions. While the best method depends heavily on the type of assumption you are testing, this library should be a solid starting point:

Testing desirability

Testing viability, testing feasibility, testing usability.

There’s no way to test desirability without interacting with your users. Get out of the door, one way or another, and see if the solution is something your users truly want.

Techniques for assessing the desirability of a solution include:

Landing pages

Crowdfunding, alpha and beta testing.

One of the fastest and most insightful desirability validation techniques is to interview your target users .

You don’t want to ask users upfront because doing so produces skewed answers. Instead, you want to understand the user’s problem, how they describe it, and the most significant pain points they have. You can then look at your proposed solution and judge whether it could potentially solve the problems users mentioned.

You can create a product landing page even if you don’t yet have the product. By monitoring the engagement on the site, you can gauge the overall interest in the solution; if users bounce from the site after a few seconds, they are probably not interested.

You can take it a step further and include the option to subscribe to a waitlist. Signing up would be a powerful signal that users are genuinely interested.

If you are building a B2B solution, you can try to actually sell it to potential clients. There are three ways to approach this:

• Mock sales — A sales simulation when you try to sell the solution but don’t commit to an actual sale
• Letter of intent — You ask your potential client to sign a letter of intent to buy the solution once it’s live
• Actual sale — In some cases, you might be able to finalize the sale before the product is even live, with an option to revert the sale if you decide not to pursue the direction after all

If people are willing to pay for the solution before it is even created, the desirability is really high.

Crowdfunding is a presale option for mass B2C consumers. However, it’s viable mostly for brand-new products.

By promoting your idea on sites like Kickstarter, you can not only gauge overall desirability but also capture funding to improve the viability of the idea.

The most powerful yet expensive way of testing desirability is to build a minimal version of the solution. You can then conduct alpha and beta tests to see actual user engagement and gather real-time feedback on the further direction.

Due to the cost, this method is recommended after you have some initial confirmation with other validation techniques.

You can test the viability of assumptions by taking a closer look at the business side of things to evaluate whether the initiative fits well or contradicts with other areas.

Techniques for testing the viability of your product include:

Business model review

Strategy canvas, business case.

The first step in assessing initiative viability is to review your current business model and see how it would fit there:

Does the solution connect well to your current value proposition and distribution channel? Do you have key resources and partners to pull it off? Does it sync well with key activities you are performing?

Ideally, your initiative will not only not disrupt your business model but also contribute to it as a whole.

A viable solution helps you build a competitive advantage in the market. One way to evaluate viability is to map a strategy canvas of your competitive alternatives and judge whether the initiative will help you strengthen your advantage or reduce your weaknesses:

A great solution helps you maintain and expand your competitive edge on the market.

With basic viability tested, it’s worth investing some time to build a robust business case.

Gather all relevant input and try to build well-informed projections:

• How many people can you reach?
• How expensive the solution is going to be?
• What’s the expected long-term revenue gain and maintenance cost?
• What is the anticipated ROI over time?

A strong business case will also help you pitch the idea to key stakeholders and compare the business viability of various initiatives and solutions to choose the most impactful one.

Validating whether a solution is possible to implement usually requires a team of subject matter experts to do a deep dive into potential implementation details. Two common approaches are

Technical research

Proof of concept (poc).

This step includes researching various implementation methods and limitations to determine whether a solution is feasible.

For example, suppose you are considering a range of trial lengths for various user segments in your mobile product. In that case, you might need to review app store policy and limitations to see if it’s allowed out of the box or if any external solution is necessary.

If an external solution is needed, you might investigate whether there’s an SDK supporting that or it requires building from scratch (thus increasing complexity and reducing the viability of the solution).

For more complex initiatives, you might need to develop a proof ofconcept. One could call it a “technical MVP”. It includes building the minimal version of the most uncertain part of the solution and evaluating if it even works. Proof of concept might vary from a few lines of code for simple tests to a fully-fledged development for the most complex initiatives.

Usability is the most straightforward thing to test. You want to put the solution in front of the user to see if they understand how to use it and what potential friction points are.

There are two common ways to do this:

Analytics review

Prototypes are at the forefront of usability testing. Build a simulation of the experience you want to provide, ask the user to finish a specific task, and observe how they interact with the product.

Depending on the level of uncertainty and the investment you want to make, prototypes can vary from quick-and-dirty paper prototypes to fully interactive, no-code solutions.

If you are already at an MVP stage, you have the benefit of having actual data on how the solution is used. Analyze this data closely to evaluate how discoverable the product is, how much time it takes for users to complete specific tasks, and what are the most common dropout moments.

Combining quantitative data review with qualitative insights from prototypes will help you validate most of your usability assumptions.

Every initiative you pursue is based on a set of underlying assumptions — that is, a set of preconceived beliefs we have when deciding which direction to pursue.

Validating these beliefs is a critical part of product management. After all, it’s easier and cheaper to test individual assumptions than to test solutions as a whole.

Make sure you identify your main desirability, viability, feasibility, and usability assumptions and test them before committing to a fully-fledged solution.

I recommend you store the insights from assumptions tests for future reference. Many solutions tend to share similar assumptions, so the insights might help you speed up your validation process in the future.

Featured image source: IconScout

LogRocket generates product insights that lead to meaningful action

Get your teams on the same page — try LogRocket today.

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Reddit (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to share on Facebook (Opens in new window)
• #product strategy

Stop guessing about your digital experience with LogRocket

Recent posts:.

How PMs can best work with UX designers

With a well-built collaborative working environment you can successfully deliver customer centric products.

Leader Spotlight: Evaluating data in aggregate, with Christina Trampota

Christina Trampota shares how looking at data in aggregate can help you understand if you are building the right product for your audience.

What is marketing myopia? Definition, causes, and solutions

Combat marketing myopia by observing market trends and by allocating sufficient resources to research, development, and marketing.

Leader Spotlight: How features evolve from wants to necessities, with David LoPresti

David LoPresti, Director, U-Haul Apps at U-Haul, talks about how certain product features have evolved from wants to needs.

Leave a Reply Cancel reply

Product Talk

Make better product decisions.

The 14 Most Common Hypothesis Testing Mistakes Product Teams Make (And How to Avoid Them)

Originally published: September 5, 2014 by Teresa Torres | Last updated: December 8, 2018

I’ve been working with a product team on how to get better at hypothesis testing. It’s a lot of fun.

They were introduced to  dual-track Agile  by Marty Cagan and are doing a great job of putting it into practice.

As they explore how to support backlog items with research in the discovery track, they are finding that hypothesis testing isn’t as easy as it sounds.

Very few of us have had to formulate hypotheses and design experiments since perhaps our elementary school science fair days.

And while the scientific method conceptually is easy to grasp, putting it into practice can be much more challenging.

Why You Should Get Better at Hypothesis Testing

Across the Internet industry, we are seeing a shift from the “executive knows all” mindset to an experimentation mindset where we support ideas with research before investing in them.

More companies are running A/B tests, conducting usability studies, and engaging customers in discovery interviews than ever before.

We are investing in tools like Optimizely, Visual Website Optimizer, UserTesting.com, KissMetrics, MixPanel, and Qualaroo.

But teams who invest in research are quickly finding that their experiments are only as good as their hypotheses and experiment design.

It’s a classic case of garbage in, garbage out.

You can waste thousands of dollars, hundreds of hours, and countless sprints running experiments that don’t matter. That don’t net any meaningful results.

Experimentation is like Agile. It’s a tool in our toolbox, but we still need to do the strategic work to get value out of it.

Agile will help us move through a backlog quicker, but it won’t help us put the right stories in the backlog. – Tweet This

Experimentation will help us support or refute a hypothesis, but we have to do the work to design a good hypothesis and a good experiment.

Avoid These 14 Common Pitfalls

You don’t have to be a rocket scientist to know how to formulate a good hypothesis or design a good experiment, but you do want to avoid these common mistakes.

1. Not knowing what you want to learn.

Too many teams test anything and everything. See: Why Testing Everything Doesn’t Work .

If you want to get meaningful results, you need to be clear about what you are trying to learn and design your experiment to learn just that.

In the product world, we can experiment at different levels of analysis.

We can test our value propositions.

We can test whether or not a feature delivers on a specific value proposition.

We can test a variety of designs for each feature.

And we can test the feasibility of each solution.

Too often people test all of these layers at once. This makes it hard to know which layer is working and which is not and often leads to faulty conclusions.

Instead, be clear about what you are testing and when. This will simplify your experiment design and fuel your rate of learning.

2. Using quantitative methods to answer qualitative questions (and vice versa). 

Qualitative methods such as interviews, usability tests, and diary projects help us understand context.

They are great for helping us to understand why something may or may not be happening. They expose confusing interface elements and gaps in our mental models and metaphors.

However, with qualitative research, it can be hard to generalize our findings beyond the specific contexts we observe.

Quantitative methods, on the other hand, allow us to go broad, collecting large amounts of data from broad samples.

Think A/B tests, multivariate tests, and user surveys.

Quantitative research is great for uncovering how a large population behaves, but it can be challenging to uncover the why behind their actions.

An A/B test will tell you which design converts better but it won’t tell you why.

The best product teams mix-and-match the right methods to meet their learning goals. – Tweet This

3. Starting with untestable hypotheses. 

It’s easy to be sloppy with your hypotheses. This might be the most common mistake of all.

Have you found yourself writing either of the following:

• Design A will improve the overall user experience.
• Feature X will drive user engagement.

How will you measure improvements in user experience or user engagement?

You need to be more specific.

A testable hypothesis includes a specific impact that is measurable. – Tweet This

At the end of your test it should be crystal clear whether your hypothesis passed or failed.

4. Not having a reason for why your change will have the desired impact.

You might know what you want to learn, but not know why you think the change will have the desired impact.

This is common with design changes. You test whether a blue or green button converts better, but you don’t have a theory as to why one might convert better than the other.

Or you think a feature will increase return visits, but you aren’t quite sure why. You just like it.

The problem with these types of experiments is that they increase the risk of false positives.

A false positive is an experiment result where one design looks like it converts better than another, but the results are just due to chance.

Internet companies are running enough experiments now that we need to start taking false positives seriously.

Always start with an insight as to why your change might drive the desired impact.

Need further convincing? Spend some time over at  Spurious Correlations .

5. Testing too many variations. 

Google tested 41 shades of blue. Yahoo tested 30 logos in 30 days.

Don’t mimic these tests. I suspect these companies had other reasons for running these experiments other than finding the best shade of blue or the best logo.

Each variation of blue has a 5% chance of being a false positive. Same with each logo. If you want to increase the odds that your experiment results reflect reality, test fewer variations.

Suppose you are testing different headlines for an article.  You don’t want to test 25 different headlines. You want to identify the 2 or 3 best headlines where you have a strong argument for why each might win and test just those variations.

More variations leads to more false positives. You don’t  have to understand the math, but you do need to understand the implications of the math.

Here’s the key takeaway:

Run fewer variations and have a good reason for testing each one. – Tweet This

6. Running your experiment with the wrong participants.

Who you test with is just as important as what you test. – Tweet This

This is often overlooked.

If Apple is trying to understand the buying habits of iPhone customers, they shouldn’t interview price-sensitive buyers. The iPhone is a high-end product. Apple needs to interview buyers who value quality over price.

If you are marketing to new moms, don’t run your tests with experienced moms. Don’t run your tests with people who don’t have kids. Their opinions and behaviors don’t matter.

This can be trickier than it seems. Often times the who is implied in the hypothesis. Do the work to make it explicit so you don’t make these errors.

7. Forgetting to draw a line in the sand.

It’s easy after the fact to call mediocre good. But nobody wants to be mediocre.

The best way to avoid this is to determine up front what you consider good.

With every hypothesis, you are assuming a desired impact. Quantify it. – Tweet This

For quantitative experiments, draw a hard line in the sand. How much improvement do you expect to see?

Draw the line as if it’s a must-have threshold. In other words, if you don’t meet the threshold, the hypothesis doesn’t pass.

This takes discipline, but you’ll get much better outcomes if you stick with it.

8. Stopping your test at the wrong time. 

Many people make the mistake of stopping their quantitative tests as soon as the results are statistically significant. This is a problem. It will lead to many false positives.

Determine ahead of time for how long to run your test. Don’t look at the results until that time has elapsed. – Tweet This

Use a duration calculator . Again, you don’t need to understand the math, you just need to know how to apply it.

And be sure to take into account seasonality for your business. For most internet businesses Monday traffic is better than Thursday traffic. This will impact your test results.

If you want a statistical explanation for why fixing the duration of your test ahead of time matters, read this article .

9. Underestimating the risk or harm of the experiment. 

Experimenting is good.

Ignoring the impact your experiment might have is bad.

Yes, we can and should make data-informed decisions. But this doesn’t mean that we should take unnecessary risks.

For each experiment, we need to understand the risk to the user and the risk to the business. – Tweet This

And then we need to do what we can to mitigate the risk to both.

10. Collecting the wrong data.

You need to collect the right data in the right form.

This one sounds obvious in the abstract, but can be hard to do in practice.

Before you start collecting data, start by thinking through what data you need.

• Do you need to collect the number of actions taken or the number of people who action?
• Are you tracking visits, sessions, or page views?

Thinking through  how you will make decisions with this data will help you make sure you get it right. Ask yourself:

• What would the data need to look like for me to refute this hypothesis?
• What would the data need to look like to support this hypothesis?

The more you think through how you might use the data to drive decisions, the more likely you will collect usable data.

11. Drawing the wrong conclusions. 

It’s easy to draw the wrong conclusions from our experiments.

There are two things to keep in mind.

First, experiments can refute or support hypotheses but they never prove them.

We live in a world where nothing is certain. If you want to be a good experimenter, you have to accept this.

Don’t be dogmatic about your results. What didn’t work last year, might work this year. And vice versa.

Second, know what you are testing and make sure your conclusions remain within the scope of that test.

For example, if I am testing the impact of a new feature and it doesn’t have the desired impact that I had hoped, I might conclude that the feature isn’t good.

But I might be wrong. It also could be that the design of the feature wasn’t adequate. Or that the feature was buggy. Or the content that supports the feature was a mismatch.

Before you draw a conclusion, you need to ask, “What else could explain this result?” – Tweet This

12. Blindly following the data. 

It’s comforting to think that you can run an experiment and have the results tell you what to do. But it rarely happens this way.

More often than not there are other factors to consider.

We live in a complex world. Your experiment results are only one input of many. You need to use your best judgement to come to a conclusion.

Over relying on your data to tell you the right answer leads to implementing false positives and over optimization.

Don’t forget you are a human. Keep using your brain.

13. Spreading Yourself Too Thin

Just as we look forward to the next iPhone and crave the next episode of Breaking Bad, we also chase after more tools in our toolbox.

I often see teams new to A/B testing, rush to try multivariate testing.

Others jump from interviews to diary projects to Qualaroo surveys.

Each method requires developing a skill-set appropriate for each. Dive deep. Learn a method inside and out before moving on to the next one.

You can get more value from going deep with A/B testing than you will from only understanding the basics of both A/B testing and multivariate testing.

You’ve got a long career ahead of you. Go for depth before breadth. – Tweet This

14. Not Understanding How the Tools Work

Know your tools.

Know how to set up conversions correctly.

Know whether they are tracking actions vs. people, visits vs sessions.

Know whether your funnels are page to page or steps at any point in a session.

Understand where they draw the line for statistical significance.

Some consider 80% confidence significant – this means that 1 out 5 tests are likely a false positive. Understand how this impacts your product decisions.

You can try every product on the market, but if you don’t understand how they work, you won’t make good product decisions.

Over the next couple of months, we’ll dive deep into each of these mistakes and look at how you can avoid them.

We’ll explore real examples and get specific so that you can take what you learn and put it into practice right away.

Don’t miss out, subscribe to the Product Talk mailing list to get new articles delivered straight to your inbox.

Get the latest from Product Talk right in your inbox.

Join 41,000+ product people. Never miss an article.

[…] For more tips, see her article on The 14 Most Common Hypothesis Testing Mistakes (And How to Avoid Them) […]

[…] The 14 Most Common Hypothesis Testing Mistakes Product Teams Make (And How to Avoid Them) Forming and testing a hypothesis may seem easy, but it’s even easier to get wrong. Teresa Torres talks about 14 common mistakes people make when doing Lean Product testing. […]

Popular Resources

• Product Discovery Basics: Everything You Need to Know
• Product Trios: What They Are, Why They Matter, and How to Get Started
• Visualize Your Thinking with Opportunity Solution Trees
• Shifting from Outputs to Outcomes: Why It Matters and How to Get Started
• Customer Interviews: How to Recruit, What to Ask, and How to Synthesize What You Learn
• Assumption Testing: Everything You Need to Know to Get Started

Recent Posts

• Product in Practice: How Botify Broke Down Silos and Moved to Product Trios
• Ask Teresa: For Customer Interviews, Who Counts as a Customer?
• Product in Practice: Why Ramsey Solutions Rotates Engineers in Their Product Trios
• Tools of the Trade: Switching from Miro to Jira Product Discovery for Opportunity Solution Trees

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Offense and defense between streamers and customers in live commerce marketing: Protection motivation and information overload

Roles Conceptualization, Formal analysis, Methodology, Writing – original draft

Affiliation School of Business, Yangzhou University, Yangzhou, China

Roles Data curation, Formal analysis, Writing – original draft, Writing – review & editing

Roles Conceptualization, Project administration, Writing – review & editing

* E-mail: [email protected]

Roles Formal analysis, Investigation, Methodology, Writing – review & editing

Affiliation School of Business, Xinyang Normal University, Xinyang, China

Roles Funding acquisition, Supervision, Writing – review & editing

Affiliation School of Flight, Anyang Institute of Technology, Anyang, China

• Junwei Cao, 
• Lingling Zhong, 
• Dong Liu, 
• Guihua Zhang, 

• Published: September 6, 2024
• https://doi.org/10.1371/journal.pone.0305585
• Reader Comments

While live commerce provides consumers with a new shopping experience, it also leads them to experience shopping failures and to develop a self-protection mechanism to prevent wrong purchases. To address this issue, merchants have attempted to explore new marketing methods for live commerce, giving rise to an offense and defense game between streamers and consumers. In this study, we sought to confirm the effectiveness of consumer protection mechanisms and the impact of streamers’ information overload marketing strategy in live commerce. Accordingly, we constructed a hypothetical model based on protection motive theory and information overload theory. In addition, we analyzed the data from the simulated live streaming marketing on seven hundred people through partial least squares structural equation modeling. The results indicate that product utilitarian value uncertainty, consumers’ experiential efficacy, and response costs, which are the main factors in the formation of consumer protection mechanisms, influence consumers’ intention to stop their purchases. Streamers can circumvent consumer self-protection mechanisms through information overload marketing by reducing utilitarian value uncertainty and consumers’ experiential efficacy and increasing consumers’ response costs. However, consumers would be able to rebuild their self-protection mechanism through consumer resilience, which moderates the effects of information overload. This study’s results provide important theoretical perspectives and new ideas for formulating marketing strategies for live commerce.

Citation: Cao J, Zhong L, Liu D, Zhang G, Shang M (2024) Offense and defense between streamers and customers in live commerce marketing: Protection motivation and information overload. PLoS ONE 19(9): e0305585. https://doi.org/10.1371/journal.pone.0305585

Editor: Wojciech Trzebiński, SGH Warsaw School of Economics: Szkola Glowna Handlowa w Warszawie, POLAND

Received: October 18, 2023; Accepted: June 1, 2024; Published: September 6, 2024

Copyright: © 2024 Cao et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: Prof.Meng Shang received funded by Henan Province Philosophy and Social Science Planning Project (grant number: 2021CJJ122). The Science and Technology Development Project of Henan Province in 2021 (Soft Science Research) (grant number212400410251). The R&D and Promotion Key Program of Anyang in 2020 (grant number 2020-256). The Development Program for University Key Teacher of Henan Province (grant number 2020GGJS233). Scientific research and cultivation fund of Anyang Institute of Technology (grant number YPY2020035). Funders were involved in the experiment and data collection. Ms. Lingling Zhong received funded by Research and Innovation Project of Yangzhou University Business School "Study on the Impact of Information Overload on User Decision Making in E-commerce Live Streaming Environment". (grant number:SXYYJSKC202421).

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

As digital transformation continues to evolve, digital platforms are becoming increasingly important for corporate performance growth [ 1 ]. Alongside the advancements in information technology, digital business models are continuously being updated. Especially with the development of information technology, e-commerce has gradually shifted to social commerce, the most representative form of which today is live commerce [ 2 ]. Live commerce not only facilitates product demonstrations and guided shopping but also creates an unprecedented platform that supports real-time communication about goods between streamers and consumers [ 3 ]. In live commerce, streamers interact with the audience in real-time and establish a relationship that stimulates consumers’ desire to participate. This interaction creates a better shopping experience for participants, making them more willing to purchase than they might be with traditional marketing methods [ 4 , 5 ]. Consequently, consumers are increasingly favoring live streaming [ 6 , 7 ].

China’s live streaming market is growing faster than that in any other region of the world. The size of China’s live commerce market exceeded $128.5 billion in 2020 [ 6 ]. According to a 2020 report released by Baidu.com , the frequency of searches for the keyword ’live streaming’ increased by 120% in one year. Some famous streamers have demonstrated significant business value; for example, a live shopping session can attract tens of millions of users and generate more than ten million dollars in revenue [ 8 ]. However, with the rapid growth of China’s live-streaming market, some problems have emerged.

In live commerce, consumers typically make consumption decisions through self-evaluation after receiving streamers’ recommendations and product descriptions [ 9 ]. However, there is an increasing number of cases where consumers exhibit impulsive buying behaviors, influenced by streamers to make unplanned purchases [ 10 ]. Many of these purchase behaviors are impulsive and conforming, leading consumers to acquire items they do not need or whose actual value is lower than expected [ 2 , 11 ]. Over time, consumers develop self-protection mechanisms; that is, during subsequent live streaming sessions, they may hesitate or refrain from making purchases to avoid regrettable decisions [ 6 , 12 ]. However, the psychological processes underlying these self-protective mechanisms are not fully understood. Hence, this study seeks to address the following question:

• RQ1: What factors contribute to the emergence of consumer self-protection mechanisms in live commerce, and how do these mechanisms function to prevent incorrect purchasing decisions?

Notwithstanding such self-protection mechanisms, streamers can still influence consumer behavior through sophisticated communication strategies. For instance, streamers engage consumers with frequent interactions to retain them in the live broadcast room, subsequently providing comprehensive product information from various angles such as efficacy, design, price, usage, and suitability, ensuring consumers perceive a match between their needs and the product [ 7 ]. When consumers exhibit hesitation, streamers may create a false sense of urgency, emphasizing the product’s benefits and price advantages, promising additional free products, and using countdown timers to pressure consumers into making immediate purchases [ 5 ]. Moreover, streamers leverage parasocial relationships, persuading consumers by establishing a persona that garners recognition and sympathy, thereby fostering a sense of social connection [ 13 ]. The streamer is also good at creating a persona to gain consumer recognition and sympathy, which allows consumers to purchase in order to make social connections. They also ensure that the consumer treats them not as a provider of goods, but as a close friend. In addition, they can win consumers’ trust by influencing their emotional attachment or building intimacy [ 4 , 12 , 14 , 15 ]. As a result, consumers may decide to follow the streamer’s shopping advice. They buy goods recommended by the streamer to maintain the relationship, even if they do not need the goods [ 5 , 16 ]. Overall, streamers provide consumers with a lot of information in a short period by using a professional customer communication script to overcome the consumer’s protection mechanism against wrong purchases, reduce consumer hesitation, and lead them to make purchases. This marketing method based on "pushing” products in a "short time” and with a "large amount of information" meets the definition of information overload [ 17 ].

Merchants can influence consumer decision-making by affecting users’ psychological ownership. Information overload may serve as one method of control [ 18 ]. Numerous studies have explored the role of information overload in online marketing, focusing particularly on its relationship with consumers’ purchase intentions. However, the findings are mixed [ 19 ]. Some researchers argue that reducing consumer price sensitivity and increasing trust can encourage purchases [ 20 ], while others suggest that information overload heightens perceived risk among online consumers, reducing their intention to buy [ 19 , 21 ]. Additionally, the impact of information overload on purchase intention may exhibit an inverted-U shape, varying with the level of overload [ 22 ]. Typically, the adverse subjective state induced by information overload can lead to suboptimal purchasing decisions [ 23 ]. Although the exact mechanism remains unclear, it could depend on consumers’ psychological evaluation processes in various shopping contexts. A recent study suggests that information overload indirectly influences purchasing behavior by affecting these psychological evaluations, such as inducing panic buying [ 24 ]. When confronted with information overload, consumers encounter more data than they can process, which heightens uncertainty and hampers decision-making. Some research suggests that postponing decisions, as a strategy to manage overload, fosters hopefulness. This newfound hope, resulting from extra time to deliberate and make informed choices, often leads to a preference for delayed but larger rewards, indicating a shift toward more patient and reflective decision-making [ 25 ]. Another study points out that virtual product displays in e-commerce introduce new challenges of information asymmetry, with the excess of digital information complicating consumer purchasing decisions [ 26 ]. For example, in online tourism promotion, the number of images in reviews, the frequency of merchant responses, and the length of these responses positively impact tourism product sales [ 27 ]. This leads to the inquiry: Does a similar mechanism operate in live commerce? How does information overload influence consumer self-protection against incorrect purchases in live commerce settings? Accordingly, the second question this study seeks to answer is as follows:

• RQ2: In what ways does information overload marketing by streamers influence the effectiveness of consumer self-protection mechanisms in live commerce, and what are the specific tactics used by streamers to overcome these mechanisms?

In the context of information overload, individuals typically progress through three stages: compliance, acceptance, and resistance. Over time, consumers gradually mitigate the pressure of information overload and eventually re-establish their self-protection mechanisms to avoid erroneous purchases. To evaluate consumers’ capability to withstand information overload-induced marketing tactics, this study introduces the concept of resilience.

Resilience is acknowledged as an effective shield against stress. In the domain of information systems, prior research indicates that consumer resilience significantly contributes to alleviating stress [ 28 ]. This notion has also gained attention in consumer behavior research, suggesting its relevance in various contexts [ 29 ]. In live commerce, while merchants might breach consumers’ psychological defenses using information overload-centric marketing strategies, the potency of such strategies could be attenuated by consumer resilience. Prior investigations affirm that resilience can diminish the impacts of information overload [ 28 ]. However, in live commerce, when consumers are already under information overload, the role of resilience in restoring their self-protection mechanisms against wrong purchases remains unclear, which leads to the last research question of this study:

• RQ3: How does consumer resilience interact with information overload in live commerce to affect the stability and effectiveness of consumer self-protection mechanisms, and what role does consumer resilience play in moderating the impact of streamer marketing tactics?

This study aims to investigate the dynamic interactions between streamers (sellers) and consumers within the live commerce environment, focusing on the streamers’ offensive strategies and the consumers’ defensive mechanisms. Offensively, streamers employ tactics designed to overcome consumer hesitation and resistance. These tactics include providing an excess of product information, creating a sense of urgency (e.g., limited-time offers), and emphasizing the benefits and features of the products. Such actions are considered offensive as they aim to break down the protective barriers consumers may have erected to prevent impulsive or misguided purchasing decisions. Defensively, consumers develop mechanisms to protect themselves from potential regrets associated with purchases. These include hesitating or abstaining from buying due to skepticism towards the streamer’s presentation and coping with the streamer’s aggressive sales tactics, often characterized by "information overload." The study explores how streamers successfully influence consumer decisions (offense) and how consumers, in turn, protect themselves from making decisions they might regret (defense). This dynamic interplay of "offense" and "defense" forms the core of this research. By analyzing these interactions, the study seeks to reveal how these mechanisms jointly affect consumer behavior on live commerce platforms and how they shape the unique consumer experience in live commerce. It aims to provide a comprehensive understanding of how consumer protection mechanisms are established and function and how they impact the streamer’s information delivery strategies.

As live commerce swiftly expands, consumers often find themselves overwhelmed by the vast amount of product information available. This study aims to explore the interaction between information overload and consumer resilience, and how these factors influence consumer purchasing decisions. By developing a theoretical model, this research not only clarifies how consumers establish protective mechanisms (defense) to counteract aggressive marketing strategies in a live streaming environment but also examines how streamers can influence these mechanisms through their information strategies (offense). This approach offers a new perspective on consumer behavior in the digital marketplace.

The innovative aspects of this study are manifold. 1) it provides empirical evidence of the existence of psychological self-protection mechanisms among consumers in live commerce, thus enriching our understanding of consumer behavior within the context of social commerce. These insights reveal how consumers protect themselves from making erroneous purchases and navigate the rapidly changing shopping environment. 2) the study enhances our comprehension of the role of information overload in influencing consumer behavior, contributing valuable insights to the marketing literature. It offers strategic recommendations for e-commerce platforms on how to balance the provision of information with consumers’ ability to process it, aiming to enhance consumer purchase intentions and satisfaction. 3) By identifying consumer resilience as a moderating factor in the dynamic interaction between streamers and consumers, this research extends existing knowledge within consumer behavior studies. It highlights the importance of resilience in helping consumers withstand persuasive marketing tactics, providing practical insights for developing more effective marketing strategies that tackle the challenges of information overload in digital environments. Overall, this study not only aids in crafting more humane marketing strategies but also promotes the protection of consumer rights, contributing to the sustainable development of the e-commerce industry.

2. Theoretical background and hypothesis development

2.1 consumer’s defense mechanisms: protection motivation theory.

Protection motivation theory (PMT) was proposed by Rogers [ 30 ]. It divides the process by which people become motivated to protect themselves into three stages: threat appraisal, coping appraisal, and protective behavior. Individuals often develop risk-averse protective intentions to protect themselves from harm (e.g., natural disasters, the threat of global climate change). However, before taking any final protective actions, they usually weigh the benefits and risks of the action by comparing the level of environmental threat and their ability to cope [ 30 ].

Threat appraisal is a cognitive process through which individuals measure the threat level. It includes two aspects: perceived severity and perceived susceptibility [ 31 ]. The perceived severity of threat refers to an individual’s judgment of the severity of harm. Perceived susceptibility reflects an individual’s perception of the possibility of harm. Individuals’ perceptions of threat severity and susceptibility can motivate protective behavior [ 30 ].

Coping appraisal refers to the assessment of an individual’s ability to exhibit risk-prevention behaviors. It includes three aspects: self-efficacy, response efficacy, and response cost [ 30 ]. Self-efficacy is the individual’s judgment of their ability to display the desired behavior [ 32 ]. Response efficacy is the expectation regarding the outcome of an individual’s protective action [ 30 ]. Response costs are the benefits lost by individuals who engage in risk-prevention behaviors [ 30 ]. The sum of an individual’s self-efficacy and response efficacy minus the required response cost constitutes the results of the coping appraisal. The higher the response efficacy and self-efficacy and the lower the response cost, the more likely it is that the individual will decide to engage in protective behaviors [ 33 ].

PMT was mainly used for research in healthcare areas such as vaccination [ 34 , 35 ] and disease management [ 36 ]. It was later extended to the study of management information systems, such as network security behavior [ 37 ] and information security behavior [ 38 , 39 ]. It has also recently been used in studies of consumer purchasing behavior. For example, an investigation suggested that COVID-19 influences customers’ willingness to buy clothes by affecting perceived severity and self-efficacy in relation to the disease [ 40 ]. Another study suggested that the perceived severity of environmental problems and the effectiveness of environment protection responses motivate consumers to engage in protection behaviors and ultimately influence their purchasing behavior in relation to green products [ 41 ]. Another PMT-based study indicated that perceived risk positively influences consumers’ motivation to spend money in luxury restaurants [ 42 ]. In summary, these previous studies confirm that risk factors from the living environment affect consumers’ consumption attitudes. Risk factors induce product uncertainty, and consumers are more likely to be uncertain about the value of products in such cases. For this reason, consumers may decide to protect themselves by considering the risks and benefits of their purchases [ 42 ], which may cause them to stop buying or to indulge in panic buying [ 43 ].

From the PMT perspective, in live streaming, the consumer protection mechanism against wrong purchases consists of three components: an assessment of the severity of the threat factors that may lead to wrong purchases; an assessment of the possibility of a response; and ultimately, the act of interrupting the purchase. With a wide variety of products appearing on live streaming, it is understandable for consumers to be uncertain whether the purchased products will be worthwhile. Therefore, they have to evaluate the risk of uncertainty, analyze the advantages and disadvantages to buying or not buying the product, and protect themselves from making incorrect purchases.

2.1.1 Threat appraisal.

The essence of live commerce is the sale of goods. Live commerce can display the utilitarian value of goods in a superior way, compared to traditional e-commerce. Streamers can reduce the uncertainty regarding product value based on social and product attributes. They improve customers’ purchase intentions through real-time interaction and the display of product information [ 3 , 6 ]. The streamer can use various methods to provide consumers with detailed and vivid product information. Consumers obtain product information that seems immediate, easily comprehensible, and inspirational. They immerse themselves in a pleasant online shopping atmosphere and eventually engage in purchase behavior [ 2 , 7 , 13 ]. In addition, streamers can quickly establish social relationships with customers based on mutual benefits by communicating with consumers from their perspective and responding to consumers’ needs instantly [ 13 ]. This social relationship increases consumers’ trust in the streamer, supports the hedonic value consumers derive from live streaming, creates emotional commitment and attachment to the streamer in consumers, and promotes their willingness to purchase [ 4 , 12 ].

Since live commerce demonstrates the value of products to consumers efficiently and reduces the uncertainty in consumers’ perception of the value of products, consumers’ willingness to purchase improves [ 3 , 6 ]. For example, when a consumer feels that the product does not justify the value and that the purchase of the product does not necessarily increase the intimacy with the streamer, they reduce their purchase expectations and stop the purchase. From the PMT perspective, the higher the uncertainty in consumers about the value of a product, the more they realize that the current shopping environment is causing them to make wrong purchases, and the higher the possibility that they will suffer losses if they continue to buy.

Product value uncertainty is defined as the degree of difficulty consumers have in assessing product attributes and predicting future product performance [ 44 ]. Product uncertainty is a major barrier in online shopping. It has a significant impact on consumers’ willingness to buy [ 45 ]. In a live commerce environment, consumers tend to be unsure of whether the products are worth purchasing, and therefore, they may evaluate the risks in continuing to purchase [ 42 ].

Product value uncertainty, a multifaceted construct, encompasses uncertainties related to product description, fit, and performance. Specifically, product description uncertainty arises when the seller’s presentation fails to adequately encapsulate the product’s characteristics. Product performance uncertainty is the consumer’s apprehension about the actual performance aligning with their expectations [ 44 ], while fit uncertainty pertains to the consumer’s concern about the product meeting their specific needs [ 46 ]. In the realm of live commerce, where the immediacy and interactivity of the shopping experience are pronounced, the clarity of product value, particularly its utilitarian aspect, is paramount. Utilitarian value, the assessment of a product’s functional benefits [ 47 ], significantly influences consumer purchase behavior [ 48 ] and is even more critical in the dynamic environment of live commerce [ 49 ]. The unique capabilities of live commerce to demonstrate products and deliver information efficiently are pivotal in eliciting consumers’ utilitarian shopping motivation [ 50 ].

However, when faced with product value uncertainty in this live interactive setting, consumers engage in a threat appraisal process, as postulated by Protection Motivation Theory (PMT). This process involves evaluating the severity of the potential threat (e.g., the risk of dissatisfaction or regret from an incorrect purchase) and their susceptibility to this threat (e.g., the likelihood of making a poor purchase decision due to inadequate product information). Consequently, this perceived threat may catalyze a protective behavioral response, wherein consumers are inclined to halt their purchasing decision to safeguard against potential adverse outcomes. Therefore, integrating the constructs of PMT with the dynamics of live commerce, we propose the following hypothesis:

H1a: In live commerce, uncertainty about a product’s utilitarian value positively affects consumers’ intention to stop purchase.

Live commerce, distinct from traditional e-commerce, thrives on the interactive dynamics between streamers and consumers, often resembling a celebrity-fan relationship [ 50 ]. This interactive milieu fosters a hedonic value proposition, where the enjoyment and experiential benefits derived from the engagement are paramount [ 51 ].

The hedonic value in live commerce is not merely about the product but also about the relational experience with the streamer, enhancing trust, emotional commitment, and a sense of community among consumers (Hu & Chaudhry, 2020; Park & Lin, 2020). Previous research has confirmed that hedonic value is an important motivation in consumers’ purchase decisions during live commerce [ 50 , 52 ]. Live commerce transactions with a hedonic value can influence customer attitudes and behavioral responses. For example, when consumers perceive the integrity and kindness of a streamer through live streaming, they begin to trust the streamer. They believe that they can help streamers by purchasing products [ 53 ].

In the framework of Protection Motivation Theory (PMT), the concept of hedonic value uncertainty in live commerce can be interpreted as a form of threat appraisal. When consumers are unsure about the hedonic benefits of their engagement in live commerce—whether due to ambiguous streamer-consumer interactions, inconsistent content quality, or unclear emotional rewards—they undergo a cognitive process of assessing the severity of this uncertainty and their vulnerability to potential dissatisfaction or regret associated with their purchase decisions.

Given this backdrop, hedonic value uncertainty can trigger protective motivations, prompting consumers to reconsider or halt their purchase intentions as a mechanism to shield themselves from the anticipated dissonance of unmet emotional expectations or the lack of perceived relational value from the live commerce experience. Therefore, synthesizing the aspects of PMT and the hedonic nuances of live commerce, we propose the following hypothesis:

H1b: In live commerce, hedonic value uncertainty positively impacts consumers’ intention to stop purchase.

2.1.2 Coping appraisal.

In the context of live commerce, consumers’ decision-making processes are profoundly influenced by their past purchasing experiences, which shape their perceptions of self-efficacy and response efficacy—key components of the coping appraisal mechanism as delineated in PMT. Self-efficacy in this realm refers to consumers’ belief in their ability to make informed purchasing decisions, while response efficacy pertains to their assessment of the effectiveness of these decisions in yielding satisfactory outcomes [ 54 ]. Experiential efficacy, a construct synthesizing self-efficacy and response efficacy, encapsulates consumers’ confidence and perceived effectiveness based on their historical interactions within the live commerce environment. This concept aligns with PMT’s coping appraisal, where individuals evaluate their capability to mitigate or avoid perceived threats (in this case, the threat of unsatisfactory purchases).

Empirical studies underscore the link between past purchasing experiences and future purchasing behaviors, Studies suggesting that positive experiences enhance consumers’ confidence and perceived control over future transactions, potentially reducing their likelihood to halt purchases due to uncertainty [ 55 , 56 ]. Based on previous research, we combined self-efficacy and response efficacy as experiential efficacy in this study. Therefore, we propose the following hypothesis:

H2a: In live commerce, consumers’ experiential efficacy has a positive impact on their intention to stop purchase.

Protection Motivation Theory (PMT) provides a robust framework for understanding how individuals assess and respond to threats, incorporating elements of threat appraisal and coping appraisal. In the context of live commerce, consumers engage in a threat appraisal process, where they evaluate the potential risks associated with their purchasing decisions. Concurrently, the coping appraisal process involves an assessment of the response costs associated with taking protective actions to mitigate these risks. According to PMT, the likelihood of an individual engaging in a protective behavior increases when the perceived severity and vulnerability associated with the threat are high, and when the perceived response efficacy and self-efficacy are significant enough to outweigh the response costs.

In live commerce settings, streamers create value bonds with consumers by offering extra benefits such as personalized recommendations, product trials, and exclusive offers, as highlighted by [ 13 ]. These value bonds can be viewed as factors that lower the perceived response costs of continuing a purchase, as they enhance the perceived benefits of engaging with the live commerce platform and reduce the perceived gains of discontinuing a purchase. When consumers contemplate stopping a purchase, they weigh the response costs, which now include potential losses of additional benefits and the emotional connection with the streamer. If these perceived response costs of protective action (i.e., stopping the purchase) are viewed as high relative to the benefits of risky behavior (continuing the purchase despite potential risks), the consumer’s motivation to adopt self-protective behavior is likely to diminish. Therefore, we propose the following hypothesis that directly links the PMT constructs with the context of live commerce:

H2b: In live commerce, consumers’ perceived response costs have a negative impact on their intention to stop buying.

2.2 Streamer offense strategy: Information overload

2.2.1 information overload and marketing..

In recent years, terms such as information asymmetry, data smog, and information overload have frequently appeared in research reports. However, information overload has long been an important issue in daily life. Information overload means that the amount of information processing exceeds the capacity for information processing [ 57 ]. In the process of information retrieval, analysis, and decision making, information overload occurs when the amount of information that needs to be processed is greater than an individual’s ability to process information in a short period of time [ 58 ]. The degree of individual information overload depends not only on the amount of information received but also on the nature of the information and the individual’s experience reserve. When an individual’s experience is insufficient to cope with uncertain and complex information, the information overload phenomenon is more significant [ 59 ]. An information overload is harmful to individuals. It confuses them, affects their ability to prioritize, and makes it difficult to use previous information effectively. As a result, it leads to poor decision-making, dysfunction, and anxiety [ 59 , 60 ]. In particular, information overload from social media is effective in changing consumer attitudes and convincing them to believe the message maker’s opinion [ 61 ].

Early research on information overload in traditional retail marketing suggests that the relationship between the quantity of information and the quality of purchase decisions has an inverted U-shaped relationship [ 62 ]. However, subsequent studies have pointed out that an increase in the amount of information does not influence consumer decision-making and actual purchase behavior [ 63 , 64 ]. However, an increase in the quantity of information can reduce the accuracy of purchase decisions [ 65 ].

With the development of e-commerce, the impact of information overload in the online environment on consumers’ purchase intentions has been studied. Chen, Shang and Kao [ 23 ] suggested that e-retailers can deliver rich information to customers, but information overload caused by too much information could cause consumers to slip into a poor subjective state when making decisions. A study suggested that more product information, whether about the product or its price, would increase consumer trust, reduce consumer price awareness, and lead to consumer purchases [ 20 ]. Interestingly, a survey of 1,396 online shoppers in Spain confirmed that information overload positively influences consumers’ willingness to buy online, but also increases perceived risk, and indirectly decreases their willingness to buy [ 19 ]. According to the study’s authors, their findings "add some controversy to the relationship between information overload and customer purchase intentions" [ 19 ]. Subsequently, [ 22 ] conducted a study on the purchase intention toward online experience services and suggested an inverted U-shaped relationship between information load, trust, and purchase intention. That is, low information load is ineffective in fostering trust and purchase intention; medium information load is effective in fostering trust and purchase intention; and high information load is less effective than medium information load in fostering trust and purchase intention. However, a subsequent study showed that information overload reduces consumer trust and purchase intention, especially in online mobile shopping [ 21 ]. In summary, there are conflicting views on the role of information overload marketing in the field of e-commerce, and further research is needed.

2.2.2 Information overload and live commerce.

Product uncertainty is a major barrier to online shopping, and product descriptions and third-party product warranties help reduce product uncertainty [ 44 ]. Live commerce allows streamers to present detailed and vivid product information to consumers as if they were presenting this information in the presence of the consumer [ 2 , 7 , 13 ]. This ultimately reduces the uncertainty about the utilitarian value of the product.

Streamers also communicate with consumers from the consumer’s perspective and establish a social relationship of mutual understanding [ 13 ], thereby delivering the hedonic value of live streaming to consumers. Moreover, consumers gradually form emotional commitments and attachments to streamers [ 4 , 12 ], following which their hedonic value uncertainty decreases, and they become willing to purchase goods to enhance their relationship with the streamer.

When streamers find that consumers are hesitant to buy, they urge them to buy, repeatedly emphasizing the effectiveness of the product and the price advantage (utilitarian information), so that consumers are afraid of missing the opportunity to buy. Streamers deliberately limit the number of discounted products, replenish goods on the grounds of intimacy, and use rounds of surprises to increase consumers’ emotional attachment (hedonic information). In this time limitation created by streamers, the utilitarian and hedonic information greatly exceeds the consumer’s information processing capacity, and consumers enter a state of information overload.

Information Overload Theory posits that an excess of information can overwhelm consumers, impairing their decision-making capabilities [ 59 ]. In the context of live commerce, streamers can exacerbate information overload by rapidly presenting product information, thereby challenging consumers’ ability to process and evaluate this information effectively. This overload can trigger mechanisms like information avoidance and the information cocoon effect, where consumers limit their information sources to those that are most immediately accessible or reassuring—in this case, the streamer [ 66 ].

From the perspective of PMT, information overload can impact the threat and coping appraisal processes. When consumers face information overload, their ability to assess the potential threat (e.g., making a wrong purchase decision) and their efficacy in coping with this threat (e.g., evaluating product value accurately) can be compromised. Consequently, they may rely more heavily on the streamer’s guidance, which can reduce their perception of uncertainty regarding the product’s utilitarian and hedonic values. Thus, information overload in live commerce can paradoxically reduce consumers’ perceived uncertainty about a product’s value by nudging them towards a simplified decision-making process that leans heavily on the streamer’s input. This dynamic suggests that information overload might inadvertently diminish consumers’ perceived utilitarian and hedonic value uncertainty, as they become more dependent on the streamer’s narratives and less inclined to seek out additional information or evaluate their options critically. Therefore, the following hypothesis was proposed in this study:

• H3a: Information overload has a negative effect on utilitarian value uncertainty.
• H3b: Information overload has a negative effect on hedonic value uncertainty.

Protection Motivation Theory (PMT) provides a framework for understanding how individuals appraise threats and their coping responses. In the context of live commerce, experiential efficacy—consumers’ belief in their ability to make informed purchase decisions based on past experiences—is a crucial component of the coping appraisal process. Information Overload Theory suggests that an excess of information can impede individuals’ ability to process and make decisions, potentially undermining their experiential efficacy [ 59 ]. When consumers encounter information overload in live commerce, their ability to leverage past purchasing experiences may be compromised, leading to cognitive dissonance and a reduction in self-efficacy [ 67 ]. This phenomenon is exacerbated by streamers who rapidly disseminate product information, creating a sense of urgency and compelling consumers to make quick decisions, often sidelining their past experiences and judgment.

Additionally, response cost, a concept from PMT that denotes the perceived cost associated with engaging in a protective behavior, can be influenced by information overload. In live commerce settings, streamers accentuate discounts and limited-time offers, inflating the perceived cost of not making a purchase and thus potentially reducing consumers’ likelihood of engaging in protective behaviors like delaying or forgoing a purchase. Therefore, this study proposes the following hypotheses:

• H4a: Information overload has a negative effect on experiential efficacy.
• H4b: Information overload has a positive effect on response cost.

2.3 Consumer defense mechanism recovery: Consumer resilience

Resilience, defined as the ability of an individual to recover in the face of pressure or adversity, exhibits significant individual differences [ 68 ]. In the business realm, particularly in consumer behavior research, resilience has become a critical concept. In the business environment, marketing strategies may constrain consumers’ freedom, prompting them to face various temptations, with consumer resilience emerging as a key force to resist these market temptations and maintain individual freedom of choice [ 69 ]. Furthermore, consumer resilience displayed in shopping environments is often influenced by personal intrinsic traits and family environment, which together constitute important dimensions affecting consumer decisions [ 68 ].

In the emerging field of live commerce, consumer resilience manifests as particularly complex, influenced by multiple factors. Chen and Yang [ 70 ] study reveals the intrinsic connection between consumer experience and purchase intent, emphasizing the mediating role of network structural embeddedness in this process, pointing out that optimizing the usability of network interfaces and relationship services significantly impacts enhancing consumer resilience. Additionally, Xu, Cui and Lyu [ 26 ] explore the influence of host attributes and consumer interaction on purchasing behavior in live e-commerce, demonstrating how consumer trust and social capital play crucial roles in building consumer resilience. Zhang, Qi and Lyu [ 9 ] and Zhang, Yang and Bei [ 18 ] approach from the perspective of virtual communities, exploring the roles of knowledge sharing and social capital in promoting consumer resilience. Wei, Hai, Zhu and Lyu [ 25 ] emphasize the importance of perceived information integrity in shaping consumer resilience in studies of consumer delay behavior. Finally, Helin, Donglu, Shaoying, Decheng and Bei [ 27 ] analyze from the perspective of online reviews and merchant interaction responses how these factors influence consumer trust in brands and willingness to continue purchasing, providing new insights into understanding consumer resilience in live commerce. Together, these studies constitute an in-depth understanding of consumer resilience in live commerce, revealing its multiple influencing factors.

There may also be a relationship between information overload and consumer resilience. Information overload makes consumers dyscognitive dissonance, vulnerable, and susceptible; however, its impact can be reduced by consumer resilience [ 28 ]. When consumers experience information pressure, their original thought equilibrium may be disrupted. However, resilience can reduce information pressure. Consumers with high resilience feel less information pressure [ 28 ]. In particular, consumers with high resilience may recover more quickly from a potentially stressful event (e.g., service failure) compared to low resilience consumers [ 71 ].

In the field of consumer behavior research, the role of consumer resilience in reducing product uncertainty is unclear. However, a study in healthcare suggested that resilience in healthcare systems is beneficial in reducing risk uncertainty [ 72 ], who found resilience to be beneficial in reducing risk uncertainty in healthcare systems, we suggest that resilience in consumers can similarly reduce uncertainty in purchasing decisions by enhancing their ability to process overwhelming information and make clear evaluations of potential risks and benefits. Further, [ 73 ]. highlight the role of resilience in reducing the cost of adopting reliability behaviors. Extending this insight to the consumer domain, we argue that resilience enables consumers to better cope with information overload, thereby reducing the cognitive costs associated with evaluating information and making purchasing decisions. Consequently, resilient consumers are better equipped to discern the utilitarian and hedonic values of products and to assess the efficacy of their purchasing decisions, even in the face of excessive information. Therefore, we propose the following hypotheses:

• H5a: Consumer resilience reduces the negative effect of information overload on utilitarian value uncertainty.
• H5b: Consumer resilience reduces the negative effect of information overload on hedonic value uncertainty.
• H5c: Consumer resilience reduces the negative effect of information overload on experiential efficacy.
• H5d: Consumer resilience reduces the positive relationship of information overload on response costs.

3. Research model and investigation design

3.1 research model.

Based on the theoretical foundations and hypotheses discussed thus far, this study proposes a research model as shown in Fig 1 . First, we construct a consumer protection mechanism to prevent incorrect purchases in live commerce based on the PMT. The PMT proposes that consumers’ perceived threat severity and susceptibility can be used for threat appraisal. Self-efficacy, response efficacy, and response cost can be used for coping appraisal. In this study, in the context of live commerce, utilitarian and hedonic value uncertainty were used to assess the severity of and susceptibility to wrong purchases, and experiential efficacy and response cost were used to measure consumer coping appraisal. Together, these factors influence consumers’ self-protective behavior. However, the investigation of consumer behavior through self-reporting may bias the results. Conversely, the intention of consumers can easily be measured through self-reported data [ 67 , 74 ]. Ajzen [ 75 ] indicated that the variables in PMT influence consumers’ behavior by affecting their intentions, and consumers’ intentions significantly predict their behavior. Accordingly, this study focused on consumers’ intentions to stop purchases rather than their actual behavior. Second, information overload, as an attacking tool used by streamers to break through consumers’ psychological protection, was introduced into this model. Its function in live commerce has been confirmed. Finally, consumer resilience was introduced into the model as a moderating variable, and its role in the consumer’s reconstruction of self-protection mechanisms was also confirmed.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0305585.g001

3.2 Simulation environment design and survey

Due to the fact that retrospective surveys are detached from the actual situation, they can introduce some biases into the research. Therefore, to minimize errors, we designed and simulated a live streaming marketing environment to aid the survey., and the effectiveness of this method has been proved by previous studies of consumer psychology [ 26 ]. As consumers are not always in an information overload environment, asking them to respond to a questionnaire by recalling their most recent experience of information overload would result in measurement bias. The longer they are away from the information overload environment, the greater the bias. Additionally, in a pressure environment, it is necessary to accurately measure the effect of consumer resilience. Therefore, we simulated a live streaming environment to validate the model proposed in this study. In the full simulation environment, participants were completely anonymized, participants were informed of the purpose of the simulated live streaming marketing, only the necessary data were collected and kept strictly confidential, and certain rewards were given after the simulated live streaming marketing; informed consent was obtained by having a representative of the participant sign an informed consent form.

Simulation environment. (1) Streamer marketing encompasses three stages: interaction, product introduction, and transaction promotion. During the interaction and product introduction stages, the streamer must create an atmosphere to promote interaction and introduce the product. In the stage of promoting the transaction, the streamer must focus on utilitarian and hedonic information; they also use professional customer communication scripts by repeatedly emphasizing the effects of the product, thereby creating an atmosphere of a limited-time discount, urging consumers to immediately place orders, and as far as possible, exposing consumers to an information overload. According to these different stages, we compiled sample phrases for streamer reference (see supporting information S1 Appendix ). (2) According to Baidu index data ( http://index.baidu.com ), people aged 20–39 years in Jiangsu province of China were most concerned about live commerce in December 2021. So, this part of the population was taken as a recruitment target for participation in the live streaming marketing simulation. (3) Two streamers (one male and one female) were recruited. They were well-versed with using professional customer communication scripts for live streaming and had six to nine months of live commerce experience; each person received a commission of 1000 RMB (about 150 USD). (4) Seven imperceptibly branded products that are suitable for both men and women were used in the simulated live streaming environment. (5) Two online meeting channels, A and B, were prepared to simulate a live room. (6) Questionnaire scales validated in previous studies were used in the present study, with the questions adapted to the present context. The participants chose responses on a 5-point Likert scale; the questionnaire was reviewed and revised by experts in the field, yielding the final questionnaire used as shown in S2 Appendix .

Simulating the process of live streaming. (1) The participants entered the online meeting channel A, and the two streamers worked with each other to interact with the participants on the channel, as the staff provided each participant with a reward of 10 RMB (about 1.5 USD). When the number of participants reached 100, we closed the entry to Channel A, and the simulated live streaming marketing began. (2) The two streamers began to work with each other to introduce the product for 5 minutes. During this time, if participants thought that the product may not be suitable for them and they intended to stop purchase, they were asked to inform the staff and were invited to wait in online meeting channel B. They were then told that they could participate in a prize-draw by completing the next phase of the trial. (3) When the activity of channel A was finished, the two streamers entered channel B and started marketing by "pushing the deal.” They had to do their best to allow consumers to experience information overload. About 3 minutes later, participants were listening to streamer and filling out the questionnaires we prepared at the same time. After completing the questionnaires, the participants were given a gift.

We conducted seven simulations of live streaming marketing in January 2022; the details of simulations of live streaming marketing are listed in Table 1 . A total of 391 valid questionnaires were returned.

https://doi.org/10.1371/journal.pone.0305585.t001

4. Empirical analyses

4.1 analysis methods.

We derived the descriptive statistics of the sample and assessed the indicators related to data quality. The proposed hypotheses were tested using a research model.

There are two types of structural equation models: the covariance-based structural equation model (CB-SEM) and variance-based partial least squares structural equation model (PLS-SEM). In this study, PLS-SEM and the corresponding software package SmartPls 3.0 were used for data analysis [ 76 ]. The main reasons are: (1) Compared with CB-SEM, PLS-SEM is more suitable for measuring complex models, especially those with more than six variables [ 77 ]. There are seven variables in this study. (2) Compared to CB-SEM, PLS-SEM can calculate non-normal distribution data more effectively [ 77 ]. A multivariate normality analysis was performed on the data using a web-based calculator ( http://www.biosoft.hacettepe.edu.tr/MVN/ ) [ 78 ]. The results showed Mardie’s multivariate skewness (β = 29.297, p <0.05) and multivariate kurtosis (β = 470.706, p <0.001), indicating that the data in this study were multivariate non-normal [ 79 ]. (3) PLS-SEM is more suitable for small-sample measurements [ 77 ]. In summary, PLS-SEM was more suitable for analysis in this study.

4.2 Measurement

Prior to developing scales to measure the variables, we operationally defined the variables. 1) Product Utilitarian Value Uncertainty: This variable refers to the consumer’s perceived difficulty in assessing the attributes and future performance of a product showcased in live commerce. It signifies the degree to which consumers feel uncertain about the practical and functional value of a product. 2) Hedonic Value Uncertainty: This variable measures the consumer’s perceived uncertainty regarding the enjoyment and experiential benefits they will receive from purchasing a product during a live commerce session. It reflects the ambiguity consumers feel about the emotional and experiential returns of their purchase. 3) Experiential Efficacy: This combines self-efficacy and response efficacy, representing the consumer’s belief in their ability to make correct purchase decisions based on past experiences in similar contexts. 4) Response Costs: This variable reflects the perceived losses or costs associated with halting a purchase during a live commerce session. It includes the potential loss of additional benefits or emotional connections with the streamer. 5) Information Overload: In the context of this study, information overload refers to a marketing strategy employed by streamers where a vast amount of product information is delivered rapidly, surpassing the consumer’s ability to process it effectively. This overload aims to influence purchase decisions by creating a sense of urgency and reducing the consumer’s ability to evaluate the product critically. 6) Consumer Resilience: This variable measures the consumer’s ability to withstand and recover from the stress induced by information overload in live commerce. It indicates the consumer’s capacity to maintain or regain their self-protection mechanisms against incorrect purchases despite the overwhelming information presented by streamers.

To enhance the methodological rigor and ensure the content validity of our research, the scales implemented in this study were meticulously derived from extant scholarly literature, with necessary terminological modifications to tailor them to our specific research context. Specifically, the constructs of utilitarian value uncertainty and hedonic value uncertainty were adapted from Lu and Chen [ 6 ] and Park and Lin [ 12 ]. Additionally, the concept of experience efficacy was redefined based on the frameworks proposed by Farooq, Laato, Islam and Isoaho [ 32 ] and Tsai, Jiang, Alhabash, LaRose, Rifon and Cotten [ 37 ], while the measure of response cost drew upon the operationalization by Farooq, Laato, Islam and Isoaho [ 32 ]. Furthermore, the construct of Purchase Interruption Intention was refined following the methodology of Park and Lin [ 12 ], with the notion of information overload being recalibrated based on Farooq, Laato, Islam and Isoaho [ 32 ]. The dimension of consumer resilience was reconceptualized in accordance with Bermes [ 28 ].

Acknowledging the geographical specificity of our data collection in China, we employed the back-translation method to ensure linguistic and conceptual equivalence. Initially, the first author translated the original English questionnaire into Chinese, which was then back-translated into English by an independent translator unfamiliar with the study’s objectives. This iterative process facilitated a meticulous comparison of the two English versions, confirming their consistency without significant discrepancies.

To further ascertain the face validity of the instrument, we engaged three doctoral candidates specializing in Marketing and two experts in marketing to scrutinize each item for potential ambiguities. Their insights contributed to the refinement of the instrument. Subsequently, a pilot study involving 61 participants with prior experience in live commerce was conducted to validate the scale’s reliability. The details of the refined scale and its validation are documented in S2 Appendix of the Supporting Information.

According to Baidu index data ( http://index.baidu.com ), people aged 20–39 years in Jiangsu province of China were most concerned about live commerce in December 2021. Therefore, to obtain a representative sample, this study employed a combined methodology of random sampling and snowball sampling for data collection. During the random sampling phase, the research team randomly selected live streaming rooms and SNS platforms, engaging with the audience during or after the live streaming session to inquire about their willingness to participate in this study. This process was designed to ensure the randomness of the sampling, allowing the sample to broadly reflect the characteristics of the target population. In the snowball sampling approach, participants who consented to partake in the study were encouraged to invite their friends to join the research. These friends were required to meet the same participation criteria: being between the ages of 20 and 29 and having experience with live commerce. Through this method, the research team was able to reach a broader audience, particularly targeting individuals who might not frequently appear in the randomly selected live streaming rooms. Ultimately, study pre-recruited 1326 people aged 20–39 from Jiangsu province who have previously experienced wrong purchases during live streaming.

4.3 Demographics and bias test

A total of 391 valid questionnaires were collected. Among the participants, 202 (51.7%) were male and 189 (48.3%) were female; 182 (46.5%) were 20–29 years old and 209 (53.5%) were 30–39 years old; 132 (33.8%) had a college degree and 103 (26.3%) had a bachelor’s degree; and the largest number of people earned RMB 0–1999–117 people (29.9%)—while 106 people (27.1%) earned between RMB 2000 and 3999. There were 252 people who often shopped through live streaming in the TikTok app (64.5%) and 139 people who often shopped through live streaming in the Taobao app (35.5%).

To avoid nonresponse bias, we performed a paired t-test on the demographic data of the first and last thirty people who answered the questionnaire. The results showed no significant difference; therefore, nonresponse was not a serious problem in this study.

Common method bias is a common issue in questionnaires, and two methods were used to measure it in this study. First, Harman’s single-factor analysis was conducted [ 80 ]. The results showed that the percentage of extracted single variables was 26.10% (less than 40%). The common method bias in PLS-SEM was measured according to FULL-VIF [ 79 , 81 ], in which all VIF values were below 3.3. The results of both testing methods indicated that common method bias was not a serious problem.

4.4 Measurement model

To evaluate the measurement model, we evaluated composite reliability (CR), average variance extracted (AVE), discriminant validity, and outer loading. As shown in Table 2 , the variables’ composite reliability was >0.7, and Cronbach’s alpha was >0.7, indicating that the internal consistency of the data in this study is satisfactory. The AVE value of >0.5 and outloadings of >0.7 indicate that the convergent validity of the data is also acceptable [ 77 ]. Discriminant validity was measured using Fornell and Larcker’s test and the heterotrait-monotrait ratio (HTMT) test. As shown in Table 3 , the HTMT values were below the 0.85 threshold and the square root of each variable’s AVE was also greater than the correlation between variables [ 77 ]. Cross-loading is an additional criterion for measuring discriminant validity, and the results are presented in supporting information S3 Appendix . The results indicate that this study has good reliability, convergent validity, and discriminant validity.

https://doi.org/10.1371/journal.pone.0305585.t002

https://doi.org/10.1371/journal.pone.0305585.t003

4.5 Structural model

First, we checked for collinearity. The VIF values of the variables were all below 5; therefore, collinearity was not a major issue in this study. After ensuring that the reliability, validity, and collinearity of the model were not a problem, we analyzed the structural model to verify the hypothesis. The path coefficients and significance test results of the structural model are presented in Table 4 and Fig 2 . Utilitarian value uncertainty (β = 0.238, p < 0.001) and experiential efficacy (β = 0.116, p < 0.05) have a significant positive effect on consumers’ intention to stop purchase; thus, H1a and H2a are supported. Response cost (β = -0.349, p < 0.001) has a significant negative effect on consumers’ intention to stop purchase, supporting H2b. Information overload has a significant negative effect on utilitarian value uncertainty (β = -0.390, p < 0.001) and experiential efficacy (β = -0.165, p < 0.01), supporting H3a and H4a. Information overload has a significant positive effect on response cost (β = 0.374, p < 0.001), supporting H4b. However, the effect of hedonic value uncertainty (β = 0.012, p>0.05) on consumers’ intention to discontinue a purchase is not significant; thus, H1b is not supported. The effect of information overload on hedonic value uncertainty (β = -0.071, p < 0.064) is not significant; therefore, H3b is also not supported. This study also measured the effect of control variables (subject’s age, gender, income, education level, and platform use) on stop purchase intention, and none of them were significantly related. In addition, there was no significant effect of different product categories on stop purchase intention.

https://doi.org/10.1371/journal.pone.0305585.g002

https://doi.org/10.1371/journal.pone.0305585.t004

Finally, we tested the goodness of fit of the model. We used the standardized root mean square residual (SRMR) values to check the goodness of fit of the model in this study. The SRMR value was 0.063, which met the requirement of being less than the threshold value of 0.08. We conclude that the fit of this study is satisfactory [ 77 ].

4.6 Moderation effects

In this study, "consumer resilience" was proposed as a moderator and its moderating effect was tested. The test was performed in two parts. First, the significance of the moderating effect was measured. Following this, the strength of the moderating effect was measured by calculating F 2 , which was calculated as (R 2 interaction model–R 2 main effects model) / (1–R 2 main effects model). An F 2 value between 0.02 and 0.15 indicates a small moderating effect, a value between 0.15 and 0.35 indicates a moderate moderating effect, and a value greater than 0.35 indicates a high moderating effect [ 82 ].

The moderation effects are shown in Table 5 and Fig 3 . Consumer resilience significantly moderates the negative effect of information overload on utilitarian value uncertainty (β = 0.082, p<0.05) with a small effect size (F 2 = 0.143). Consumer resilience also significantly moderates the negative effect of information overload on experiential efficacy (β = 0.124, p<0.01), with a small effect size (F 2 = 0.047). Consumer resilience significantly moderates the positive effect of information overload on response costs (β = -0.100, p<0.05), and the effect was medium (F 2 = 0.203). There was no significant moderating effect of consumer resilience on information overload and hedonic uncertainty (β = -0.100, n.s).

Abbreviations: UVU, utilitarian value uncertainty; HUV, hedonic value uncertainty; EEY, experiential efficacy; RCT, response cost; IOD, information overload; CRE, consumer resilience.

https://doi.org/10.1371/journal.pone.0305585.g003

https://doi.org/10.1371/journal.pone.0305585.t005

5. Discussion and conclusion

5.1 key findings.

In this study, we have meticulously explored the intricate dynamics of consumer behavior within the sphere of live commerce, guided by the foundational principles of Protection Motivation Theory. Our investigation reveals that both threat and coping appraisals serve as essential mechanisms through which consumers navigate their purchasing decisions, particularly under the influence of the live streamers’ interactive strategies. Notably, we found that the uncertainty regarding product quality significantly influences consumers’ hesitation and propensity to abort potential purchases, highlighting the critical role of perceived product value in the digital shopping realm. Moreover, our findings illuminate the profound impact of past purchasing experiences on consumer behavior; positive experiences tend to bolster consumers’ confidence and likelihood to engage in repeat purchases, while negative experiences enhance their protective instincts, deterring future purchasing activities. This behavioral pattern is accentuated by the strategies employed by streamers, who often emphasize product value to stimulate spending, effectively mitigating information asymmetry and enhancing consumer purchase intentions. Interestingly, our research also uncovered some divergences from existing studies, particularly concerning the influence of hedonic value uncertainty and information overload, which did not significantly affect purchase decisions as previously theorized. This discrepancy may stem from the unique characteristics of the live commerce platforms in China, where the distinction between e-commerce and entertainment-focused platforms shapes consumer expectations and engagement levels. Our simulated live streaming marketing design, involving streamers and consumers without prior interaction, likely did not replicate the depth of emotional engagement typically observed on entertainment-centric platforms, thereby influencing the outcomes.

The following discussion will focus on several specific objectives of this study. The primary objective of this investigation was to elucidate the mechanisms by which consumers safeguard themselves from erroneous purchases within the realm of live commerce. Drawing upon Protection Motivation Theory, this study delineates how threat and coping appraisals act as pivotal determinants in fostering protection motivation among consumers [ 30 ]. Within the specific milieu of live commerce, our analysis substantiates that these appraisals can be discerned through more granular factors, which in turn, orchestrate a consumer protection mechanism, influencing their purchasing decisions. Our empirical findings illuminate that product uncertainty exerts a positive influence on consumers’ inclination to halt purchases. Such uncertainty, recognized as a utilitarian value in the digital shopping domain, amplifies consumers’ hesitancy regarding the congruence between product quality and their requirements, thereby diminishing their purchasing intent—a phenomenon that is accentuated in online shopping environments [ 6 ]. This apprehension, born out of uncertainty, propels consumers towards developing a self-protection intent. Furthermore, this study elucidates the role of coping appraisal, where the impact of experience effectiveness and response costs are meticulously examined. Aligning with Lin [ 56 ] observations, our findings suggest that consumers with positive past purchasing experiences are more predisposed to repeat purchases. Conversely, negative experiences galvanize a robust self-protection stance, potentially curtailing future purchasing behaviors. Additionally, the study delves into response costs, a crucial element in the consumer’s coping appraisal, emphasizing how streamers in live commerce frequently accentuate the added value of products to stimulate consumer spending [ 13 ]. In concordance with Xu, Cui and Lyu [ 26 ], our study corroborates the notion that real-time interaction in live commerce can mitigate information asymmetry, thereby enhancing consumers’ purchase intentions by alleviating product uncertainties. Concurrently, our insights on response costs find a parallel in Chen and Yang [ 70 ] discourse on the significance of network structural embeddedness in shaping consumer purchase intentions in cross-border e-commerce, offering a complementary viewpoint on consumer decision-making online. Moreover, our analysis on the efficacy of past experiences finds resonance with Helin, Donglu, Shaoying, Decheng and Bei [ 27 ] investigation into the impact of online comments and merchant responses on tourism product sales, underscoring the interplay between various facets of online commerce. The study by Zhang, Yang and Bei [ 18 ] further enriches our discussion, presenting a nuanced perspective on how psychological ownership and social capital influence consumer behavior in virtual communities, thereby offering strategic insights to mitigate incorrect purchasing in live commerce. This comprehensive discussion, grounded in empirical evidence and theoretical frameworks, not only amplifies our understanding of consumer self-protection mechanisms in live commerce but also establishes a confluence with extant research, thereby contributing to the broader discourse on consumer behavior in online shopping environments.

The Second objective of the present study was to investigate the potential of information overload, propagated by live streamers in e-commerce environments, to circumvent the mechanisms consumers employ to protect themselves from erroneous purchases. The analysis conducted confirms that information overload does, indeed, compromise these self-protection mechanisms by attenuating threat appraisal, thereby reducing consumer uncertainty about the practical value of products. This finding aligns with the research conducted by Xu, Cui and Lyu [ 26 ] which suggested that the direct interaction between streamers and consumers during live sessions may decrease information asymmetry and increase the intention to purchase. However, the authors also highlighted the critical influence of streamer professionalism and the parasocial relationship with the viewer on this dynamic. Additionally, our findings indicate that information overload can weaken coping appraisals within consumer self-protection frameworks, evidenced by decreased effectiveness of experience and increased consumer response costs. This observation is consistent with the study by Wei, Hai, Zhu and Lyu [ 25 ], which examined the effect of consumers’ deferral of choices on their preferences in intertemporal decision-making, emphasizing the vital role of perceived information integrity in influencing consumer preferences. Complementing our insights, the application of the stimulus-organism-response model by Zhang, Qi and Lyu [ 9 ] illustrates how knowledge sharing within virtual communities can shape consumer-brand relationships, thereby underscoring the significance of information quality and community interaction in influencing consumer perceptions and decision-making processes. Furthermore, our exploration into the direct impacts of information overload is enriched by the findings of Zhang, Yang and Bei [ 18 ], who investigated the roles of social capital and psychological ownership in virtual communities on knowledge sharing among consumers. Their study suggests an indirect influence on consumer decision-making and a potential buffering effect against information overload. In essence, these studies collectively illuminate the intricate dynamics between information presentation, consumer perception, and decision-making in the realm of digital commerce. They underscore the necessity of a balanced information delivery approach and the cultivation of positive consumer relationships and vibrant community interactions to counteract the detrimental effects of information overload. Such strategies not only aid in consumer decision-making but also enhance the overall efficacy of live commerce platforms. Notably, streamers’ creation of a false sense of urgency and the subsequent consumer difficulty in accurately assessing product value, as highlighted by Cao, Liu, Shang and Zhou [ 59 ], Soroya, Farooq, Mahmood, Isoaho and Zara [ 66 ], Sun, Shao, Li, Guo and Nie [ 2 ], Wongkitrungrueng and Assarut [ 7 ], and Zhang, Sun, Qin and Wang [ 13 ], accentuates the critical need for mitigating these effects to foster informed consumer choices and enhance the integrity of live commerce environments.

The third objective of our research was to delve into how consumer resilience influences the reactivation of self-protection mechanisms against unsuitable purchases during instances of information overload from streamers in live commerce environments. Our results substantiate the critical function of resilience in alleviating information-induced stress, echoing [ 28 ] observations that, within the live shopping sphere, the deluge of information tends to erode consumer resistance to temptation, yet resilience can counterbalance this effect by diminishing the impact of information overload. Additionally, our investigation validates that resilience mitigates the negative ramifications of information overload on the uncertainty associated with a product’s practical value and on the effectiveness of experiences, while concurrently having a beneficial impact on response costs. This aligns with the findings of Helin, Donglu, Shaoying, Decheng and Bei [ 27 ], who unravel the intricate interplay between online feedback, merchant reactions, and the sales metrics of tourism offerings, underlining the capacity of consumer engagement strategies to shape perceptions and influence decision-making. Such evidence intimates that resilience, similar to proactive consumer engagement, may act as a protective mechanism within the information processing continuum, augmenting the consumer’s proficiency in selectively assimilating and evaluating information. Furthermore, the analysis conducted by Zhang, Qi and Lyu [ 9 ] utilizing the stimulus–organism–response paradigm within virtual communities accentuates the manner in which external stimuli, such as information overload, can affect consumer-brand relationships. Their insights into the importance of the quality of knowledge exchange in molding consumer perceptions highlight the utility of resilience in aiding consumers to traverse information-dense environments more adeptly, thus fostering more robust consumer-brand connections despite the prevalence of information overload. Moreover, the research by Xu, Cui and Lyu [ 26 ] concerning the interplay between a streamer’s professionalism and the parasocial relationship with viewers in live commerce provides an illustrative backdrop wherein consumer resilience may be particularly advantageous. In scenarios where the professional conduct of streamers and relational dynamics are pivotal, resilience emerges as a key trait enabling consumers to critically assess information, thereby bolstering their decision-making capabilities amidst persuasive tactics utilized by streamers. This discourse underlines the indispensability of resilience in the contemporary consumer’s toolkit, offering a shield against the barrage of information inherent in the digital commerce landscape.

In this study, we encountered findings that diverge from existing research, particularly in the domain of consumer behavior in live commerce settings. Firstly, the anticipated influence of hedonic value uncertainty on disrupting purchase decisions did not align with the established consumer protection mechanisms against incorrect purchases identified in prior studies. Secondly, the anticipated adverse impact of information overload on hedonic value uncertainty failed to materialize in our analysis. This discrepancy invites a nuanced interpretation, possibly tied to the distinct nature of live commerce platforms in China, as identified by Cai, Wohn, Mittal and Sureshbabu [ 50 ]. The landscape of live commerce in China bifurcates into two primary categories: platforms that are extensions of traditional e-commerce services and those that evolve from live streaming entertainment platforms. This distinction is not trivial, as it underpins the varying consumer motivations across these platforms. On e-commerce-centric platforms, the consumer’s focus is predominantly on the utilitarian aspects of the product. Conversely, on platforms with roots in live streaming entertainment, the hedonic value derived from interactions with streamers tends to take precedence. The simulated live streaming marketing design of our study, which involved streamers and consumers who were unfamiliar with each other prior to the experiment, might not have effectively replicated the depth of emotional engagement typically observed on live streaming entertainment platforms. Despite efforts to induce hedonic value through designed information stimuli, the absence of pre-existing emotional connections likely attenuated the potential for forming strong attachment and trust bonds within the limited timeframe of the simulated live streaming marketing. Given that the simulated live streaming marketing setup mirrored the context of an e-commerce platform incorporating live commerce features, participant focus was likely skewed towards product utility rather than hedonic value. Consequently, the anticipated influence of hedonic value uncertainty on purchase interruptions did not manifest significantly, suggesting that the context and nature of consumer-streamer interactions play a critical role in shaping purchase behaviors in live commerce environments.

5.2 Theoretical contributions

First, the model proposed in this study confirms the existence of an offensive and defensive game between streamers and consumers in commercial marketing, which provides a new theoretical perspective regarding the operation of live commerce and enriches the marketing literature. (1) Consumers measure the level of potential harm from a product purchase during live stream by assessing the uncertainty in utilitarian value and then decide whether to stop purchase by combining experience effectiveness and response cost, to protect themselves from being harmed by a wrong purchase. (2) The streamer can effectively break through the consumer’s self-protection mechanism by using an information overload strategy. However, consumer resilience can also mitigate the impact of this strategy.

Second, PMT was applied to live commerce to analyze the protection psychology of consumers, thereby expanding the scope of application of PMT. The results showed that (1) consumers can measure the degree of harm from wrong purchases by assessing a product’s utilitarian value uncertainty in live commerce, which enriches the dimension of threat appraisal in PMT; (2) consumers assess their ability to adopt protective behaviors by weighing the pros and cons of adopting protective behaviors based on their experience and existing abilities, which enriches the dimension of coping appraisal in PMT.

Third, the important role of information overload in the field of marketing was confirmed by introducing information overload into PMT, which extends the scope of application of information overload theory and enriches the literature in the field of marketing. (1) The effect of information overload on consumer uncertainty about product value was confirmed, enriching the antecedents of threat appraisal in PMT. (2) The effect of information overload on experiential efficacy and response cost was confirmed, which enriches the antecedents of coping appraisal in PMT.

Finally, this study makes some contributions to resolving the controversy over the relationship between information overload and consumers’ purchase intentions. 1) Previous studies have pointed out that information overload can both positively influence consumers’ purchase intentions[ 19 , 20 ], and negatively affect them [ 19 , 21 ], From the results of this study, the reason for this controversy can be explained as follows: consumers have a psychological defense mechanism, when the information overload marketing from the seller is sufficient to break the consumer’s psychological defense mechanism, it actively pushes the consumer to purchase; if the information overload is not sufficient to break the consumer’s defense mechanism, the consumer will not purchase. 2) Previous studies have also suggested an inverted U-shaped relationship between information overload and consumers’ purchase intentions[ 22 , 62 ], The results of this study can also explain such results: When information overload is low, users can make high-quality decisions, purchase intentions are less influenced by marketing, and purchase decisions are rational; When the degree of information overload reaches a certain level, the quality of users’ decisions gradually decreases, and they gradually become more influenced by sellers’ marketing and are easily prone to make irrational purchases. However, because of the existence of consumer resilience, consumers will gradually get used to the information overload, even if the information overload given by sellers becomes higher and higher, the impact on consumers is more and more limited, and consumers’ purchase decisions tend to be rational again. In addition, this results also confirm that consumer resilience can weaken the impact of merchant marketing. This provides a new perspective in the study of resilience in the field of marketing and enriches the PMT.

5.3 Practical contributions

The findings of the research underscore the nuanced interplay between streamer strategies and consumer responses within the live commerce context, highlighting the need for a holistic and ethical approach to enhance both the effectiveness and the ethical standards of live commerce. Streamers play a crucial role in shaping the consumer experience, and their actions can either empower consumers or lead them into decision-making traps spurred by information overload.

To navigate this delicate balance, streamers require advanced training that goes beyond mere communication skills and script usage. They need to be imbued with ethical marketing practices, ensuring they present information in a way that is clear, engaging, and not overwhelming. This approach helps in creating an environment where consumers are informed and involved, rather than being led into an information prison where they are more susceptible to making impulsive or uninformed decisions.

On the other side, empowering consumers is equally vital. The research findings suggest that consumers equipped with the right knowledge and tools can critically assess the information presented to them during live commerce sessions. Awareness campaigns and educational initiatives are crucial in helping consumers recognize signs of information overload and guiding them to use external platforms for additional verification of product details. Such informed consumers are more resilient to aggressive marketing tactics and can make autonomous purchasing decisions.

Resilience among consumers emerges as a key theme in the study, indicating that when consumers have resources to enhance their knowledge and decision-making capabilities, they are better positioned to withstand marketing pressures. This resilience is further supported when live-commerce platforms and streamers commit to ethical marketing practices, prioritizing consumer well-being and ensuring transparency and honesty in their communications.

Continuous monitoring and gathering consumer feedback are essential to ensure that live commerce evolves in a direction that aligns with consumer preferences and tolerances for information. By integrating these strategies, live commerce can strike an optimal balance between engaging marketing and consumer well-being, creating an environment conducive to informed choices and respected consumer autonomy.

Thus, the research findings advocate for a comprehensive approach that incorporates streamer training, consumer empowerment, resilience building, ethical marketing, and continuous feedback. This approach not only enhances the effectiveness of live commerce but also upholds its ethical standards, ensuring a sustainable and consumer-friendly live commerce ecosystem.

5.4 Limitations and future directions

While this study provides valuable insights into consumer behavior and streamer interactions in live commerce, it also presents several limitations that future research should address. Firstly, the simulated live streaming marketing design, particularly the simulation environment for live shopping, lacked emotional bonds between streamers and consumers. The participants were not familiar with the streamers, who were not prominent Internet celebrities. This setup overlooked the critical ’fan effect’ in live commerce, which could influence consumption motivations tied to hedonic values. Future studies should consider the simulated live streaming marketing involving well-known personalities to better capture this dynamic.

Additionally, the study did not account for interactions among consumers, which can be a significant factor in live commerce environments. Consumer-to-consumer interactions can sometimes mitigate or exacerbate the effects of information overload, suggesting that future research should explore this aspect more thoroughly. The study also focused solely on the streamer-to-consumer relationship, neglecting other forms of overload such as system, communication, and social overload, which warrant further investigation.

In terms of consumer protection mechanisms, this study primarily drew from protection motivation theory to identify key factors. However, there might be other relevant theories and factors that could provide a more comprehensive understanding of consumer behavior in live commerce settings. Future research could explore additional theoretical frameworks to uncover other potential mechanisms of consumer protection. Furthermore, the study’s geographic and demographic scope was limited to Jiangsu Province, China, with a relatively small sample size that may not fully represent the broader population. This limitation raises concerns about the generalizability of the findings to other regions or demographics. Future research should aim to include a more diverse and larger sample to enhance the external validity of the findings.

Supporting information

S1 appendix. examples of customer communication script..

https://doi.org/10.1371/journal.pone.0305585.s001

S2 Appendix. Measurement items.

https://doi.org/10.1371/journal.pone.0305585.s002

S3 Appendix. Discriminant validity- cross loading.

https://doi.org/10.1371/journal.pone.0305585.s003

https://doi.org/10.1371/journal.pone.0305585.s004

• View Article
• Google Scholar
• PubMed/NCBI
• 50. Cai J.; Wohn D.Y.; Mittal A.; Sureshbabu D. in Utilitarian and Hedonic Motivations for Live Streaming Shopping, Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online Video, New York, USA, 2018-1-1, 2018; ACM: New York, USA, 2018.
• 53. Bojun L. in How is the Purchase Intention of Consumers Affected in the Environment of E-commerce Live Streaming? Proceedings of the 2021 International Conference on Financial Management and Economic Transition (FMET 2021), Paris, France, 2021-1-1, 2021; Atlantis Press: Paris, France, 2021.
• 54. Bandura A. Social Foundations of Thought and Action. Prentice Hall: Upper Saddle River, NJ, United States, 1986; p.
• 67. Farooq A.; Ndiege J.R.A.; Isoaho J. Factors Affecting Security Behavior of Kenyan Students: An Integration of Protection Motivation Theory and Theory of Planned Behavior. In 2019 IEEE AFRICON., Accra, Ghana, 2019; pp 1–8.