Update Feb 18, 2025: The experiment described under "the first A/B test" below is now running.
Users with low rep, typically new to Stack Overflow, frequently chafe against the limitations they encounter when they first try to use the website. Reports of this feeling are virtually countless across Meta, throughout interviews with users, across our surveys, in our support inbox, and throughout conversations with employees who use the network for the first time (e.g. during community-a-thon).
Right now, when a user tries to perform an action they can’t do, our system tells them quite plainly that they are not able to participate in that way (yet). For example, a user might see:
But “get 50 rep” is not something a user can just tick off—it’s not quick and not necessarily all that easy, either. In essence, this message boils down to, “No, you can’t do that right now.” (We’ve got a related experiment in progress for comments as well.)
What if the user tries to vote?
The wording is softer, but the underlying meaning is the same. The UI actually reverts the action the user tried to perform, returning the vote button to its neutral color, visually showing the user that their vote has not been accepted.
These automated messages - well, they’re not bad. You gotta admit, they’re simple, and they do get the job done. Messages like these appear in a number of places across the network, usually associated with early participation and privileges.
What if our systems instead offered users the option to do something else - something they did have permission to do? That’s the question of the day.
When a user tries to participate and is refused, we believe this changes how they view the network (and not usually for the better).
When we converse with network users, both on the platform and in user interviews, they consistently point out that the system restricts them from participating in the way they naturally want to. Separately, we find that many newer users, or users with lower rep, are unsure how to participate successfully on the network; many are not even sure how to effectively use the features that are available to them. From our research, we know that users often create accounts to perform specific actions (upvote, comment, edit e.g.), only to see these messages after creation and learn that they are not permitted to do what they set out to do.
The original rationale behind the wording of the error messages above was reasonable: Users who encounter restrictions might be encouraged to do something about it if those restrictions are explained to them simply and sufficiently well. In most cases, this means the rationale suggests the user will go out and get the rep they need, potentially becoming active participants in the process. However, a growing body of evidence suggests that users do not respond this way. Instead, when these users attempt to participate on the network, and their attempt is refused by the system, they are inclined to feel shut out. They may learn that…:
- …the system opposes what they naturally want to do.
- …the system is difficult to engage with.
- …they are not trusted to engage in basic activities.
If a user feels these statements about the network are true, then we believe this makes it less likely for them to return to the network and participate in the future.
As with any assessment of users’ thinking, this set of beliefs has its limitations. It is informed by both research and discussion, and neither of those is foolproof. Some of our beliefs about this state of affairs could be wrong. But the evidence strongly suggests there is something worth testing here.
I want to be clear up front that we are not proceeding on the assumption that it is appropriate to simply remove the associated restrictions. For example, while we could remove the 50-rep commenting restriction, the design choice is load-bearing: we can’t remove it wholesale without expecting some unwelcome knock-on consequences for curators and Stack Exchange systems. We could certainly find a safe way to remove that rep requirement, but it would necessitate a deeper rethinking of the way the network operates. That’s a larger proposition, and it would be hard to test nondisruptively. Even if it ends up being the right thing to do, we’d be wise to attempt to validate our beliefs a bit first. So…:
We are running a series of A/B tests to better understand how, and whether, to redirect users to alternative actions.
While it’s very difficult to measure what goes on in a user’s head, we can test against what actions a user performs after they see a modal. When a user attempts to perform an action they can’t yet do, we expect that if we redirect that user toward a related action they can do, they will be more likely to return and contribute in the future than if they’d been shown an error message.
We are going to run a series of experiments surrounding this theory, which all involve redirecting users from an action they can’t (yet) perform towards an action they can currently perform. However, we’re going to take it a step at a time. What we learn from earlier tests informs what we try in later tests. So for now, let’s just look at:
The first A/B test
All of our planned experiments involve redirecting users from an action they can’t do towards an action they can do - one that should be as closely related to the original action as possible.
The first test involves redirecting users who attempt to upvote a post to save that post for later instead. During this A/B test, users in the ‘B’ group will see the following modal upon voting:
We plan to measure the difference in future participation between the A and B groups.
We expect that the exact nature of what we test and measure will change between each experiment that we run. However, in general, you can expect us to run a fairly standard A/B test.
Our goal will be to produce a control group and a test group that have approximately the same demographics (reputation, contribution history, creation time), and measure how their activity changes. The control group will see the current error message, and the test group will see a redirection modal.
Between the control and test group, we are particularly interested in comparing:
- The fraction of users who later return to the network.
- The fraction of users who later perform specific contribution actions (ask, answer, edit, e.g.).
We’ll also be monitoring how often users engage with the redirection modals in order to make sure that it’s working the way we intend (though this is not, strictly speaking, a measure of success).
Cleanup: timeline and your requests
This is an open-ended experiment series. We expect to make modifications to what we plan to test as we go. Though some of our experiments may become long-term changes to the platform, our primary aim is not to release a specific change - yet. Our immediate interest is in understanding how users respond to redirection across a few different designs and actions. This also means the timeline for these experiments is fluid as we run tests that we hope validate the concept, and if so, help us settle on a final design.
The first A/B test is scheduled to begin the week of February 18th, but after that - we’ll see.
This also means that if you’ve got specific requests for redirections we should test, we’d be happy to consider them in answers below. (We may or may not be able to do them, depending on their complexity, though!)
