Pop-up Focus
"[Navigating a website is] kind of like walking through a mall with multiple kids and they all see something they want."
– Isabel, a high-confidence screen reader user and teacher
"[Navigating a website is] kind of like walking through a mall with multiple kids and they all see something they want."
– Isabel, a high-confidence screen reader user and teacher
This functional specification establishes the need for a machine learning (ML) dataset and algorithm in order to identify pop-ups and inform screen-reader users of a pop-up’s existence, type, and exit mechanism so that they can close the pop-up if they desire. As well as justifying the dataset creation, this document outlines a method and guidelines for creating said dataset. Ultimately, this project will serve as a step in making the internet more accessible to screen reader users who are blind.
Our project focuses on screen readers and how they interact with pop-ups on the Internet. As a result of extensive user research with screen reader users who are blind, we learned that pop-ups present challenges in web navigation that are often annoying and sometimes completely blocking. In order to appreciate the issues that arise between screen readers and pop-ups, it is important to understand what screen readers are and how they work.
Screen readers are one of the most widely used accessibility tools for navigating the Internet. Screen readers navigate the web based on the site's code rather than the visual cues that sighted people can pick up on. An example to illustrate this interaction, shown in figure 1, is that in the case of web search, screen reader users can navigate by clicking "skip to main content," and it takes them to the first search result link. The search engine's code designates search result links as heading level one, so users can easily skip the elaboration text for each result. They can also choose to drill down into the detailed content below the header at any point.
Figure 1 : Screen Reader Demonstration
The screen reader has a focus that shifts upon commands from the user.
In order to fully understand this report, it is necessary to define exactly what a pop-up is. We define pop-ups as a graphical user interface (GUI) display area, usually a small window, that suddenly appears in the foreground of the visual interface. This can range from a small drop-down banner to a large pop-up window that covers most of the screen.
From users, we heard that one of the biggest desires when it comes to pop-ups is to simply know that it is there. There are two distinct issues with pop-ups. First, when a user is navigating a website with a screen reader, a pop-up may appear that contains meaningful information such as a login, and the screen reader is unable to detect the pop-up. This leads to screen reader users not being able to access information vital to their web navigation process. On the other hand, pop-ups can appear unexpectedly and instantly change the focus of the screen reader to another portion of the webpage. This quick shift can cause disorientation and confusion especially for lower confidence screen reader users. This issue is compounded when there is no clear mechanism to exit the pop-up. For example some pop-ups do not appear with an “x” or “Dismiss” button and must be closed by clicking outside of the pop-up, which is difficult for screen reader users who navigate the web using almost exclusively keyboard input.
As a result of a six-month study including a survey of screen-reader users and interviews with both accessibility experts and people who are blind, we identified that interacting with pop-ups is disproportionately difficult for screen reader users. This is largely because pop-ups do not conform to the linear progression of screen-reader focus as one navigates a page. Additionally, pop-ups are intended as a visual component of web design with the goal of shifting the user’s focus away from the main content of the page.
In order to illustrate some of the experiences we heard about during our interview and survey process, this section contains two stories outlining the two most common types of issues with pop-ups for screen reader users. As a disclaimer, these stories are composites of experiences relayed to us through several user interviews and pseudonyms are used to protect our interviewees’ identities.
Chris (He/Him) is a screen reader user who is blind, and has been using screen reader technology for years. His job requires some work to be done online and he is able to expertly navigate the internet to complete those tasks.
One aspect of his job involves ordering parts for office repairs, and recently a door hinge broke requiring a new door hinge to be ordered. He knows the exact type of door hinge that needs to be ordered, and as a result of a google search quickly finds that this hinge is available to order on the Home Depot website.
Chris runs into trouble however when he looks for the account login button, that he can’t find. On the old website version, to log in to his account all he had to do was navigate to the top right corner of the webpage and there is a login entry there. Chris has spent the better part of an hour searching for this user login but simply cannot find it. Ultimately, Chris calls over one of his sighted co-workers to help him locate the user login. This was easy for the sighted co-worker as Home Depot had changed their login to appear as a pop-up that covers most of the screen. Chris’ screen reader focus however, did not detect the pop-up and Chris was navigating through the information behind the pop-up, and not interacting with the pop-up itself even though it took up a large part of his screen. Chris was then able to perform the rest of this task without issue. In describing this situation, Chris said, “I lacked a lot of self-sufficiency".
This example involving Chris illustrates a few important aspects of how pop-ups can impede web navigation of screen reader users. It is important to point out that even though Chris is a highly competent screen reader user, some pop-ups are simply not detectable to screen reader users unless a visual cue is given. This is a complete blocker of progress and can prevent users from performing vital tasks. This situation is easily remedied by implementing the first aspect of the upcoming solution overview which is to inform the screen reader user that a pop-up is present. With that information, Chris would be able to then find the pop-up, input the correct information, and order the door hinge. We heard from many of our interviewees that simply knowing of a pop-up’s existence is crucial in navigating past these blockers.
Sam (They/Them) is a local DJ who is blind and has used screen readers for a little while. They would describe themself as a low-confidence screen reader user. They often use a variety of websites to get new samples and music to incorporate into their DJ sets. Most of the time this works pretty well, they go to a site, test all the potential sounds, and download the ones they want.
Sam was trying out a site they had heard about from friends, but had never tried before. At the beginning everything was going great, the website layout was intuitive and Sam was able to navigate around the site without issue. Then, suddenly, the screen reader started outputting content that was not at all related to what Sam was doing, and Sam had no idea what was going on.
After trying to get back to the main content of the site, unsuccessfully, they decided to ask their sighted roommate for help to figure out what was going on. It turned out that a pop-up advertisement had appeared and started playing a video. Sam had no idea that a pop-up was on their screen, and because of the sound from the video was not able to navigate around the pop up. Sam’s roommate was able to simply close the pop-up using the mouse and Sam was able to return to their previous page.
This type of pop-up is different from the first example because instead of not appearing to screen reader users, this one appeared and took up all of the audio content. While this pop-up was a complete blocker for a low-confidence user such as Sam, more advanced screen reader users have often built up the intuition that shows them that this is a pop-up. The proposed solution overview is built on scenarios like this one which make it vital for screen reader users to know that a pop-up has appeared, the type of pop-up that has appeared, and the exit mechanism so they can return to their original activities like testing out sounds for their DJ set.
This solution overview section will detail the desired output of the machine learning and computer vision algorithm, as well as a method for how to collect the data necessary to create a dataset that informs that algorithm.
This section will describe how our user interviews and survey data informed the solution for Pop-ups causing barriers for people who are blind. In addition to describing how this desired output was reached, this section also describes in detail what the user would expect to see as a result of the algorithm built from the dataset.
Throughout our interview process, we heard that the biggest desire when it comes to pop-ups is to simply know that it is there. As illustrated in the user scenarios, there are two main distinct issues with pop-ups. First, when a user is navigating a website with a screen reader and a pop-up appears that contains meaningful information, and the screen reader is unable to detect the pop-up. Additionally there are pop-ups that can appear unexpectedly and instantly change the focus of the screen reader to another portion of the webpage. We used these two scenarios to design a desired output from an algorithm that would best address the two issues.
From our user research, we discerned three main pieces of information that would be most useful to screen reader users when pop-ups appear:
This information is important so that screen reader users can be alerted if a pop-up appears, which is useful both so that users aren’t missing important information and so that they know the cause of disorientation if the screen reader focus changes. The type of pop-up is also important for users to know so that they are able to understand how the webpage has changed and how their navigation has been interrupted. And lastly the exit mechanism is important for users to know so that they aren’t stuck on a pop-up.
Therefore, our desired output would be an audio indication that a pop-up has appeared along with a description of the type and exit mechanism. The pop-up type is defined as the form in which it appears on the screen, such as “floating bar”, or “fullscreen”. Further definitions of pop-ups types with detailed descriptions and example images are included in the appendix of this document. The exit mechanism describes the action that the user would have to take to exit the pop-up. There are only three major types of exit mechanisms, an “X” button (typically in the top right corner), a text exit (Usually in the form of an underlined link that says, “ Dismiss ” or “ No, I’m not interested ”), and in some cases the only way to exit a pop-up is to click outside of the boundary of the pop-up. Additionally, there are pop-ups that the user should be notified of, but no exit mechanism required. For example with a floating bar pop-up, and this exit mechanism would be labeled as “None”.
This quick summary would be informed by the algorithm designed based on the dataset created as a result of this functional specification. As an example, the screen reader audio output would be something like this:
“A full screen pop-up has appeared with a dismiss button.”
This sort of output would fulfill the user requirements of alerting the user of the pop-up’s presence, describing its type, and giving screen reader users an indication of how to find the exit mechanism.
This categorization section outlines the several distinct categories by which pop-ups can be classified. There are two categories explored: Type of Pop-up and Pop-up Exit Mechanism. A combination of both of these pieces of information is what the end users would find most valuable for moving past this barrier. In order to effectively label and create the dataset for an algorithm to tackle the problem of pop-up interactions with screen readers, a standardized categorization of pop-ups is necessary.
The following table outlines the variation in types for each of these categories. These categories were generated from internet research into the different types of, and situations in which you would find pop-ups during regular internet use. Determining the categories for exit mechanisms involved looking at many different examples of pop-ups as they appear in the everyday use of the internet, as well as researching several marketing companies’ protocols for pop-up advertisement creation. While there are variations on each of the categories listed below, such as text exit being a button or a link, all pop-ups have a mechanism that fall into one of the four distinct categories. A similar process was conducted for determining the type of pop-up, but involved more extensive research into web development guidelines such as JavaScript pop-up creation guides and analysis of website creator pop-up options (such as Squarespace or Wix).
|
Pop-Up Types |
Exit Mechanisms |
|
Lightbox |
“X” button |
|
Scroll-in |
Text exit |
|
Floating Bar |
Click Outside |
|
Fullscreen |
None (must proceed) |
|
On-Click |
|
Note: The pop-up types and exit mechanisms listed above do not correspond horizontally.
All of these pop-up type descriptions have corresponding example images in the first appendix of this report.
Lightbox pop-ups appear in a smaller new window over the web page you’re currently browsing. When a lightbox pop-up shows up, it makes the background of your browser look darker to really grab the user’s attention. Though lightbox popups can be used for many purposes, they’re mostly used for advertising purposes.
These are small sideboxes located at the bottom of the page that usually appear on a webpage to give information or promotional offers to the visitor. Scroll-in pop-ups are activated by scrolling down a webpage.
Floating bar pop-ups often don’t cover the whole screen, but can appear at the top or bottom of a page giving new information.
A fullscreen pop-up is similar to a lightbox pop-up but covers the entire screen. There is no darker background, it simply covers the entire webpage.
When clicking buttons such as, “Try it free” a pop-up appears prompting further action. On-click pop-ups appear exclusively as a result of user interaction on a web page.
All of these exit mechanism descriptions have corresponding example images in the first appendix of this report.
An “X” button exit mechanism is a way to exit a pop-up that involves activating a button with an “x” in it. These are usually located in one of the top corners of the pop-up, but the location can vary depending on the specific pop-up.
A text exit is similar to the “X” button, but is a piece of text that is linked to close the pop-up. It often appears in the form of “ Dismiss ” or “ Close ”. This exit mechanism is oftentimes located at the end of the body of text within the pop-up, and sometimes in a bottom corner.
The Click Outside exit mechanism is when there is no graphical element that indicates a closing mechanism. To close pop-ups of this type, the user must click outside of the are of the pop-up (this type of exit mechanism is most common with lightbox pop-ups)
Finally, there are pop-ups that do not have an exit mechanism and in order to continue navigation on the web page users must proceed with the action prompted by the pop-up.
Example images for each of these pop-up types and exit mechanisms can be found in the appendix of this functional specification. Altogether there are 20 unique combinations that can arise from this set of types and exit mechanisms. Through the data collection process, large numbers of screenshots of pop-ups will be documented and labeled according to this system of categorization.
In order to build the dataset, a strict data collection process is needed. First all data labelers should read over the solution overview portion of this functional specification and ensure they read over Appendix A containing the descriptions and example images for pop-up types and exit mechanisms so that there aren’t any discrepancies between different users’ labeling practices.
It is important to note that none of the individuals involved in the development of this functional specification are blind or visually impaired. There was however, an intensive human-centered design research effort with screen reader users who are blind that form the basis of the decision-making process used to select and develop this functional specification. While lots of evidence points towards this project being a useful addition to screen readers, extensive user testing will have to take place to ensure that it is beneficial to the web navigation process and not an intrusive add-on that provides unnecessary information. One of the greatest insights from our research process is that screen reader users want to be involved in the beta-testing phase so that potential pitfalls are avoided. The detection/type/exit mechanism organization of information is created from insights during our human-centered design research process, but given our small sample size of interviewees, our results may have been biased and these may not be representative of all screen reader user’s top priorities.
The tool resulting from this dataset will be most effective if it is preceded by a human-centered design process that pairs continuous iteration with co-designing with screen reader users. Beta-testing and other types of UX testing will also need to take place to ensure a smooth rollout and integration into screen readers as they exist now.
There are many potential risks of developing this technology. Machine learning as a field has many different potential applications with varying intent. This process and resulting dataset could be used for purposes other than accessibility and could be used for malicious purposes such as gathering sensitive data from screenshots of digital images.
We assume that this project will continue to be addressed by Danna Gurari’s team at the University of Texas at Austin. A potential risk is that if a different team picks up this project, the implementation descriptions to follow may not match how the team decides to address the issue.
We assume that the dataset will capture enough of the variety represented by web forms to identify and caption error messages with a high confidence interval. A potential risk is that the data is heavily biased to serve one site or issue well at the expense of effectively resolving issues across sites or interaction problems. In order to remediate this potential risk, testing throughout dataset creation is necessary to ensure adequate variety representation in the data collected.
There is also the assumption that screen reader companies will want to adopt this feature into their programs, and questions of screen reader feasibility are not covered in this research or specification. Additionally, based on extensive competitive landscape research, it does not appear that this issue has been solved already or is being solved by another group at this time, but this is entirely possible.
While the full implementation of the solution overview described would be most impactful based on the human-centered design research performed as part of this project, there are multiple levels of potential implementation. The detection of pop-ups would be very useful for screen reader users and acts as the first step of this project. The simple state change feedback can inform users of a change in the visual content of their screen, even if screen readers do not automatically detect it. This can ease confusion and disorientation that is often involved with unexpected screen reader focus change.
The next most important aspect of this implementation is the description of the exit mechanism. This is essential to give users autonomy and the option to exit the pop-up if they desire. This is the bare minimum for giving users the power to navigate pop-ups.
Following exit mechanisms, the type of pop-up is important in order to provide a more complete understanding of what kind of pop-up has appeared. For example, a user’s response to a full screen pop-up will likely be different from the response to a floating bar pop-up. The end goal is to make visual information that sighted web users see as accessible as possible for screen reader users, and this means providing useful information that they can use in their web navigation process.
The data collection process outlines a number of different website types and examples that the algorithm will be based on, and this information is in a complementary document called “Website Type Lists” that contains a comprehensive list of different website types and examples that data collectors and labelers will use to form the basis of the dataset.
The success of this project can be measured in a few different ways. Similar to other machine learning and computer vision projects, this algorithm could be tested against pre-labeled examples of pop-up screenshots to calculate its effectiveness and determine a confidence interval. Aside from quantitative metrics of success, a large part of the success of this project is how well it is accepted and used by screen reader users who are blind. User experience design is a huge determinant of success and will likely determine whether this project is adopted into everyday use by screen reader users.
A complete success would entail screen readers including this project in their code and screen reader users finding the indication, type, and exit mechanism of a pop-up useful and non-inhibitive to their web navigation patterns. A quantitative measure of this would be in the form of screen reader companies getting fewer reports of pop-ups being an issue.
Pictures and descriptions of different pop-up types and exit mechanisms.
As an extension to this project, there is the opportunity to inform screen reader users of the goal of the pop-up as well as the type and exit mechanism.
The pop-up goal is the sentiment or intention behind the pop-up. The goals vary widely and can be used as functional aspects of the website with features such as logins, or as a non-functional addition with features like advertisements. The goal is important to identify for the user so that they can decide whether they want to interact with the pop-up or simply continue the task that they were performing beforehand. Similarly to pop-up type and exit mechanism, the categorization of pop-up goals was mainly pulled together from different marketing firms’ pop-up advertisement offerings and webpage creation guidelines.
The goal aspect complicates this challenge immensely because it both adds many categories numerically, and makes the data collection process more vague in terms of what data collectors determine to be the goal. Below are some preliminary categories for pop-up goals with some examples and brief descriptions.
