Our quantitative usability testing of hidden menus (such as hamburger icons) and visible menus (such as links across the top of pages) reveals that:
- Hidden navigation is less discoverable than visible or partially visible navigation.
- When navigation is hidden, users are less likely to use navigation.
- If people use hidden navigation, they do so later in the task than if it were visible.
- Hidden navigation provides a worse user experience than visible or partially visible navigation does, in both mobile phones and desktop user interfaces. This finding holds true across multiple UX metrics including users’ assessment of task difficulty, time spent on task, and task success.
- On desktops, hiding navigation degrades the experience and the navigation discoverability more than it does on the phones.
- Hiding the navigation mostly affects content that is not directly accessible through an in-page link.
Why Study Hidden and Visible Navigation
Responsive design has many merits; however, one of its less fortunate effects has been an excessive popularity of mobile-first designs that translate poorly onto larger desktop screens. To be clear, “mobile-first” itself is not the issue. Rather, the issue is taking sufficiently good mobile designs and porting them to desktops where they do not work as well. Mobile-first should not equal mobile-only.
(Obviously, there’s a general principle here: if a design technique is intended to span platforms X and Y, then an uncompromising X-first approach will make Y an afterthought and harm users on Y. To design for X and Y, you need to consider the strengths and needs of both and adapt the two versions of the website accordingly.)
Hidden navigation, such as the hamburger menu, is one of the many patterns inspired by mobile designs. Screen space is a precious commodity on mobile. To meet the challenge of prioritizing content while still making navigation (and other chrome) accessible, designers commonly rely on hiding the navigation under a menu — often indicated by the infamous hamburger icon. Like a cheap fast food chain, it got designers addicted to its convenience, and now serves millions each day, both on mobile devices and on desktops.
While our qualitative user testing has repeatedly shown that navigation hidden under a drop-down menu is less discoverable on the desktop, we wanted to measure the size of this effect in a quantitative study and assess the relative impact of hidden navigation on the desktop versus mobile. To do so, we partnered with WhatUsersDo (a company specializing in remote usability testing) to run a study with 179 participants who completed tasks on 6 different sites, both on smartphones and desktops. The results of this large study are reported in this article. An additional article includes our detailed research methodology and images of the menus tested.
Measures: Navigation Usage, Content Discoverability, Time, Task Difficulty
To determine the effectiveness and usability of the different menus, we ran usability tests on the desktop and mobile versions of 6 different live websites with different types of navigation designs. Here are the types of navigation that we considered:
- Hidden navigation: the main navigation links were placed under an icon (such as the hamburger) or button and required an explicit user action to be displayed.
- Visible navigation: the main navigation links were shown in a visible navigation bar and could thus be seen by a simple glance without needing any prior action.
- Combo navigation: some of the main navigation links were visible and some were hidden under a menu.
Note that, for mobile devices, we did not fully test the visible-navigation condition. (Visible navigation is less common on mobile devices, and, due to other constraints in our study, we were not able to include this condition.) We tested the hidden and combo navigation conditions on mobile.
We wrote 2 simple tasks for each website. Some tasks could be completed without the use of the general site navigation, while other tasks required the use of navigation (or search). Here’s an example of a task: “Go to http://www.bbc.com and see if there is any information on nature or wildlife you are interested in. Once you find a particular article that is interesting, please say which one it is.”
To measure the effectiveness of these designs, we collected 5 metrics:
- Navigation use: whether participants used the navigation links at all
- Time to navigation: the time from the beginning of the task until the first use of the navigation in that task
- Task difficulty: participants’ assessment of task difficulty
- Content discoverability: whether participants were able to discover the content on the site
- Task time: the time it took people to complete the tasks
The first 2 metrics are related to navigation usage, while the last 3 refer to the quality of the user experience.
(More on different types of UX metrics and how to collect and analyze them in our full-day course Measuring User Experience.)
Navigation Usage: Hidden Navigation Used Less and Later in the Task
To quantify the navigation usage, we looked at (1) navigation use: whether navigation was used during a task; and (2) time to navigation: the time from the beginning of the task until the moment when the navigation was used. (We only counted time to navigation if the navigation was used.)
These metrics tell us whether the navigation is helpful and also whether it is easy to discover. If the navigation is salient, you’d expect it to be used more and sooner than if it was hard to discover.
Our tasks did not direct participants to use the navigation, search, page links, or any other interface features. Rather, people could use whatever they wanted in the interface to do the task. We assumed that if navigation is visible and seems helpful to users, they will click it; if not, they will try other things. Thus, we recognize that not clicking a menu can simply reflect that users perceived something else to be a better choice, and not that they did not notice or recognize the menu.
For example, on certain pages and tasks, an in-line link may have high information scent — think of looking for a movie on a newspaper site: if the first headline that you see on the page is exactly about that movie, chances are that you’ll pick it without using navigation or search.
Our results indicate that, across both these measures, hidden navigation was significantly less discoverable than visible or combination navigation:
- Both on mobile and on desktop, people were significantly more likely to use the navigation when all or some of the navigation options were visible (that is, in the visible and combo conditions).
- On desktop, people used the hidden menus in only 27% of the cases, while they used visible or combo navigation almost twice as much: in 48% and 50% of the cases, respectively.
- On mobile people used the hidden navigation in 57% of the cases, and the combo navigation in 86% of the cases — 1.5 times more.
We ran a mixed-effects logistic regression and found that these differences were statistically significant. This result is not surprising: when (all or some of) the navigation options are visible, they are more salient to users. If nothing else, the screen footprint of visible or combo navigation is larger than that of a menu represented by just an icon or a label, so people will notice it more.
2. People were significantly more likely to use the navigation (whether hidden or partially visible) on mobile than on desktop.
There are many potential explanations for this finding:
- Navigation on mobile (whether hidden under a menu or partially visible) takes a larger proportion of the screen space than on the desktop, simply because the screen is smaller. Thus, people are more likely to notice it and use it. Indeed, on the desktop we witnessed many sessions where people did not even see the menu icon. Some test participants even said that the sites with hidden navigation didn’t have a menu.
- Because hidden navigation is more common on mobile than on desktop, people may be more familiar with it and more likely to think of using it.
- Mobile pages tend to require more scrolling, have less content visible above the fold, and sometimes be slower to load, so people may resort to using a menu instead of looking on the page and exploring whether there is any content that matches their interest. On the desktop, page-load speeds are high and the larger screen offers a better preview of the content of the page. (In other words, the interaction cost of accessing a desired piece of information is higher on mobile than on the desktop.)
Typing is harder and more error prone on mobile, and people may be more likely to prefer navigation instead of searching, when the navigation is available.
- On both desktop and mobile, when the navigation was hidden, it took participants longer to use it than when it was visible or partly visible. Our mixed-effects generalized model showed that this result was marginally significant (p < 0.1).
Although the numbers seem to indicate a device effect (that is, that participants were faster to use the hidden navigation on mobile than on desktop) or an interaction (that is, that the impact of hidden navigation was higher on desktop than on mobile), these effects were not statistically significant.
An additional few seconds spent may seem petty, and we agree that it will not be the difference between a fine and a terrible user experience. But in those few moments the user may begin to experience frustration and self-doubt. Add on a few more seconds and the user may abandon the task, or at least the menu.
Hidden Navigation Leads to Worse User Experience
The other 3 metrics that we collected focused on the quality of the user experience:
- Content discoverability. Our tasks were fairly simple and gave users a fair amount of freedom (e.g., “Find a technology article that interests you”), so people were actually able to complete them most of the time. However, given the focus of our study, we used a more nuanced measure of success (content discoverability) that took into account not only whether people completed the task, but also how they completed it. Thus, content discoverability measured whether users were able to find the content they were looking for by using navigation (and not search) in those cases when the content wasn’t directly linked from the homepage.
- Task-difficulty rating. At the end of each task, we asked participants to rate how easy or difficult the task was on a scale from 1 to 7, with 1 being easy and 7 being difficult. The task difficulty is a subjective metric; it measures users’ self-reported assessment of the task. It usually reflects their overall experience in using the site, so a high estimated difficulty rating will indirectly indicate actual difficulty in locating the navigation and navigating through the site.
- Time on task. This metric represented the time it took participants to complete the task, whether successfully or not. A menu can add or decrease task time, if it is easy or difficult to find, open, or use, so longer task times also reflect the added interaction cost due to a less usable navigation.
Our findings show that, across all 3 different metrics, hidden navigation significantly decreases user experience both on mobile and on desktop.
1. Both on mobile and desktop, the content discoverability was significantly lower when the navigation was hidden. This measure showed a more than 20% drop in discoverability on sites with hidden navigation, compared with sites with visible or combo navigation. In other words, visible or combo navigation made people more likely to complete the task successfully and without relying on search.
- Hidden navigation led to significantly higher difficulty ratings than visible or combo navigation: the difficulty rating showed a 21% increase in the hidden-navigation condition compared to visible, and 11% increase when compared to combo navigation. This result is consistent with our other findings and strengthened the conclusion that hiding navigation makes it harder for users to complete their tasks. Visible navigation was ranked as least difficult, most likely because it was easy to discover and users could immediately scan the navigation options and select one of them. The other navigation types hide navigation options and increase the interaction cost.
- Users took significantly longer to complete tasks on sites with hidden navigation than they did on sites with visible or combo navigation on both desktop and mobile.
- On the desktop, people were at least 39% slower when the navigation was hidden (compared with the visible or combo conditions).
- On mobile people were 15% slower when the navigation was hidden (compared with the combo condition).
Differences Between the Desktop and Mobile User Experience
In the first section of this article, we reported that mobile users were significantly more likely to use the navigation than desktop users (see Figure 1). Here we discuss desktop–mobile differences related to the user-experience metrics.
- Hidden navigation is more likely to reduce content discoverability on desktop than on mobile. In other words, when users had to discover content that required the use of navigation, they were more likely to fail on the desktop than on mobile. Moreover, the drop in success was greater when the navigation was hidden. This result can also be explained by desktop users’ increased propensity to use search: search is more salient (most of the time) on desktop and users may be more inclined to use it when a path to the content is not immediately apparent.
- Desktop users were faster than mobile users (when combining all types of navigation) On average, mobile users took 12% more time to complete the tasks. This result is not surprising: due to the small screen and the narrow communication channel between the user and the device, getting the same content on mobile usually requires more interaction.
Why Hidden Navigation Is Less Effective
Why does hidden navigation have these effects? Throughout the article, we hinted at several answers:
- Low salience: A small icon is harder to notice on a large screen size (and even on a smaller mobile one).
- Low information scent: The menu icon or label doesn’t usually tell people what’s inside it, so they have no idea if they’ll find what they need by clicking on it.
- Extra work: To figure out what’s inside the menu, people must expand it. That increases the interaction cost for users and makes them less likely to do it, or, if they do it, they may take longer.
- Lack of standards: Hidden navigation is implemented in different ways by different sites. Some sites use it, some don’t. On mobile, patterns are starting to form, but on desktop there is a lot of variability and inconsistency in the placement and the labeling of hidden navigation.
- Low familiarity: Especially on desktops, hiding navigation is not a common pattern and people may not expect to find global navigation under an expandable menu. Some people may also still be unfamiliar with the hamburger icon that is frequently used for such menus. This low familiarity is exacerbated by the lack of standards which reduce long-term learnability (as further discussed in our course on The Human Mind and Usability, learning is facilitated by repeated exposure to the same pattern.)
Of these 5 points against hidden navigation, only the last one might decline in future years. Hypothetically, increased design standardization could also occur (which would also increase familiarity), but our experience observing the web design evolve over the decades doesn’t make us optimistic that strong design standards will emerge anytime soon. The 3 biggest points of low salience, low information scent, and extra work will always be a curse on hidden navigation that will reduce its usability.
Navigation Recommendations
What are the takeaways of this research for your next design project?
Desktop: There is plenty of screen real estate for displaying navigation options, and we recommend:
- Do not use hidden navigation (such as hamburger icons) in desktop user interfaces.
- Instead display the top-level navigation options, usually across the top of the page or down the left side.
Mobile: For phones, there is no hard-and-fast rule. But our general recommendations are:
- If your site has 4 or fewer top-level navigation links, display them as visible links.
- If your site has more than 4 top-level navigation links, the only reasonable solution is to hide some of these. We did find that the usability penalty imposed by hidden navigation is very present on mobile, but it is smaller than on the desktop. Thus, we recommend making this design tradeoff because the usability of an expandable navigation menu is far better than that of alternative designs.
- Provide in-page links to important information on your site, or use other methods of supporting the hamburger (or some other hidden) menu.
Remember: mobile-first is not the same as mobile-only. Plus, of course, follow the other long-established UX guidelines for menu design.
