Valuable Usability Testing: A Case Study of the Qatari Government Portal
Hukoomi has come a long way! As Qatar's primary government portal, it is the single place businesses can visit to complete transactions like applying for work visas, exit permits, residence permits, and much more. Now Qatari citizens and residents can pay traffic violations, renew passports, and renew health cards. Since Hukoomi continues to add more transactions and services every year, it is increasingly important that their audiences can effectively use their site. Enter usability testing.
Usability Testing - Round 1
Hukoomi's homepage in 2009
Forum One conducted the first round of usability testing in 2009 in Doha. The Hukoomi site looked very different then. The center of the page was filled with a decorative navigation for different audiences that might visit the site and the left column contained additional, commonly used links. But while the site was backed with some powerful services, users struggled to find and use those services.
During usability testing, we brought in business people, citizens, residents, and visitors – the primary audiences for Hukoomi. These participants attempted to complete ten of the most common tasks on the site and gave us their overall impressions of the site. The results were powerful. Few participants could complete the tasks. Some participants had strong opinions about the difficulty they faced using the site, although others were just grateful they didn't have to stand in line all day to complete a transaction in person. In the end, we learned a lot about user perception and areas that needed improvement.
After completing this usability testing, we provided recommendations for improvement. The list was long, but we learned that users struggled with five main areas:
- Access and Site Performance
Example: It was difficult to access the site in some browsers and outside of Qatar. - Content and Naming
Example: Instructions were often too technical for users to understand. - Navigation
Example: To access content, users had to know about and navigate via a single path to get to content. It was difficult for users to understand where they were in the site and how to get to another section. - Visual Design
Example: Links did not use a consistent style throughout the site. - Translation
Example: The translations in English and Arabic were sometimes different.
Generally, participants loved the idea of Hukoomi. Completing transactions online often saved them time since they did not have to travel to a government building and wait in line. However, when using the website, it was difficult to find services or complete a transaction. With room for improvement, the Hukoomi team took our report and recommendations and revamped the site.
Usability Testing - Round 2
Hokoomi's homepage 2011
Two years later, in 2011, we conducted a second round of usability testing. The results were once again powerful, but for a very different reason! This time, the issues users faced were minor and largely related to fine-tuning the existing interactions. For example, in the 2011 testing, we found that when a participant completed an exit visa form, he/she could not easily start another exit visa transaction. Without a link to start a new transaction, a user had to either click the back button six times or restart the whole process by going back to the homepage and navigating to the exit visa page again. To think one link could make such a huge difference! In this case, business people (a primary audience) may complete 400 transactions on Hukoomi every day. That link could save a significant amount of time, and it's possible that we would not have discovered this without talking to these audiences and completing usability testing. This highlights the very real opportunity Hukoomi has to make transactions even more current.
Striking Results
Table of Improvements after usability testing
Since 2009, Hukoomi has made huge strides in the right direction, improving usability and the average user perception of the Hukoomi experience. In the 2011 testing, users were able to complete more tasks in less time. Nearly all participants who have contacted the call center commended Hukoomi for providing excellent customer service and follow-up. The team behind Hukoomi should be commended for engaging in this usability testing – it underscores the importance placed on user needs in evaluating the effectiveness and reach of government services.
Usability Testing and ROI
By regularly doing usability testing and reaching out to users, Hukoomi can maintain a user-centric approach and prioritize its organizational objectives for the site, give thought to which content and services are of the highest value to users in support of those objectives, and refine the quality of the information and the online experience provided. In the end, the difference is striking. Understanding your audience's needs, opinions, and interactions on your website is powerful, not only when trying to improve experiences and change perceptions, but also in saving time and money.
Comments
Interesting case study; thank
Interesting case study; thank you for sharing. As a way to benchmark improvements, I'm wondering if users were given the same ten same tasks to complete in 2011 as they were in 2009. Also, were you able to quantify the amount of improvement for each issue? For instance, were you able to show if navigation had been greatly improved vs only slightly improved? I'm also curious if there were data that showed the popularity of the top 10 most completed tasks. I'm making up numbers here, but if three of the tasks were responsible for 75% of the usage, was there any effort to hone in on these tasks more so than the other 7 tasks that (hypothetically) made up only 25% of the usage? Thanks again for sharing! Sounds like it was a rewarding project.
-Emily Cunningham
Some answers
Hi Emily, Great questions! Q: I'm wondering if users were given the same ten same tasks to complete in 2011 as they were in 2009. A: Yes, the participants were given the same set of tasks in both years. We did remove one task in 2011 because the page/information was removed when the site was redesigned. Q: Also, were you able to quantify the amount of improvement for each issue? For instance, were you able to show if navigation had been greatly improved vs only slightly improved? A: Such an interesting question! The short answer is "no", I did not quantify the amount of improvement for each issue. I could have, but I didn't because I felt such numbers would be misleading and wouldn't tell the full story (sort of like a USA Today statistic). The improvement chart (with the green boxes) in the blog post was based on three things: - task completion, - time to complete tasks, and - user perceptions. While this data gives us some numbers to work with, I didn't feel that crunching these numbers would accurately show a level of improvement. If we had done A/B testing, there would have been a control and only one element would have changed. It would be much easier to confidently quantify the level of improvement with data from A/B testing. With usability testing, we're receiving more information than A/B testing, but the lack of a control makes it harder to make accurate quantitative claims. I'll give an example. Let's say that tasks related to navigation were all completed in a short amount of time, but users didn't understand the terminology for most of the navigation links. The data would say that the navigation was 100% improved, but I know that we still need to greatly improve the language in the navigation. Additionally, navigation is much harder to quantify, because it is comprised of multiple elements (placement, terms, design, number of items, interaction of links, etc). When all navigation elements change, we have no control to compare with. I don't want to get too far into the weeds, but I've worked with researchers in other fields and I know that if I present numbers I should be able to explain and stand beside them. Our goal was really to identify big problems and suggest solutions. Nielsen recently wrote about Accuracy vs. Insights in Quantitative Usability - http://www.useit.com/alertbox/study-accuracy-confidence.html Q: I'm also curious if there were data that showed the popularity of the top 10 most completed tasks. A: Site analytics could help us determine the top 10 tasks. For Hukoomi, the task list based on: - top pages visited (from site analytics) - areas that client stakeholders were particularly interested in - areas we want more users to visit Sorry for the long answers. Let me know if you have other questions!
- Courtney Clark
Post new comment