THE PROBLEM:
CostcoGrocery has been one of the website's fastest-growing sections, particularly during the 2020 pandemic, and presents a business opportunity to strengthen brand loyalty and convert new customers into members. Our team was interested to observe how users learn and shop on the site, whether they are aware of the various grocery delivery services provided, and how they feel about the overall site experience.
This project was coursework for SJSU’s ISE215 - Usability Evaluation & Testing, Spring 2021.
MY ROLE:
Researcher & Team Leader
My work on this 4-person team project included:
background research into Costco’s business strategy and website
heuristic evaluation of the relevant website pages
writing test scripts
recruiting and screening participants, and distributing compensation
executing the study in each of three key roles (moderator, a/v technician, data logger)
analyzing data and summarizing insights
The final report including our testing protocol (designed as if we were a professional testing lab), and our actual research findings and recommendations is presented in CostcoGrocery: Usability Study Report.
Protocol:
Participants were recruited through word-of-mouth and postings in online social groups on Facebook, and the r/Costco subreddit. Candidates were phone-screened to categorize and filter for experience with online grocery shopping, familiarity with the Costco brand, overall technical capability, and vision impairments that may hinder their ability to participate. All study sessions were conducted over Zoom.
During each session, a moderator guided the participant through three script sections: an introduction and exploration of previous experience with online grocery shopping, three explicit tasks using the Costco website, and a few minutes of free play & final impressions. Two data loggers recorded notes into a prepared spreadsheet, capturing measurements of performance (completion, accuracy, and path/sequence), objective observations (body language, facial expressions, unprompted verbal comments), and subjective feedback (responses to specific probing questions). An AV technician was responsible for recording the session and supporting the moderator and participant via chat.
Image: Screenshot of a remote moderated session conducted via Zoom
insights:
Observations of our measures of participants’ performance, objective and subjective responses revealed three key insights: the site suffers from visual clutter, poor user flow for non-members, and a lack of adequate information for decision making. We then compiled our findings and insights into a scorecard that rated the site’s Learnability, Usability, and User Satisfaction among inexperienced and experienced participants.
Table: CostcoGrocery Usability Scorecard
RECOMMENDATIONS:
Our team concluded that the following recommendations would address a large percentage of the site’s usability problems:
Reduce the number of products and services on each page, and increase the amount of white space between categories to help the user focus their attention when browsing or searching.
Group content into categories that match the users’ mental model to help them skim more efficiently.
Show all pricing on the product page.
For all-access products, make the benefits of membership apparent by showing both the member and non-member prices.
For members-only products, include a one-click option for non-members to add the item and a basic membership to their cart.
Reduce the amount of required user input to access the “Contact Us” chat function
Addressing these issues in a future redesign would create a significant opportunity for business growth, as they will significantly improve the user experience and reduce common user complaints.
wrap up:
What I learned from this project:
Develop a phone-screener that filters for necessary characteristics with subtlety, so that the research team can select participants without candidates self-selecting in/out of the study.
Start recruiting participants as early as possible, to ensure that you have many candidates to choose from.
Formative usability testing aims to ‘fill the circle’ of knowledge, so moderators need to be flexible and alert for opportunities to collect valuable information by modifying the script when appropriate.
Video conferencing tools like Zoom can be useful for observing remote usability testing, but it’s important to review the software’s recording layout settings to ensure that you capture the participant’s video as well as their shared screen for post-session review.