Project Overview

Client: Fidelity Investments

Study Type: A/B Study with various UX metrics

My Role: Analyzed data from study, and made recommendations.

The Problem: Some Fidelity clients with brokerage accounts are unfamiliar with the industry terminology. This study evaluated three different designs which provide online help to users.

Recommendation (Conclusion): Based on the data, Version 1 (V1) is the most successful in delivering users the answers they seek, and should be implemented into the Fidelity website.

Version 1

Every industry-specific term has a blue question mark next to it. The user clicks on the question mark for a definition of the term.

Version 2

Similar to version 1, but the question marks only appear upon mousing over the term.

Version 3

Clicking on the “Balance Definitions” button at the top of the screen brings up a list of all definitions, on a single screen.

Methodology

The study was conducted online.
312 people with a minimum level of financial expertise completed the study.
Each participant received the same 5 tasks, in a random order.
- They had to choose the correct answer to the task from a drop-down list.
- The answer could be found using the help feature.
- Data collected for each task:
  - Accuracy (0=Wrong, 1=Correct)
  - Time (in seconds)
  - Task Ease Rating (1-5, with 5 being easiest)
Each participant was randomly assigned to one of the three help designs being tested (V1, V2, or V3).
System Usability Scale (SUS) was administered at the end (285 completions).
Participants were asked (not required) to provide comments about:
- Challenging aspects of the interface
- Effective aspects of the interface

Results

Accuracy

Overall, participants had higher accuracy (success rate) when using V1, compared to V2, and V3.
- The error bars of V1 and V2 overlap slightly. A t-test showed that there is a 2% chance that the difference in accuracy could be due to chance. Since the t-test value is smaller than the alpha value, the difference in accuracy of V1 and V2 is statistically significant.
- The error bars of V1 and V3 overlap slightly. There is a 0.6% chance that the difference in accuracy could be due to chance, meaning that the two versions are statistically significant.
On a task by task basis, V1 most often produced higher accuracy rates by task, when compared to V2, and V3.