©2016 by Dawn Procopio.

this page is under construction!

ENABLON

Heuristic Review

& Prototypes

Enablon is a EHS Software company which helps businesses manage risk, data and content more easily. The company wanted a design of 3 new views for their Dashboards, but also wanted them to meet W3C accessibility standards.

OVERVIEW

UX RESEARCHER

2 days

Sketch, W3C HCI Literature, Google Material Design Style Guide, Nielsen Heuristics and UX Research experience 

OBJECTIVE
  • Prototype an new design for: 

    • The “Dashboard" - The first view for the user

    • “Add a Report” - This pop-up is critical to add reports from a library to the dashboard

    • “Delete an Action Plan”: These dialog boxes are displayed to users when they do actions that will have a serious impact on the data

  • Identify what elements of the previous dashboard design are good and areas of improvement

  • A brief summary that explains choices for the dashboard based on market best practices or personal experience

Scenario 1

User makes a purchase using the sales funnel experience. 

Scenario 2

User sets up their app for continual usage monitoring and service engagement. 

Scenario 3

User receives their bill and attempts to understand charges. 

Challenge 2

Additionally, the phone service plan offered for this journey was not a conventional data plan. The phone plan allowed users to use GB of data on an ad hoc basis. Essentially, there was no pre-planned data limit. Users would be billed as they used the data, without pre-approval from the user. Users could set up a limit for data usage, but not a minimum data plan. The unique function of the app was that it served as both a historical tool and a management tool in which users could see and change data usage.  The challenge was to convey this unique message within a friendly and encouraging experience.  

My objective

  • Identify user pain points and expectations within the entire customer journey.

  • Enhance clarity and transparency in all billing processes

  • Gauge user comprehension & sentiment regarding the online purchased service & phone

  • Discover opportunities to enhance progressive disclosure, error prevention & recovery, and control & freedom

PROCESS

THE DASHBOARD

Identifying elements of the client's design that were good

COMPETITIVE ANALYSIS

The initial information architecture of online sales funnel was vague and tedious. I took the client's top 2 competitors and analyzed user friendly trends and similar pitfalls. I came up with 5 principles that were satisfied based on the competitor's surveyed: 

1. Design induced Ethos with Visibility, Clarity & Transparency 

2. Logos - designs are logical in that they use familiar symbols and information

3. Presenting Pathos with Testimonials

4. Low Cognitive Workloads with Progressive Disclosure 

5. Rapport Building with Confirmation & Feedback

HEURISTICS

Evaluation of Current Website & App | Formulating Hypotheses

The initial information architecture and design elements of the online sales funnel & app lacked in almost all 5 principles. The biggest problem within the app revolved on an unclear way to explain and predict future charges. For this study there were 3 distinct scenarios being tested. Together they formed a type of customer journey.  For the purposes of brevity in this case study, I will discuss 3 main hypothesis regarding my heuristic evaluation of the client's prototypes. 

 

Recap on customer journey:  

Scenario 1: User makes a purchase using the sales funnel experience. 

Scenario 2: User sets up their app for continual usage monitoring and service engagement. 

Scenario 3: User receives their bill and attempts to understand charges. 

Hypothesis 1 

A shopping cart that combined the purchase price of the phone and the service was not clear.

Regarding Scenario 1, the client's desktop prototype for the sales funnel did not explain the unique feature of the service and combined the phone price with the phone service plan. This was confusing since users' expectations were not properly set. Not only did the data plan remain unexplained but the initial service plan fees spoke to user's previous mental models of data plans.

 

 

From the user's standpoint the shopping cart may incite a feeling of unethical sales practices since charges were seemed lack transparency. This was a deviation from the principle of Ethos UX declared in my competitive analysis. The sales funnel started without a clear and transparent explanation of the carrier's unique data plan. If my hypothesis is supported, this would create a deficit in relationship building and cause a PR nightmare. 

Hypothesis 2

The design element that indicated the amount of data used induced negative transfer from similar designs from the client's current app.

 

In Scenario 2 the user has downloaded the app and has already used their phone's data plan. The home screen of the app had a large number graphic containing which represented how much data was used with a tag line that told the user "how many days to go." However, the client's current app had a similar display, in which the number represented how much data was remaining. My hypothesis regards a psychological phenomenon called "negative transfer" where in users will not be able to sufficiently comprehend the design element since a previous mental model will interfere with the current design experience. 

This would violate the user's sense of logos (principle 2), since by their logic, the same carriere should have the same graphical representation for similar information.  While this could be somewhat remedied by the user's knowledge of the unique data plan ( i.e. the user's plan has no data limit therefore a number would never represent remaining data usage) I contended that it would still be better for the design to speak to current mental models, especially those models which were sourced by the client. 

Hypothesis 3

Reconciling the receipt page, usage odometer and statements page required a high cognitive workload. 

Within Scenario 3 users have had their app for awhile and have presumably managed their data usage by receiving data usage notifications, set data limits, looked at rewards and planned a number of other service related functions. Now they have received their first phone bill. The prototype explained their bill with 3 screens: the main data usage odometer, statements, and receipts. While each of these screens may serve separate purposes with their own scenarios, I explored the scenario where one would want to see if all three screens "add up." In other words, do all three screens tell a narrative that explains the final bill?

 

It was essential that each screen did tell the "final bill" story in different ways, allowing the user to focus on different purchases based on the type of screen. However, what happened if one screen indicated a different total than the other screen. Would users notice? If they did, was there a logical explanation for a different total? Would users depend on any one of the screens to tell the same story for the bottom line? I imagined if a user viewed one screen, and came to the conclusion that that was the screen with all the information for the final bill, when in fact that screen did not have the final bill's total then users would be surprised and trust would be violated. 

Not only was it difficult to practically make sure all the numbers on the three different screens added up, but it was also impossible to get the same number. The client's prototype did not intend for the 3 screens to add up to the same number or be reconciled against each other. My hypothesis not only presumes that these pages should be reconcilable, but that they should also add up to the same number. Otherwise, the screens containing numbers without all the numbers should be clearly marked: "This is not your total bill."  

TESTING

Hypotheses

USER TESTING

 

Background 

 

I conducted user tests based on the client's target market: millennials. The Observation room and lab was set in an office building in downtown Chicago in early 2017. It consisted of 3 cameras, a two way mirror, 3 desktop computers (two for the moderator and one for the participant) and a smart phone (OS varied on participant's comfort.) Projections of the users interactions were displayed in the observation room as well as each participant's face and hand movements (for mobile). One of the computers in the lab was for the moderator and allowed communication from me, acting Assessor. 

The session was a semi-structured interview and usability test. We interviewed 35 users in a 1 week period. Sessions lasted approximately 1 hour long. Each participant was paid $60 for their time, and was assured that their performance was not the basis of their compensation, they were free to leave at anytime, and that their names were kept confidential from the client and reports. 

Writing A Discussion Guide (a.k.a Mod Guide)

Writing a discussion guide needs to overcome two challenges: time constraints and hypothesis testing. With only an hour for each participant only a certain amount of questions and impromptu investigation can be exercised.

Each participant was verbally told how the data plan works from a script, which they were asked to read aloud. The copy from the script was meant to serve as knowledge they would have if commercials or the website had explained the plan. The script was then given to the participant for their reference at anytime. The main points of the copy included:

 

1. no brick and mortar stores

2. no planned data purchases

3. optional data limit setting from the app

4. unlimited data as you use it

Then the participants were asked to answer and perform a series of questions intertwined with tasks. They were also asked to think aloud whenever they were performing a task.  The main prompts and questions included: 

[Prompt user to pick out a phone, configure it and arrive at the shopping cart. Record click path, interactions and comments.]

 

Did you expect a credit check during this process? 

What were your thoughts on the configuration options. I noticed that you did X. Did you see this choice? 

Open the app and go through it as you would at home. [Record click path, interactions and comments.]

If you had to pay your entire bill today, what is the total amount? 

Describe the main screen.

What do you like or dislike? 

Is this your final bill? 
 

Go to statements.

How do you feel about the statement display? 

How would you explain the charges on your statement. 

Open your receipt. 

What is your final bill? 

How would you explain your data plan to a friend or family member? 

How much data have you used? 

How many days are left in your billing cycle?

What do you expect to pay when your bill is due? 

What features do you like or dislike most about the data plan? 

From the words in this deck (see image), what words would describe this app? 

What would you say about the tone and color of the app? 

How do you think we can improve the app or any of the screens you saw today? 

 

 

Whiteboarding: Getting Stakeholder Buy-In

After each user testing session teams of designers, developers and stakeholders gathered in a room and white boarded what we saw. My research partner (moderator) and I led the discussion and offered our observations as well. This process helped us gained a type of buy-in from our clients and helped us further understand what mattered most to our clients. The most important benefit of this was for the final report. This way, when results and conclusions were delivered in the final report no one was surprised by our findings.

DATA ANALYSIS

There were many issues with the app and website after the user testing sessions were. Some of them were unanticipated. This was good news since it proved the power of usability testing. After the sessions the user data was analyzed with three key methods were used to model predicted pain points using OLS Regression, Averages, and the qualitative statements. While qualitative insights could be easily reached immediately after the sessions, these mathematical tools could help us make generalizable statements to the population, i.e. estimate what could happen in the real world without any changes to the design. 

Below are the qualitative insights that pertain to the hypotheses mentioned above. Other insights were recorded as either quotes and qualitative statements like (see below). The qualitative statements helped us focus our efforts on where to perform deeper research dives. Not all the qualitative statements were analyzed with econometric models, since most statements delivered enough understanding by themselves. 

To avoid an illegitimate inferences from a small sample size (n = 35), the words "all," "some," "most" and "few" are used instead of fractions (e.g. 3 of 35, 27 of 35).  This way recipients of the report can focus on the qualitative significance instead of the quantitative, which is reserved for the next phase.

Key Qualitative Statements

All users were able to complete the sales funnel with zero assistance. 

 

Some users did not expect a credit check during the sales funnel.

Most users did not understand all of the configuration options. Some users did not see the options. 

Most users were able to reach all pages of the app. Some users did not understand and verbalized confusion for some features. 

Most users did not know the correct final bill total. 

All users described the main screen with positive attributes. 

Most users liked the color and tone of the app design. 

Most users pointed the main screen as the place where they would expect to have all the information about their final bill. 
 

All users like the statements screen and specifically verbalized appreciation for the expandable panels as a way to find more details about a specific charge. 

How would you explain the charges on your statement. 

Most users tried to add all the charges up from the receipt screen. 

Most users expected receipt total to match the main screen. 

Most users failed to communicate the auto-spend function of unlimited data. 

Some users had difficulty reconciling how much data they spent with the incorrect belief that the main screen gave information about how much data they had left. 

All users understood how many days they had left in their billing cycle. 

Most users did not calculate their phone installations and extra charges into their final bill. 

Most users did not like that there were three different screens in which to calculate monthly bills.

Quantitative Methods

After I distilled the data into the key qualitative statements (above) I began to organize answers, paths, behaviors and interactions into variables. I then conducted an online user testing sessions with 4,634 participants using UserTesting.com 

 

The sessions were modeled in a similar fashion fashion as the live sessions. Below is the econometric model that was applied to the dataset to see if similar pain points were observed which will help me predict how severe the causal relationship is to any given pain point. The formula is called Ordinary Least Squares (OLS) Multivariate Regression. This is the most common mathematical model for economists and social science researchers.

 

percentage of times X interaction correlates to Y pain point

mean user behavior on interaction X

each user behavior on interaction X

expanded equation - something SPSS & STATA do for you

Observed pain point minus mean pain point

simplified equivalents

squared error for each variable

This formula allows one to control for many other variables like age, race, education, gender and income. Since most of our demographic is homogeneous on many of these variables, and some were theoretically irrelevant to our goals, we controlled for a unique demographic score called the Millennial Questionnaire Score (MQS). Our third party recruiters were asked to recruit an equal amount of participants who scored high, medium and low on the questionnaire. The questionnaire was meant to gauge users on how sensitive they were to data constraints relative to pricing plans. The variables score is simply added to the end of the model which, in effect standardizes the prediction against data-to-price sensitivity. This way we could be sure that, for instance, that a pain point was not solely due to a user's sensitivity to how much a plan costs. 

RESULTS

Here I will discuss findings that can support my hypotheses. For tables with the full models and descriptive statistics please contact me

Hypothesis 1

A shopping cart that combined the purchase price of the phone and the service was not clear. 

For the user testing session we created an A/B testing scenario where half the users were given 2 side by side shopping carts (users A), and the other half were given the current model (users B).  Users B were associated with a 0.64 increase in reporting a pain point in understanding the cart, on average controlling for all other variables in the model. This was significant at the 0.05 level which was ideological intensity. The F-value of 35.10 indicates that the model's variables are jointly statistically significant at the p < 0.001 level.

Hypothesis 2

The design element that indicated the amount of data used induced negative transfer from similar designs with unrelated purposes.

One the key findings from the model showed that participants who had already had the carrier's plan and used the app were associated with a 0.416 increase in reporting a pain point with the "data used" design element on the home screen of the app, relative to respondents who are not customers with the app, on average controlling for all other variables in the model. This variable (current versus new users) had an F-value value of 16.93 which indicates that they were jointly statistically significant at the p < 0.001 level.

Hypothesis 3

Reconciling the receipt page, usage odometer and statements page required a high cognitive workload. 

 

For the online user testing session we created an A/B testing scenario where half the users used an app with only one statements page that also acted as the main screen and their receipt. The other half were given the current prototype (users B). All users were asked to report what their final bill would be and what their bill current was.  Users B were associated with a 0.768 increase in reporting the wrong answer for both questions, relative to users A, on average controlling for all other variables in the model, with a p-value < 0.01.  This variable (users A & B) had an F-value value of 30.48 which indicates that they were jointly statistically significant at the p < 0.001 level.

IDEATION

IDEATING

After the online testing and live sessions my partner and I wrote down the problem areas of the design and possible solutions. We used post it notes to organize areas. Initially we wrote our ideas down separately and then combined them in a final meeting. We decided to create a red routes matrix for the third scenario since re-design would be essentially require new information architecture. 

expected home sreen to show final bill

set data limit

FAQs

chat icon

export a reciept

tapped charge icons

checked family usage

calculated receipt total

tapped "data used" graphic

hard pulldown on homescreen

check extra charges

check rewards

calculated statements total

menu

confused data used with data remaining

calculated homescreen total

The ideas relating to my hypothesis are shown below.  We decided that we should create a low fidelity prototype to demonstrate a solution supported by the data. Click an icon below to view a prototype.

Re-designed Shopping Cart page

Re-designed "Data Used" Display

Re-designed Statements page