Abstract. This report includes the need-finding results using participant observation, user interviews, and post-event protocols. The need-finding focuses on the pain point of using the iPhone mailbox app to check email in all situations, especially when available time is limited (under 10 minutes time span). The clear need for functionalities that facilitate organizing and filtering was identified. Users consider “smart sorter” to be a great-to-have functionality for the specific purpose, conditioned on additional requirements, such as “smart sorter needs to be truly smart”. The below sessions give a high-level summary of the need-finding exercise and the results. Detailed transcripts are included in the appendix.
Need finding exercise 1: participant observation
According to the plan I laid out during assignment M1, I went through the exercise of email checking in 3 different environments and recorded my feedbacks real-time.
The 3 environments I tested are morning in the bed, during lunchtime, and in between meetings.
Morning in the bed
- The context of the task
This is the time I can actually concentrate on checking the emails without much distraction, although my time is normally very limited (5–10 minutes).
- What is my major purpose?
My major purpose of checking emails at this time is to prepare for the day.
- What do I need to achieve the purpose?
I need to remember all the important tasks I need to accomplish both at work and in personal life.
- What specific tasks I do? What are the subtasks?
My high-level task is to quickly look through the new emails until the point I reach an email that I already checked yesterday morning.
Subtasks include: I make notes in the notes app on my cell phone if there is something particularly important I need to deal with either today or in the near future. I forward the important personal information to my husband. I quickly reply work-related emails to schedule meetings if I want to deal with it in the morning. I search for important people/companies’ names and titles just to make sure I am not missing anything from them.
- What’s my thoughts and complaint?
So many unimportant information. I am sure I am missing something important because of all the noises, but I just don’t have the time and energy to care. I really wish there is some smart sorting to help me out and automatically organize the emails. I also have to switch back and forward to make notes. I hope there is a better way to help me track important info.
- What hacks did I use to help achieve the goal?
I use the notes app as the note-taking tool, so I have to switch back and forth. I use my husband as the “reminder”, which is very effective, but I also get yelled at a lot. I reply to colleagues saying “please schedule a meeting to discuss with me” instead of actually schedule a meeting myself. I can’t really do it when the email is coming from client or leadership, so I have to switch to note-making for those ones. This definitely is not the most efficient way, but the scheduling integration is just not that convenient for me to deal with it when I need to finish checking all emails in 5 minutes.
During lunchtime and in between meetings (simplified to shorten this report)
Under these environments and contexts, I normally just have 1–5 minutes to check my emails. I feel even more strongly annoyed by the emails from promotion and social media. I scan through emails very quickly and ignore everything that is not work-related nor crucial personal information. I keep sending important personal emails to my husband, who acts like my “personal assistant”. I think a smart sorting with the functionality of notetaking and scheduling would help me immensely.
Avoid the bias
I am not my user, so I am going to use other need-finding methods to further explore others’ pain points without presumption. I also let myself be more objectively observing myself, so I was actually able to identify additional needs such as “notes taking” and “schedule meeting” besides needing help to filter information (smart sorting).
Need finding execution 2: user interviews
Figure 1: workflow and final script of the user interview
My second need-finding exercise was a user interviews. I recruited users who use iPhone and mailboxes app from my colleagues and my husband’s company, refined the scripts with my UX design and research colleagues, and finally conducted the interview (Figure 1). I enlisted the same 5 users for both user interviews and the 3rd need finding “post-event protocols”.
The detailed transcripts are available in the appendix. The results are summarized in Table 1 in the Data Inventory session.
4 out of 5 users voluntarily mentioned organizing emails is difficult with the current interface. The other user thinks it is manageable but not as convenient. All users voluntarily mentioned that they have too many promotional emails. All users mentioned searching for important emails is a problem, partially due to not robust searching function, as well as the amount of “unimportant” emails. All users mentioned the pain point missing important emails due to the unimportant emails.
“Painful. Very much. All the time. (I was just) waiting for quotes from the house improvement contractor. Didn’t realize it was already there.”
2 users mentioned that they want a timeline reminder. 2 users mentioned cell phone signal is less than ideal, which impacts their user experience of email checking. 1 user mentioned want a “pin email to the top” function. 2 users complained that the interface is not user friendly.
When talking about smart sorter, all users mentioned that they need to be “truly smart”. 4 out of 5 users want to be able to customize the folders. 3 out of 5 mentioned that the folders shouldn’t be too granular. 3 users mentioned that their needs change dynamically, so the folder structure needs to change accordingly. 2 users mentioned they want to be able to customize the sorting rules themselves.
To sum up, there is an overwhelming need for functionality/interface to help organize emails. Users want help to distinguish the important emails from unimportant ones. At the same time, they also want to be able to customize the sorting. Most users also demand a low error rate.
Avoid the bias
To avoid bias, I checked with colleagues who specialize in user research to make sure that my questions are reasonable and not leading questions. I also let the users go on tangent lines when time allows, so they can provide more spontaneous feedbacks. I let the interviewees do most of the talking, so I can objectively collect data without confirmation bias. I mentioned smart sorter but intentionally avoided showing my own attitude and my effort to developing one.
Need finding execution 3: post-event protocols
Right after the user interviews described in need of finding 2, I asked the users to spend less than 2 minutes to check their emails with their iPhone and mailboxes app. Immediately after that, I asked them for feedback about the app usage and pain points. The benefit of arranging interviews this way is that the feeling of using the app is still fresh in their mind, but I wouldn’t introduce bias by interrupting the normal usage pattern like in the “think aloud” format. I didn’t recruit for new users for this activity partly because of the cost (difficulty to find users who fit the profile and are willing to spend the time). One benefit of using the same users is that they will pay more attention to the topics we just discussed, which makes their feedback more relevant. This, at the same time, is a potential bias. The user’s focus may be more narrowed down than freshly recruited candidates.
Based on the result, the benefit outweighs the risk of bias. The users indeed give feedback relevant to the smart sorter and organizing the email box, but at the same time, they also gave additional feedback on items that were not restricted in this area. The major feedback points include suboptimal function discoverability, user’s detailed normal workflow, unsubscribe function, audio function to read out email aloud, poor cell phone signal, preference of mobile platform over the web interface, auto-reply suggestions functionality, the preview of attachment functionality, auto-extract important information and links from email bodies, etc. Many of these feedbacks can be important functionality of a broad definition of a smart sorter, such as unsubscribe function, the preview of attachment functionality, auto-extract important information and links from email bodies.
Overall, the users all feel managing important and unimportant emails is a challenging task. They recognize the value of smart sorters. They also provided feedback that can help the future flesh out the broad definition of a smart sorter.
Data Inventory
Table 1 data inventory is built based on the user interview and post-event protocols. The results of participant observation are already described in detail in the need-finding 1, so were not included here to save space. Besides the required 7 categories, I also included project-specific categories, such as pain points and smart sorter expectations. All the users recruited are right within the target profiles, so the specific feedback of pain points and smart sorters are closely relevant to the project. The only bias I want to point out while putting this table together is that the users I recruited are skewed towards the high technology population and high-income population. This is largely due to the accessibility of the users but could be an important watch out in the future data gathering and conclusions.
Table 1: data inventory from need finding 2 (user interview) and 3 (post event protocols)
Defining Requirements
After the user interview, I want to change my topic of “smart sorter” to “smart manager” to broaden the scope and address user needs more closely. It should include functions that help users to 1) sort emails into customized folders 2) making notes while reading emails 3) adding calendar events easily 4) summarize email and attachment content 5) extract important links such as unsubscription, account activation, etc. 6) highlight good deals.
Matrics I would uses include but not limited to 1) user engagement matrix (checking frequency, length of usage per app opening) 2) direct user feedback/rating (usability, error rate, discoverability, etc.) 3) efficiency: a path analysis of steps and time users take to achieve the same goal 4) versatility (the number of different tasks users achieve within the app).
Since I am targeting medium and power users, I would emphasize user experience (direct feedback) and efficiency.
Continued need-finding
I would like to conduct another round of user interviews, this time focusing more on usability. I plan to brainstorm more about potential design to address users’ needs first. Then I would like to use some low fidelity prototypes (sketched) to inspire more thoughts from the interviewees.
At the same time, I am concerned about the skewness of my users towards high tech and high-income population. I would like to use a survey to get a better distribution within my targeted user profiles.
My new questions are mostly focusing on the discovery of functional needs from the first round 3 need-finding exercises. I would like to investigate how broadly the needs are among more users. I will also focus more on relatively quantitative questions, such as “how many folders you think will be the right amount?”, “how many good deals you want to be recommended to you every day?”, etc.
Appendix
Transcripts
User 1
Profile
Age: 25
Education: Master degree
Income level: 75–100k
Occupation: Data Scientist
App using expertise: Intermedium level
Motivation: Keep contact with important but not very close people
Interview
- How often you use cell phone to check email?
- every other day
- Under what circumstance?
- When outside urgent
- What pain point do you have when using the mailbox app?
- Hard to open attachment. Switch to the another app to open it. Like to open it in the app
- Don’t know how to use some function such as add attachment
- Internet is slow. Didn’t know the photo is stuck in the draft. Like it to be cancelled at the moment. Then will switch to PC to do it.
- Interface is not as convenient as on PC.
- Screen is small. Junk trash are not very visible. Need to go back and forth.
- Hard to find the email if it is long time ago. Searching is not easy
- not sorting very often so hard to find the email
- How many emails you receive everyday?
- 4–6 emails, useful email once for 2–3 cdays. Use phone call and WeChat to contact with friend. Important info to email. Have promotional emails. Like Amazon’s promotion. Don’t look much.
- How you feel about mixing important emails and unimportant ones?
- Because don’t look at email much, so junk accumulate.
- What do you think about smart sorting?
- Smart sorting is going to be useful.
- What expectations you have for a smart sorting?
- All ad together. Look at it 1/2weeks
- Bills together
- School and work
- Conversation
- Job posting and interview related
- different time care about different things. Dynamically change your focus
- Deadline automatic identification
- Do you have any specific requirement related to smart sorting accuracy?
- Don’t Don’t put important email to advertisement.
- Not too many categories.
- Customize the sorting. And smart help. Like iPhone photo auto sorting.
Post event protocols
- Don’t know why not showing every emails. Emails at the Bottom doesn’t show.
- Just discovered that there is a “search”
- Too many discover card email. All over the place. Can sort into a folder based on name
- Need to go back and forth to select sent and inbox.
- Don’t know what the filter function is about. Never discovered it.
User 2
Profile
Age: 37
Education: PhD degree
Income level: 175–200k
Occupation: Quantitative Analyst
App using expertise: Intermedium level
Motivation: Make sure I am on top of everything
Interview
- How often you use cell phone to check email?
- Everyday more than 10 times
- Under what circumstance?
- In the middle of work, at lunch, morning before work, walking my dog. Any time except when driving
- What pain point do you have when using the mailbox app?
- Search can not find the desired result
- useful information goes to spam.
- How many emails you receive everyday?
- 30 emails every day.
- How you feel about mixing important emails and unimportant ones?
- No don’t feel important emails berried in unimportant ones.
- What do you think about smart sorting?
- Smart sorting must be smart enough to be useful. Need to be accurate. Hate when there is mistake
- What expectations you have for a smart sorting?
- Don’t mind unimportant things to be in important things. But don’t sort my important things into unimportant things
- No other thoughts.
- Hate web version
- want to be able to customize category
- Category shouldn’t be too granular. Don’t want to manage that many folders
- low error rate. Sort into the folder only when there is a high confidence
Post event protocols
- Open the inbox, look at every unread email from top to down
- Everything that is ads, delete
- Everything that want to read more will keep. Until reach last time’s reading point.
- Then go back to read all the details of the emails want to read
- Also keep the emails that knows need to keep, but won’t read now.
- Repeat this 10 -20 times. Less than 5 minutes every time.
- Why I subscribed to so many junk. Maybe my wife did it. But yeah, sometimes there will be one that is useful. It’s actually worth it to get all the junks just for that few useful ones.
- What if I want to unsubscribe but was not successful, then just throw them all into spam.
User 3
Profile
Age: 40
Education: PhD degree
Income level: 200–225K
Occupation: Quantitative Analyst
App using expertise: Intermedium level
Motivation: Make sure I don’t miss important emails including good deal
Interview
- How often you use cell phone to check email?
- 3 times per day
- Under what circumstance?
- work break, and morning, evening
- What pain point do you have when using the mailbox app?
- Too many promotional emails, hard to manage.
- Don’t want to do anything about them. Just leave them their. But there is a risk to have useful emails berried in them.
- Don’t want to spend time delete things. Too time consuming
- How many emails you receive everyday?
- 50–100 emails every day
- How you feel about mixing important emails and unimportant ones?
- Painful. Very much. All the time. Waiting for quote from house improvement contractor. Didn’t realized it was already there.
- What do you think about smart sorting?
- Smart sorting useful
- Would help me manage.
- Investment house. Business need.
- What expectations you have for a smart sorting?
- Get rid of or organize the seemingly junk emails
- Automatically detect the type of the email. Decide which folder to go into.
- Recommend to me to read or not.
- Management result easy to navigate. Intuitive presentation
- Minimize the click number. These are all chore for me. The fastest way to help me get through them.
- Do you have any specific requirement related to smart sorting accuracy?
- As accurate as possible. But save me time is more important.
Post event protocols
- Didn’t have enough time to go through all the emails
- Hated the amount of promotional emails, but liked the content. Some are very appealing. Awesome deals. Got distracted…
- A smart management is very much desired.
- ATT signal is not very good. waited too long to get it refreshed.
- Hope it has an audio function can read the emails for me.
User 4
Profile
Age: 37
Education: Bachelor degree
Income level: 150k — 175k
Occupation: Front end developer
App using expertise: Intermedium level
Motivation: Get important informations
Interview
- How often you use cell phone to check email?
- Daily
- Under what circumstance?
- Every where
- What pain point do you have when using the mailbox app?
- You check for the company email. Best way is the automatic notification.
- sound notification. I don’t like the sound
- Doing what it suppose to do.
- You can not do bulk mail
- Some time accidentally move to trash.
- How many emails you receive everyday?
- 8–10 emails.
- How you feel about mixing important emails and unimportant ones?
- Recruiter
- Subscriptions
- Marketing
- Don’t go through emails one by one.
- Lots of unwanted emails
- What do you think about smart sorting?
- That would be a good solution
- A new email account
- Create folders, automatically send to those folders.
- What expectations you have for a smart sorting?
- Smarter mechanism to sort, don’t want important emails to go to spam.
Post event protocols
- Nobody want to go to computer to check it. Easy to use
- a lot of unwanted emails. How they got my emails?!
- iPhone app is nice.
User 5
Profile
Age: 29
Education: Master degree
Income level: 125- 150k
Occupation: Data Analyst
App using expertise: Power user
Motivation: Keep track of all social activities
Interview
- How often you use cell phone to check email?
- 5–10 times / day
- Under what circumstance?
- At work at home first thing in the morning last thing at night.
- What pain point do you have when using the mailbox app?
- No push notification. Disabled. Frustrated initial, but when don’t have, you find it good.
- UI has been the same for the past years.
- Functionality not good
- A bug: when click in box, search, click on the first result, go back, it doesn’t go back to the search page, go back to general page. Randomly happening
- Multi mail box set up. Big mess.
- Managing not very intuitive.
- Idea of having multiple is good.
- 10 mail boxes, how to be very intuitive to mark some email box that is inavtive. Only push notify only some mail box. Priority
- Prioritize different mail box.
- you create account to separate your life. Now it’s all mixed together.
- Only go to job looking account when looking for job.
- UI not good.
- Not appealing
- Can’t create folder
- Can’t organize mailbox
- Have to go to the actual web mail box.
- It is light weight not very comprehensive
- Heavy user version, advance functionality
- How many emails you receive everyday?
- 1000/day 5–8 mailbox
- How you feel about mixing important emails and unimportant ones?
- Email types
- News letters
- Recruiters
- Promotion, marketing
- Personal
- Work
- Organizations
- School
- Select and delete, quick look to see whether you missing anything important
- Filters
- Smart sorter
- Important emails berried, miss important email because of these.
- Have to constantly go unsubscribe unwanted subscriptions. Small letter to unsubscribe. Select a couple of domains and automatically go to there web and unsubscribe. That would be so cool.
- Distinct sender. See some statistics. See which domain is putting all the junks in my emails.
- What do you think about smart sorting?
- (Skip. He mentioned smart sorting spontaneously)
- What expectations you have for a smart sorting?
- Creating rules from the app. No mailbox have it now. Custom folders. Want to customize rules from the app
- Folder for a specific mail box
- Only open iCloud email on phone
- Common function across emails.
- (Voluntarily gave a lot of other thoughts)
- Sudo email box, combine and show everything on that.
- Can not search all mail box at once. No advance search. No to someone, from someone. Other than that search is good. Type a name shows the name card.
Post event protocols
- As a heavy user, never-ending mail. Pin email at the top would be great.
- Reply auto suggest would be good.
- if have time in the email, would like a link to create reminder
- preview/summary of the attachment. Inbox app
- Linking other apps, reminders, show in the mailbox. Google keep notes integration.
- Shopping: shop now directly takes you to the app.
- Check score:
- Hyper link handy. Don’t even need to open the email.
- Share file easily. Gmail can do google drive easily.
- Change design. also give you the options.
- Smart folder
- Flight info extraction from the long email. Pin and set reminder for certain time. Already have google flight.
- Concert have similar functionality.
