Smart Glasses & Streaming

Remote Assist

Project Overview
I designed the Remote Assist app, which is used by Forge to connect their skilled trades workers for real-time job support and quality assurance. I designed two iterations of this product, the first being a native iOS app that leverages LiDAR sensors, and the second a smart glasses app on the Vuzix Blade that connects to a web app.
Platforms
Smart Glasses
Responsive Web
Role
Lead Designer

Goals

At Forge, a tech-enabled home renovations company, a significant portion of the workforce consists of junior and mid-level skilled trades workers. To efficiently address complex on-site challenges, the team requires streamlined access to senior crew members, who are often dedicated to higher-complexity projects. I led this project which has two phases:

1

2

Build out a prototype to validate assumptions and prioritize quick learnings.

Refine product and ship a solution to be used in the field to continue gathering feedback.

We explored various hardwares choices and decided to build the solution on the Vuzix Blade which has a front-facing camera.

Project Complexities

Environmental Factors
This product required considering the pro's work environment and the challenges of a physical job that often times involve using both hands.
Hardware Selection
The experience depended heavily on which hardware we decide to go with, and its unique capabilities. We had to consider the form factor, weight, camera quality, and battery life.
Ambuiguity
This project had a small team that worked collaboratively to figure out each step. We had to make quick and strategic decisions to move the project forward.

One of the most common use cases for windows replacement is troubleshooting the issues that are found after an old window has been removed.

Research

I spoke with crew members of various skill levels to understand what type of issues commonly require a senior team member to help resolve. In addition, I shadowed on jobs to observe how the crews work.

1x1 Interviews
These conversations are structured around when issues commonly arise, how they troubleshoot, and what are the senior crew's role in troubleshooting.
Field Observations
I hung out with the crew members and observed the process for window replacement jobs, which are the most common type of jobs.

Main Takeaways

Windows Removal
It is most common to encounter issues after an older window is removed. This uncovers any wood rot, mold, or termite issues which will either halt or delay projects significantly.
Troubleshooting
There are two general types of troubleshooting. The first involves helping figure out a creative solution to work around the job's unique issues. The second involves making a judgement whether to delay a job or not.
Project Coordinators
In addition to senior crew members, I discovered that project coordinators are also closely intertwined in the workflow. They need to know of any additional materials, time, or scope changes that comes with the issues.
Quality Control
The junior team members mentioned that they "don't know what they don't know" yet, and would appreciate feedback on whether they completed the job well.  

Iterations

iPhone (with LiDAR)

Our first explorations were a streaming and viewing experience on the iPhone 12 Pro Max, which has LiDAR sensors. I envisioned the UI to have three main functionality: tools, clipping of video, and messaging.

The tools are used to highlight specific areas in the 3D space. Video clips help the team flag important things that were discussed. Messaging helps facilitate any additional context the users need to share.

Clip Moments

The video clips are collected for both crew leads and project coordinators to keep tabs on the issues and solutions.

Vuzix Blade

In the second iteration, we switched hardware because we found our team needed to be hands-free a lot of the times. In these explorations, I tested out a flow the allows the user to start a stream directly on the Vuzix's heads up display.

Feedback Summary

Hands-free
In testing our prototype designed for the iPhone, we observed that often times pros needed to use both hands to demonstrate and troubleshoot at the same time.
Larger Video Feed
In the prototype, we designed both the viewing and streaming experience on the iPhone, anticipating that our crews will need be mobile. However, we saw that the crew leads wanted to zoom in to the video to look at the details.
Color Contrast
When wearing the Vuzix, the colors in the HUD is actually quite muted. The hardware auto adjusts the contrast to fit the environment's lighting. Colors and certain UI elements are hard to discern.

Solution Highlights

Contrast

To ensure user safety and optimize usability in unpredictable environments, I prioritized high contrast for all core functionality. Bright green shades which have been proven to have the best visibility to the human eye is favored over our brand colors.

Responsive Web App

I decided to go with a responsive web app for the viewing experience. This would allow flexibility for our crew leads to join either via their mobile phone, or on the company iPad, which will give them a larger view. This also opens up the stream for the project coordinators who are commonly looped into the troubleshooting process.

Consolidated Job Data

Leveraging the front camera of the Vuzix, we allow the streamer to both stream and take snap shots at the same time. This viewing experience allows the user to flip between video and photo views, as well as document notes. Job information, images, and notes are tied together via a job ID in the database.

Bookmarks

We included a bookmarking feature that allows certain moments to be flagged. This is especially helpful when the crew lead and project coordinator can not join the call at the same time, ensuring critical information is still captured and shared.

The Results

Increased Utilization
Senior pros spent less time traveling to job sites and were able to be allocated to more jobs, which led to more overall revenue.
Quality Control
Crew leads are able review every job which helped uphold Forge's craft quality. Junior team members also reported feeling more confident in their work.
Informed Scheduling
Project coordinators gain visibility to every job progress which helps with managing client expectations and job scheduling.

Notes & Thoughts

The team and I felt strongly that there was something to be solved with a hands-free experience for our pros in the field. The Remote Assist experience is a good first step in the right direction, but I'm curious to know where the product can be pushed with the right hardware. There were issues with battery life, connectivity, and camera quality that didn't make the experience frictionless. In addition, because the Vuzix Blades didn't have the same sensors and processing power as the iPhone, the current solution lost the 3D mapping functionalities we originally had. I'm excited to see how AR hardware evolves in the next few years.

Project Artifacts

Kathy Chau
LINKEDIN