Grabango

LabelTool is Grabango's internal platform. It is a machine learning platform used by a large group of Humans in the Loop, or Quality Trainers. LabelTool is distributed in a logical number of tasks and sub-tasks to be completed in order by the quality trainers. The tasks (in order) and thier descriptions are:

• Person/People Dotting: dotting up to one person, and all people, given up to 3 keystone images at 2hz

• Track Validation:
   Consisting of three sub-tasks:
   • Single Track Validation — for a camtrack video, confirm this is one and only one person
   • Bulk Track Validation — confirm that the images have the same person in the keystone images
   • Track Match — confirm that two camtracks videos have the same person

• Product ID:
   Consisting of four sub-tasks:
   •Event Identification — confirm when events happen and what kind
   •Event Location — confirm where (aisle, section, shelf) an event happens
   •Product Identification — confirm what product is asociated eith the event
   •Product Quantity — confirm number of products in the event

• Receipt Revue: validation that all the events from Product ID are labeled correctly

Machine learning is a large part of the process at Grabango. Grabango relies on Quality Trainers, also referred to as Humans in the Loop, to teach the computer.

Grabango's platform uses software that tracks and collects data that the Quality Trainers provide. Many of the tasks are very simple and menial, but necessary for machine learning. I took a user-centered approach to the designs. To fully understand how to design the LabelTool for the Quality Trainers, I sat down to interview and observe their workflows.

The research was an observational study. I had the quality trainers go through their process from starting a task to completing it. I observed while asking questions and taking lots of notes. I also had them complete several tasks - this was to observe the different methods the quality trainers employed, as well as any "hacks" they discovered.

After the research part of the design process, I synthesized the data and started sketching lots of flows and a few interfaces.

Grabango's LabelTool is based in flows - workflows, task flows, and user flows - and the actions (and failures) that a user can take (or experience). A Quality Trainer works through each task, and the data collected is used for machine learning. Once enough data is collected and used to identify certain events, the manual task will be left to the computer, and the quality trainer will move on to more complex tasks.

Actions on a particular task are compiled into a list, along with the failures that can happen at any time: a mislabeled event, item, or quantity, a incorrect person being tracked, or simply just a bad track. After actions are defined, I then create flows.

Flows are an extremely important part of UX - many things are discovered during this part of the process, and leaves little room for error later. After the overall task flow of the LabelTool was defined, I focused on the time-consuming tasks performed by the Quality Trainers. These were broken down into several steps and informed the design of LabelTool's user interface.

Then, the flows start to get more granular. I create the happy path for a Quality Trainer: starting with the first task and completing the task without any friction. From there, I create flows for each type of user journey, and then they start to get more complex. I start adding failures and their solutions. Once those have been flushed out, I try to capture as many edge cases as possible.

Quality Trainers and Managers of Quality Trainers both need a dashboard. A dashboard is the entry point to LabelTool. It tells a user the current status of their system, and answers the questions: 'What do I need to do next?' and 'What do I care about?'. The Quality Trainer and Manager of Quality Trainers have different needs that were displayed in their dashboard.

For the Manager of Quality Trainers, the dashboard is a high-level view into the overall status and progress of tasks and people. Important metrics for the QT Manager are:

• Total number of tasks across team
• Number of tasks in progress
• Number of specific tasks
• Average time to complete a task
• Average time to complete a specific task
• Improvement over time

For the Quality Trainer, the dashboard serves two purposes. It serves as an action-based entry point to tasks and as a view into their progress and status. Quality Trainers need to know:

• Where did I leave off?
• What is my next action item?
• What needs to be done?
• How am I doing? (Compared to teammates)
• What is my score? (Based off of time)
• Improvement over time

Person Dotting is the first task for a Quality Trainer. The Quality Trainer must first label when a shopper has entered the store. The Quality Trainer then clicks (dots) the shopper's head every 5 seconds as the shopper walks through the store. The result is a “track” - a list of coordinates that pair a 2d point with a timestamp, representing the path of the shopper through the view of the selected sensor. The final part of this task is to label when the shopper has exited the store.

People Dotting is a task that is the similar to Person Dotting, but the Quality Trainer is asked to provide an individual track for every person who walks through the sensor’s view.

Single Track Validation is the second task in LabelTool and asks the Quality Trainer to evaluate an supplied track. The Quality Trainer must confirm that the track matches the person in the frame, confirm that the track show a person, or if the track is malformed that it should be discarded.

In Bulk Track Validation, the images at the top of the screen are stills or video of the focused shopper on checkout. The images below are sampled from the track that describes the path of the shopper. The Qualtiy Trainer responds to the task by selecting any images that do not accurately show the shopper.

Product ID asks the Quality Trainer to validate and capture four parts of an event: identification, location, product, and quantity. Event location is a task that allows shopping events to be added to a shopping trip. One shopper is selected in this task, and the trainer watches and marks the events that occur during the time provided. If an event is identified, the trainer is also asked to add location markers in the next view so that we can generate a 3D location of the product that is being interacted with. Event Identification can either be a pick up, a put down, an uncertain pick up, or an uncertain put down.

For Product ID, a quality trainer is given a view of a single event from the shopping trip, and identifies the product and the quantity from the trip.

Receipt Review is the final task in LabelTool. The Quality Trainer reviews all of the events that are included in the receipt and validate that the product is correctly identified and that the event is described correctly. If something is wrong with the receipt, the Quality Trainer has the opportunity to fix the issue.

For Event Identification, the Quality Trainer must confirm that the event, product, and quantity are identified correctly. In the case that an event, product, or quantity was identified incorrectly, I designed flows to address all cases.

In Event Identification, there are three options: correct event, incorrect Event, and missing event. In Product Identification, ther are also three options: correct product, incorrect product, and missed product. For incorrect quantity, the Qulaity Trainer simply updates the quantity inline.

User management for LabelTool is important for granting access to specific users. In LabelTool, administrators can manage user access to various resources within LabelTool. User management also enables administrators to control user access and on-board and off-board users to and from LabelTool.

The purpose of the Grabango Control Panel is to allow users to manage visits beyond the LabelTool dashboard and create more visibility into labeled work across the organization.

The Control Panel ultimately serves 3 user groups: Quality Trainer Managers, Computer Vision teams, and engineering roles.

Each user group has a slightly tailored view from the tool’s mainpage, and this is referred to as the dashboard. The needs are determined by group:

Quality Trainer Managers need to be able to:
• Track visits in flight
• Process payment / monitor payment
• Make data-driven optimizations
• Perform Data Management / tagging

Computer Vision teams need to be able to:
• Optimize
• Manage Data (Eg. identifying useful visits for training set)
• Scale Projections

Engineering roles needs to be able to:
• Optimize
• Debug visits

Planning, designing, testing and coding are all streamlined to reduce wasting time. Incorporating a design system for Grabango's LabelTool platform saves time for design and engineering. BY breaking design elements into homogenized components, the design system made the whole process of building the platform more efficient.