Machine Learning Bridge and External Compute¶
Spot provides capabilities to perform computation off-board and easily integrate it into your API programs. For example, you might want to run a deep-learning-based bounding box detector to identify a ball that you want to command the robot to pick up.
The network_compute_bridge
API can help you by coordinating the networking between various components and adding pre- and post-processing to your image calls, capturing and rotating images for you as needed. The tablet application has a built-in capability to view the output of bounding-box detectors online while driving (Menu > Utilities > ML Model Viewer
).
Example Concept¶
You have trained a model to detect tennis balls. You want to build a script to walk to every tennis ball in an area. Because your tennis ball detector is computationally expensive, you decide to have a big server running on the same WiFi network as Spot to do the computation.
Here’s how you might set that up:
Large server is running in a server room, on the same WiFi network as Spot.
Your client laptop has a script that queries the
Network Compute Bridge
to:capture an image
run the tennis ball detector on that image
return results to the script
With the results, your script commands the robot to walk to a tennis ball.
Components of the system¶
Spot robot
Server running computation
mounted on Spot or externally networked
Client computer
robot tablet
external computer
same computer as is running the server
Discovering Servers and Models¶
Computation servers register with Spot’s Directory.
Clients can use the directory to look up available servers (see: listing services).
Servers should implement the
ListAvailableModels
RPC to inform clients of their available computation.
Example Clients and Servers¶
Examples are found in the network compute bridge examples folder.
Pre- and Post-Processing¶
If you are working with images, you can:
Ask the robot to capture an image
Send an already-captured image for computation
If you choose (1), you can optionally have the robot rotate the image to align the horizontal. This is particularly useful for the robot’s built-in cameras that are not all mounted horizontally. Many ML detectors are biased towards horizontally-captured images, so this is a good way to improve your results.
If you use the rotation functions, the robot will:
Capture image
Rotate the image to the horizontal
Send to the external server
Receive the bounding-box results and un-rotate the boxes so they line up with the original image
Send you the original image with the corresponding bounding boxes.
RPC Definitions: NetworkComputeBridge
and NetworkComputeBridgeWorker
¶
NetworkComputeBridge
is the API interface between your API client and the robot.NetworkComputeBridgeWorker
is the API interface between the robot and a compute server
Other Data¶
The API provides an Any
field in the proto to allow you to pack arbitrary input and output data. This field allows you to perform external computation on any data. However, the tablet application will not be able to interpret this custom data.