Machine Learning Bridge and External Compute¶
Spot provides capabilities to perform computation off-board and easily integrate it into your API programs. For example, you might want to run a deep-learning-based bounding box detector to identify a ball that you want to command the robot to pick up.
network_compute_bridge API can help you by coordinating the networking between various components and adding pre- and post-processing to your image calls, capturing and rotating images for you as needed. The tablet application has a built-in capability to view the output of bounding-box detectors online while driving (
Menu > Utilities > ML Model Viewer).
You have trained a model to detect tennis balls. You want to build a script to walk to every tennis ball in an area. Because your tennis ball detector is computationally expensive, you decide to have a big server running on the same WiFi network as Spot to do the computation.
Here’s how you might set that up:
Large server is running in a server room, on the same WiFi network as Spot.
Your client laptop has a script that queries the
Network Compute Bridgeto:
capture an image
run the tennis ball detector on that image
return results to the script
With the results, your script commands the robot to walk to a tennis ball.
Components of the system¶
Server running computation
mounted on Spot or externally networked
same computer as is running the server
Discovering Servers and Models¶
Example Clients and Servers¶
Examples are found in the network compute bridge examples folder.
Pre- and Post-Processing¶
If you are working with images, you can:
Ask the robot to capture an image
Send an already-captured image for computation
If you choose (1), you can optionally have the robot rotate the image to align the horizontal. This is particularly useful for the robot’s built-in cameras that are not all mounted horizontally. Many ML detectors are biased towards horizontally-captured images, so this is a good way to improve your results.
If you use the rotation functions, the robot will:
Rotate the image to the horizontal
Send to the external server
Receive the bounding-box results and un-rotate the boxes so they line up with the original image
Send you the original image with the corresponding bounding boxes.
NetworkComputeBridgeis the API interface between your API client and the robot.
NetworkComputeBridgeWorkeris the API interface between the robot and a compute server
The API provides an
Any field in the proto to allow you to pack arbitrary input and output data. This field allows you to perform external computation on any data. However, the tablet application will not be able to interpret this custom data.