Data Buffer Overview

The “Data Buffer” is a system that enables API clients to log data to the robot using the API. This data is combined with data logged by internal robot services, offering users a better understanding of robot operation and offering developers better tools for developing and debugging their new services. Clients are able to query the robot data service for information about the data stored on the robot as well as download the data from an HTTP server. Data acquisition services use the data buffer as a mechanism for storing acquired sensor data.

The robot allocates 60 gigabytes for data buffer storage. Logged data is saved to persistent storage in data pages. In Spot v2.1 all data pages are BDDF files on disk. When the limit of 60 gigabytes is exceeded, data files are deleted in oldest-first order until disk space used by these files falls below that threshold.

Supported Data Types

Although API clients may log arbitrary data, the Data Buffer service enables clients to log specific structured types that may be handled specially and simplify later analysis.

Text Messages

Basic text logs supporting Info, Error, Warning, and Debug levels, as well as file and line number of origin. API clients and services may use the data buffer as a backend to their own text logging systems. Clients may log text messages via the RecordTextMessages RPC.

Operator Comments

Text-based messages, but primarily intended as a mechanism for robot operators to insert comments describing, for example, notable robot or mission observations, or instances where the robot did not perform as expected. Operator comments do not have a level, file, or line number. These messages are shown on the timeline in the log-download page. Also, internal log data near the time of log comments is preserved for a longer time so that it may be selected for sending to Boston Dynamics for debugging and failure analysis. Clients may log operator comments via the RecordOperatorComments RPC.

Events

Event messages are appended to an on-robot event diary, enabling a robot or mission timeline to be built. Typical usage will include events for robot falls or calibration, but clients may add any context-building events that are useful when reviewing missions or robot performance. Each event is logged with a “level” enum designating the importance or criticality of the event. Most events will be enumerated with LOW, MEDIUM, or HIGH level, but the most important events may be enumerated as MISSION_CRITICAL or SYSTEM_CRITICAL. Clients may choose any appropriate level, but SYSTEM_CRITICAL events are quasi-reserved for internal events generated by Boston Dynamics. Clients may log events via the RecordEvents RPC.

Signals

Signals messages enable logging of numerical time series data. Clients must first log a “schema” message describing the time series data to be logged, including the name and type of each variable. This enables post-processing tools to properly parse the downloaded signals logs. The robot returns a schema id and the client may proceed to log the time varying values associated with that schema. Clients may register schemas via the RegisterSignalSchema RPC and log signals via the RecordSignalTicks RPC. Signals data is not well-supported in release 2.1, but tools for using this data should become available in future releases.

Blobs

Blob messages enable clients to log arbitrary binary message data to the robot. This may include, for example, sensor data products, user-defined structures and messages, whole files, etc. The data buffer itself does not parse the binary data, but requires that the client specify a type_id and channel in the logging request. The type_id specifies the type of data stored in the opaque blob, required for post-processing. The channel is a name for a sequence of messages. Clients may log blobs via the RecordDataBlobs RPC. The Data Acquisition system logs acquired data to the data buffer as blobs for later retrieval.

Internally Logged Data

Internally, the robot logs a subset of its data to the data buffer which may be accessed by external clients. This data set may expand in future releases. The current set includes events for robot falls, SpotCheck, comms loss, along with all the gRPC requests/responses, and Spot IO data acquisitions. For improved performance, large fields in the gRPC messages may be stripped from the log.

Data Service

The data service is a gRPC service that allows clients to query for information about data that has been logged to the robot. The user may:

  • Query for pages containing data within a specified time range

  • Query for events and comments logged to the robot

  • Query for overall data buffer status, including the amount of data currently logged to the robot

  • Delete individual data pages on the robot

Data Download

Raw Data Download

Data may be downloaded as BDDF files by querying the robot’s HTTPS REST interface with a GET request at the URL /v1/data-buffer/bddf/. Queries are limited to one hour. A single start time must be specified in either the from_sec or from_nsec parameters. Likewise, a single end time must be specified in either the to_sec or to_nsec parameters. If the time range is unspecified or exceeds 3600 seconds, the query fails and HTTP 403 (Bad Request) is returned. The event_level_min parameter can only be specified once. All other parameters may be specified zero or more times in combination.

  • from_sec or from_nsec: Download data after the specified epoch time if set, in seconds and nanoseconds respectively.

  • to_sec or to_nsec: Download data before the specified epoch time if set, in seconds and nanoseconds respectively.

  • type: Filter on the type_id of blobs.

  • channel: Filter on the channel of blobs.

  • schema_name: Filter on the schema name for schema and signals messages.

  • schema_source: Filter on the schema source for signals messages.

  • grpc_service: Filter grpc services by name.

  • event_level_min: Filter events by level.

Data Acquisition Download

As noted, Data Acquisition services use the data buffer for storing captured data. The data is retrieved by querying the HTTPS server with a GET request at the URL /v1/data-buffer/daq-data/. These requests return data blobs of type bosdyn.api.DataAcquisitionObject and bosdyn.api.AssociatedMetadata as a ZIP file. By default, all such objects are returned, but can be filtered by the following parameters:

  • from_sec or from_nsec: Download data after the specified epoch time if set, in seconds and nanoseconds respectively. These are mutually exclusive, and may only be specified once.

  • to_sec or to_nsec: Download data before the specified epoch time if set, in seconds and nanoseconds respectively. These are mutually exclusive, and may only be specified once.

  • channel: Filter on the channel associated with the logged blobs. May be specified zero or more times.

Authorization

All requests to download data from the robot must be authorized with a user token included in the HTTP header as a bearer token. User tokens may be acquired by authorizing with the API auth service.

Example

An example data download, using cURL, is shown below. This example assumes the user has a valid user token in the file user.token and that the robot hostname or IP address is stored in the $ROBOT environment variable. This downloads Data Acquisition data between the epoch timestamps 1601524800 and 1601611200, corresponding to Thursday, October 1, 2020 12:00:00 AM GMT-04:00, through Friday, October 2, 2020 12:00:00 AM GMT-04:00.

curl -O -J \
    -H "Authorization: Bearer $(cat user.token)"
    "https://$ROBOT/v1/data-buffer/daq-data/?from_sec=1601524800&to_sec=1601611200"

BDDF (Boston Dynamics Data Format)

Raw data downloaded off the robot will generally be returned in BDDF (Bosdyn Download Data Format) files. BDDF is an open file format developed by Boston Dynamics that can hold arbitrary POD (plain old data) or message style data. BDDF files are simple to read and write, offer efficient random access to embedded messages or can be streamed, and are secure in that they require little or no parsing logic. Additional details can be found in the BDDF documentation.