Data Buffer Overview¶
The “Data Buffer” is a system that enables API clients to log data to the robot using the API. This data is combined with data logged by internal robot services, offering users a better understanding of robot operation and offering developers better tools for developing and debugging their new services. Clients are able to query the robot data service for information about the data stored on the robot as well as download the data from an HTTP server. Data acquisition services use the data buffer as a mechanism for storing acquired sensor data.
The robot allocates 60 gigabytes for data buffer storage. Logged data is saved to persistent storage in data pages. In Spot v2.1 all data pages are BDDF files on disk. When the limit of 60 gigabytes is exceeded, data files are deleted in oldest-first order until disk space used by these files falls below that threshold.
Supported Data Types¶
Basic text logs supporting Info, Error, Warning, and Debug levels, as well as file and line number of origin. API clients and services may use the data buffer as a backend to their own text logging systems. Clients may log text messages via the
Text-based messages, but primarily intended as a mechanism for robot operators to insert comments describing, for example, notable robot or mission observations, or instances where the robot did not perform as expected. Operator comments do not have a level, file, or line number. These messages are shown on the timeline in the log-download page. Also, internal log data near the time of log comments is preserved for a longer time so that it may be selected for sending to Boston Dynamics for debugging and failure analysis. Clients may log operator comments via the
Event messages are appended to an on-robot event diary, enabling a robot or mission timeline to be built. Typical usage will include events for robot falls or calibration, but clients may add any context-building events that are useful when reviewing missions or robot performance. Each event is logged with a “level” enum designating the importance or criticality of the event. Most events will be enumerated with LOW, MEDIUM, or HIGH level, but the most important events may be enumerated as MISSION_CRITICAL or SYSTEM_CRITICAL. Clients may choose any appropriate level, but SYSTEM_CRITICAL events are quasi-reserved for iternal events generated by Boston Dynamics. Clients may log events via the
Signals messages enable logging of numerical time series data. Clients must first log a “schema” message describing the time series data to be logged, including the name and type of each variable. This enables post-processing tools to properly parse the downloaded signals logs. The robot returns a schema id and the client may proceed to log the time varying values associated with that schema. Clients may register schemas via the
RegisterSignalSchema RPC and log signals via the
RecordSignalTicks RPC. Signals data is not well supported in release 2.1, but tools for using this data should become available in future releases.
Blob messages enable clients to log arbitrary binary message data to the robot. This may include, for example, sensor data products, user-defined structures and messages, whole files, etc. The data buffer itself does not parse the binary data, but requires that the client specify a
channel in the logging request. The
type_id specifies the type of data stored in the opaque blob, required for post-processing. The
channel is a name for a sequence of messages. Clients may log blobs via the
RecordDataBlobs RPC. The Data Acquisition system logs acquired data to the data buffer as blobs for later retrieval.
Internally Logged Data¶
Internally, the robot logs a subset of its data to the data buffer which may be accessed by external clients. This data set may expand in future releases. The current set includes events for robot falls, SpotCheck, comms loss, along with all the gRPC requests/responses, and Spot IO data acquisitions. For improved performance, large fields in the gRPC messages may be stripped from the log.
The data service is a gRPC service that allows clients to query for information about data that has been logged to the robot. The user may:
Query for pages containing data within a specified time range
Query for events and comments logged to the robot
Query for overall data buffer status, including the amount of data currently logged to the robot
Delete individual data pages on the robot
Raw Data Download¶
Data may be downloaded as BDDF files by querying the robot’s HTTPS REST interface with a GET request at the URL
/v1/data-buffer/bddf/. Queries are limited to one hour. A single start time must be specified in either the
from_nsec parameters. Likewise, a single end time must be specified in either the
to_nsec parameters. If the time range is unspecified or exceeds 3600 seconds, the query fails and HTTP 403 (Bad Request) is returned. The
event_level_min parameter can only be specified once. All other parameters may be specified zero or more times in combination.
from_nsec: Download data after the specified epoch time if set, in seconds and nanoseconds respectively.
to_nsec: Download data before the specified epoch time if set, in seconds and nanoseconds respectively.
type: Filter on the
channel: Filter on the
schema_name: Filter on the the schema name for schema and signals messages.
schema_source: Filter on the schema source for signals messages.
grpc_service: Filter grpc services by name.
event_level_min: Filter events by level.
Data Acquisition Download¶
As noted, Data Acquisition services use the data buffer for storing captured data. The data is retrieved by querying the HTTPS server with a GET request at the URL
/v1/data-buffer/daq-data/. These requests return data blobs of type
bosdyn.api.AssociatedMetadata as a ZIP file. By default, all such objects are returned, but can be filtered by the following parameters:
from_nsec: Download data after the specified epoch time if set, in seconds and nanoseconds respectively. These are mutually exclusive, and may only be specified once.
to_nsec: Download data before the specified epoch time if set, in seconds and nanoseconds respectively. These are mutually exclusive, and may only be specified once.
channel: Filter on the
channelassociated with the logged blobs. May be specified zero or more times.
An example data download, using cURL, is shown below. This example assumes the user has a valid user token in the file
user.token and that the robot hostname or IP address is stored in the
$ROBOT environment variable. This downloads Data Acquisition data between the epoch timestamps
1601611200, corresponding to Thursday, October 1, 2020 12:00:00 AM GMT-04:00, through Friday, October 2, 2020 12:00:00 AM GMT-04:00.
curl -O -J \ -H "Authorization: Bearer $(cat user.token)" "https://$ROBOT/v1/data-buffer/daq-data/?from_sec=1601524800&to_sec=1601611200"
BDDF (Boston Dynamics Data Format)¶
Raw data downloaded off the robot will generally be returned in BDDF (Bosdyn Download Data Format) files. BDDF is an open file format developed by Boston Dynamics that can hold arbitrary POD (plain old data) or message style data. BDDF files are simple to read and write, offer efficient random access to embedded messages or can be streamed, and are secure in that they require little or no parsing logic. Additional details can be found in the BDDF documentation.