Bddf

Code for reading and writing ‘bddf’ data files.

exception bosdyn.client.bddf.DataError[source]

Bases: Exception

Errors related to the DataWriter/DataReader system.

exception bosdyn.client.bddf.AddSeriesError[source]

Bases: Exception

Errors related to registering a series in a DataWriter.

exception bosdyn.client.bddf.SeriesNotUniqueError[source]

Bases: bosdyn.client.bddf.AddSeriesError

The series_spec is not unique within the file.

exception bosdyn.client.bddf.ChecksumError[source]

Bases: bosdyn.client.bddf.DataError

The file checksum does not match the computed value.

exception bosdyn.client.bddf.DataFormatError[source]

Bases: bosdyn.client.bddf.DataError

Data to be stored has the wrong format.

exception bosdyn.client.bddf.ParseError[source]

Bases: bosdyn.client.bddf.DataError

Data file has incorrect format.

class bosdyn.client.bddf.FileIndexer[source]

Bases: object

An object which keeps an index of series and blocks within series.

It can write a block index at the end of a data file.

property file_index

Get the FileIndex proto used which describes how to access data in the file.

property descriptor_index

Get the Descriptor proto containing the FileIndex.

property series_block_indexes

series_index -> SeriesBlockIndex.

Type

Returns the current list of SeriesBlockIndexes

series_descriptor(series_index)[source]

Return SeriesDescriptor for given series index.

static series_identifier_to_hash(series_identifier)[source]

Given a SeriesIdentifier, return a 64-bit hash.

add_series_descriptor(series_descriptor, series_block_file_offset)[source]

Add the given series_descriptor to the index, with the given file offset.

Parameters
  • SeriesDescriptor to add to the index (series_descriptor) –

  • Location in file where SeriesDescriptor will be written (series_block_file_offset) – was read from.

  • or – was read from.

add_series(series_type, series_spec, message_type, pod_type, annotations, additional_index_names, writer)[source]

Register a new series for messages for a DataWriter.

Parameters
  • series_type – the kind of spec, corresponding to the set of keys expected in series_spec.

  • series_spec – dict of {key (string) -> value (string)| describing the series.

  • message_type – MessageTypeDescriptor (need EITHER this OR pod_type)

  • pod_type – PodTypeDescriptor (need EITHER this OR pod_type)

  • annotations – optional dict of key (string) -> value (string) pairs to associate with the message channel

  • additional_index_names – names of additional timestamps to store with each message (list of string).

  • writer – BlockWriter owned by the DataWriter.

Returns series id (int).

Raises SeriesNotUniqueError if a series matching series_spec is already added.

index_data_block(series_index, timestamp_nsec, file_offset, nbytes, additional_indexes)[source]

Add a an entry to the data block index of the series identified by series_index.

make_data_descriptor(series_index, timestamp_nsec, additional_indexes)[source]

Return DataDescriptor for writing a data block, and add the block to the series index.

write_index(block_writer)[source]

Write all the indexes of the data file, and the file end.

class bosdyn.client.bddf.BlockWriter(outfile)[source]

Bases: object

Writes data structures in the data file.

tell()[source]

Return location from start of file.

write_descriptor_block(block)[source]

Write a DescriptorBlock to the file.

write_data_block(desc_block, data)[source]

Write a block of data to the file.

close()[source]

Close the file, if not already closed.

property closed

Returns True if the writer has been closed.

write_header(annotations)[source]

Write the header of the data file, including annotations.

write_file_end(index_offset)[source]

Write the end of the data file.

class bosdyn.client.bddf.DataWriter(outfile, annotations=None)[source]

Bases: object

Class for writing data to a file.

property file_index

Get the FileIndex proto used which describes how to access data in the file.

add_message_series(series_type, series_spec, content_type, type_name, is_metadata=False, annotations=None, additional_index_names=None)[source]

Add a new series for storing message data. Message data is variable-sized binary data.

Parameters
  • series_type – the kind of spec, corresponding to the set of keys expected in series_spec.

  • series_spec – dict of {key (string) -> value (string)| describing the series.

  • content_type – data encoding, like http content-type header (string)

  • type_name – string describing the kind of data

  • is_metadata – Metadata messages are needed to interpret other messages which may be stored in the file. If the file is split into parts, metadata messages must be duplicated into each part. (default=False)

  • annotations – optional dict of key (string) -> value (string) pairs to associate with the message channel

  • additional_index_names – names of additional timestamps to store with each message (list of string).

Returns series id (int).

add_pod_series(series_type, series_spec, type_enum, dimension=None, annotations=None)[source]

Add a new series for storing data POD data (float, double, int, etc….).

Parameters
  • series_type – the kind of spec, corresponding to the set of keys expected in series_spec.

  • series_spec – dict of {key (string) -> value (string)} describing the series.

  • type_enum – kind of values stored in the file (PodTypeEnum).

  • dimensions – None or empty-array means elements are single values, [3] means vectors of size 3, [4, 4] is a 4x4 matrix, etc….

  • annotations – optional dict of key (string) -> value (string) pairs to associate with the message channel

Returns series id (int).

add_series(series_type, series_spec, message_type=None, pod_type=None, annotations=None, additional_index_names=None)[source]

Register a new series for messages.

Parameters
  • series_type – the kind of spec, corresponding to the set of keys expected in series_spec.

  • series_spec – dict of {key (string) -> value (string)| describing the series.

  • message_type – MessageTypeDescriptor (need EITHER this OR pod_type)

  • pod_type – PodTypeDescriptor (need EITHER this OR pod_type)

  • annotations – optional dict of key (string) -> value (string) pairs to associate with the message channel

  • additional_index_names – names of additional timestamps to store with each message (list of string).

Returns series id (int).

Raises SeriesNotUniqueError if a series matching series_spec is already added.

write_data(series_index, timestamp_nsec, data, additional_indexes=None)[source]

Store binary data into the file, under a previously-defined channel.

Parameters
  • series_index – integer returned when series was registered with the file.

  • timestamp_nsec – nsec since unix epoch to timestamp the data

  • data – binary data to store

  • additional_indexss – additional timestamps if needed for this channel

Raises DataFormatError if the data or additional_indexes are not valid for this series.

run_on_close(thunk)[source]

Register a function to be called when file is closed, before index is written.

class bosdyn.client.bddf.ProtoSeriesWriter(data_writer, protobuf_type, channel_name=None, is_metadata=False, annotations=None, additional_index_names=None)[source]

Bases: object

A class for registering a series which stores protobuf messages in a message series.

The series is named by a ‘channel_name’ which defaults to the full type name of the

protobuf type.

write(timestamp_nsec, protobuf, additional_indexs=None)[source]

Store protobuf in the file.

Args

timestamp_nsec: nsec since unix epoch to timestamp the data protobuf: a protobuf message, not serialized. additional_indexes: additional timestamps if needed for this channel

Raises DataFormatError if the data or additional_indexes are not valid for this series.

property series_type

Return the series type string.

property series_spec

Return the series_spec for the series.

class bosdyn.client.bddf.PodSeriesWriter(data_writer, series_type, series_spec, pod_type, dimensions=None, annotations=None, data_block_size=2048)[source]

Bases: object

A class to assist with writing POD data values into a series, within a DataWriter.

write(timestamp_nsec, sample)[source]

Add sample to data block, and write block if block is full.

A block is full if there is no room for an additional sample within the data_block_size.

Parameters
  • timestamp_nsec – nsec since unix epoch to timestamp the data

  • sample – array/vector of POD values to write

Raises DataFormatError if the data is invalid for this series.

finish_block()[source]

If there are samples which haven’t been written to the file, write them now.

property series_type

Return the series_type (string) with which the series was registered.

property series_spec

Return the series_spec ({key -> value}) with which the series was registered.

class bosdyn.client.bddf.BaseDataReader(outfile)[source]

Bases: object

Shared parent class for DataReader and StreamedDataReader.

property file_descriptor

Return the file descriptor from the start of the file/stream.

property version

Return file version as a bosdyn.api.FileFormatVersion proto.

property annotations

Return {key -> value} dict for file annotations.

property file_index

Get the FileIndex proto used which describes how to access data in the file.

property checksum

160-bit checksum read from the end of the file, or None if not yet read.

property read_checksum

Override to compute checksum on reading, in stream-readers.

series_spec_to_index(series_spec)[source]

Given a series spec (map {key -> value}), return the series index for that series.

Raises ValueError if no such series exists.

class bosdyn.client.bddf.DataReader(outfile)[source]

Bases: bosdyn.client.bddf.BaseDataReader

Class for reading data from a file-like object which is seekable.

Methods raise ParseError if there is a problem with the format of the file.

series_descriptor(series_index)[source]

Return SeriesDescriptor for given series index, loading it if necessary.

num_data_blocks(series_index)[source]

Returns the number of data blocks for a given series in the file.

total_bytes(series_index)[source]

Returns the total number of bytes for data in a given series in the file.

read(series_index, index_in_series)[source]

Retreives a message and related information from the file.

Parameters
  • series_index – int selecting from which series to read the message.

  • index – The index number of the message within the channel.

Returns: DataTypeDescriptor for channel, timestamp_nsec (int), message-data (bytes)

Raises ParseError if there is a problem with the format of the file.

series_block_index(series_index)[source]

Returns the SeriesBlockIndexes for the given series_index, loading it as needed.

class bosdyn.client.bddf.ProtobufReader(data_reader)[source]

Bases: object

A class for reading Protobuf data from a DataFile.

Methods raise ParseError if there is a problem with the format of the file.

property data_reader

Return underlying DataReader this object is using.

series_index(channel_name)[source]

Return the series index (int) by which SeriesDescriptors and messages can be accessed.

Parameters

channel_name – name of the channel of messsages.

series_index_to_descriptor(series_index)[source]

Given a series index, return the associated SeriesDescriptor

Parameters

series_index – index (int) from the series_index() call

get_message(series_index, protobuf_type, index_in_series)[source]

Return a deserialized protobuf from bytes stored in the file.

Parameters
  • series_index – index (int) from the series_index() call

  • protobuf_type – class of the protobuf we want to deserialize

  • index_in_series – the index of the message within the series

Returns: DataTypeDescriptor for channel, timestamp_nsec (int), deserialized protobuf object

class bosdyn.client.bddf.ProtobufChannelReader(protobuf_reader, protobuf_type, channel_name=None)[source]

Bases: object

A class for reading a single channel of Protobuf data from a DataFile.

property series_descriptor

SeriesDescriptor for this series

property num_messages

Number of messages in this series.

get_message(index_in_series)[source]

Get the specified message in the series, as a deserialized protobuf.

Parameters

index_in_series – the index of the message within the series

Returns: timestamp_nsec (int), deserialized protobuf object

class bosdyn.client.bddf.PodSeriesReader(data_reader, series_spec)[source]

Bases: object

A class for reading a series of POD data from a DataFile.

Methods raise ParseError if there is a problem with the format of the file.

property pod_type

Return the PodTypeDescriptor for the series.

property series_descriptor

Return the SeriesDescriptor for the series.

property num_data_blocks

Number of data blocks in this series.

read_samples(index_in_series)[source]

Return the POD data values from the data block of the given index.

Returns: timestamp_nsec (int), POD data values (array of (array … (of POD values)))

class bosdyn.client.bddf.StreamDataReader(outfile)[source]

Bases: bosdyn.client.bddf.BaseDataReader

Data reader which reads the file format from a stream, without seeking.

property read_checksum

64-bit checksum read from the end of the file, or None if not yet read.

property stream_file_index

Return the file index as parsed from the stream.

series_descriptor(series_index)[source]

Return SeriesDescriptor for given series index.

Returns KeyError if no such series exists.

read_data_block()[source]

Read and return next data block.

Returns: DataDescriptor, SeriesDescriptor, data (bytes)

Raises ParseError if there is a problem with the format of the file,

EOFError if the end of the file is reached.

read_next_block()[source]

Read and return next block.

Returns: True, DataDescriptor, data (bytes) for data block Returns: False, DescriptorBlock, None for descriptor block

Raises ParseError if there is a problem with the format of the file,

EOFError if the end of the file is reached.

property series_block_indexes

series_index -> SeriesBlockIndex.

Type

Returns the current list of SeriesBlockIndexes

series_block_index(series_index)[source]

Returns the SeriesBlockIndexes for the given series_index.

property eof

Returns True if all blocks in the file have been read.