API reference: pb_decode.h

Nanopb functionality related to decoding protobuf messages is provided in the pb_decode.h header. It contains the definitions for the decoding context, reading from input streams and decoding the message data into C structures.

The decoding module also handles dynamic memory allocation, if it is enabled by build options.

Message decoding

pb_decode

Read and decode all fields of a structure. Reads until EOF on input stream.

#define pb_decode(ctx, msgdesc, dest_struct) \
    pb_decode_s(ctx, msgdesc, dest_struct, sizeof(*dest_struct))

This is the main function for the decoding process. It will read one protobuf field tag from the input at a time, find the corresponding field in the descriptor and decode the data into the C structure.

To provide some type safety, the function is implemented as a macro wrapper around pb_decode_s(). The macro uses sizeof() to ensure that the size of the structure passed matches the size in the descriptor. This catches most bugs where a descriptor for wrong message type is accidentally given.

Message length

Usually the message length is provided when calling pb_init_decode_ctx_for_buffer(). For callback streams, the length can alternatively be indicated by the callback reporting EOF condition when it detects end of data from an external source.

In Protocol Buffers binary format, end-of-file is only allowed between fields. If it happens anywhere else, pb_decode will return false. On failure return the structure will have an internally consistent state, but the field contents may be partial.

Control flags and field initialization

The decoder behavior can be controlled by modifying ctx->flags before calling it.

By default pb_decode() initializes all fields, either to the default value declared in .proto or to a zero value. Setting the context flag PB_DECODE_CTX_FLAG_NOINIT skips the initialization. This can be used for merging protobuf messages or if you have already initialized the structure.

Dynamic allocation

If the generator options have been used to set fields to FT_POINTER type, this function will allocate storage for them. The allocation will use either the ctx->allocator or the default allocator (by default realloc()).

For messages containing pointers (those that have PB_MSGFLAG_R_HAS_PTRS), you must call pb_release() to release the allocation after you are done with the structure. On error, pb_decode() will call pb_release() before returning false.

pb_decode_s

The actual implementation of the decoding function. Prefer pb_decode() macro when possible. Calling pb_decode_s() directly may be necessary if the structure type is not available at the point of code where the call happens.

bool pb_decode_s(pb_decode_ctx_t *ctx, const pb_msgdesc_t *msgdesc,
                 void *dest_struct, size_t struct_size);

Dynamic memory allocation

pb_release

Releases all dynamically allocated fields in a message.

#define pb_release(ctx, msgdesc, dest_struct) ...
bool pb_release_s(pb_decode_ctx_t *ctx, const pb_msgdesc_t *msgdesc,
                  void *dest_struct, size_t struct_size);

Similarly to pb_decode(), pb_release() has a wrapping macro to check the structure size. The actual implementation is in pb_release_s().

The release function goes through all FT_POINTER type fields in the message, releases the storage associated with them and sets the pointer to NULL. Either the allocator given by ctx or the default allocator is used.

If the build option PB_NO_MALLOC is defined, this function does nothing. Similarly if the message descriptor does not have PB_MSGFLAG_R_HAS_PTRS or if the dest_struct pointer is NULL, the function returns true without doing anything.

This function is safe to call multiple times, calling it again does nothing.

The release function normally always succeeds. It can return failure if the message structure is deeper than maximum nesting depth or if struct_size does not match msgdesc->struct_size.

pb_allocate_field

Allocate or reallocate storage for a single field.

bool pb_allocate_field(pb_decode_ctx_t *ctx, void **ptr, pb_size_t data_size, pb_size_t array_size);

This function either newly allocates memory or changes the size of an existing allocation. The total allocation size is data_size * array_size, but this function internally checks for overflow in the multiplication.

This function uses either the default allocator, or the allocator defined by ctx->allocator.

The initial value of *ptr should be NULL if there is no previous allocation. Otherwise the previous allocation is resized using realloc(), which may also change the value of *ptr.

On allocation failure, false is returned and *ptr remains unchanged.

Example usage:

typedef struct MyMessage {
    pb_size_t myarray_count;
    int32_t *myarray;
} MyMessage;

MyMessage msg = MyMessage_init_zero;
msg.myarray_count = 10;

if (!pb_allocate_field(ctx, &msg.myarray, sizeof(int32_t), msg.myarray_count))
    return false;

pb_release_field

Release storage previously allocated by pb_allocate_field().

void pb_release_field(pb_decode_ctx_t *ctx, void *ptr);

This function uses either the default allocator, or the allocator defined by ctx->allocator.

Input stream management

Protobuf data to be decoded is read from an input stream, which can be either a memory buffer or a custom callback function. It is possible to decode messages larger than the available RAM if you provide a stream callback function that directly reads from some external communication interface.

pb_decode_ctx_t

The decode context contains the input stream definition and other data associated with the decoding process. Use one of pb_init_decode_ctx_* functions to initialize it.

struct pb_decode_ctx_s
{
    pb_decode_ctx_flags_t flags;

    pb_size_t bytes_left;
    const pb_byte_t *rdpos;

    pb_walk_state_t *walk_state;

    pb_decode_ctx_read_callback_t stream_callback;
    void *stream_callback_state;
    pb_byte_t *buffer;
    pb_size_t buffer_size;

    const char *errmsg;

    pb_allocator_t *allocator;

    pb_decode_ctx_field_callback_t field_callback;
    void *field_callback_state;
};

pb_decode_ctx_flags_t

Decoding context flags control the behavior of stream output and decoding functions. They can be set by user before calling pb_decode().

Stream bytes_left and read position

The bytes_left is an upper bound on the remaining bytes in the stream. For the top-level stream with memory buffers it is the total length of the input message. For submessages, nanopb will temporarily set bytes_left to the length of the submessage data. For callback streams, the callback function can indicate end of message before bytes_left has reached 0.

For memory buffers, the rdpos pointer is the current read position in the buffer. For callback streams it can be either NULL or point to a temporary input buffer.

Walk state

The walk_state field is internally set by the first pb_decode() call. It is used to share the pb_walk() stack when user callbacks recursively call pb_decode().

Stream callback

The stream callback is a user-provided function, which reads data from some external communication interface such as network, serial port or filesystem. Nanopb will then decode this data.

pb_size_t my_stream_read_callback(pb_decode_ctx_t *ctx, pb_byte_t *buf, pb_size_t count);

The callback should read up to count bytes into buf from the external source, and return the number of bytes read. Value of count is always smaller than PB_SIZE_MAX and at most ctx->bytes_left. Any return value less than count is interpreted as end-of-stream.

On errors, return PB_READ_ERROR. This will cause decoding to abort. You can optionally set ctx->errmsg using PB_SET_ERROR(). Otherwise pb_read() will set it to "io error".

The callback can use stream_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The context variables such as bytes_left are updated by pb_read() after the callback returns.

There is an example of a custom stream callback in examples/network_server/common.c.

Error message

The primary status indication is the bool return value for success or failure. There is an optional error message in the ctx->errmsg, but this is for debugging purposes only and can be disabled with the PB_NO_ERRMSG build option.

Field callback

Nanopb uses field callbacks to handle message fields with unknown or large size. The callback function is provided by user code and decodes data directly from the input stream.

The field_callback function specified in the decoding context receives a call for each field that has FT_CALLBACK type. It can read data using the manual decoding functions. The callback should return false on errors, which will terminate the decoding process.

The field tag has already been decoded before the callback is called, and the ctx->bytes_left is set to the payload length of the field.

The context field callback is also called for any unknown fields which are not present in the descriptor generated from .proto file. For unknown fields, the field tag is available in field->tag as normal, and field->type is set to a PB_LTYPE value corresponding to the wire type, but other values such field->pData are just zeroed. The callback can simply return true for any fields it doesn’t know, and nanopb will discard the field data.

The callback can use ctx->field_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The data in the C structure can be written using pointers in pb_field_iter_t. The callback can determine which field is being decoded by comparing field->descriptor and field->tag. For example:

bool my_field_callback(pb_decode_ctx_t *ctx, const pb_field_iter_t *field)
{
    if (field->descriptor == &SubMessage_msg && field->tag == SubMessage_stringvalue_tag)
    {
        // Decode our callback field
        uint64_t value;
        if (!pb_decode_varint(ctx, &value))
            return false;

        printf("Value: %d\n", (int)value);
        return true;
    }

    // Unknown field, just skip it
    return true;
}

....

pb_decode_ctx_t ctx;
pb_init_decode_ctx_for_buffer(&ctx, ...);
ctx.field_callback = &my_field_callback;

pb_init_decode_ctx_for_buffer

Initializes an encoding context for reading from a memory buffer. The length of the message data must be known beforehand.

void pb_init_decode_ctx_for_buffer(pb_decode_ctx_t *ctx, const pb_byte_t *buf, pb_size_t msglen);

Note

msglen is not the length of the buffer, it must be the length of actual message data. Protobuf messages contain no indication of where the message ends, so the length must be known by other means.

pb_init_decode_ctx_for_callback

Initializes an encoding context for passing data to a custom callback function.

void pb_init_decode_ctx_for_callback(pb_decode_ctx_t *ctx,
    pb_decode_ctx_read_callback_t stream_callback, void *stream_callback_state,
    pb_size_t max_msglen, pb_byte_t *buf, pb_size_t bufsize);

Providing a temporary buffer is optional. If provided, it is used to coalesce read requests so that a larger amount of data can be read in one IO operation. It is also used as a read-ahead buffer for optimizing memory allocation sizes when dynamic allocation is used.

The max_msglen is an upper bound for the message size. The callback can indicate end-of-stream earlier by its return value. It is recommended to set a reasonable maximum length to avoid denial-of-service through excessively long input messages from e.g. a network socket.

Maximum possible length for callback stream is PB_SIZE_MAX - 1, because the highest value is reserved for PB_READ_ERROR. This initialization function automatically limits the max_msglen value.

pb_read

Read raw data from input stream.

bool pb_read(pb_decode_ctx_t *ctx, pb_byte_t *buf, pb_size_t count);

End of stream is signalled by stream->bytes_left being zero after pb_read() returns false.

For callback streams, any data in the temporary buffer is returned first, and the stream callback is called for any remaining requested count. The pb_read() call never puts data into the temporary buffer. The buffer is only filled internally at points where the remaining length of a protobuf element is known.

Manual field decoding

The functions with names pb_decode_<datatype> are used for decoding single fields manually. Typically this is used when writing custom field callbacks, but it can also be used for fully manual control over the decoding process.

For callback fields, the field tag is decoded by nanopb before the callback function is called. For fully manual decoding, pb_decode_tag() can be used to read the field identifier, after which the field data can be decoded.

To decode numeric (including enumerated and boolean) values, use pb_decode_varint, pb_decode_svarint, pb_decode_fixed32 and pb_decode_fixed64. They take a pointer to a 32- or 64-bit C variable, which you may then cast to smaller datatype for storage.

For decoding strings and bytes fields, the length has already been decoded and the callback function is given a length-limited substream. You can therefore check the total length in stream->bytes_left and read the data using pb_read.

Finally, for decoding submessages in a callback, use pb_decode() and pass it the submessage message descriptor.

Important

See Google Protobuf Encoding Format Documentation for background information on the Protobuf wire format.

pb_decode_tag

Decode the tag that comes before each field in the protobuf format:

bool pb_decode_tag(pb_decode_ctx_t *stream, pb_wire_type_t *wire_type, pb_tag_t *tag, bool *eof);

When the message (stream) ends, this function will return false and set eof to true. On other errors, eof will be set to false.

pb_skip_field

Discard the data for a field from the stream, without actually decoding it:

bool pb_skip_field(pb_decode_ctx_t *stream, pb_wire_type_t wire_type);

This function determines the amount of bytes to read based on the wire type. For PB_WT_STRING, it will read the length prefix of a string or submessage to determine its length.

pb_decode_varint

Read and decode a varint encoded integer.

bool pb_decode_varint(pb_decode_ctx_t *stream, uint64_t *dest);

pb_decode_varint32

Same as pb_decode_varint, but limits the value to 32 bits:

bool pb_decode_varint32(pb_decode_ctx_t *stream, uint32_t *dest);

Parameters are the same as pb_decode_varint. This function can be used for decoding lengths and other commonly occurring elements that you know shouldn’t be larger than 32 bit. It will return an error if the value exceeds the uint32_t datatype.

pb_decode_bool

Same as pb_decode_varint, but converts value to boolean.

bool pb_decode_bool(pb_decode_ctx_t *ctx, bool *dest);

Any non-zero value is regarded as true.

pb_decode_svarint

Similar to pb_decode_varint, except that it performs zigzag-decoding on the value. This corresponds to the Protocol Buffers sint32 and sint64 datatypes. :

bool pb_decode_svarint(pb_decode_ctx_t *stream, int64_t *dest);

(parameters are the same as pb_decode_varint)

pb_decode_fixed32

Decode a fixed32, sfixed32 or float value.

bool pb_decode_fixed32(pb_decode_ctx_t *stream, void *dest);

This function reads 4 bytes from the input stream. On big endian architectures, it then reverses the order of the bytes. Finally, it writes the bytes to dest.

pb_decode_fixed64

Decode a fixed64, sfixed64 or double value. :

bool pb_decode_fixed64(pb_decode_ctx_t *stream, void *dest);

Same as pb_decode_fixed32, except this reads 8 bytes.

pb_decode_double_as_float

Decodes a 64-bit double value into a 32-bit float variable. This is sometimes needed when platforms like AVR that do not support 64-bit double need to communicate using a message type that contains double fields. Counterpart of pb_encode_float_as_double().

bool pb_decode_double_as_float(pb_decode_ctx_t *stream, float *dest);

This function is only included if PB_CONVERT_DOUBLE_FLOAT build option is set.

pb_decode_open_substream

Decode the length for a field with wire type PB_WT_STRING and open a length-limited substream for reading the data.

bool pb_decode_open_substream(pb_decode_ctx_t *ctx, pb_size_t *old_length);

This function uses pb_decode_varint to read an integer from the stream, which is the length of the STRING type protobuf element. The value is first compared against the remaining bytes_left, and if not enough data is remaining, error is returned. Otherwise ctx->bytes_left is set to the data length.

The remaining length in original stream after the read is stored into old_length for restoring.

pb_decode_close_substream

Close the substream opened with pb_decode_open_substream()

bool pb_decode_close_substream(pb_decode_ctx_t *ctx, pb_size_t old_length)

This function discards any remaining data in the substream and then restores bytes_left to old_length.