API reference: pb_encode.h

Nanopb functionality related to encoding protobuf messages is provided in the pb_encode.h header. It contains the definitions for the encoding context, writing to output streams and encoding the data for message fields.

Message encoding

pb_encode

Encodes the contents of a structure as a protocol buffers message and writes it to output stream.

#define pb_encode(ctx, msgdesc, src_struct) \
    pb_encode_s(ctx, msgdesc, src_struct, sizeof(*src_struct))

This is the main function for the encoding process. It will walk through all fields in the message structure and encodes them to the output stream.

To provide some type safety, the function is implemented as a macro wrapper around pb_encode_s(). The macro uses sizeof() to ensure that the size of the structure passed matches the size in the descriptor. This catches most bugs where a descriptor for wrong message type is accidentally given.

The encoder behavior can be controlled by modifying ctx->flags before calling it.

pb_encode_s

The actual implementation of the encoding function. Prefer pb_encode() macro when possible. Calling pb_encode_s() directly may be necessary if the structure type is not available at the point of code where pb_encode() is called.

bool pb_encode_s(pb_encode_ctx_t *ctx, const pb_msgdesc_t *msgdesc,
                const void *src_struct, size_t struct_size);

pb_get_encoded_size

Calculate the size of a message, but do not store the encoded data.

#define pb_get_encoded_size(size, msgdesc, src_struct) ...
bool pb_get_encoded_size_s(size_t *size, const pb_msgdesc_t *msgdesc,
                       const void *src_struct, size_t struct_size);

Equivalent to using pb_init_encode_ctx_sizing() and pb_encode().

Output stream management

Encoded data is written to an output stream, which can either be a memory buffer or a custom callback function. It is possible to encode messages larger than the available RAM if you provide a stream callback function that directly writes to some external communication interface.

pb_encode_ctx_t

The encode context contains the output stream definition, and other data associated with the encoding process. Use one of the pb_init_encode_ctx_* functions to initialize it.

After initialization, you can manually set e.g. field_callback or flags.

typedef struct pb_encode_ctx_s pb_encode_ctx_t;
struct pb_encode_ctx_s
{
    pb_encode_ctx_flags_t flags;

    pb_size_t max_size;
    pb_size_t bytes_written;
    pb_byte_t *buffer;

    pb_walk_state_t *walk_state;

    pb_encode_ctx_write_callback_t stream_callback;
    void *stream_callback_state;
    pb_size_t buffer_size;
    pb_size_t buffer_count;

    const char *errmsg;

    pb_encode_ctx_field_callback_t field_callback;
    void *field_callback_state;
};

pb_encode_ctx_flags_t

Encoding context flags control the behavior of stream output functions. Some of them are automatically set internally, but they can also be set by user code.

Stream max_size and bytes_written

The max_size and bytes_written track the total number of bytes written to the output stream.

When encoding happens into a memory buffer, the max_size equals the size of the buffer and bytes_written is the count of bytes written so far. The buffer always points to the beginning of the output buffer.

If a stream callback is provided, the buffer acts as temporary storage. Separate buffer_size and buffer_count track the number of bytes currently in the buffer. Once the buffer fills up, pb_flush_write_buffer() calls stream_callback to write out the data, and buffer_count resets to 0.

The temporary storage speeds up encoding of any submessages small enough to fit in it. Larger submessages need to be encoded twice: first to determine the submessage length and then to write out the data.

Walk state

The walk_state field is internally set by the first pb_encode() call. It is used to share the pb_walk() stack when user callbacks recursively call pb_encode_submessage().

Stream callback

The stream callback is a user-provided function, which takes encoded data from nanopb and writes it to some external communication interface such as network, serial port or filesystem.

bool my_stream_write_callback(pb_encode_ctx_t *ctx, const pb_byte_t *buf, pb_size_t count);

The callback should write exactly count bytes from buf to the external interface. Value of count is at most ctx->max_size - ctx->bytes_written.

The callback should return true on success and false on IO errors. Error return will cause encoding to abort. The callback can optionally set ctx->errmsg using PB_RETURN_ERROR(), otherwise pb_write() will set it to "io error".

The callback can use stream_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The context variables such as bytes_written are updated by pb_write() after the callback returns.

There is an example of a custom stream callback in examples/network_server/common.c.

Error message

The primary status indication is the bool return value for success or failure. There is an optional error message in the ctx->errmsg, but this is for debugging purposes only and can be disabled with the PB_NO_ERRMSG build option.

Field callback

Nanopb uses field callbacks to handle message fields with unknown or large size. The callback function is provided by user code and can directly write to the output stream.

The field_callback function specified in the encoding context receives a call for each field that has FT_CALLBACK type. It can write data using the manual encoding functions, or just do nothing. The callback should return false on errors, which will terminate the encoding process.

The callback can use ctx->field_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The data in the C structure can be accessed by pointers in pb_field_iter_t. The callback can determine which field is being encoded by comparing field->descriptor and field->tag. For example:

bool my_field_callback(pb_encode_ctx_t *ctx, const pb_field_iter_t *field)
{
    if (field->descriptor == &SubMessage_msg && field->tag == SubMessage_stringvalue_tag)
    {
        return pb_encode_tag_for_field(ctx, field) &&
               pb_encode_varint(ctx, 42);
    }

    return true;
}

....

pb_encode_ctx_t ctx;
pb_init_encode_ctx_for_buffer(&ctx, ...);
ctx.field_callback = &my_field_callback;

pb_init_encode_ctx_for_buffer

Initializes an encoding context for writing into a memory buffer. The output buffer has to be large enough to contain the whole encoded message.

void pb_init_encode_ctx_for_buffer(pb_encode_ctx_t *ctx, pb_byte_t *buf, pb_size_t bufsize);

After writing, you can check ctx.bytes_written to find out how much valid data there is in the buffer. This should be passed as the message length on decoding side.

pb_init_encode_ctx_for_callback

Initializes an encoding context for passing data to a custom callback function.

void pb_init_encode_ctx_for_callback(pb_encode_ctx_t *ctx,
    pb_encode_ctx_write_callback_t stream_callback, void *stream_callback_state,
    pb_size_t max_size, pb_byte_t *buf, pb_size_t bufsize);

Providing a temporary buffer is optional. If the buffer is provided, it is used to temporarily store encoded submessages, so that their length prefix can be written after the encoded length is known. Without a temporary buffer, submessages will are internally processed twice to determine the length and then write the data.

pb_init_encode_ctx_sizing

Initializes an encoding context for just counting the number of bytes that would have been written. The encoded data is discarded.

void pb_init_encode_ctx_sizing(pb_encode_ctx_t *ctx);

After encoding, the number of encoded bytes can be read from ctx.bytes_written.

pb_flush_write_buffer

For encoding context using a stream callback and a temporary buffer, the data might not be immediately passed to the callback. The flushing function empties the temporary buffer. This function has no effect for contexts that write to a memory buffer directly.

bool pb_flush_write_buffer(pb_encode_ctx_t *ctx);

The flushing is automatically done at the end of a pb_encode() call. Calling this function manually is only necessary if you call pb_write() manually.

pb_write

Writes data to the output stream associated with the context.

bool pb_write(pb_encode_ctx_t *ctx, const pb_byte_t *buf, pb_size_t count);

Before writing the data, this function checks that the stream max_size wouldn’t be exceeded. If an error happens, false is returned, errmsg is set and bytes_written is not incremented.

For memory streams, this copies the data to the output buffer and increments the bytes_written count.

For sizing streams, only bytes_written is incremented.

For callback streams, the data may be either copied to a temporary buffer or passed directly to the stream callback function. If a temporary buffer was provided, calling pb_flush_write_buffer() is needed to make sure the data will be passed to the callback function after a manual pb_write() call. The pb_encode() automatically flushes the buffer at the end of the message.

Manual field encoding

The functions with names pb_encode_<datatype> are used for encoding single fields manually. Typically this is used when writing custom field callbacks, but it can also be used for fully manual control over the encoding process.

For example an array of unlimited size could be encoded like this:

while(1)
{
    const char *str = generate_next_string();
    if (!str) break;

    if (!pb_encode_tag_for_field(ctx, field))
        return false;

    if (!pb_encode_string(ctx, str, strlen(str)))
        return false;
}

This will work equivalently both when used in field callbacks, or when used without actually calling pb_encode() at all.

The tag of a field must be encoded first with pb_encode_tag_for_field() or pb_encode_tag(). After that, you can call exactly one of the content-writing functions to encode the payload of the field. For repeated fields, you can repeat the tag + payload combination multiple times.

Writing packed arrays is a little bit more involved: you need to use pb_encode_tag and specify PB_WT_STRING as the wire type. Then you need to know exactly how much data you are going to write, and use pb_encode_varint to write out the number of bytes before writing the actual data.

Important

See Google Protobuf Encoding Format Documentation for background information on the Protobuf wire format.

pb_encode_tag

Starts a field in the Protocol Buffers binary format: encodes the field number and the wire type of the data.

bool pb_encode_tag(pb_encode_ctx_t *ctx, pb_wire_type_t wiretype, pb_tag_t field_number);

The tag number is defined in the .proto file. In nanopb, it is available from pb_field_iter_t, or from the MyMessage_myfield_tag defines in the generated .pb.h file.

pb_encode_tag_for_field

Same as pb_encode_tag(), except takes the parameters from a pb_field_iter_t structure.

bool pb_encode_tag_for_field(pb_encode_ctx_t *ctx, const pb_field_iter_t *field);

This function only considers the PB_LTYPE of the field pb_type_t.

Wire type mapping is as follows:

pb_encode_varint

Encodes a signed or unsigned integer in the varint format. Works for fields of type bool, enum, int32, int64, uint32 and uint64:

bool pb_encode_varint(pb_encode_ctx_t *ctx, uint64_t value);

Note

Value will be converted to uint64_t in the argument. To encode signed values, the argument should be cast to int64_t first for correct sign extension.

pb_encode_svarint

Encodes a signed integer in the zig-zagged format. Works for fields of type sint32 and sint64:

bool pb_encode_svarint(pb_encode_ctx_t *ctx, int64_t value);

Parameters are the same as for pb_encode_varint()

pb_encode_string

Writes the length of a string as varint and then contents of the string. Works for fields of type bytes and string:

bool pb_encode_string(pb_encode_ctx_t *ctx, const pb_byte_t *buffer, size_t size);

This function does not perform UTF-8 validation, which is separately available using pb_validate_utf8().

pb_encode_fixed32

Writes 4 bytes to stream and swaps bytes on big-endian architectures. Works for fields of type fixed32, sfixed32 and float:

bool pb_encode_fixed32(pb_encode_ctx_t *ctx, const void *value);

pb_encode_fixed64

Writes 8 bytes to stream and swaps bytes on big-endian architecture. Works for fields of type fixed64, sfixed64 and double:

bool pb_encode_fixed64(pb_encode_ctx_t *ctx, const void *value);

pb_encode_float_as_double

Encodes a 32-bit float value so that it appears like a 64-bit double in the encoded message. This is sometimes needed when platforms like AVR that do not support 64-bit double need to communicate using a message type that contains double fields.

bool pb_encode_float_as_double(pb_encode_ctx_t *ctx, float value);

This function is only included if PB_CONVERT_DOUBLE_FLOAT build option is set.

pb_encode_submessage

Encodes a submessage field, including the size header for it. Works for fields of any message type.

bool pb_encode_submessage(pb_encode_ctx_t *ctx, const pb_msgdesc_t *msgdesc, const void *src_struct);

This function is similar to calling pb_encode() with PB_ENCODE_CTX_FLAG_DELIMITED, but it differs in that it attempts to reuse the same pb_walk() context. This reduces the total stack space necessary for deep message hierarchies.

In Protocol Buffers format, the submessage size must be written before the submessage contents. For efficiency, this function attempts to write the encoded data to memory and then prepend the size afterwards. This requires only one pass over the message data.

When stream callbacks are used, it may be that the submessage does not fit in the provided temporary buffer. In this case the function resorts to a two-pass approach, where the first pass only calculates the size. If the submessage contains callback fields, the callback function might misbehave and write out a different amount of data on the second pass. This situation is recognized and false is returned, but garbage will be written to the output before the problem is detected.