pb_common.h

The pb_common module contains shared support functions that are used by both pb_decode.c and pb_encode.c.

User code rarely calls these functions directly, but they can be used for a limited kind of reflection over the field information stored in the message descriptors.

Message descriptor handling

Nanopb stores information about each field in the protobuf messages in a compact representation. The data is written by the generator and the PB_BIND() macro and is stored in a pb_msgdesc_t structure. The functions shown below access this data and provide data for each field as a pb_field_iter_t.

pb_field_iter_begin

Begins iterating over the fields in a message type:

bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_msgdesc_t *msgdesc, void *message);

After successful return, the iterator contains the data for the first field in the message. See pb_field_iter_t in pb.h for definition of the iterator.

pb_field_iter_next

Advance to the next field in the message:

bool pb_field_iter_next(pb_field_iter_t *iter);

When the last field in the message has been processed, this function will return false and initialize iter back to the first field in the message.

pb_field_iter_find

Find a field specified by tag number in the message:

bool pb_field_iter_find(pb_field_iter_t *iter, pb_tag_t tag, pb_extension_t **ext);

This function is functionally identical to calling pb_field_iter_next() until iter.tag equals the searched value. Internally this function avoids fully processing the descriptor for intermediate fields.

If ext parameter is given and no match was found in normal fields, the function searches for a matching extension field. If such is found, iter will point to the field with PB_LTYPE_EXTENSION and *ext will point to the pb_extension_t matching the tag.

pb_field_iter_load_extension

Load field iterator values from an extension:

bool pb_field_iter_load_extension(pb_field_iter_t *iter, pb_extension_t *extension);

This function temporarily sets the values in iter to match the first (and only) field in the extension. The field can then be processed as normal, and next call to pb_field_iter_next() will restart from the beginning of the original message.

pb_get_extension_data_ptr

Returns a pointer to the data for an extension.

void *pb_get_extension_data_ptr(pb_extension_t *extension);

For fields with PB_ATYPE_POINTER, this returns &extension->dest, as the payload pointer is directly stored in the extension structure. This avoids unnecessary extra indirection.

For other field types, this returns extension->dest, which points to the data payload.

pb_ltype_to_wire_type

Returns the wire type in protobuf encoding which corresponds to the field pb_type_t:

pb_wire_type_t pb_ltype_to_wire_type(pb_type_t type);

Returns PB_WT_VARINT, PB_WT_32BIT, PB_WT_64BIT, PB_WT_STRING or PB_WT_INVALID.

Note that for fields with ltype less than PB_LTYPE_LAST_PACKABLE, PB_WT_STRING is also a valid wire type.

Message hierarchy processing

Protobuf messages and their submessages form a tree-like structure. Both encoding and decoding processes need to walk this tree.

Using C recursion for processing the hierarchy increases stack usage, as each level will save return addresses and other runtime information on the stack.

The pb_walk() implements the visitor design pattern, where a callback function is called for each field. The return value then determines whether the processing descends into the submessage or proceeds to next field.

For each message level, minimum amount of information is stored, such as the descriptor and field index of the parent message. A statically sized buffer for up to PB_MESSAGE_NESTING levels is allocated on the stack. Unless recursion is disabled with PB_NO_RECURSION, pb_walk() will automatically call itself to obtain more stack space if the hierarchy is deeper. See stack usage options for details.

A simplified sequence could be like this:

1. Initialization:
   • pb_encode() calls pb_walk_init() and provides the message descriptor
   • pb_encode() calls pb_walk()
2. Top level message processing:
   • pb_walk() calls pb_encode_walk_cb() with state->iter pointing to the first field in the top-level message
   • pb_encode_walk_cb() determines that the field is a submessage, encodes the tag and returns PB_WALK_IN
3. Submessage processing:
   • pb_walk() stores stack frame and calls pb_encode_walk_cb() with the first field in the submessage
   • pb_encode_walk_cb() encodes the submessage field tag and data and returns PB_WALK_OUT
   • pb_walk() restores stack frame and calls pb_encode_walk_cb() with the first field in the top-level message
4. Return to top level message processing:
   • pb_encode_walk_cb() detects state->retval == PB_WALK_OUT and returns PB_WALK_NEXT_ITEM
   • pb_walk() advances the iterator and notices this was the last field. It calls pb_encode_walk_cb() with tag 0.
5. End of message:
   • pb_encode_walk_cb() detects tag 0 as a marker of end of message and returns PB_WALK_OUT.
   • pb_walk() notices it is on top level and exits with success.

pb_walk_retval_t

Return value from the user callback, indicates the action pb_walk() should take next.

• PB_WALK_EXIT_OK: Return from pb_walk() with true status.
• PB_WALK_EXIT_ERR: Return from pb_walk() with false status.
• PB_WALK_IN: Descend into a submessage and invoke the callback on its first field.
• PB_WALK_OUT: Return from a submessage and invoke the callback on the field it was on before PB_WALK_IN.
• PB_WALK_NEXT_ITEM: If the field is an array or an extension, go to the next item in it. Otherwise go to the next field.
• PB_WALK_NEXT_FIELD: Go to the next field. At the end of the message, iter.tag is set to 0.

pb_walk_state_t

The state of the walking operation is stored in this structure. The pb_encode_submessage() and pb_decode() functions attempt to reuse the same state if they are recursively called from a user-provided field callback.

typedef struct pb_walk_state_s pb_walk_state_t;
struct pb_walk_state_s {
    pb_field_iter_t iter;
    pb_walk_cb_t callback;

    void *stack;
    pb_walk_stacksize_t stacksize;
    pb_walk_stacksize_t stack_remain;
    pb_walk_stacksize_t next_stacksize;

    void *ctx;
    uint32_t flags;

    pb_fieldidx_t depth;
    pb_fieldidx_t max_depth;

    pb_walk_retval_t retval;

    const char *errmsg;
};

Field iterator

The iter member contains a field iterator for the field that is currently being processed. The user-provided walk callback function can use and advance the iterator.

When the callback returns, pb_walk() continues from where the callback left the iterator.

Stack management

The stack is a pointer to a memory buffer where information for each message level is stored. It is used as a full-descending stack: the initial value points to the end of the buffer, and it is decremented for each level.

The user code can request storage for the callback by setting next_stacksize, which can be modified for each level. The callback can check how much stack space it has available by checking stacksize, which it shouldn’t modify.

The remaining stack is tracked by stack_remain, which is automatically updated by pb_walk().

Callback context and flags

The ctx and flags variables are for free use by the walk callback implementation, they are not used by the pb_walk() code.

Typically the ctx will store either pb_encode_ctx_t or pb_decode_ctx_t.

The flags can be used to convey information between message levels, such as “this is the first field in a submessage”.

Depth limit

The depth and max_depth variables track the level in the hierarchy. The depth is automatically updated by pb_walk(), and an error is returned if it exceeds max_depth.

User code is allowed to increment and decrement depth if it performs recursive actions that are not visible to pb_walk(). The only special value is 0, at which level pb_walk() will return with success.

Return value and error message

The retval stores the latest return value from the walk callback. This can be read by the next call, to determine what was the previous action taken. Typically it is used to advance into next field after a PB_WALK_OUT return from a submessage.

The errmsg is used for storing informative error messages for error conditions inside pb_walk(). The caller can combine this information with the ctx->errmsg.

pb_walk_init

This function initializes the state for the walking operation.

bool pb_walk_init(pb_walk_state_t *state, const pb_msgdesc_t *msgdesc,
    const void *message, pb_walk_cb_t callback);

After the initialization, caller can modify state to set e.g. initial stack allocation, the ctx or the flags.

pb_walk

Iterate over the message hierarchy until the callback returns an exit value.

bool pb_walk(pb_walk_state_t *state);

The walk callback is repeatedly called, and iterator is moved according to the return value.

If the callback returns PB_WALK_IN, then pb_walk() stores information of current iterator location and restarts it using state->iter.submsg_desc and state->iter.pData. If there is not sufficient space available, C recursion is used to allocate more from stack unless PB_NO_RECURSION is set.

If the callback returns PB_WALK_OUT, then pb_walk() restores previously stored iterator state. If this was the top level, it acts like PB_WALK_EXIT_OK.

pb_walk_into

Helper function for descending into a message different from the current iterator location.

pb_walk_retval_t pb_walk_into(pb_walk_state_t *state, const pb_msgdesc_t *msgdesc, void *message)

This can be used for descending into messages outside the normal hierarchy, such as extension messages.

Other utility functions

pb_validate_utf8

Validates an UTF8 encoded string:

bool pb_validate_utf8(const char *s);

The protobuf standard requires that string fields only contain valid UTF-8 encoded text, while bytes fields can contain arbitrary data.

By default nanopb validates all strings on encoding and decoding. This can be disabled by defining PB_NO_VALIDATE_UTF8 build option or the PB_ENCODE_CTX_FLAG_NO_VALIDATE_UTF8 or PB_DECODE_CTX_FLAG_NO_VALIDATE_UTF8 context options.

User code can call this function to validate strings in e.g. custom callbacks.