Overview

Nanopb is an ANSI-C library for encoding and decoding messages in Google’s Protocol Buffers format with minimal requirements for RAM and code space. It is primarily suitable for 32-bit microcontrollers.

Documentation version

This documentation applies for nanopb 1.0 and later versions. For documentation of older releases, see:

• Nanopb 0.1.x documentation
• Nanopb 0.2.x documentation
• Nanopb 0.3.x documentation
• Nanopb 0.4.x documentation

Overall structure

Protocol Buffers is designed as a portable data format for communication between programs written in any programming language. The format of the communication is specified in .proto files, which look like this:

message MyMessage {
    required int32 my_numeric_value = 1;
    optional string my_text = 2;
}

Protocol Buffers libraries then convert between native data representation in the programming language and the standard protobuf encoded format. For nanopb, the native representation is C structures:

typedef struct MyMessage {
    int32_t my_numeric_value;
    bool has_my_text;
    char my_text[MyMessage_my_text_max_size];
} MyMessage;

To convert between this C structure and the encoded message format, two functions are used:

bool pb_decode(pb_decode_ctx_t *ctx, const pb_msgdesc_t *msgdesc, void *dest_struct);
bool pb_encode(pb_encode_ctx_t *ctx, const pb_msgdesc_t *msgdesc, void *src_struct);

The encoding or decoding context ctx defines the storage for the encoded data. This can be a simple memory buffer, or the data can be directly read or written to e.g. filesystem or serial port.

The user code processes the C structure MyMessage, and provides a pointer to it as dest_struct or src_struct.

The final part is the middle parameter, msgdesc, which has the necessary information about the structure layout and the .proto field definitions to perform the encoding. This metadata and the C structure definition are automatically created by the nanopb generator:

Parts of the library

The generator/nanopb_generator.py script is run on the build machine using Python. It takes as the input the .proto files that define the message types. As the output it writes .pb.c and .pb.h files that get included in your C program.

The main contents of the generated .pb.h files is the C data structure definitions, and a FIELDLIST macro for each message:

#define MyMessage_FIELDLIST(X, a) \
  X(a, STATIC,   REQUIRED, INT32,    my_numeric_value,   1) \
  X(a, STATIC,   OPTIONAL, STRING,   my_text,            2)

This list of fields is turned into the pb_msgdesc_t type at compile time using C preprocessor macros defined in pb.h. It uses standard C features such as sizeof() and offsetof() to describe the data layout in the structure, so that it can be processed at runtime.

The C code for nanopb library itself is in src/pb_decode.c, src/pb_encode.c and src/pb_common.c. If you only need to encode messages or only need to decode them, you can only include that part and the common logic. This code reads the pb_msgdesc_t description and serializes the data structures to protobuf on-the-wire encoded format.

So a typical project might include these files:

1. Nanopb runtime library:

   • include/nanopb: header files
   • src/pb_common.c: Common utility code, always needed
   • src/pb_decode.c: Needed for decoding messages
   • src/pb_encode.c: Needed for encoding messages

2. Protocol description (you can have many):

   • my_protocol.proto (the protocol definition)
   • my_protocol.pb.h (autogenerated, contains type declarations and macros)
   • my_protocol.pb.c (autogenerated, contains message descriptors)

Features and limitations

The main goal of nanopb is to permit processing protobuf messages on microcontroller platforms. This requires a different kind of implementation than in a protobuf library targeted at PCs.

The differentiating characteristics of nanopb are:

Features

1. Pure C runtime
2. Small code size (5–20 kB depending on processor and compilation options, plus any message definitions)
3. Small ram usage (typically ~1 kB stack, plus any message structs)
4. Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
5. No malloc needed: everything can be allocated statically or on the stack. Optional malloc support available.
6. You can use either encoder or decoder alone to cut the code size in half.
7. Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, oneofs, packed arrays, extension fields.
8. Callback mechanism for handling messages larger than can fit in available RAM.
9. Extensive set of tests.

Limitations

1. Some speed has been sacrificed for code size.
2. The deprecated Protocol Buffers feature called “groups” is not supported.
3. Unknown fields are not preserved when decoding and re-encoding a message.
4. Reflection (runtime introspection) is not supported. E.g. you can’t request a field by giving its name in a string.
5. Cyclic references between messages are supported only in callback and malloc mode.

Compiler requirements

Nanopb 1.0 requires a C compiler supporting the ISO C99 standard or a later version. Support for C11 is recommended. Versions up to 0.4.x supported older C89 compilers.

Nanopb does not require a full libc to be available, but it does require some basic definitions and utility functions:

1. string.h, with these functions: strlen(), memcpy(), memset(), memmove()
2. stdint.h, for definitions of int32_t etc.
3. stddef.h, for definition of size_t
4. stdbool.h, for definition of bool
5. stdlib.h, for definition of realloc() and free(), unless PB_NO_MALLOC build option is used.
6. limits.h, for definition of CHAR_BIT

If these header files do not come with your compiler, you can use the file extra/pb_syshdr.h instead. It contains an example of how to provide the dependencies. You may have to edit it a bit to suit your custom platform.

To use the pb_syshdr.h, define PB_SYSTEM_HEADER_NAME as pb_syshdr.h.

Examples

Tutorial projects are in the examples folder:

• examples/simple: A basic hello-world level of example, showing the encoding and decoding of one message.
• examples/network_server: More advanced example, which provides directory listings over a network connection.
• examples/lowlevel: Example of using the low-level encoding and decoding functions for manual processing of protobuf data.

In addition to the examples, the test cases under tests cover all nanopb features. Some of the more interesting ones are:

• tests/alltypes: A large message type that shows every supported field type.
• tests/any_type: Handling variant-type google.protobuf.Any messages.
• tests/cyclic_messages: Handling cyclic message structure using callbacks.
• tests/extensions: Handling proto2 format extension fields.
• tests/framing: How to mark message start and end over e.g. serial port connection.
• tests/map: Handling Map fields.
• tests/typename_mangling: Remove protobuf package names from C identifiers.

Running the test cases

Extensive unittests and test cases are included under the tests folder.

To build the tests, you will need the scons build system and the nanopb generator dependencies.

The tests should be runnable on most platforms. Windows and Linux builds are regularly tested. The tests also support embedded targets: STM32 (ARM Cortex-M) and AVR builds are regularly tested.

Easiest way to install scons is to use the Python package manager pip, which works on all platforms supported by Python:

$ cd nanopb/tests
$ python3 -m venv venv
$ venv/bin/pip install scons protobuf grpcio-tools
$ venv/bin/scons
...
scons: done building targets.

Installation

The nanopb project consists of two parts:

1. Generator, which runs on developer’s computer.
2. C library, which runs on the embedded system.

The generator is Python based and is run during project compilation to convert .proto files into C source code and headers. It can be used on macOS, Windows, Linux, BSD, and other platforms where Python 3 is available.

The C library is compiled for the embedded target and handles the runtime processing of the messages. It is compatible with practically any platform that has a C99-conformant C compiler available.

Installing the generator

There are three ways to install the nanopb generator:

1. Install depedencies and generator from pip.
2. Install dependencies, then run generator from nanopb source folder.
3. Run generator from nanopb binary package without installation.

Installing from pip

This is the easiest way to get latest version of nanopb generator. It is recommended to use a Python virtual environment (venv) for the installation, as that separates the packages installed from any other projects on the same computer.

To create a virtual environment and install the generator, use:

python3 -m venv venv
venv/bin/pip install nanopb grpcio_tools

By default the latest version is installed. You can optionally specify the version you require by using e.g. nanopb==1.0.0.

The generator is then available at venv/bin/nanopb_generator and can be used to convert .proto files. It is not necessary to manually activate the virtual environment, as the installed script automatically uses it.

$ venv/bin/nanopb_generator test.proto
Writing to test.pb.h and test.pb.c

Running from nanopb source folder

The generator script is in generator folder in nanopb source distribution. To run it, you need to have Python 3, python-protobuf and the protobuf compiler protoc available. Protoc can be installed either separately or as part of the grpcio_tools package.

It doesn’t matter much which method you use to install the dependencies, but using pip generally gives the most up to date versions if you need the latest protobuf features:

• Preferred for any system: Install Python 3, then use venv/bin/pip install protobuf grpcio_tools
• Alternative for Ubuntu / Debian: Use apt install python3-protobuf protobuf-compiler
• Alternative for macOS: Install homebrew, then use brew install protobuf

The generator can then be run through python:

$ python3 nanopb/generator/nanopb_generator.py test.proto
Writing to test.pb.h and test.pb.c

By default some internal files are generated and cached under the generator/proto folder. If you require that no extra files are created into the source folder, you can set environment variable NANOPB_PB2_TEMP_DIR to configure a custom build folder.

Running from binary package

Nanopb binary packages are available for Linux, Windows and macOS. They include the Python interpreter and the protobuf libraries. The binary package includes generator source code in generator folder, and the binary version under generator-bin:

$ nanopb/generator-bin/nanopb_generator test.proto
Writing to test.pb.h and test.pb.c

Troubleshooting generator installation

The most common problem with nanopb generator installation is that some protoc versions are incompatible with some python-protobuf versions. Python-protobuf 3.20.0 and later have improved compatibility.

There may be multiple copies of python-protobuf and protoc installed on your system. To check which paths nanopb is using, give -vV (or --verbose --version) argument:

$ nanopb_generator -Vv
nanopb-1.0.0
Using grpcio-tools protoc from /.../venv/lib/python3.13/site-packages/grpc_tools/protoc.py
libprotoc 31.1
protoc builtin include path: ['/.../venv/lib/python3.13/site-packages/nanopb/generator/proto', '/.../venv/lib/python3.13/site-packages/grpc_tools/_proto']
Python version 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0]
Using python-protobuf from /.../venv/lib/python3.13/site-packages/google/protobuf/__init__.py
Python-protobuf version: 6.33.6

The compatibility matrix of protoc vs. python-protobuf versions is available in Google’s Protobuf Version Support document.

Building the C library

The nanopb C library is built and linked with your own C source code. There are three ways to include it in the build:

1. Compile nanopb as part of your project build.
2. Compile a nanopb static library and link it into your project.
3. Compile a nanopb shared library and use runtime dynamic linking.

For embedded targets, either 1. or 2. is usually preferred.

For any of these you need the nanopb source code, which you can either download separately, or include as a git submodule.

Compile as part of your project build

To build nanopb as part of your project, add the source code files and include directory to project configuration:

• Add source files from nanopb/src/ into the project
• Add nanopb/include to C compiler include path

Compile as a static library

A static library can be compiled using the CMake build system:

$ cmake -B build
$ cmake --build build
...
[100%] Built target protobuf-nanopb-static

For embedded targets you need to specify cross compilation options:

$ cmake -B build -DCMAKE_C_FLAGS="-Os -Wall" -DCMAKE_C_COMPILER="arm-none-eabi-gcc" -DCMAKE_TRY_COMPILE_TARGET_TYPE="STATIC_LIBRARY"
$ cmake --build build    # Add -v to see compilation commands for troubleshooting
...
[100%] Built target protobuf-nanopb-static

This produces build/libprotobuf-nanopb.a, which you can then link into your own project.

Compile as a shared library

Shared libraries are a reasonable option for platforms with a higher level operating system where dynamic linking is supported. CMake can be configured to build a dynamic library by setting BUILD_SHARED_LIBS:

$ cmake -B build -DBUILD_SHARED_LIBS=1
$ cmake --build build
...
[ 50%] Linking C shared library libprotobuf-nanopb.so
...

After this you can link against the build/libprotobuf-nanopb.so and copy it into the shared library folder on the target device.

Build system integration

There exist community-maintained support to help integrating nanopb with various build systems.

For example PlatformIO build rules support automatically installing the required Python dependencies and running the generator.

Getting stated with nanopb

Protobuf basics

The basic idea in protobuf is to define Messages, which can then be encoded and transferred between systems. All protobuf libraries use a common .proto format for describing the message formats and the encoded message data is compatible.

Protobuf libraries for different programming languages vary in how the data is represented to the user program. For nanopb, the data is stored in C structs.

Generated structures

For starters, consider this simple message:

message Example {
    required int32 value = 1;
}

Save this in message.proto and run the nanopb generator:

user@host:~$ python3 nanopb/generator/nanopb_generator.py message.proto
Writing to message.pb.h and message.pb.c

You should now have message.pb.h which contains, among other definitions, the structure:

typedef struct Example {
   int32_t value;
} Example;

Encoding messages

To use nanopb from your own code, include the nanopb library header and the generated message header:

#include <pb_encode.h>
#include "message.pb.h"

Then in your main function fill in the data for the structure:

Example mymessage = {};
mymessage.value = 42;

It is recommended to initialize message structures before use, to ensure that uninitialized fields do not contain garbage values. Either = {} initializer, the generated Example_init_zero macro or memset() can be used.

Before encoding the message, you need to create an encoding context that defines where the output data will be written. In this example we will use a simple memory buffer. The automatically generated macro Example_size gives the maximum length of the message.

pb_encode_ctx_t stream;
uint8_t buffer[Example_size];
pb_init_encode_ctx_for_buffer(&stream, buffer, sizeof(buffer));

To process the structure, a matching message descriptor is needed. This is declared in the automatically generated .pb.h header, and is called MessageName_fields.

Finally, we will call pb_encode() to convert data from the structure mymessage into binary protobuf data in buffer. The function will return success or error status, which you should check.

if (!pb_encode(&stream, Example_fields, &mymessage))
{
    printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
}

The encoded message may be shorter than the provided buffer. Check stream.bytes_written to determine the actual message length. The result data can then be e.g. written to a file:

FILE *file = fopen("example.pb", "wb");
fwrite(buffer, 1, stream.bytes_written, file);
fclose(file);

You can feed the message to protoc --decode=Example message.proto to verify its validity.

Decoding messages

Decoding received message data works very similarly. Include the nanopb decoder header and the message header:

#include <pb_decode.h>
#include "message.pb.h"

Load the data to be decoded. In this case it comes from a file, but you can use any communication interface:

uint8_t buffer[Example_size];
FILE *file = fopen("example.pb", "rb");
size_t msglen = fread(buffer, 1, sizeof(buffer), file);
fclose(file)

Prepare the input stream. Note that you need to pass the actual message length, not the size of the buffer:

pb_decode_ctx_t stream;
pb_init_decode_ctx_for_buffer(&stream, buffer, msglen);

Prepare the structure where the decoded data is stored:

Example mymessage = {};

And finally call pb_decode() and check for errors:

if (!pb_decode(&stream, Example_fields, &mymessage))
{
    printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
}

After the decoding, you can access the fields in the structure to process the data:

printf("We received value: %d\n", (int)mymessage.value);

Further reading

For a complete example of the simple case, see examples/simple/simple.c. For a more complex example with network interface, see the examples/network_server subdirectory.

Nanopb: Basic concepts

The things outlined here are the underlying concepts of the nanopb design.

Proto files

All Protocol Buffers implementations use .proto files to describe the message format. The point of these files is to be a portable interface description language.

Compiling .proto files for nanopb

Nanopb comes with a Python script to generate .pb.c and .pb.h files from the .proto definition:

user@host:~$ nanopb/generator/nanopb_generator.py message.proto
Writing to message.pb.h and message.pb.c

Internally this script uses Google protoc to parse the input file. If you do not have it available, you may receive an error message. You can install either grpcio-tools Python package using pip, or the protoc compiler itself from protobuf-compiler distribution package. Generally the Python package is recommended, because nanopb requires protoc version 3.6 or newer to support all features, and some distributions come with an older version.

Modifying generator behaviour

Using generator options, you can set maximum sizes for fields in order to allocate them statically. The preferred way to do this is to create an .options file with the same name as your .proto file:

# Foo.proto
message Foo {
   required string name = 1;
}

# Foo.options
Foo.name max_size:16

For more information on this, see the Proto file options section in the reference manual.

Streams

Nanopb uses streams for accessing the data in encoded format. The stream abstraction is very lightweight, and consists of a structure (pb_encode_ctx_t or pb_decode_ctx_t) which contains a pointer to a callback function.

There are a few generic rules for callback functions:

1. Return false on IO errors. The encoding or decoding process will abort immediately.
2. Use state to store your own data, such as a file descriptor.
3. bytes_written and bytes_left are updated by pb_write and pb_read.
4. Your callback may be used with substreams. In this case bytes_left, bytes_written and max_size have smaller values than the original stream. Don’t use these values to calculate pointers.
5. Always read or write the full requested length of data. For example, POSIX recv() needs the MSG_WAITALL parameter to accomplish this.

Output streams

struct _pb_encode_ctx_t
{
   bool (*callback)(pb_encode_ctx_t *stream, const uint8_t *buf, size_t count);
   void *state;
   size_t max_size;
   size_t bytes_written;
};

The callback for output stream may be NULL, in which case the stream simply counts the number of bytes written. In this case, max_size is ignored.

Otherwise, if bytes_written + bytes_to_be_written is larger than max_size, pb_write returns false before doing anything else. If you don't want to limit the size of the stream, pass SIZE_MAX.

Example 1:

This is the way to get the size of the message without storing it anywhere:

Person myperson = ...;
pb_encode_ctx_t sizestream = {0};
pb_encode(&sizestream, Person_fields, &myperson);
printf("Encoded size is %d\n", sizestream.bytes_written);

Example 2:

Writing to stdout:

bool callback(pb_encode_ctx_t `stream, const uint8_t `buf, size_t count)
{
   FILE *file = (FILE*) stream->state;
   return fwrite(buf, 1, count, file) == count;
}

pb_encode_ctx_t stdoutstream = {&callback, stdout, SIZE_MAX, 0};

Input streams

For input streams, there is one extra rule:

6. You don’t need to know the length of the message in advance. After getting EOF error when reading, set bytes_left to 0 and return false. pb_decode() will detect this and if the EOF was in a proper position, it will return true.

Here is the structure:

struct _pb_decode_ctx_t
{
   bool (*callback)(pb_decode_ctx_t *stream, uint8_t *buf, size_t count);
   void *state;
   size_t bytes_left;
};

The callback must always be a function pointer. Bytes_left is an upper limit on the number of bytes that will be read. You can use SIZE_MAX if your callback handles EOF as described above.

Example:

This function binds an input stream to stdin:

bool callback(pb_decode_ctx_t *stream, uint8_t *buf, size_t count)
{
   FILE *file = (FILE*)stream->state;
   bool status;

   if (buf == NULL)
   {
       while (count-- && fgetc(file) != EOF);
       return count == 0;
   }

   status = (fread(buf, 1, count, file) == count);

   if (feof(file))
       stream->bytes_left = 0;

   return status;
}

pb_decode_ctx_t stdinstream = {&callback, stdin, SIZE_MAX};

Data types

Most Protocol Buffers datatypes have directly corresponding C datatypes, such as int32 is int32_t, float is float and bool is bool. However, the variable-length datatypes are more complex:

1. Strings, bytes and repeated fields of any type map to callback functions by default.
2. If there is a special option (nanopb).max_length or (nanopb).max_size specified in the .proto file, string maps to null-terminated char array and bytes map to a structure containing a char array and a size field.
3. If (nanopb).fixed_length is set to true and (nanopb).max_size is also set, then bytes map to an inline byte array of fixed size.
4. If there is a special option (nanopb).max_count specified on a repeated field, it maps to an array of whatever type is being repeated. Another field will be created for the actual number of entries stored.
5. If (nanopb).fixed_count is set to true and (nanopb).max_count is also set, the field for the actual number of entries will not by created as the count is always assumed to be max count.

Examples of .proto specifications vs. generated structure

Simple integer field:
.proto: int32 age = 1;
.pb.h: int32_t age;

String with unknown length:
.proto: string name = 1;
.pb.h: pb_callback_t name;

String with known maximum length:
.proto: string name = 1 [(nanopb).max_length = 40];
.pb.h: char name[41];

Repeated string with unknown count:
.proto: repeated string names = 1;
.pb.h: pb_callback_t names;

Repeated string with known maximum count and size:
.proto: repeated string names = 1 [(nanopb).max_length = 40, (nanopb).max_count = 5];
.pb.h: size_t names_count; char names[5][41];

Bytes field with known maximum size:
.proto: bytes data = 1 [(nanopb).max_size = 16];
.pb.h: PB_BYTES_ARRAY_T(16) data;, where the struct contains {pb_size_t size; pb_byte_t bytes[n];}

Bytes field with fixed length:
.proto: bytes data = 1 [(nanopb).max_size = 16, (nanopb).fixed_length = true];
.pb.h: pb_byte_t data[16];

Repeated integer array with known maximum size:
.proto: repeated int32 numbers = 1 [(nanopb).max_count = 5];
.pb.h: pb_size_t numbers_count; int32_t numbers[5];

Repeated integer array with fixed count:
.proto: repeated int32 numbers = 1 [(nanopb).max_count = 5, (nanopb).fixed_count = true];
.pb.h: int32_t numbers[5];

The maximum lengths are checked in runtime. If string/bytes/array exceeds the allocated length, pb_decode() will return false.

Note: For the bytes datatype, the field length checking may not be exact. The compiler may add some padding to the pb_bytes_t structure, and the nanopb runtime doesn’t know how much of the structure size is padding. Therefore it uses the whole length of the structure for storing data, which is not very smart but shouldn’t cause problems. In practise, this means that if you specify (nanopb).max_size=5 on a bytes field, you may be able to store 6 bytes there. For the string field type, the length limit is exact.

Note: The decoder only keeps track of one fixed_count repeated field at a time. Usually this it not an issue because all elements of a repeated field occur end-to-end. Interleaved array elements of several fixed_count repeated fields would be a valid protobuf message, but would get rejected by nanopb decoder with error "wrong size for fixed count field".

Field callbacks

The easiest way to handle repeated fields is to specify a maximum size for them, as shown in the previous section. However, sometimes you need to be able to handle arrays with unlimited length, possibly larger than available RAM memory.

For these cases, nanopb provides a callback interface. Nanopb core invokes the callback function when it gets to the specific field in the message. Your code can then handle the field in custom ways, for example decode the data piece-by-piece and store to filesystem.

The pb_callback_t structure contains a function pointer and a void pointer called arg you can use for passing data to the callback. If the function pointer is NULL, the field will be skipped. A pointer to the arg is passed to the function, so that it can modify it and retrieve the value.

The actual behavior of the callback function is different in encoding and decoding modes. In encoding mode, the callback is called once and should write out everything, including field tags. In decoding mode, the callback is called repeatedly for every data item.

To write more complex field callbacks, it is recommended to read the Google Protobuf Encoding Specification.

Encoding callbacks

bool (*encode)(pb_encode_ctx_t *stream, const pb_field_iter_t *field, void * const *arg);

When encoding, the callback should write out complete fields, including the wire type and field number tag. It can write as many or as few fields as it likes. For example, if you want to write out an array as repeated field, you should do it all in a single call.

Usually you can use pb_encode_tag_for_field to encode the wire type and tag number of the field. However, if you want to encode a repeated field as a packed array, you must call pb_encode_tag instead to specify a wire type of PB_WT_STRING.

If the callback is used in a submessage, it will be called multiple times during a single call to pb_encode. In this case, it must produce the same amount of data every time. If the callback is directly in the main message, it is called only once.

This callback writes out a dynamically sized string:

bool write_string(pb_encode_ctx_t *stream, const pb_field_iter_t *field, void * const *arg)
{
    char *str = get_string_from_somewhere();
    if (!pb_encode_tag_for_field(stream, field))
        return false;

    return pb_encode_string(stream, (uint8_t*)str, strlen(str));
}

Decoding callbacks

bool (*decode)(pb_decode_ctx_t *stream, const pb_field_iter_t *field, void **arg);

When decoding, the callback receives a length-limited substring that reads the contents of a single field. The field tag has already been read. For string and bytes, the length value has already been parsed, and is available at stream->bytes_left.

The callback will be called multiple times for repeated fields. For packed fields, you can either read multiple values until the stream ends, or leave it to pb_decode to call your function over and over until all values have been read.

This callback reads multiple integers and prints them:

bool read_ints(pb_decode_ctx_t *stream, const pb_field_iter_t *field, void **arg)
{
    while (stream->bytes_left)
    {
        uint64_t value;
        if (!pb_decode_varint(stream, &value))
            return false;
        printf("%lld\n", value);
    }
    return true;
}

Function name bound callbacks

bool MyMessage_callback(pb_decode_ctx_t *istream, pb_encode_ctx_t *ostream, const pb_field_iter_t *field);

Storing function pointer in pb_callback_t fields inside the message requires extra storage space and is often cumbersome. As an alternative, the generator options callback_function and callback_datatype can be used to bind a callback function based on its name.

Typically this feature is used by setting callback_datatype to e.g. void\* or even a struct type used to store encoded or decoded data. The generator will automatically set callback_function to MessageName_callback and produce a prototype for it in generated .pb.h. By implementing this function in your own code, you will receive callbacks for fields without having to separately set function pointers.

If you want to use function name bound callbacks for some fields and pb_callback_t for other fields, you can call pb_default_field_callback from the message-level callback. It will then read a function pointer from pb_callback_t and call it.

Message descriptor

For using the pb_encode() and pb_decode() functions, you need a description of all the fields contained in a message. This description is usually autogenerated from .proto file.

For example this submessage in the Person.proto file:

message Person {
    message PhoneNumber {
        required string number = 1 [(nanopb).max_size = 40];
        optional PhoneType type = 2 [default = HOME];
    }
}

This in turn generates a macro list in the .pb.h file:

#define Person_PhoneNumber_FIELDLIST(X, a) \
X(a, STATIC,   REQUIRED, STRING,   number,            1) \
X(a, STATIC,   OPTIONAL, UENUM,    type,              2)

Inside the .pb.c file there is a macro call to PB_BIND:

PB_BIND(Person_PhoneNumber, Person_PhoneNumber, AUTO)

These macros will in combination generate pb_msgdesc_t structure and associated lists:

const uint32_t Person_PhoneNumber_field_info[] = { ... };
const pb_msgdesc_t * const Person_PhoneNumber_submsg_info[] = { ... };
const pb_msgdesc_t Person_PhoneNumber_msg = {
  2,
  Person_PhoneNumber_field_info,
  Person_PhoneNumber_submsg_info,
  Person_PhoneNumber_DEFAULT,
  NULL,
};

The encoding and decoding functions take a pointer to this structure and use it to process each field in the message.

Oneof

Protocol Buffers supports oneof sections, where only one of the fields contained within can be present. Here is an example of oneof usage:

message MsgType1 {
    required int32 value = 1;
}

message MsgType2 {
    required bool value = 1;
}

message MsgType3 {
    required int32 value1 = 1;
    required int32 value2 = 2;
} 

message MyMessage {
    required uint32 uid = 1;
    required uint32 pid = 2;
    required uint32 utime = 3;

    oneof payload {
        MsgType1 msg1 = 4;
        MsgType2 msg2 = 5;
        MsgType3 msg3 = 6;
    }
}

Nanopb will generate payload as a C union and add an additional field which_payload:

typedef struct _MyMessage {
  uint32_t uid;
  uint32_t pid;
  uint32_t utime;
  pb_size_t which_payload;
  union {
      MsgType1 msg1;
      MsgType2 msg2;
      MsgType3 msg3;
  } payload;
} MyMessage;

which_payload indicates which of the oneof fields is actually set. The user is expected to set the field manually using the correct field tag:

MyMessage msg = MyMessage_init_zero;
msg.payload.msg2.value = true;
msg.which_payload = MyMessage_msg2_tag;

Notice that neither which_payload field nor the unused fields in payload will consume any space in the resulting encoded message.

When a field inside oneof contains pb_callback_t fields, the callback values cannot be set before decoding. This is because the different fields share the same storage space in C union. Instead either function name bound callbacks or a separate message level callback can be used. See tests/oneof_callback for an example on this.

Extension fields

Protocol Buffers supports a concept of extension fields, which are additional fields to a message, but defined outside the actual message. The definition can even be in a completely separate .proto file.

The base message is declared as extensible by keyword extensions in the .proto file:

message MyMessage {
    .. fields ..
    extensions 100 to 199;
}

For each extensible message, nanopb_generator.py declares an additional callback field called extensions. The field and associated datatype pb_extension_t forms a linked list of handlers. When an unknown field is encountered, the decoder calls each handler in turn until either one of them handles the field, or the list is exhausted.

The actual extensions are declared using the extend keyword in the .proto, and are in the global namespace:

extend MyMessage {
    optional int32 myextension = 100;
}

For each extension, nanopb_generator.py creates a constant of type pb_extension_type_t. To link together the base message and the extension, you have to:

1. Allocate storage for your field, matching the datatype in the .proto. For example, for a int32 field, you need a int32_t variable to store the value.
2. Create a pb_extension_t constant, with pointers to your variable and to the generated pb_extension_type_t.
3. Set the message.extensions pointer to point to the pb_extension_t.

An example of this is available in tests/test_encode_extensions.c and tests/test_decode_extensions.c.

Default values

Protobuf has two syntax variants, proto2 and proto3. Of these proto2 has user definable default values that can be given in .proto file:

message MyMessage {
    optional bytes foo = 1 [default = "ABC\x01\x02\x03"];
    optional string bar = 2 [default = "åäö"];
}

Nanopb will generate both static and runtime initialization for the default values. In myproto.pb.h there will be a #define MyMessage_init_default {...} that can be used to initialize whole message into default values:

MyMessage msg = MyMessage_init_default;

In addition to this, pb_decode() will initialize message fields to defaults at runtime. If this is not desired, pb_decode_ex() can be used instead.

Message framing

Protocol Buffers does not specify a method of framing the messages for transmission. This is something that must be provided by the library user, as there is no one-size-fits-all solution. Typical needs for a framing format are to:

1. Encode the message length.
2. Encode the message type.
3. Perform any synchronization and error checking that may be needed depending on application.

For example UDP packets already fulfill all the requirements, and TCP streams typically only need a way to identify the message length and type. Lower level interfaces such as serial ports may need a more robust frame format, such as HDLC (high-level data link control).

Nanopb provides a few helpers to facilitate implementing framing formats:

1. Functions pb_encode_ex and pb_decode_ex prefix the message data with a varint-encoded length.
2. Union messages and oneofs are supported in order to implement top-level container messages.
3. Message IDs can be specified using the (nanopb_msgopt).msgid option and can then be accessed from the header.

Return values and error handling

Most functions in nanopb return bool: true means success, false means failure.

The encoding and decoding contexts have a errmsg variable for a textual error message. This is meant primarily as a debugging aid, and can be disabled with the build option PB_NO_ERRMSG. There are helper macros PB_GET_ERROR, PB_SET_ERROR and PB_RETURN_ERROR for accessing the errmsg member variable.

Common error messages are:

• “array overflow”: Number of array entries exceeds the specified max_count.
• “bytes overflow”: Length of a bytes field exceeds the specified max_size.
• “callback failed”: Field callback returned false without setting errmsg.
• “end-of-stream”: Input data ended at the middle of a protobuf element.
• “failed to set defaults”: Internal error when decoding the default value for a field.
• “fixed_count mismatch”: Decoded length didn’t match the size of a fixed_count array.
• “fixed_length mismatch”: Decoded length didn’t match a fixed_length bytes field.
• “invalid data_size”: Internal error or using uint64_t with PB_WITHOUT_64BIT set.
• “invalid extension”: The pb_extension_t in a message did not point to a valid descriptor.
• “invalid field type”: Internal error or corrupted message descriptor data.
• “invalid oneof tag”: The value of which_ for a oneof doesn’t match any member.
• “invalid utf8”: String data was not valid UTF-8 encoded text.
• “io error”: Stream callback returned false without setting errmsg.
• “max_depth exceeded”: Message hierarchy depth exceeded PB_MESSAGE_NESTING_MAX.
• “missing required field”: A field with proto2 required specifier was not present in a message.
• “no allocator”: PB_NO_DEFAULT_ALLOCATOR was set and no context allocator was provided.
• “no malloc support”: PB_NO_MALLOC was set and FT_POINTER field was being decoded.
• “null descriptor”: Internal error or corrupted message descriptor data.
• “onepass flush”: pb_flush_write_buffer() call during one-pass sizing (usually internal error).
• “realloc failed”: Allocator returned NULL, typically indicating out-of-memory condition.
• “recursion disabled”: PB_NO_RECURSION is set and PB_MESSAGE_NESTING was exceeded.
• “size too large”: Memory allocation would exceed the limits of pb_size_t type.
• “sizing failed”: Internal error when computing submessage size.
• “stackframe too large”: Internal error, single frame exceeded PB_WALK_STACK_SIZE.
• “stream full”: Stream max_size was hit when encoding.
• “string overflow”: Length of a string field exceeds the specified max_size.
• “struct_size mismatch”: Structure passed to pb_en/decode() did not match the descriptor.
• “submsg size changed”: Field callback wrote different amount of data on second pass.
• “unterminated string”: C string in the message structure does not have terminating 0-byte.
• “varint overflow”: Value of decoded varint exceeds the limits of the structure field datatype.
• “wrong tag”: Internal error when decoding a repeated pointer field.
• “wrong wire type”: Decoded wire_type didn’t match the expected type of the field.
• “zero tag”: Tag number 0 is forbidden in protobuf (maybe msglen is wrong?)

Errors that cannot necessarily be detected by nanopb include:

• Invalid protobuf data: Many types of corruption will just result in incorrect data contents being decoded.
• Incorrect message length: If the message ends between protobuf elements, it is a valid message with just some fields missing.
• Stack overflow: There is no standard C mechanism to detect running out of stack. PB_MAX_MESSAGE_NESTING and PB_NO_RECURSION can be used to limit stack usage.

Static assertions

Nanopb code uses static assertions to check size of structures at the compile time. The PB_STATIC_ASSERT macro is defined in pb.h. If ISO C11 standard is available, the C standard _Static_assert keyword is used, otherwise a negative sized array definition trick is used.

Common reasons for static assertion errors are:

1. FIELDINFO_DOES_NOT_FIT_width2 with width1 or width2: Message that is larger than 256 bytes, but nanopb generator does not detect it for some reason. Often resolved by giving all .proto files as argument to nanopb_generator.py at the same time, to ensure submessage definitions are found. Alternatively (nanopb).descriptorsize = DS_4 option can be given manually.

2. FIELDINFO_DOES_NOT_FIT_width4 with width4: Message that is larger than 64 kilobytes. There will be a better error message for this in a future nanopb version, but currently it asserts here. The compile time option PB_FIELD_32BIT should be specified either on C compiler command line or by editing pb.h. This will increase the sizes of integer types used internally in nanopb code.

3. DOUBLE_MUST_BE_8_BYTES: Some platforms, most notably AVR, do not support the 64-bit double type, only 32-bit float. The compile time option PB_CONVERT_DOUBLE_FLOAT can be defined to convert between the types automatically. The conversion results in small rounding errors and takes unnecessary space in transmission, so changing the .proto to use float type is often better.

4. INT64_T_WRONG_SIZE: The stdint.h system header is incorrect for the C compiler being used. This can result from erroneous compiler include path. If the compiler actually does not support 64-bit types, the compile time option PB_WITHOUT_64BIT can be used.

5. variably modified array size: The compiler used has problems resolving the array-based static assert at compile time. Try setting the compiler to C11 standard mode if possible. If static assertions cannot be made to work on the compiler used, the compile-time option PB_NO_STATIC_ASSERT can be specified to turn them off.

Extensible structures

TODO: Describe how pb_encode_ctx_t, pb_allocator_t etc. can be extended.

Build options

Compilation options affect the functionality included in the nanopb core C code. The options can be specified in one of two ways:

1. Using the -D switch on the C compiler command line.
2. Using a #define in a pb_config.h file.

You can find a list of all the flags in the beginning of pb.h, or in the pb_config_example.h file.

Note

You must have the same compilation options for the nanopb library and all code that includes nanopb headers. For shared libraries, ABI compatibility between nanopb versions is only tested for the fully-featured default configuration.

System and configuration header names

If the compiler supports __has_include() preprocessor directive, nanopb will automatically include a file named pb_config.h if it exists. There is an example pb_config_example.h that you can copy to your own project.

Alternatively you can define PB_CONFIG_HEADER_NAME to the name of a configuration header you want to include. For example, you can use compiler command line argument -DPB_CONFIG_HEADER_NAME=my_nanopb_config.h.

By default nanopb includes the standard C headers such as stdint.h and string.h. Alternatively PB_SYSTEM_HEADER or PB_SYSTEM_HEADER_NAME can be specified to provide a custom header which provides the necessary definitions.

The PB_SYSTEM_HEADER value must include brackets or quotes around the text, while PB_SYSTEM_HEADER_NAME adds quotes automatically. The latter is easier to set from compiler command line parameters, as the command line shell may eat the quotes.

API version compatibility flag

Backwards compatibility functions and macros have been included to ease porting user code from older nanopb versions to the 1.0 version.

By default these compatibility features are disabled. To enable them, define PB_API_VERSION to a value corresponding to the nanopb version the code has been written against. For example, for nanopb-0.4.x compatibility set PB_API_VERSION = 40

The latest API remains fully usable even if compatibility features are enabled. This permits piece-by-piece migration of old code.

Feature disable flags

These flags can be used to disable individual nanopb features. Disabling features generally reduces code size and RAM usage, but benefits vary between platforms.

To disable a feature, define its PB_NO_xxxxx flag to 1.

Alternatively, define PB_MINIMAL to 1, which disables all optional features by default. Individual features can then be enabled by setting their PB_NO_xxxxx flags to 0.

Memory allocation

• PB_NO_MALLOC: Disable support for dynamically allocated fields.

• PB_NO_DEFAULT_ALLOCATOR: Disable support for default allocator. Dynamic allocation is enabled only if ctx->allocator is set by user code.

• PB_NO_CONTEXT_ALLOCATOR: Disable support for context-specific allocator. Only default allocator (pb_realloc()) is supported, the pb_decode_ctx_t allocator field is disabled.

Context and stream management

• PB_NO_STREAM_CALLBACK: Only support input/output to memory buffers. Disables using callback functions for IO streams. This option was called PB_BUFFER_ONLY in nanopb-0.4.x

• PB_NO_ERRMSG: Disable support for descriptive error messages. Only success/failure status is provided.

• PB_NO_RECURSION: Disable recursive function calls in nanopb core. Only up to PB_MESSAGE_NESTING levels of nested messages can be processed. User callbacks can still invoke recursion.

• PB_NO_OPT_ASSERT: Disable optional assertions in code. These are mainly to more easily catch bugs during development.

Protobuf feature support

• PB_NO_LARGEMSG: Disable support for messages over 4 kB. Tag numbers are limited to max 4095 and arrays to max 255 items.

• PB_NO_VALIDATE_UTF8: Do not validate string encoding. Protobuf spec requires strings to be valid UTF-8. This setting disables string validation in nanopb.

• PB_NO_EXTENSIONS: Disable support for proto2 extension fields

• PB_NO_CONTEXT_FIELD_CALLBACK: Disable support for field callbacks using ctx->field_callback() mechanism.

• PB_NO_NAME_FIELD_CALLBACK: Disable support for name-bound field callbacks using generator ‘callback_function’ option.

• PB_NO_STRUCT_FIELD_CALLBACK: Disable support for field callbacks defined using the pb_callback_t mechanism.

• PB_NO_DEFAULT_VALUES: Disable support for runtime-initialization of field default values. MyMessage_init_default macro is still available.

Platform-specific options

Some embedded platforms have special limitations, which can require special compilation options. For most platforms there is no need to define these, as they are automatically detected from C compiler features.

Data types

• PB_WITHOUT_64BIT: Disable usage of 64-bit data types in the code. For compilers or CPUs that do not support uint64_t.

• PB_LITTLE_ENDIAN_8BIT: Specify memory layout compatibility. This can be defined if CPU uses 8-bit bytes and has little-endian memory layout. If undefined (default), the support is automatically detected from compiler type.

• PB_NO_PACKED_STRUCTS: Never use ‘packed’ attribute on structures. Note that the attribute is only specified when requested in .proto file options. This define allows globally disabling it on platforms that do not support unaligned memory access.

• PB_WALK_STACK_ALIGN_TYPE: Alignment requirement for pb_walk() stack. By default this is void*, which should have large enough alignment for storage of any pointer or 32-bit integer. Special platforms could require e.g. uint32_t here.

• PB_BYTE_T_OVERRIDE: Override type used to access byte buffers. On most platforms, pb_byte_t = uint8_t. On platforms without uint8_t, by default unsigned char is used. Alternatively this option can be used to set a custom type.

• PB_SIZE_T_OVERRIDE: Override size type used for messages and streams. On 8-32 bit platforms, pb_size_t defaults to size_t. On 64 bit platforms, pb_size_t defaults to uint32_t. This option can be used to override the type.

Static assertions

• PB_C99_STATIC_ASSERT: Force use of older, C99 static assertion mechanism. This is for compilers that do not support _Static_assert() keyword that was introduced in C11 standard. Many compilers supported it before that.

• PB_NO_STATIC_ASSERT: Disable compile-time assertions in the code. This is for compilers where the PB_STATIC_ASSERT macro does not work. It’s preferable to either change compiler to C11 standards mode or to define PB_C99_STATIC_ASSERT.

Memory and code attributes

• PB_NO_FUNCTION_POINTERS: Disable usage of function pointers in the library. Useful for platforms where function pointers are either expensive, or for compatibility with code safety standards such as MISRA-C. Removes support for any callback-based features.

• PB_PROGMEM: Attribute and access method for storing constants in ROM. This is automatically enabled for AVR platform. It can be used on platforms where const variables are not automatically stored in ROM.

• PB_WEAK_FUNCTION: Attribute for weak function declarations. Used only with PB_NO_FUNCTION_POINTERS and other special features. By default autodetected by compiler type.

Protobuf compatibility options

The options below enable interoperability features that can be useful for communicating with externally defined protobuf schema.

• PB_ENCODE_ARRAYS_UNPACKED: Use ‘unpacked’ array format for all fields. Normally the more efficient ‘packed’ array format is used for field types that support it. This option forces ‘unpacked’ format. In particular, it is needed when communicating with protobuf.js versions before 2020.

• PB_CONVERT_DOUBLE_FLOAT: Convert 64-bit doubles to 32-bit floats. AVR platform only supports 32-bit floats. If you need to use a .proto that has ‘double’ fields, this option will convert the encoded binary format. The precision of values will be limited to 32-bit.

Stack usage options

Nanopb uses a hybrid approach to handling recursive messages. Instead of C recursion, the core uses a memory buffer to store minimal amount of information for each message level. For this storage, a constant-sized buffer is allocated on stack. This initial reservation is enough for PB_MESSAGE_NESTING levels.

Once the buffer fills up, more memory is allocated from stack using C recursion. This can be disabled with PB_NO_RECURSION.

In any case, message nesting is limited to maximum of PB_MESSAGE_NESTING_MAX levels, after which runtime error is returned.

• PB_MESSAGE_NESTING: Expected depth of message hierarchy. Encode and decode calls initially reserve enough stack space to handle this number of nested message levels.

• PB_MESSAGE_NESTING_MAX: Runtime limit of message nesting. If recursion is enabled, up to this many nested message levels can be processed by dynamically allocating more stack space.

• PB_WALK_STACK_SIZE: Block size of recursive memory reservation. Once the initial stack allocation is exhausted, pb_walk() will reserve more stack in blocks of this many bytes.

• PB_MAX_REQUIRED_FIELDS: Expected number of required fields per message. This is only used for calculating the initial stack allocation. At runtime, memory is allocated based on actual number of required fields in each message.

Nanopb generator API reference

Generator options

Generator options affect how the .proto files get converted to .pb.c and .pb.h. files.

Most options are related to specific message or field in .proto file. The full set of available options is defined in nanopb.proto. Here is a list of the most common options, but see the file for a full list:

• max_size: Allocated maximum size for bytes and string fields. For strings, this includes the terminating zero.
• max_length: Maximum length for string fields. Setting this is equivalent to setting max_size to a value of length + 1.
• max_count: Allocated maximum number of entries in arrays (repeated fields).
• type: Select how memory is allocated for the generated field. Default value is FT_DEFAULT, which defaults to FT_STATIC when possible and FT_CALLBACK if not possible. You can use FT_CALLBACK, FT_POINTER, FT_STATIC or FT_IGNORE to select a callback field, a dynamically allocate dfield, a statically allocated field or to completely ignore the field.
• long_names: Prefix the enum name to the enum value in definitions, i.e. EnumName_EnumValue. Enabled by default.
• packed_struct: Make the generated structures packed, which saves some RAM space but slows down execution. This can only be used if the CPU supports unaligned access to variables.
• skip_message: Skip a whole message from generation. Can be used to remove message types that are not needed in an application.
• no_unions: Generate oneof fields as multiple optional fields instead of a C union {}.
• anonymous_oneof: Generate oneof fields as an anonymous union.
• msgid: Specifies a unique id for this message type. Can be used by user code as an identifier.
• fixed_length: Generate bytes fields with a constant length defined by max_size. A separate .size field will then not be generated.
• fixed_count: Generate arrays with constant length defined by max_count.
• package: Package name that applies only for nanopb generator. Defaults to name defined by package keyword in .proto file, which applies for all languages.
• int_size: Override the integer type of a field. For example, specify int_size = IS_8 to convert int32 from protocol definition into int8_t in the structure. When used with enum types, the size of the generated enum can be specified (C++ only)

These options can be defined for the .proto files before they are converted using the nanopb-generator.py. There are three ways to define the options:

1. Using a separate .options file. This allows using wildcards for applying same options to multiple fields.
2. Defining the options on the command line of nanopb_generator.py. This only makes sense for settings that apply to a whole file.
3. Defining the options in the .proto file using the nanopb extensions. This keeps the options close to the fields they apply to, but can be problematic if the same .proto file is shared with many projects.

The effect of the options is the same no matter how they are given. The most common purpose is to define maximum size for string fields in order to statically allocate them.

Defining the options in a .options file

The preferred way to define options is to have a separate file ‘myproto.options’ in the same directory as the ‘myproto.proto’. :

# myproto.proto
message MyMessage {
    required string name = 1;
    repeated int32 ids = 4;
}

# myproto.options
MyMessage.name         max_size:40
MyMessage.ids          max_count:5

The generator will automatically search for this file and read the options from it. The file format is as follows:

• Lines starting with # or // are regarded as comments.
• Blank lines are ignored.
• All other lines should start with a field name pattern, followed by one or more options. For example: MyMessage.myfield max_size:5 max_count:10.
• The field name pattern is matched against a string of form Message.field. For nested messages, the string is Message.SubMessage.field. A whole file can be matched by its filename dir/file.proto.
• The field name pattern may use the notation recognized by Python fnmatch():
  • * matches any part of string, like Message.* for all fields
  • ? matches any single character
  • [seq] matches any of characters s, e and q
  • [!seq] matches any other character
• The options are written as option_name:option_value and several options can be defined on same line, separated by whitespace.
• Options defined later in the file override the ones specified earlier, so it makes sense to define wildcard options first in the file and more specific ones later.

To debug problems in applying the options, you can use the -v option for the nanopb generator. With protoc, plugin options are specified with --nanopb_opt:

nanopb_generator -v message.proto           # When invoked directly
protoc ... --nanopb_opt=-v --nanopb_out=. message.proto  # When invoked through protoc

Protoc doesn’t currently pass include path into plugins. Therefore if your .proto is in a subdirectory, nanopb may have trouble finding the associated .options file. A workaround is to specify include path separately to the nanopb plugin, like:

protoc -Isubdir --nanopb_opt=-Isubdir --nanopb_out=. message.proto

If preferred, the name of the options file can be set using generator argument -f.

Defining the options in the .proto file

The .proto file format allows defining custom options for the fields. The nanopb library comes with nanopb.proto which does exactly that, allowing you do define the options directly in the .proto file:

import "nanopb.proto";

message MyMessage {
    required string name = 1 [(nanopb).max_size = 40];
    repeated int32 ids = 4   [(nanopb).max_count = 5];
}

A small complication is that you have to set the include path of protoc so that nanopb.proto can be found. Therefore, to compile a .proto file which uses options, use a protoc command similar to:

protoc -Inanopb/generator/proto -I. --nanopb_out=. message.proto

The options can be defined in file, message and field scopes:

option (nanopb_fileopt).max_size = 20; // File scope
message Message
{
    option (nanopb_msgopt).max_size = 30; // Message scope
    required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
}

Defining the options on command line

The nanopb_generator.py has a simple command line option -s OPTION:VALUE. The setting applies to the whole file that is being processed.

There are also a few command line options that cannot be applied using the other mechanisms, as they affect the whole generation:

• --c-style: Modify symbol names to better match C naming conventions.
• --custom-style: Modify symbol names by providing your own styler implementation.
• --no-timestamp: Do not add timestamp to generated files.
• --strip-path: Remove relative path from generated #include directives.
• --cpp-descriptors: Generate extra convenience definitions for use from C++

For a full list of generator command line options, use nanopb_generator.py --help:

Usage: nanopb_generator.py [options] file.pb ...

Options:
-h, --help            show this help message and exit
-V, --version         Show version info and exit (add -v for protoc version
                        info)
-x FILE               Exclude file from generated #include list.
-e EXTENSION, --extension=EXTENSION
                        Set extension to use instead of '.pb' for generated
                        files. [default: .pb]
-H EXTENSION, --header-extension=EXTENSION
                        Set extension to use for generated header files.
                        [default: .h]
-S EXTENSION, --source-extension=EXTENSION
                        Set extension to use for generated source files.
                        [default: .c]
-f FILE, --options-file=FILE
                        Set name of a separate generator options file.
-I DIR, --options-path=DIR, --proto-path=DIR
                        Search path for .options and .proto files. Also
                        determines relative paths for output directory
                        structure.
--error-on-unmatched  Stop generation if there are unmatched fields in
                        options file
--no-error-on-unmatched
                        Continue generation if there are unmatched fields in
                        options file (default)
-D OUTPUTDIR, --output-dir=OUTPUTDIR
                        Output directory of .pb.h and .pb.c files
-Q FORMAT, --generated-include-format=FORMAT
                        Set format string to use for including other .pb.h
                        files. Value can be 'quote', 'bracket' or a format
                        string. [default: #include "%s"]
-L FORMAT, --library-include-format=FORMAT
                        Set format string to use for including the nanopb pb.h
                        header. Value can be 'quote', 'bracket' or a format
                        string. [default: #include <%s>]
--strip-path          Strip directory path from #included .pb.h file name
--no-strip-path       Opposite of --strip-path (default since 0.4.0)
--cpp-descriptors     Generate C++ descriptors to lookup by type (e.g.
                        pb_field_t for a message)
-T, --no-timestamp    Don't add timestamp to .pb.h and .pb.c preambles
                        (default since 0.4.0)
-t, --timestamp       Add timestamp to .pb.h and .pb.c preambles
-q, --quiet           Don't print anything except errors.
-v, --verbose         Print more information.
-s OPTION:VALUE       Set generator option (max_size, max_count etc.).
--protoc-opt=OPTION   Pass an option to protoc when compiling .proto files
--protoc-insertion-points
                        Include insertion point comments in output for use by
                        custom protoc plugins
-C, --c-style         Use C naming convention.
--custom-style=MODULE.CLASS
                      Use a custom naming convention from a module/class
                      that defines the methods from the NamingStyle class to
                      be overridden. When paired with the -C/--c-style
                      option, the NamingStyleC class is the fallback,
                      otherwise it's the NamingStyle class.

Compile file.pb from file.proto by: 'protoc -ofile.pb file.proto'. Output will
be written to file.pb.h and file.pb.c.

Security model

Importance of security in a Protocol Buffers library

In the context of protocol buffers, security comes into play when decoding untrusted data. Naturally, if the attacker can modify the contents of a protocol buffers message, he can feed the application any values possible. Therefore the application itself must be prepared to receive untrusted values.

Where nanopb plays a part is preventing the attacker from running arbitrary code on the target system. Mostly this means that there must not be any possibility to cause buffer overruns, memory corruption or invalid pointers by the means of crafting a malicious message.

Division of trusted and untrusted data

The following data is regarded as trusted. It must be under the control of the application writer. Malicious data in these structures could cause security issues, such as execution of arbitrary code:

1. Callback, pointer and extension fields in message structures given to pb_encode() and pb_decode(). These fields are memory pointers, and are generated depending on the message definition in the .proto file.
2. The automatically generated field definitions, i.e. pb_msgdesc_t.
3. The pb_decode_ctx_t and pb_encode_ctx_t structures (this does not mean the contents of the stream itself, just the pointers and other fields in the context structure).

The following data is regarded as untrusted. Invalid/malicious data in these will cause “garbage in, garbage out” behaviour. It will not cause buffer overflows, information disclosure or other security problems:

1. All data read from the input stream represented by pb_decode_ctx_t.
2. All fields in message structures, except:
   • callbacks (pb_callback_t structures)
   • pointer fields and _count fields for pointers
   • extensions (pb_extension_t structures)

Invariants

The following invariants are maintained during operation, even if the untrusted data has been maliciously crafted:

1. Nanopb will never read more than bytes_left bytes from pb_decode_ctx_t.
2. Nanopb will never write more than max_size bytes to pb_encode_ctx_t.
3. Nanopb will never access memory out of bounds of the message structure.
4. After pb_decode() returns successfully, the message structure will be internally consistent:
   • The count fields of arrays will not exceed the array size.
   • The size field of bytes will not exceed the allocated size.
   • All string fields will have null terminator.
   • bool fields will have valid true/false values
   • pointer fields will be either NULL or point to valid data
5. After pb_encode() returns successfully, the resulting message is a valid protocol buffers message. (Except if user-defined callbacks write incorrect data.)
6. All memory allocated by pb_decode() will be released by a subsequent call to pb_release() on the same message.

Further considerations

Even if the nanopb library is free of any security issues, there are still several possible attack vectors that the application author must consider. The following list is not comprehensive:

1. Stack usage may depend on the contents of the message. The message definition places an upper bound on how much stack will be used. Tests should be run with all fields present, to record the maximum possible stack usage.
2. Callbacks can do anything. The code for the callbacks must be carefully checked if they are used with untrusted data.
3. If using stream input, a maximum size should be set in pb_decode_ctx_t to stop a denial of service attack from using an infinite message.
4. If using network sockets as streams, a timeout should be set to stop denial of service attacks.
5. If using malloc() support, some method of limiting memory use should be employed. This can be done by defining custom pb_realloc() function. Nanopb will properly detect and handle failed memory allocations.

Security hardening

Following steps will add deference-in-depth protections that make successful exploitation harder, even if there were an undiscovered vulnerability in nanopb or in the user application:

1. Enable compiler memory protection features, such as -fstack-protector or -fsanitize=address.

2. Disable unneeded features using compilation option PB_MINIMAL or the individual feature disables. This reduces the attack surface. In particular the following options should be considered for security-critical applications:

   • PB_NO_STRUCT_FIELD_CALLBACK = 1: Storing callbacks inline with structure data creates a pathway to exploitation on a buffer overflow.
   • PB_NO_EXTENSIONS = 1: To a lesser extent, the pointer to extension definition can be used as exploitation pathway after another bug has caused memory corruption.
   • PB_NO_RECURSION = 1: Disabling recursion makes the stack usage bounded and largely independent of message contents.
   • PB_NO_FUNCTION_POINTERS = 1: Disabling function-pointer usage reduces exploitation methods if a buffer overflow were to occur.
   • PB_NO_MALLOC = 1: Disable dynamic allocation if it is not needed. Alternatively, use a custom arena allocator to reduce risks of heap corruption or malicious heap fragmentation.
   • PB_NO_OPT_ASSERT = 0: Keeping optional asserts enables can detect internal errors before they lead to a security problem.

3. Set max_size limits to all output streams and bytes_left limits to input streams. The generator creates MyMessage_size defines that indicate the maximum expected size for a particular message type.

4. Perform fuzzing against your application. Start with valid messages and use a profiling-guided fuzzer such as afl-fuzz to trigger otherwise dormant code paths.

Nanopb: API reference

The sections below document the full user-facing API of the nanopb library.

3. pb.h header: Data types and utility macros used by the whole project.
4. pb_encode.h header: Functions used for encoding protobuf messages.
5. pb_decode.h header: Functions used for decoding protobuf messages and for memory allocation.
6. pb_common.h header: Functions used for processing message descriptors and common utility functions.

API reference: pb.h

The pb.h file contains type and macro definitions shared between nanopb components. It also validates feature enables for preprocessor #if directives and includes the needed system headers.

Data types

pb_allocator_t

An extensible structure for providing a custom allocator to nanob decoder functions. The allocator is used when generator options set a field type to FT_POINTER.

typedef struct pb_allocator_s pb_allocator_t;
struct pb_allocator_s {
    void* (*realloc)(pb_allocator_t *actx, void *ptr, size_t size);
    void (*free)(pb_allocator_t *actx, void *ptr);
    void *state;
};

The two functions follow the behavior of standard C realloc() and free() functions.

The actx arguments is a pointer to the pb_allocator_t structure, which the allocator implementation can use to store its state. The free pointer field state is for user code, it is not used by nanopb. Alternatively the user code can extend the structure definition.

See [tests/custom_allocator](tests/custom_allocator] for an example implementation of an arena allocator.

Custom realloc() implementation

If the ptr argument is NULL, realloc() will make a new memory allocation of size bytes.

If the ptr argument is not NULL, the size of the existing allocation is adjusted to new size. Up to size bytes of data from the old allocation are retained. The realloc() implementation is allowed to make a new allocation, copy the data to the new location and release the old allocation.

Nanopb does not call realloc() with size = 0.

The function returns either NULL on failed allocation, or a valid pointer to an allocation of at least size bytes long.

Custom free() implementation

Releases a previously made allocation. The ptr argument is the return value from a previous call to realloc().

Nanopb does not call free() with ptr = NULL.

pb_byte_t

Type used for storing byte-sized data, such as raw binary input and bytes-type fields.

typedef uint_least8_t pb_byte_t;

For most platforms this is equivalent to uint8_t. Some platforms however do not support 8-bit variables, and on those platforms 16 or 32 bits need to be used for each byte.

pb_tag_t

Type used for storing field tag numbers. By default this is uint32_t:

typedef uint32_t pb_tag_t;

If PB_NO_LARGEMSG build option is given, then uint_least16_t is used instead. This saves some stack space and is sufficient due to the 4095 tag number limit for small messages.

pb_size_t

Type used for storing stream, field and array sizes.tag numbers and sizes of message fields. By default this is equivalent to size_t:

typedef size_t pb_size_t;

If PB_NO_LARGEMSG build option is given, then uint_least16_t is used instead. This saves some stack space and is sufficient due to the 4 kB limit for small messages.

Optionally PB_SIZE_T_OVERRIDE can be used to customize the type. For example, uint8_t could be used for processing tiny messages on a 8-bit platform.

pb_fieldidx_t

Type used for storing index of a field in the protobuf message descriptors. Equivalent to uint_least16_t:

typedef uint_least16_t pb_fieldidx_t;

The value is multiplied by 5 when accessing large format descriptors. This makes a 16-bit type sufficient for up to 13000 fields. According to Google’s Proto Limits, typical protobuf implementations are limited to about 4000 fields per message, so this should be sufficient.

pb_type_t

Type used to store the type of each field, to control the encoder/decoder behaviour.

typedef uint_least16_t pb_type_t;

The macros PB_LTYPE(), PB_HTYPE() and PB_ATYPE() can be used to access individual components of the field type.

The low-order nibble of the enumeration values defines the function that can be used for encoding and decoding the field data:

The bits 4-5 define whether the field is required, optional or repeated. There are separate definitions for semantically different modes, even though some of them share values and are distinguished based on values of other fields:

The bits 6-7 define the how the storage for the field is allocated:

Upper bits 8-10 are available in the large-format message descriptors, but they are reserved for future use.

pb_msgdesc_t

Autogenerated structure that contains information about a message and pointers to the field descriptors. Use functions defined in pb_common.h to process the field information.

typedef struct pb_msgdesc_s pb_msgdesc_t;
struct pb_msgdesc_s {
    pb_size_t struct_size;
    pb_fieldidx_t field_count;
    pb_fieldidx_t required_field_count;
    pb_tag_t largest_tag;
    pb_msgflag_t msg_flags;
    const uint32_t *field_info;
    const pb_msgdesc_t * const * submsg_info;
    const pb_byte_t *default_value;

    bool (*field_callback)(pb_decode_ctx_t *istream, pb_encode_ctx_t *ostream, const pb_field_iter_t *field);
};

pb_msgflag_t

Message flags are stored in the pb_msgdesc_t and provide high-level metadata about the message. They are primarily used for nanopb code to skip unnecessary steps if the message does not contain particular features.

User code can use the flags to e.g. check if the message contains any pointers that need to be released.

Currently defined message flags are listed below. Flags with PB_MSGFLAG_R_ are recursive, they get set on the parent message if they apply to any submessages.

pb_field_iter_t

Describes a single structure field with memory position in relation to others. The field information is stored in a compact format and loaded into pb_field_iter_t by the functions defined in pb_common.h, such as pb_field_iter_next().

typedef struct pb_field_iter_s pb_field_iter_t;
struct pb_field_iter_s {
    const pb_msgdesc_t *descriptor;
    void *message;

    pb_fieldidx_t index;
    pb_fieldidx_t required_field_index;
    pb_fieldidx_t submessage_index;
    pb_fieldidx_t field_info_index;

    pb_tag_t tag;
    pb_size_t data_size;
    pb_size_t array_size;
    pb_type_t type;

    void *pField;
    void *pData;
    void *pSize;

    const pb_msgdesc_t *submsg_desc;
};

pb_bytes_array_t

An byte array with a field for storing the length:

typedef struct {
    pb_size_t size;
    pb_byte_t bytes[1];
} pb_bytes_array_t;

In an actual array, the length of bytes is set by generator options. The macros PB_BYTES_ARRAY_T() and PB_BYTES_ARRAY_T_ALLOCSIZE() are used to allocate variable length storage for bytes fields.

pb_callback_t

Part of a message structure, for fields with type PB_HTYPE_CALLBACK:

typedef struct pb_callback_s pb_callback_t;
struct pb_callback_s {
    union {
        bool (*decode)(pb_decode_ctx_t *stream, const pb_field_iter_t *field, void **arg);
        bool (*encode)(pb_encode_ctx_t *stream, const pb_field_iter_t *field, void * const *arg);
    } funcs;

    void *arg;
};

A pointer to the arg is passed to the callback when calling. It can be used to store any information that the callback might need. Note that this is a double pointer. If you set field.arg to point to &data in your main code, in the callback you can access it like this:

myfunction(*arg);           /* Gives pointer to data as argument */
myfunction(*(data_t*)*arg); /* Gives value of data as argument */
*arg = newdata;             /* Alters value of field.arg in structure */

When calling pb_encode(), funcs.encode is used, and similarly when calling pb_decode(), funcs.decode is used. The function pointers are stored in the same memory location but are of incompatible types. You can set the function pointer to NULL to skip the field.

pb_wire_type_t

Protocol Buffers wire types. These are used with pb_encode_tag():

typedef enum {
    PB_WT_VARINT = 0,
    PB_WT_64BIT  = 1,
    PB_WT_STRING = 2,
    PB_WT_32BIT  = 5
} pb_wire_type_t;

pb_extension_t

Ties together the extension field type and the storage for the field value. For message structs that have extensions, the generator will add a pb_extension_t* field. It should point to a linked list of extensions.

typedef struct {
    const pb_msgdesc_t *type;
    void *dest;
    pb_extension_t *next;
    bool found;
} pb_extension_t;

Utility macros

PB_GET_ERROR

Get the current error message from a context, or a placeholder string if there is no error message:

#define PB_GET_ERROR(ctx) (string expression)

The ctx can have type pb_encode_ctx_t* or pb_decode_ctx_t*.

If the error message is not set, "(none)" is returned. If error messages are disabled with PB_NO_ERRMSG, "(errmsg disabled)" is returned.

This should be used for printing errors, for example:

if (!pb_decode(&ctx, ...))
{
    printf("Decode failed: %s\n", PB_GET_ERROR(&ctx));
}

The macro only returns pointers to constant strings (in code memory), so that there is no need to release the returned pointer.

PB_SET_ERROR

Set the error message if it has not been set yet.

#define PB_SET_ERROR(ctx, msg) (set errmsg if it is null)

If multiple errors occur (for example IO error followed by failed decoding), the first error message will persist.

The msg parameter must be a constant string.

PB_RETURN_ERROR

Set the error message if it is not already set, and return false:

#define PB_RETURN_ERROR(ctx, msg) (PB_SET_ERROR() and returns false)

This should be used to handle error conditions inside nanopb functions and user callback functions:

if (error_condition)
{
    PB_RETURN_ERROR(ctx, "something went wrong");
}

The msg parameter must be a constant string.

PB_READ_ERROR

Sentinel value used by pb_decode_ctx_t stream callbacks to indicate error condition.

#define PB_READ_ERROR PB_SIZE_MAX

The largest value representable by pb_size_t is reserved for error indication by pb_init_decode_ctx_for_callback() limiting the stream length to PB_SIZE_MAX - 1.

PB_BIND

This macro generates the pb_msgdesc_t and associated arrays, based on a list of fields in X-macro format. :

#define PB_BIND(msgname, structname, width) ...

This macro is automatically invoked inside the autogenerated .pb.c files. User code can also call it to bind message types with custom structures or class types.

pb_arraysize()

Get the number of entries in an array-type member of a structure. Example usage:

MyMessage msg;
for (int i = 0; i < pb_arraysize(MyMessage, my_repeated_int); i++)
{
    printf("%d\n", msg.my_repeated_int[i]);
}

PB_CONST_CAST

Nanopb encoding functions only read from the message structure and thus take const pointer to it. But internally a part of the logic is shared, resulting in a need to cast away the const qualifier. This is done with care, so that no writes are done through the pointer.

The macro implementation uses uintptr_t as an intermediate type to avoid warnings on most compilers:

#define PB_CONST_CAST(x) ((void*)(uintptr_t)(x))

PB_UNUSED

This macro is used to supress compiler warnings about unused function arguments. Typically needed when disabled features result in some arguments being ignored.

#define PB_UNUSED(x) (void)(x)

PB_OPT_ASSERT

Optional assertions that are useful for early detection of problems. This is in particular targeted for nanopb developers, though the assertions do not have significant runtime cost either. All security properties are fulfilled even if assertions are disabled using PB_NO_OPT_ASSERT.

#define PB_OPT_ASSERT(cond) assert(cond)

PB_STATIC_ASSERT

Compile-time assertions, using either C11 _Static_assert keyword or the negative-size-array mechanism in C99.

#define PB_STATIC_ASSERT(COND,MSG) _Static_assert(COND,#MSG);

The implementation of static assertions can be controlled using build options PB_NO_STATIC_ASSERT and PB_C99_STATIC_ASSERT.

API reference: pb_encode.h

Nanopb functionality related to encoding protobuf messages is provided in the pb_encode.h header. It contains the definitions for the encoding context, writing to output streams and encoding the data for message fields.

Message encoding

pb_encode

Encodes the contents of a structure as a protocol buffers message and writes it to output stream.

#define pb_encode(ctx, msgdesc, src_struct) \
    pb_encode_s(ctx, msgdesc, src_struct, sizeof(*src_struct))

This is the main function for the encoding process. It will walk through all fields in the message structure and encodes them to the output stream.

To provide some type safety, the function is implemented as a macro wrapper around pb_encode_s(). The macro uses sizeof() to ensure that the size of the structure passed matches the size in the descriptor. This catches most bugs where a descriptor for wrong message type is accidentally given.

The encoder behavior can be controlled by modifying ctx->flags before calling it.

pb_encode_s

The actual implementation of the encoding function. Prefer pb_encode() macro when possible. Calling pb_encode_s() directly may be necessary if the structure type is not available at the point of code where pb_encode() is called.

bool pb_encode_s(pb_encode_ctx_t *ctx, const pb_msgdesc_t *msgdesc,
                const void *src_struct, size_t struct_size);

pb_get_encoded_size

Calculate the size of a message, but do not store the encoded data.

#define pb_get_encoded_size(size, msgdesc, src_struct) ...
bool pb_get_encoded_size_s(size_t *size, const pb_msgdesc_t *msgdesc,
                       const void *src_struct, size_t struct_size);

Equivalent to using pb_init_encode_ctx_sizing() and pb_encode().

Output stream management

Encoded data is written to an output stream, which can either be a memory buffer or a custom callback function. It is possible to encode messages larger than the available RAM if you provide a stream callback function that directly writes to some external communication interface.

pb_encode_ctx_t

The encode context contains the output stream definition, and other data associated with the encoding process. Use one of the pb_init_encode_ctx_* functions to initialize it.

After initialization, you can manually set e.g. field_callback or flags.

typedef struct pb_encode_ctx_s pb_encode_ctx_t;
struct pb_encode_ctx_s
{
    pb_encode_ctx_flags_t flags;

    pb_size_t max_size;
    pb_size_t bytes_written;
    pb_byte_t *buffer;

    pb_walk_state_t *walk_state;

    pb_encode_ctx_write_callback_t stream_callback;
    void *stream_callback_state;
    pb_size_t buffer_size;
    pb_size_t buffer_count;

    const char *errmsg;

    pb_encode_ctx_field_callback_t field_callback;
    void *field_callback_state;
};

pb_encode_ctx_flags_t

Encoding context flags control the behavior of stream output functions. Some of them are automatically set internally, but they can also be set by user code.

Stream max_size and bytes_written

The max_size and bytes_written track the total number of bytes written to the output stream.

When encoding happens into a memory buffer, the max_size equals the size of the buffer and bytes_written is the count of bytes written so far. The buffer always points to the beginning of the output buffer.

If a stream callback is provided, the buffer acts as temporary storage. Separate buffer_size and buffer_count track the number of bytes currently in the buffer. Once the buffer fills up, pb_flush_write_buffer() calls stream_callback to write out the data, and buffer_count resets to 0.

The temporary storage speeds up encoding of any submessages small enough to fit in it. Larger submessages need to be encoded twice: first to determine the submessage length and then to write out the data.

Walk state

The walk_state field is internally set by the first pb_encode() call. It is used to share the pb_walk() stack when user callbacks recursively call pb_encode_submessage().

Stream callback

The stream callback is a user-provided function, which takes encoded data from nanopb and writes it to some external communication interface such as network, serial port or filesystem.

bool my_stream_write_callback(pb_encode_ctx_t *ctx, const pb_byte_t *buf, pb_size_t count);

The callback should write exactly count bytes from buf to the external interface. Value of count is at most ctx->max_size - ctx->bytes_written.

The callback should return true on success and false on IO errors. Error return will cause encoding to abort. The callback can optionally set ctx->errmsg using PB_RETURN_ERROR(), otherwise pb_write() will set it to "io error".

The callback can use stream_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The context variables such as bytes_written are updated by pb_write() after the callback returns.

There is an example of a custom stream callback in examples/network_server/common.c.

Error message

The primary status indication is the bool return value for success or failure. There is an optional error message in the ctx->errmsg, but this is for debugging purposes only and can be disabled with the PB_NO_ERRMSG build option.

Field callback

Nanopb uses field callbacks to handle message fields with unknown or large size. The callback function is provided by user code and can directly write to the output stream.

The field_callback function specified in the encoding context receives a call for each field that has FT_CALLBACK type. It can write data using the manual encoding functions, or just do nothing. The callback should return false on errors, which will terminate the encoding process.

The callback can use ctx->field_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The data in the C structure can be accessed by pointers in pb_field_iter_t. The callback can determine which field is being encoded by comparing field->descriptor and field->tag. For example:

bool my_field_callback(pb_encode_ctx_t *ctx, const pb_field_iter_t *field)
{
    if (field->descriptor == &SubMessage_msg && field->tag == SubMessage_stringvalue_tag)
    {
        return pb_encode_tag_for_field(ctx, field) &&
               pb_encode_varint(ctx, 42);
    }

    return true;
}

....

pb_encode_ctx_t ctx;
pb_init_encode_ctx_for_buffer(&ctx, ...);
ctx.field_callback = &my_field_callback;

pb_init_encode_ctx_for_buffer

Initializes an encoding context for writing into a memory buffer. The output buffer has to be large enough to contain the whole encoded message.

void pb_init_encode_ctx_for_buffer(pb_encode_ctx_t *ctx, pb_byte_t *buf, pb_size_t bufsize);

After writing, you can check ctx.bytes_written to find out how much valid data there is in the buffer. This should be passed as the message length on decoding side.

pb_init_encode_ctx_for_callback

Initializes an encoding context for passing data to a custom callback function.

void pb_init_encode_ctx_for_callback(pb_encode_ctx_t *ctx,
    pb_encode_ctx_write_callback_t stream_callback, void *stream_callback_state,
    pb_size_t max_size, pb_byte_t *buf, pb_size_t bufsize);

Providing a temporary buffer is optional. If the buffer is provided, it is used to temporarily store encoded submessages, so that their length prefix can be written after the encoded length is known. Without a temporary buffer, submessages will are internally processed twice to determine the length and then write the data.

pb_init_encode_ctx_sizing

Initializes an encoding context for just counting the number of bytes that would have been written. The encoded data is discarded.

void pb_init_encode_ctx_sizing(pb_encode_ctx_t *ctx);

After encoding, the number of encoded bytes can be read from ctx.bytes_written.

pb_flush_write_buffer

For encoding context using a stream callback and a temporary buffer, the data might not be immediately passed to the callback. The flushing function empties the temporary buffer. This function has no effect for contexts that write to a memory buffer directly.

bool pb_flush_write_buffer(pb_encode_ctx_t *ctx);

The flushing is automatically done at the end of a pb_encode() call. Calling this function manually is only necessary if you call pb_write() manually.

pb_write

Writes data to the output stream associated with the context.

bool pb_write(pb_encode_ctx_t *ctx, const pb_byte_t *buf, pb_size_t count);

Before writing the data, this function checks that the stream max_size wouldn’t be exceeded. If an error happens, false is returned, errmsg is set and bytes_written is not incremented.

For memory streams, this copies the data to the output buffer and increments the bytes_written count.

For sizing streams, only bytes_written is incremented.

For callback streams, the data may be either copied to a temporary buffer or passed directly to the stream callback function. If a temporary buffer was provided, calling pb_flush_write_buffer() is needed to make sure the data will be passed to the callback function after a manual pb_write() call. The pb_encode() automatically flushes the buffer at the end of the message.

Manual field encoding

The functions with names pb_encode_<datatype> are used for encoding single fields manually. Typically this is used when writing custom field callbacks, but it can also be used for fully manual control over the encoding process.

For example an array of unlimited size could be encoded like this:

while(1)
{
    const char *str = generate_next_string();
    if (!str) break;

    if (!pb_encode_tag_for_field(ctx, field))
        return false;

    if (!pb_encode_string(ctx, str, strlen(str)))
        return false;
}

This will work equivalently both when used in field callbacks, or when used without actually calling pb_encode() at all.

The tag of a field must be encoded first with pb_encode_tag_for_field() or pb_encode_tag(). After that, you can call exactly one of the content-writing functions to encode the payload of the field. For repeated fields, you can repeat the tag + payload combination multiple times.

Writing packed arrays is a little bit more involved: you need to use pb_encode_tag and specify PB_WT_STRING as the wire type. Then you need to know exactly how much data you are going to write, and use pb_encode_varint to write out the number of bytes before writing the actual data.

Important

See Google Protobuf Encoding Format Documentation for background information on the Protobuf wire format.

pb_encode_tag

Starts a field in the Protocol Buffers binary format: encodes the field number and the wire type of the data.

bool pb_encode_tag(pb_encode_ctx_t *ctx, pb_wire_type_t wiretype, pb_tag_t field_number);

The tag number is defined in the .proto file. In nanopb, it is available from pb_field_iter_t, or from the MyMessage_myfield_tag defines in the generated .pb.h file.

pb_encode_tag_for_field

Same as pb_encode_tag(), except takes the parameters from a pb_field_iter_t structure.

bool pb_encode_tag_for_field(pb_encode_ctx_t *ctx, const pb_field_iter_t *field);

This function only considers the PB_LTYPE of the field pb_type_t.

Wire type mapping is as follows:

pb_encode_varint

Encodes a signed or unsigned integer in the varint format. Works for fields of type bool, enum, int32, int64, uint32 and uint64:

bool pb_encode_varint(pb_encode_ctx_t *ctx, uint64_t value);

Note

Value will be converted to uint64_t in the argument. To encode signed values, the argument should be cast to int64_t first for correct sign extension.

pb_encode_svarint

Encodes a signed integer in the zig-zagged format. Works for fields of type sint32 and sint64:

bool pb_encode_svarint(pb_encode_ctx_t *ctx, int64_t value);

Parameters are the same as for pb_encode_varint()

pb_encode_string

Writes the length of a string as varint and then contents of the string. Works for fields of type bytes and string:

bool pb_encode_string(pb_encode_ctx_t *ctx, const pb_byte_t *buffer, size_t size);

This function does not perform UTF-8 validation, which is separately available using pb_validate_utf8().

pb_encode_fixed32

Writes 4 bytes to stream and swaps bytes on big-endian architectures. Works for fields of type fixed32, sfixed32 and float:

bool pb_encode_fixed32(pb_encode_ctx_t *ctx, const void *value);

pb_encode_fixed64

Writes 8 bytes to stream and swaps bytes on big-endian architecture. Works for fields of type fixed64, sfixed64 and double:

bool pb_encode_fixed64(pb_encode_ctx_t *ctx, const void *value);

pb_encode_float_as_double

Encodes a 32-bit float value so that it appears like a 64-bit double in the encoded message. This is sometimes needed when platforms like AVR that do not support 64-bit double need to communicate using a message type that contains double fields.

bool pb_encode_float_as_double(pb_encode_ctx_t *ctx, float value);

This function is only included if PB_CONVERT_DOUBLE_FLOAT build option is set.

pb_encode_submessage

Encodes a submessage field, including the size header for it. Works for fields of any message type.

bool pb_encode_submessage(pb_encode_ctx_t *ctx, const pb_msgdesc_t *msgdesc, const void *src_struct);

This function is similar to calling pb_encode() with PB_ENCODE_CTX_FLAG_DELIMITED, but it differs in that it attempts to reuse the same pb_walk() context. This reduces the total stack space necessary for deep message hierarchies.

In Protocol Buffers format, the submessage size must be written before the submessage contents. For efficiency, this function attempts to write the encoded data to memory and then prepend the size afterwards. This requires only one pass over the message data.

When stream callbacks are used, it may be that the submessage does not fit in the provided temporary buffer. In this case the function resorts to a two-pass approach, where the first pass only calculates the size. If the submessage contains callback fields, the callback function might misbehave and write out a different amount of data on the second pass. This situation is recognized and false is returned, but garbage will be written to the output before the problem is detected.

API reference: pb_decode.h

Nanopb functionality related to decoding protobuf messages is provided in the pb_decode.h header. It contains the definitions for the decoding context, reading from input streams and decoding the message data into C structures.

The decoding module also handles dynamic memory allocation, if it is enabled by build options.

Message decoding

pb_decode

Read and decode all fields of a structure. Reads until EOF on input stream.

#define pb_decode(ctx, msgdesc, dest_struct) \
    pb_decode_s(ctx, msgdesc, dest_struct, sizeof(*dest_struct))

This is the main function for the decoding process. It will read one protobuf field tag from the input at a time, find the corresponding field in the descriptor and decode the data into the C structure.

To provide some type safety, the function is implemented as a macro wrapper around pb_decode_s(). The macro uses sizeof() to ensure that the size of the structure passed matches the size in the descriptor. This catches most bugs where a descriptor for wrong message type is accidentally given.

Message length

Usually the message length is provided when calling pb_init_decode_ctx_for_buffer(). For callback streams, the length can alternatively be indicated by the callback reporting EOF condition when it detects end of data from an external source.

In Protocol Buffers binary format, end-of-file is only allowed between fields. If it happens anywhere else, pb_decode will return false. On failure return the structure will have an internally consistent state, but the field contents may be partial.

Control flags and field initialization

The decoder behavior can be controlled by modifying ctx->flags before calling it.

By default pb_decode() initializes all fields, either to the default value declared in .proto or to a zero value. Setting the context flag PB_DECODE_CTX_FLAG_NOINIT skips the initialization. This can be used for merging protobuf messages or if you have already initialized the structure.

Dynamic allocation

If the generator options have been used to set fields to FT_POINTER type, this function will allocate storage for them. The allocation will use either the ctx->allocator or the default allocator (by default realloc()).

For messages containing pointers (those that have PB_MSGFLAG_R_HAS_PTRS), you must call pb_release() to release the allocation after you are done with the structure. On error, pb_decode() will call pb_release() before returning false.

pb_decode_s

The actual implementation of the decoding function. Prefer pb_decode() macro when possible. Calling pb_decode_s() directly may be necessary if the structure type is not available at the point of code where the call happens.

bool pb_decode_s(pb_decode_ctx_t *ctx, const pb_msgdesc_t *msgdesc,
                 void *dest_struct, size_t struct_size);

Dynamic memory allocation

pb_release

Releases all dynamically allocated fields in a message.

#define pb_release(ctx, msgdesc, dest_struct) ...
bool pb_release_s(pb_decode_ctx_t *ctx, const pb_msgdesc_t *msgdesc,
                  void *dest_struct, size_t struct_size);

Similarly to pb_decode(), pb_release() has a wrapping macro to check the structure size. The actual implementation is in pb_release_s().

The release function goes through all FT_POINTER type fields in the message, releases the storage associated with them and sets the pointer to NULL. Either the allocator given by ctx or the default allocator is used.

If the build option PB_NO_MALLOC is defined, this function does nothing. Similarly if the message descriptor does not have PB_MSGFLAG_R_HAS_PTRS or if the dest_struct pointer is NULL, the function returns true without doing anything.

This function is safe to call multiple times, calling it again does nothing.

The release function normally always succeeds. It can return failure if the message structure is deeper than maximum nesting depth or if struct_size does not match msgdesc->struct_size.

pb_allocate_field

Allocate or reallocate storage for a single field.

bool pb_allocate_field(pb_decode_ctx_t *ctx, void **ptr, pb_size_t data_size, pb_size_t array_size);

This function either newly allocates memory or changes the size of an existing allocation. The total allocation size is data_size * array_size, but this function internally checks for overflow in the multiplication.

This function uses either the default allocator, or the allocator defined by ctx->allocator.

The initial value of *ptr should be NULL if there is no previous allocation. Otherwise the previous allocation is resized using realloc(), which may also change the value of *ptr.

On allocation failure, false is returned and *ptr remains unchanged.

Example usage:

typedef struct MyMessage {
    pb_size_t myarray_count;
    int32_t *myarray;
} MyMessage;

MyMessage msg = MyMessage_init_zero;
msg.myarray_count = 10;

if (!pb_allocate_field(ctx, &msg.myarray, sizeof(int32_t), msg.myarray_count))
    return false;

pb_release_field

Release storage previously allocated by pb_allocate_field().

void pb_release_field(pb_decode_ctx_t *ctx, void *ptr);

This function uses either the default allocator, or the allocator defined by ctx->allocator.

Input stream management

Protobuf data to be decoded is read from an input stream, which can be either a memory buffer or a custom callback function. It is possible to decode messages larger than the available RAM if you provide a stream callback function that directly reads from some external communication interface.

pb_decode_ctx_t

The decode context contains the input stream definition and other data associated with the decoding process. Use one of pb_init_decode_ctx_* functions to initialize it.

struct pb_decode_ctx_s
{
    pb_decode_ctx_flags_t flags;

    pb_size_t bytes_left;
    const pb_byte_t *rdpos;

    pb_walk_state_t *walk_state;

    pb_decode_ctx_read_callback_t stream_callback;
    void *stream_callback_state;
    pb_byte_t *buffer;
    pb_size_t buffer_size;

    const char *errmsg;

    pb_allocator_t *allocator;

    pb_decode_ctx_field_callback_t field_callback;
    void *field_callback_state;
};

pb_decode_ctx_flags_t

Decoding context flags control the behavior of stream output and decoding functions. They can be set by user before calling pb_decode().

Stream bytes_left and read position

The bytes_left is an upper bound on the remaining bytes in the stream. For the top-level stream with memory buffers it is the total length of the input message. For submessages, nanopb will temporarily set bytes_left to the length of the submessage data. For callback streams, the callback function can indicate end of message before bytes_left has reached 0.

For memory buffers, the rdpos pointer is the current read position in the buffer. For callback streams it can be either NULL or point to a temporary input buffer.

Walk state

The walk_state field is internally set by the first pb_decode() call. It is used to share the pb_walk() stack when user callbacks recursively call pb_decode().

Stream callback

The stream callback is a user-provided function, which reads data from some external communication interface such as network, serial port or filesystem. Nanopb will then decode this data.

pb_size_t my_stream_read_callback(pb_decode_ctx_t *ctx, pb_byte_t *buf, pb_size_t count);

The callback should read up to count bytes into buf from the external source, and return the number of bytes read. Value of count is always smaller than PB_SIZE_MAX and at most ctx->bytes_left. Any return value less than count is interpreted as end-of-stream.

On errors, return PB_READ_ERROR. This will cause decoding to abort. You can optionally set ctx->errmsg using PB_SET_ERROR(). Otherwise pb_read() will set it to "io error".

The callback can use stream_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The context variables such as bytes_left are updated by pb_read() after the callback returns.

There is an example of a custom stream callback in examples/network_server/common.c.

Error message

The primary status indication is the bool return value for success or failure. There is an optional error message in the ctx->errmsg, but this is for debugging purposes only and can be disabled with the PB_NO_ERRMSG build option.

Field callback

Nanopb uses field callbacks to handle message fields with unknown or large size. The callback function is provided by user code and decodes data directly from the input stream.

The field_callback function specified in the decoding context receives a call for each field that has FT_CALLBACK type. It can read data using the manual decoding functions. The callback should return false on errors, which will terminate the decoding process.

The field tag has already been decoded before the callback is called, and the ctx->bytes_left is set to the payload length of the field.

The context field callback is also called for any unknown fields which are not present in the descriptor generated from .proto file. For unknown fields, the field tag is available in field->tag as normal, and field->type is set to a PB_LTYPE value corresponding to the wire type, but other values such field->pData are just zeroed. The callback can simply return true for any fields it doesn’t know, and nanopb will discard the field data.

The callback can use ctx->field_callback_state member variable to store your own data. Alternatively you can extend the structure to add your own fields.

The data in the C structure can be written using pointers in pb_field_iter_t. The callback can determine which field is being decoded by comparing field->descriptor and field->tag. For example:

bool my_field_callback(pb_decode_ctx_t *ctx, const pb_field_iter_t *field)
{
    if (field->descriptor == &SubMessage_msg && field->tag == SubMessage_stringvalue_tag)
    {
        // Decode our callback field
        uint64_t value;
        if (!pb_decode_varint(ctx, &value))
            return false;

        printf("Value: %d\n", (int)value);
        return true;
    }

    // Unknown field, just skip it
    return true;
}

....

pb_decode_ctx_t ctx;
pb_init_decode_ctx_for_buffer(&ctx, ...);
ctx.field_callback = &my_field_callback;

pb_init_decode_ctx_for_buffer

Initializes an encoding context for reading from a memory buffer. The length of the message data must be known beforehand.

void pb_init_decode_ctx_for_buffer(pb_decode_ctx_t *ctx, const pb_byte_t *buf, pb_size_t msglen);

Note

msglen is not the length of the buffer, it must be the length of actual message data. Protobuf messages contain no indication of where the message ends, so the length must be known by other means.

pb_init_decode_ctx_for_callback

Initializes an encoding context for passing data to a custom callback function.

void pb_init_decode_ctx_for_callback(pb_decode_ctx_t *ctx,
    pb_decode_ctx_read_callback_t stream_callback, void *stream_callback_state,
    pb_size_t max_msglen, pb_byte_t *buf, pb_size_t bufsize);

Providing a temporary buffer is optional. If provided, it is used to coalesce read requests so that a larger amount of data can be read in one IO operation. It is also used as a read-ahead buffer for optimizing memory allocation sizes when dynamic allocation is used.

The max_msglen is an upper bound for the message size. The callback can indicate end-of-stream earlier by its return value. It is recommended to set a reasonable maximum length to avoid denial-of-service through excessively long input messages from e.g. a network socket.

Maximum possible length for callback stream is PB_SIZE_MAX - 1, because the highest value is reserved for PB_READ_ERROR. This initialization function automatically limits the max_msglen value.

pb_read

Read raw data from input stream.

bool pb_read(pb_decode_ctx_t *ctx, pb_byte_t *buf, pb_size_t count);

End of stream is signalled by stream->bytes_left being zero after pb_read() returns false.

For callback streams, any data in the temporary buffer is returned first, and the stream callback is called for any remaining requested count. The pb_read() call never puts data into the temporary buffer. The buffer is only filled internally at points where the remaining length of a protobuf element is known.

Manual field decoding

The functions with names pb_decode_<datatype> are used for decoding single fields manually. Typically this is used when writing custom field callbacks, but it can also be used for fully manual control over the decoding process.

For callback fields, the field tag is decoded by nanopb before the callback function is called. For fully manual decoding, pb_decode_tag() can be used to read the field identifier, after which the field data can be decoded.

To decode numeric (including enumerated and boolean) values, use pb_decode_varint, pb_decode_svarint, pb_decode_fixed32 and pb_decode_fixed64. They take a pointer to a 32- or 64-bit C variable, which you may then cast to smaller datatype for storage.

For decoding strings and bytes fields, the length has already been decoded and the callback function is given a length-limited substream. You can therefore check the total length in stream->bytes_left and read the data using pb_read.

Finally, for decoding submessages in a callback, use pb_decode() and pass it the submessage message descriptor.

Important

See Google Protobuf Encoding Format Documentation for background information on the Protobuf wire format.

pb_decode_tag

Decode the tag that comes before each field in the protobuf format:

bool pb_decode_tag(pb_decode_ctx_t *stream, pb_wire_type_t *wire_type, pb_tag_t *tag, bool *eof);

When the message (stream) ends, this function will return false and set eof to true. On other errors, eof will be set to false.

pb_skip_field

Discard the data for a field from the stream, without actually decoding it:

bool pb_skip_field(pb_decode_ctx_t *stream, pb_wire_type_t wire_type);

This function determines the amount of bytes to read based on the wire type. For PB_WT_STRING, it will read the length prefix of a string or submessage to determine its length.

pb_decode_varint

Read and decode a varint encoded integer.

bool pb_decode_varint(pb_decode_ctx_t *stream, uint64_t *dest);

pb_decode_varint32

Same as pb_decode_varint, but limits the value to 32 bits:

bool pb_decode_varint32(pb_decode_ctx_t *stream, uint32_t *dest);

Parameters are the same as pb_decode_varint. This function can be used for decoding lengths and other commonly occurring elements that you know shouldn’t be larger than 32 bit. It will return an error if the value exceeds the uint32_t datatype.

pb_decode_bool

Same as pb_decode_varint, but converts value to boolean.

bool pb_decode_bool(pb_decode_ctx_t *ctx, bool *dest);

Any non-zero value is regarded as true.

pb_decode_svarint

Similar to pb_decode_varint, except that it performs zigzag-decoding on the value. This corresponds to the Protocol Buffers sint32 and sint64 datatypes. :

bool pb_decode_svarint(pb_decode_ctx_t *stream, int64_t *dest);

(parameters are the same as pb_decode_varint)

pb_decode_fixed32

Decode a fixed32, sfixed32 or float value.

bool pb_decode_fixed32(pb_decode_ctx_t *stream, void *dest);

This function reads 4 bytes from the input stream. On big endian architectures, it then reverses the order of the bytes. Finally, it writes the bytes to dest.

pb_decode_fixed64

Decode a fixed64, sfixed64 or double value. :

bool pb_decode_fixed64(pb_decode_ctx_t *stream, void *dest);

Same as pb_decode_fixed32, except this reads 8 bytes.

pb_decode_double_as_float

Decodes a 64-bit double value into a 32-bit float variable. This is sometimes needed when platforms like AVR that do not support 64-bit double need to communicate using a message type that contains double fields. Counterpart of pb_encode_float_as_double().

bool pb_decode_double_as_float(pb_decode_ctx_t *stream, float *dest);

This function is only included if PB_CONVERT_DOUBLE_FLOAT build option is set.

pb_decode_open_substream

Decode the length for a field with wire type PB_WT_STRING and open a length-limited substream for reading the data.

bool pb_decode_open_substream(pb_decode_ctx_t *ctx, pb_size_t *old_length);

This function uses pb_decode_varint to read an integer from the stream, which is the length of the STRING type protobuf element. The value is first compared against the remaining bytes_left, and if not enough data is remaining, error is returned. Otherwise ctx->bytes_left is set to the data length.

The remaining length in original stream after the read is stored into old_length for restoring.

pb_decode_close_substream

Close the substream opened with pb_decode_open_substream()

bool pb_decode_close_substream(pb_decode_ctx_t *ctx, pb_size_t old_length)

This function discards any remaining data in the substream and then restores bytes_left to old_length.

pb_common.h

The pb_common module contains shared support functions that are used by both pb_decode.c and pb_encode.c.

User code rarely calls these functions directly, but they can be used for a limited kind of reflection over the field information stored in the message descriptors.

Message descriptor handling

Nanopb stores information about each field in the protobuf messages in a compact representation. The data is written by the generator and the PB_BIND() macro and is stored in a pb_msgdesc_t structure. The functions shown below access this data and provide data for each field as a pb_field_iter_t.

pb_field_iter_begin

Begins iterating over the fields in a message type:

bool pb_field_iter_begin(pb_field_iter_t *iter, const pb_msgdesc_t *msgdesc, void *message);

After successful return, the iterator contains the data for the first field in the message. See pb_field_iter_t in pb.h for definition of the iterator.

pb_field_iter_next

Advance to the next field in the message:

bool pb_field_iter_next(pb_field_iter_t *iter);

When the last field in the message has been processed, this function will return false and initialize iter back to the first field in the message.

pb_field_iter_find

Find a field specified by tag number in the message:

bool pb_field_iter_find(pb_field_iter_t *iter, pb_tag_t tag, pb_extension_t **ext);

This function is functionally identical to calling pb_field_iter_next() until iter.tag equals the searched value. Internally this function avoids fully processing the descriptor for intermediate fields.

If ext parameter is given and no match was found in normal fields, the function searches for a matching extension field. If such is found, iter will point to the field with PB_LTYPE_EXTENSION and *ext will point to the pb_extension_t matching the tag.

pb_field_iter_load_extension

Load field iterator values from an extension:

bool pb_field_iter_load_extension(pb_field_iter_t *iter, pb_extension_t *extension);

This function temporarily sets the values in iter to match the first (and only) field in the extension. The field can then be processed as normal, and next call to pb_field_iter_next() will restart from the beginning of the original message.

pb_get_extension_data_ptr

Returns a pointer to the data for an extension.

void *pb_get_extension_data_ptr(pb_extension_t *extension);

For fields with PB_ATYPE_POINTER, this returns &extension->dest, as the payload pointer is directly stored in the extension structure. This avoids unnecessary extra indirection.

For other field types, this returns extension->dest, which points to the data payload.

pb_ltype_to_wire_type

Returns the wire type in protobuf encoding which corresponds to the field pb_type_t:

pb_wire_type_t pb_ltype_to_wire_type(pb_type_t type);

Returns PB_WT_VARINT, PB_WT_32BIT, PB_WT_64BIT, PB_WT_STRING or PB_WT_INVALID.

Note that for fields with ltype less than PB_LTYPE_LAST_PACKABLE, PB_WT_STRING is also a valid wire type.

Message hierarchy processing

Protobuf messages and their submessages form a tree-like structure. Both encoding and decoding processes need to walk this tree.

Using C recursion for processing the hierarchy increases stack usage, as each level will save return addresses and other runtime information on the stack.

The pb_walk() implements the visitor design pattern, where a callback function is called for each field. The return value then determines whether the processing descends into the submessage or proceeds to next field.

For each message level, minimum amount of information is stored, such as the descriptor and field index of the parent message. A statically sized buffer for up to PB_MESSAGE_NESTING levels is allocated on the stack. Unless recursion is disabled with PB_NO_RECURSION, pb_walk() will automatically call itself to obtain more stack space if the hierarchy is deeper. See stack usage options for details.

A simplified sequence could be like this:

1. Initialization:
   • pb_encode() calls pb_walk_init() and provides the message descriptor
   • pb_encode() calls pb_walk()
2. Top level message processing:
   • pb_walk() calls pb_encode_walk_cb() with state->iter pointing to the first field in the top-level message
   • pb_encode_walk_cb() determines that the field is a submessage, encodes the tag and returns PB_WALK_IN
3. Submessage processing:
   • pb_walk() stores stack frame and calls pb_encode_walk_cb() with the first field in the submessage
   • pb_encode_walk_cb() encodes the submessage field tag and data and returns PB_WALK_OUT
   • pb_walk() restores stack frame and calls pb_encode_walk_cb() with the first field in the top-level message
4. Return to top level message processing:
   • pb_encode_walk_cb() detects state->retval == PB_WALK_OUT and returns PB_WALK_NEXT_ITEM
   • pb_walk() advances the iterator and notices this was the last field. It calls pb_encode_walk_cb() with tag 0.
5. End of message:
   • pb_encode_walk_cb() detects tag 0 as a marker of end of message and returns PB_WALK_OUT.
   • pb_walk() notices it is on top level and exits with success.

pb_walk_retval_t

Return value from the user callback, indicates the action pb_walk() should take next.

• PB_WALK_EXIT_OK: Return from pb_walk() with true status.
• PB_WALK_EXIT_ERR: Return from pb_walk() with false status.
• PB_WALK_IN: Descend into a submessage and invoke the callback on its first field.
• PB_WALK_OUT: Return from a submessage and invoke the callback on the field it was on before PB_WALK_IN.
• PB_WALK_NEXT_ITEM: If the field is an array or an extension, go to the next item in it. Otherwise go to the next field.
• PB_WALK_NEXT_FIELD: Go to the next field. At the end of the message, iter.tag is set to 0.

pb_walk_state_t

The state of the walking operation is stored in this structure. The pb_encode_submessage() and pb_decode() functions attempt to reuse the same state if they are recursively called from a user-provided field callback.

typedef struct pb_walk_state_s pb_walk_state_t;
struct pb_walk_state_s {
    pb_field_iter_t iter;
    pb_walk_cb_t callback;

    void *stack;
    pb_walk_stacksize_t stacksize;
    pb_walk_stacksize_t stack_remain;
    pb_walk_stacksize_t next_stacksize;

    void *ctx;
    uint32_t flags;

    pb_fieldidx_t depth;
    pb_fieldidx_t max_depth;

    pb_walk_retval_t retval;

    const char *errmsg;
};

Field iterator

The iter member contains a field iterator for the field that is currently being processed. The user-provided walk callback function can use and advance the iterator.

When the callback returns, pb_walk() continues from where the callback left the iterator.

Stack management

The stack is a pointer to a memory buffer where information for each message level is stored. It is used as a full-descending stack: the initial value points to the end of the buffer, and it is decremented for each level.

The user code can request storage for the callback by setting next_stacksize, which can be modified for each level. The callback can check how much stack space it has available by checking stacksize, which it shouldn’t modify.

The remaining stack is tracked by stack_remain, which is automatically updated by pb_walk().

Callback context and flags

The ctx and flags variables are for free use by the walk callback implementation, they are not used by the pb_walk() code.

Typically the ctx will store either pb_encode_ctx_t or pb_decode_ctx_t.

The flags can be used to convey information between message levels, such as “this is the first field in a submessage”.

Depth limit

The depth and max_depth variables track the level in the hierarchy. The depth is automatically updated by pb_walk(), and an error is returned if it exceeds max_depth.

User code is allowed to increment and decrement depth if it performs recursive actions that are not visible to pb_walk(). The only special value is 0, at which level pb_walk() will return with success.

Return value and error message

The retval stores the latest return value from the walk callback. This can be read by the next call, to determine what was the previous action taken. Typically it is used to advance into next field after a PB_WALK_OUT return from a submessage.

The errmsg is used for storing informative error messages for error conditions inside pb_walk(). The caller can combine this information with the ctx->errmsg.

pb_walk_init

This function initializes the state for the walking operation.

bool pb_walk_init(pb_walk_state_t *state, const pb_msgdesc_t *msgdesc,
    const void *message, pb_walk_cb_t callback);

After the initialization, caller can modify state to set e.g. initial stack allocation, the ctx or the flags.

pb_walk

Iterate over the message hierarchy until the callback returns an exit value.

bool pb_walk(pb_walk_state_t *state);

The walk callback is repeatedly called, and iterator is moved according to the return value.

If the callback returns PB_WALK_IN, then pb_walk() stores information of current iterator location and restarts it using state->iter.submsg_desc and state->iter.pData. If there is not sufficient space available, C recursion is used to allocate more from stack unless PB_NO_RECURSION is set.

If the callback returns PB_WALK_OUT, then pb_walk() restores previously stored iterator state. If this was the top level, it acts like PB_WALK_EXIT_OK.

pb_walk_into

Helper function for descending into a message different from the current iterator location.

pb_walk_retval_t pb_walk_into(pb_walk_state_t *state, const pb_msgdesc_t *msgdesc, void *message)

This can be used for descending into messages outside the normal hierarchy, such as extension messages.

Other utility functions

pb_validate_utf8

Validates an UTF8 encoded string:

bool pb_validate_utf8(const char *s);

The protobuf standard requires that string fields only contain valid UTF-8 encoded text, while bytes fields can contain arbitrary data.

By default nanopb validates all strings on encoding and decoding. This can be disabled by defining PB_NO_VALIDATE_UTF8 build option or the PB_ENCODE_CTX_FLAG_NO_VALIDATE_UTF8 or PB_DECODE_CTX_FLAG_NO_VALIDATE_UTF8 context options.

User code can call this function to validate strings in e.g. custom callbacks.

PlatformIO

CMake

Makefiles

Nanopb: Bazel build

The Bazel build system, is designed to be fast and correct. Nanopb provides a set of plugins for the Bazel build system allowing Nanopb to be integrated into the build.

Getting started

Add the following to your MODULE.bazel file.

# MODULE.bazel
bazel_dep(name = "nanopb", version = "0.4.9")
git_override(
    module_name = "nanopb",
    remote = "https://github.com/nanopb/nanopb.git",
    commit = "<commit>",
)

To use the Nanopb rules with in your build you can use the cc_nanopb_proto_library which works in a similar way to the native cc_proto_library rule.

# BUILD.bazel
load("@nanopb//extra/bazel:nanopb_cc_proto_library.bzl", "cc_nanopb_proto_library")

# Your native proto_library.
proto_library(
    name = "descriptor",
    srcs = [
        "generator/proto/google/protobuf/descriptor.proto",
    ],
)

# Generated library.
cc_nanopb_proto_library(
    name = "descriptor_nanopb",
    protos = [":descriptor"],
    visibility = ["//visibility:private"],
)

# Depend directly on the generated code using a cc_library.
cc_library(
    name = "uses_generated_descriptors",
    deps = [":descriptor_nanopb"],
    hdrs = ["my_header.h"],
)

If you have a custom nanopb options file, use the nanopb_options_files argument shown below.

# Generated library with options.
cc_nanopb_proto_library(
    name = "descriptor_nanopb",
    protos = [":descriptor"],
    nanopb_options_files = ["descriptor.options"],
    visibility = ["//visibility:private"],
)

Bazel Configuration Options

Use a non-default extension

By default, nanopb generates files with extension .pb, which ends up creating names such as my_proto.pb.h and my_proto.pb.c. This naming pattern conflicts with cc_proto_library rules, which generate files with the same names.

To resolve this conflict, we provide a string_flag //:nanopb_extension that currently accepts two options: .pb and .nanopb. (In the future, we could pontentially accept your choice of input, but that was tricky to do, so to start with only these two options are supported.)

To build nanopb files with the .nanopb extension (creating files of my_proto.nanopb.h and my_proto.nanopb.c) you can add the following to your command line:

bazel build --@nanopb//:nanopb_extension=".nanopb"

If you want to apply this to all nanopb files, add the above to your .bazelrc file:

# .bazelrc
build --@nanopb//:nanopb_extension=".nanopb"

Conan

Swift

New features in nanopb 1.0

This document showcases new features in nanopb 1.0 that are not immediately visible, but that you may want to take advantage of.

A lot of effort has been spent in retaining backwards and forwards compatibility with previous nanopb versions. For a list of breaking changes, see migration document.

Encode and decode contexts

Previously used pb_istream_t and pb_ostream_t types have been expanded into pb_decode_ctx_t and pb_encode_ctx_t. The substream implementation has been modified so that it is now safe to expand the contexts with custom fields. The ctx pointer passed to callbacks is always the same value that was passed to decode.

For example, this is allowed:

 typedef struct {
    pb_decode_ctx_t ctx;
    ...
    my_type_t extra_stuff;
    ...
 } my_dec_ctx_t;

 bool callback(pb_decode_ctx_t *ctx, const pb_field_t *field, void **arg)
 {
    my_dec_ctx_t *myctx = (my_dec_ctx_t*)ctx;

    ... use myctx.extra_stuff ...
 }

 bool decode(pb_byte_t *buf, size_t msglen)
 {
    my_dec_ctx_t ctx;
    MyMessage msg;
    if (!pb_init_decode_ctx_for_buffer(&ctx.ctx, buf, msglen)) return false;
    if (!pb_decode(&ctx.ctx, MyMessage_fields, &msg)) return false;

    ...
 }

Note that this does not apply to backwards compatibility functions pb_make_string_substream and pb_close_string_substream which still make a full copy of the context structure. These functions are only enabled when PB_API_VERSION build option is defined.

API updater script

Per-context memory allocator support

Per-context field callbacks

Direct pointer to raw bytes data

Single-pass encoding

Better control over stack memory usage

More granular feature disabling

Nanopb: Migration from older versions

This document details all the breaking changes that have been made to nanopb since its initial release. For each change, the rationale and required modifications of user applications are explained. Also any error indications are included, in order to make it easier to find this document.

Nanopb-1.0.0 (2025-xx-xx)

API version compatibility defines

Rationale: Several API changes have been made in nanopb-1.0.0. To make porting old code easier, a define controls whether compatibility wrappers are defined in the headers.

Changes: You can optionally define PB_API_VERSION to the nanopb version that compatibility is aimed for. Use PB_API_VERSION_v0_4 for nanopb-0.4 compatibility, and PB_API_VERSION_LATEST or PB_API_VERSION_v1_0 for new code.

Required actions: Define PB_API_VERSION or use tools/nanopb_api_updater.py to update old code.

Directory layout reorganized

Rationale: Moving nanopb source files to a subdirectory makes build system integration easier. Having headers under folder nanopb avoids namespace conflicts when including files.

Changes: Source files pb_decode.c etc. are now under src/. Header files pb_decode.h etc. are now under include/nanopb. Internally #include "nanopb/pb.h" is used to include nanopb headers.

Required actions: Change compiler include paths as needed. User code can continue to use #include <pb.h> as before if include/nanopb is added to the include path.

Streams renamed to context

Rationale: Previously streams only handled reading/writing. This has been replaced with a ctx argument that can contain e.g. allocator information. The ctx pointer is always passed as-is to callbacks, only the fields inside are modified during processing. Previously the callbacks were given varying stream pointers when substreams were processed.

Changes: pb_istream_t is now pb_decode_ctx_t and pb_ostream_t is pb_encode_ctx_t.

Required actions: Rename types or define PB_API_VERSION as PB_API_VERSION_v0_4. tools/nanopb_api_updater.py can rename types automatically.

Stream/context structure members renamed

Rationale: To distinguish it from the field callback, ctx->callback is renamed to ctx->stream_callback and its state variable to stream_callback_state.

Required actions: In custom stream callbacks, change state to stream_callback_state. For initializing pb_decode_ctx_t and pb_encode_ctx_t, use the new function such as pb_init_encode_ctx_for_callback().

Stream callback function signature changed

Rationale: Previously input stream callback only reported success/failure, and the reporting of end-of-stream condition was complicated. Now input stream callback returns the number of bytes successfully read, which simplifies EOF handling. For output stream callbacks the only difference is used of pb_size_t instead of size_t in argument.

Required actions: Update any custom input stream callbacks to follow the specification given in pb_decode.h comments.

pb_field_t is now pb_field_iter_t in field callback signature

Rationale: Back in 0.4.0, pb_field_t was renamed into pb_field_iter_t to reflect the iterator routines in pb_common.h that were introduced to access it. A typedef for pb_field_t was provided for backwards compatibility.

Changes: pb_field_t typedef is now only defined if PB_API_VERSION is defined. In particular, this affects field callbacks in user code.

Required actions: Rename type to pb_field_iter_t or define PB_API_VERSION as PB_API_VERSION_v0_4. tools/nanopb_api_updater.py can rename types automatically.

Initialization of encoding sizing streams changed

Changes: Previously a pb_ostream_t initialized to zero values, such as with = {}, acted as a sizing stream. Now the flags and max_size have to be set.

Required actions: Use pb_init_encode_ctx_sizing() to initialize the encoding context before a size calculating operation.

Built-in support for null-terminated streams is removed

Rationale: Previously nanopb implemented an option to terminate decoding on a null byte. This is a non-standard feature which does not exist in other protobuf libraries. Retaining it in the core would expand size for a fairly rarely used feature.

Required actions: Null-terminated streams can now be handled using a custom stream callback. See tests/framing/ for an example.

All features are enabled by default

Rationale: Previously some features were disabled with PB_NO_xxxx macro, while others were enabled with PB_ENABLE_xxxx macros. The disabled features got poorer compilation test coverage, and the ABI compatibility depended on what was enabled.

Changes: All supported features are enabled by default. The exception is platform-specific features and special compatibility settings. In particular, malloc support and utf8-validation are now enabled by default. The macros are now always defined as either 0 or 1, and instead of #ifdef, #if should be used. New setting PB_MINIMAL makes all features default to disabled.

Required actions: Check required features, and define e.g. PB_NO_MALLOC or PB_NO_VALIDATE_UTF8 to 1 if preferred. If user code uses #ifdef on the PB_ feature defines, convert it to #if. The nanopb_api_updater.py script can perform this automatically in most cases.

Fields keep their .proto file order by default

Rationale: For technical reasons, nanopb-0.3.x and earlier required fields to be sorted by their tag number in the generated C struct. Nanopb-0.4.x allowed using the definition order from .proto file, but kept the old default sorting.

Changes: Default value of generator setting sort_by_tag is now false. Fields in generated C structure are in the order they are listed in the .proto file.

Required actions: If custom initializers or other code relies on the order of fields, the option sort_by_tag = true can be used to restore old behavior.

Remove custom extension field callback support

Rationale: Previously extension fields used extra pb_extension_type_t indirection level. This was rarely used, not well-tested and is overlapping in functionality with the new context-based callback mechanism.

Changes: Only automatically generated extension types are supported.

Required actions: If custom extension field callback function was used, it can be converted to a context field callback. Any unknown fields are passed to the context field callback. See tests/extra_fields/decode_extra_fields.c.

Structure size check in functions

Rationale: A common mistake has been a mismatch between the fields and struct parameters given to encoding and decoding functions. This can result in access out of bounds of the structure, or just garbage data. Same problem can occur if compilation options differ between the .pb.c and user code files.

Changes: The functions that take a pb_msgdesc_t *fields arguments have been wrapped into macros, which pass sizeof(struct) as the last argument. The function itself then compares the argument to the size defined in the fields, and returns error if they differ.

Required actions: Most code works without changes. Problems occur if the type of the structure is not known at the callsite and void* is used instead. To resolve this, call the underlying function (e.g. pb_decode_s) with the size argument set to 0.

Error indications: “invalid application of ‘sizeof’ to a void type” or “you cannot dereference an operand of type ‘void’”

Change layout of pointer-type bytes fields

Rationale: Previously pointer-type bytes fields stored the length of the value in front of the data. This made it unsuitable for pointing to a block of raw bytes without length prefix.

Changes: Now instead of a direct pointer, there is pb_bytes_t structure. This contains the size and a pointer to raw byte array.

Error indications: Type mismatch compilation errors involving pb_bytes_t.

Changes to field iterator utility functions

Rationale: The functions defined in pb_common.h are rarely used in user code. Their APIs have seen small changes to better accommodate other changes in the codebase.

Changes: pb_field_iter_find() takes a third argument, which can be given as NULL.

Changed size_t to pb_size_t in structures

Rationale: Previously stream structures used platform size_t for the lengths. In practice the lengths will not be longer than pb_size_t can fit. Using pb_size_t everywhere avoids unnecessary casts and bound checks. The size_t is still retained in function call APIs that interface with system, such as read/write/realloc. The data type of pb_size_t is always smaller or equal to size_t.

Required actions: In some cases SIZE_MAX in user code may need to be replaced with PB_SIZE_MAX to silence compiler warnings.

Reserved identifiers in generated headers

Rationale: Previously nanopb generator made definitions such as typedef struct _MyMessage { ... } MyMessage; and _MyEnum_MAX. Identifiers beginning with underscore are reserved in C.

Changes: Generator now uses format typedef struct MyMessage { ... } MyMessage; and ENUM_MyEnum_MAX.

Required actions: If forward declarations or enum defines are used in user code, update the identifier names.

Error indications: _MyEnum_MIN undeclared

Default fallback type for fields without max size

Rationale: Previously string/bytes/array fields without a maximum size or count option were converted into pb_callback_t. This was often confusing to new users of the library, and led to using callbacks even when the simpler maximum size define would have been easier.

Changes: Default fallback type is now FT_STATIC with compile-time defines such as MyMessage_myfield_max_size. These can then be overridden using compiler options or (preferred) using generator options. In either case, it should make the maximum size options more discoverable.

Required actions: If you want fields to remain callback type, either add (nanopb).type = FT_CALLBACK on the single field, or (nanopb).fallback_type = FT_CALLBACK globally.

Proto3 singular submessages are unconditionally encoded

Rationale: Since nanopb-0.4.0, proto3 submessages have had a separate has_ field, consistent with behavior of other protobuf libraries. This can be omitted with generator option proto3_singular_msgs. Previously submessage was then recursively checked for non-zero values.

Changes: To simplify code, singular submessages are now unconditionally encoded as zero-length value. This decodes to the same result, but takes 2 bytes more in the encoded representation.

Remove Python 2 support

Rationale: Python 2 interpreter was deprecated in 2020. For backward compatibility, nanopb has retained support for running the generator with Python 2 for the 0.4.x series. That has required several tricks that complicate the codebase.

Changes: Removed Python 2 support files and code hacks needed to make it work.

Required actions: Upgrade to Python 3 and ensure python-protobuf is installed.

Nanopb-0.4.9 (2024-09-19)

CMake rules now default to grpcio_tools protoc

Rationale: Previously CMake rules primarily looked for protoc in system path. This was often an outdated version installed from package manager, and not necessarily compatible with python-protobuf version installed from pip.

Changes: CMake rules now default to using generator/protoc, which in turn uses grpc_tools Python package if available. If it is not available, system path is searched for protoc.

Required actions: For most users, no actions are needed. In case of version incompatibilities, pip install --user --upgrade grpcio-tools protobuf is recommended. If needed, PROTOBUF_PROTOC_EXECUTABLE can be set to override the default.

Error indications: Failed to import generator/proto/nanopb_pb2.py if versions of protoc selected by CMake is different than installed python-protobuf.

Use uint8_t for pb_byte_t when UINT8_MAX is defined

Rationale: Previously pb_byte_t was always defined as uint8_least_t. This could be annoying on some platforms without this define, or when some compiles might warn on conversion from uint8_t. However not all platforms support uint8_t sized access.

Changes: The stdint.h header will define UINT8_MAX exactly if uint8_t is available. Use it to select which type to typedef.

Required actions: Usually none. If any compiler warnings are generated, they can either be fixed or PB_BYTE_T_OVERRIDE can be defined to uint_least8_t to restore old behavior.

Error indications: Implicit conversion from uint_least8_t to uint8_t.

Migrate to bzlmod

Rationale: Due to the shortcomings of the WORKSPACE system, Bzlmod is going to replace the legacy WORKSPACE system in future Bazel releases. Therefore, nanopb has been migrated to use bzlmod to better support newer bazel versions.

Changes

• upgrade bazel deps
  • bazel_skylib: 1.7.1
  • rules_python: 0.34.0
  • rules_proto: 6.0.2
  • protobuf: 24.4
  • rules_proto_grpc: 5.0.0
• Start using bzlmod (MODULE.bazel)

Required actions: bazel build using WORKSPACE has been deprecated. To use bzlmod, adding below content to your MODULE.bazel

bazel_dep(name = "nanopb", version = "0.4.9")
git_override(
    module_name = "nanopb",
    remote = "https://github.com/nanopb/nanopb.git",
    commit = "<commit>",
)

noted that the name of the module has been changed to nanopb, to better fit the convention of bzlmod. If the old name com_github_nanopb_nanopb is preferred, can add repo_name parameter to indicate the repo name.

bazel_dep(name = "nanopb", version = "0.4.9", repo_name="com_github_nanopb_nanopb")

Separate enum_intsize setting

Rationale: Nanopb-0.4.7 extended int_size option to affect enums. This is only supported by C++11 and C23 compilers. The generation used #ifdef to limit size option to use on C++ compilers. This caused binary incompatibility when project mixed C and C++ files.

Changes: enum_intsize is now a separate option, and does not use #ifdef. If compiler does not support the setting, compilation will fail.

Required actions: If using the recently introduced int_size option on enums, update to use enum_intsize instead.

Error indications: Enum integer sizes use defaults as the old setting is ignored.

Nanopb-0.4.8 (2023-11-11)

Fix naming conflicts with CMake installation

Rationale: Previously CMakeLists.txt installed nanopb Python module under name proto and include file directly as /usr/include/pb.h. These names have potential to conflict with other libraries.

Changes: Python module is installed as nanopb and include files under /usr/include/nanopb.

Required actions: Only affects users who install nanopb using the cmake build system. Does not affect use of FindNanopb.cmake. Calling nanopb generator should work as before. Include path may need adjustment if not using nanopb-targets.cmake to determine it.

Error indications: Include file pb.h not found when compiling against a system-wide installation done with CMake.

Nanopb-0.4.7 (2022-12-11)

Add int_size option to enum fields

This option was separated to enum_intsize in nanopb-0.4.9. This migration notice has been updated to match.

Rationale: The packed_enum option does not work with MSVC due to #pragma pack not supporting enums with MSVC. To workaround this, enum sizes can be specified with the new int_size option. Note that this is only supported when generating C++.

Changes: The int_size enum_intsize option can be specified for enums.

Required actions: Any users concerned about the size of the generated C++ enums and are setting the int_size of enums via a wildcard (e.g. MyMessage.* int_size=IS_8) will need to instead set the int_size option for individual fields.

Error indications: The size of generated C++ enums has changed.

Updated include path order in FindNanopb.cmake

Changes: The include path passed to protoc by the CMake rules was updated.

Required actions: No changes needed for most users. In some specific cases it could change the directory hierarchy generated by protoc. More details in pull request #822.

Error indications: Generated .pb.c or .pb.h file not found when building with CMake rules.

Nanopb-0.4.6 (2022-05-30)

NANOPB_VERSION define is now a string

Changes: To ease NANOPB_VERSION macro usage, the value is directly a string.

Required actions: Most nanopb users probably never used that macro. If so, you certainly use the # preprocessor to convert it as string. You, now, only have to call it directly, like this for example: strcpy(myvar, NANOPB_VERSION);

FindNanopb.cmake now requires protoc 3.6.0 or newer by default

Changes: The default options passing method now uses --plugin-opt which is supported by protoc 3.6.0 and newer (released in 2018).

Required actions: Update protoc if needed, or alternatively install grpcio-tools package from pip. If neither is possible, the NANOPB_PROTOC_OLDER_THAN_3_6_0 cmake option can be used to restore the old style option passing. Note that it has problems with special characters such as :.

Error indications: “protoc: Unknown flag: --nanopb_opt”

pb.h uses C11 _Static_assert keyword by default

Rationale: The nanopb generated headers use static assertions to catch errors at compile time. There are several mechanisms to implement this. The most widely supported is C11 _Static_assert keyword. Previously the code used negative size array definition trick, which is supported already in C99 but does not work with every compiler and can produce confusing error messages.

Changes: Now _Static_assert is used by default.

Required actions: If the keyword is not recognized, set the compiler to C11 standard mode if available. If it is not available, define either PB_C99_STATIC_ASSERT or PB_NO_STATIC_ASSERT in pb.h or on compiler command line.

Error indications: Undefined identifier _Static_assert

Nanopb-0.4.4 (2020-11-25)

Remove outdated generator/nanopb/options.proto

Changes: Back in 2018, it was considered in pull request #241 to move nanopb generator options to a separate namespace. For this reason, a transitional file was added. It was later abandoned and is now removed to avoid confusion.

Required actions: Most nanopb users probably never used that transitional file at all. If your .proto files import it, change to using generator/proto/nanopb.proto.

Error indications: Errors about missing file options.proto when running the generator.

Nanopb-0.4.3 (2020-09-21)

pb_msgdesc_t struct has new fields

Changes: New fields required_field_count and largest_tag were added to pb_msgdesc_t and existing fields were reordered.

Required actions: All .pb.c files must be recompiled. Regeneration is not needed.

Error indications: Messages may fail to encode or decode, or the code can crash inside load_descriptor_values() in pb_common.c.

Nanopb-0.4.2 (2020-06-23)

Generator now uses Python 3 by default

Rationale: Previously nanopb-generator.py had hashbang of #!/usr/bin/env python, which would execute with Python 2 on most systems. Python 2 is now deprecated and many libraries are dropping support for it, which makes installing dependencies difficult. While nanopb_generator.py has worked with Python 3 for years now, and overriding the python version was possible with virtualenv, that was an extra complication.

Changes: Hashbang now uses #!/usr/bin/env python3. New file nanopb_generator.py2 can be used to run with Python 2, if necessary.

Required actions: If possible, just verify Python 3 is installed and necessary dependencies are installed for it. For example pip3 install protobuf grpcio-tools should take care of it. If this is not possible, call nanopb_generator.py2 from your build scripts instead.

Error indications: python3: command not found if Python 3 is not installed. Could not import the Google protobuf Python libraries if dependencies are only installed for Python 2.

Nanopb-0.4.0 (2019-12-20)

New field descriptor format

Rationale: Previously information about struct fields was stored as an array of pb_field_t structures. This was a straightforward method, but required allocating space for e.g. submessage type and array size for all fields, even though most fields are not submessages nor arrays.

Changes: Now field information is encoded more efficiently in uint32_t array in a variable-length format. Old pb_field_t structure has been removed and it is now a typedef for pb_field_iter_t. This retains compatibility with most old callback definitions. The field definitions in .pb.h files are now of type pb_msgdesc_t.

Required actions: If your own code accesses the low-level field information in pb_field_t, it must be modified to do so only through the functions declared in pb_common.h.

Error indications: incompatible pointer type errors relating to pb_field_t

Changes to generator default options

Rationale: Previously nanopb_generator added a timestamp header to generated files and used only basename of files in #include directives. This is different than what the protoc C++ backend does.

Changes: Now default options are --no-timestamp and --no-strip-path.

Required actions: If old behaviour is desired, add --timestamp and --strip-path options to nanopb_generator.py or on protoc command line as --nanopb_out=--timestamp,--strip-path:outdir.

Error indications: Compiler error: cannot find include file mymessage.pb.h when compiling mymessage.pb.c.

Removal of bundled plugin.proto

Rationale: Google’s Python protobuf library, which is used in nanopb generator, has included plugin_pb2 with it since version 3.1.0. It is not necessary to bundle it with nanopb anymore.

Required actions: Update python-protobuf to version 3.1.0 or newer.

Error indications: ImportError: No module named compiler.plugin_pb2

.options file is now always case-sensitive

Rationale: Previously field names in .options file were case-sensitive on Linux and case-insensitive on Windows. This was by accident. Because .proto files are case-sensitive, .options files should be too.

Changes: Now field names in .options are always case-sensitive, and matched by fnmatchcase() instead of fnmatch().

Required actions: If field names in .options are not capitalized the same as in .proto, they must be updated.

CHAR_BIT define is now needed

Rationale: To check whether the platform has 8-bit or larger chars, the C standard CHAR_BIT macro is needed.

Changes: pb.h now includes limits.h for this macro.

Required actions: If your platform doesn’t have limits.h available, you can define the macro in pb_syshdr.h. There is an example in extra directory.

Error indications: "Cannot find include file <limits.h>." or "Undefined identifier: CHAR_BIT."

Strings must now always be null-terminated

Rationale: Previously pb_encode() would accept non-terminated strings and assume that they are the full length of the defined array. However, pb_decode() would reject such messages because null terminator wouldn’t fit in the array.

Changes: pb_encode() will now return an error if null terminator is missing. Maximum encoded message size calculation is changed accordingly so that at most max_size-1 strings are assumed. New field option max_length can be used to define the maximum string length, instead of the array size.

Required actions: If your strings were previously filling the whole allocated array, increase the size of the field by 1.

Error indications: pb_encode() returns error unterminated string.

Removal of per-field default value constants

Rationale: Previously nanopb declared a fieldname_default constant variable for each field with a default value, and used these internally to initialize messages. This however used unnecessarily large amount of storage for the values. The variables were mostly for internal usage, but were available in the header file.

Changes: Default values are now stored as an encoded protobuf message.

Required actions: If your code previously used default constants, it will have to be adapted to take the default value in some other way, such as by defining static const MyMessage msg_default = MyMessage_init_default; and accessing msg_default.fieldname.

Error indications: Compiler error about fieldname_default being undeclared.

Zero tag in message now raises error by default

Rationale: Previously nanopb has allowed messages to be terminated by a null byte, which is read as zero tag value. Most other protobuf implementations don’t support this, so it is not very useful feature. It has also been noted that this can complicate debugging issues with corrupted messages.

Changes: pb_decode() now gives error when it encounters zero tag value. A new function pb_decode_ex() supports flag PB_DECODE_NULLTERMINATED that supports decoding null terminated messages.

Required actions: If application uses null termination for messages, switch it to use pb_decode_ex() and pb_encode_ex(). If compatibility with 0.3.9.x is needed, there are also pb_decode_nullterminated() and pb_encode_nullterminated() macros, which work both in 0.4.0 and 0.3.9.

Error indications: Error message from pb_decode(): zero_tag.

Submessages now have has_field in proto3 mode

Rationale: Previously nanopb considered proto3 submessages as present only when their contents was non-zero. Most other protobuf libraries allow explicit null state for submessages.

Changes: Submessages now have separate has_field in proto3 mode also.

Required actions: When using submessages in proto3 mode, user code must now set mymsg.has_submsg = true for each submessage that is present. Alternatively, the field option proto3_singular_msgs can be used to restore the old behavior.

Error indications: Submessages do not get encoded.

PB_OLD_CALLBACK_STYLE option has been removed

Rationale: Back in 2013, function signature for callbacks was changed. The PB_OLD_CALLBACK_STYLE option allowed compatibility with old code, but complicated code and testing because of the different options.

Changes: PB_OLD_CALLBACK_STYLE option no-longer has any effect.

Required actions: If PB_OLD_CALLBACK_STYLE option was in use previously, function signatures must be updated to use double pointers (void** and void * const *).

Error indications: Assignment from incompatible pointer type.

protoc insertion points are no longer included by default

Rationale: Protoc allows including comments in form @@protoc_insertion_point to identify locations for other plugins to insert their own extra content. Previously these were included by default, but they clutter the generated files and are rarely used.

Changes: Insertion points are now included only when --protoc-insertion-points option is passed to the generator.

Nanopb-0.3.9.4, 0.4.0 (2019-10-13)

Fix generation of min/max defines for enum types

Rationale: Nanopb generator makes #defines for enum minimum and maximum value. Previously these defines incorrectly had the first and last enum value, instead of the actual minimum and maximum. (issue #405)

Changes: Minimum define now always has the smallest value, and maximum define always has the largest value.

Required actions: If these defines are used and enum values in .proto file are not defined in ascending order, user code behaviour may change. Check that user code doesn't expect the old, incorrect first/last behaviour.

Fix undefined behavior related to bool fields

Rationale: In C99, bool variables are not allowed to have other values than true and false. Compilers use this fact in optimization, and constructs like int foo = msg.has_field ? 100 : 0; will give unexpected results otherwise. Previously nanopb didn't enforce that decoded bool fields had valid values.

Changes: Bool fields are now handled separately as PB_LTYPE_BOOL. The LTYPE descriptor numbers for other field types were renumbered.

Required actions: Source code files must be recompiled, but regenerating .pb.h/.pb.c files from .proto is not required. If user code directly uses the nanopb internal field representation (search for PB_LTYPE_VARINT in source), it may need updating.

Nanopb-0.3.9.1, 0.4.0 (2018-04-14)

Fix handling of string and bytes default values

Rationale: Previously nanopb didn’t properly decode special character escapes like \200 emitted by protoc. This caused these escapes to end up verbatim in the default values in .pb.c file.

Changes: Escapes are now decoded, and e.g. \200 or \x80 results in {0x80} for bytes field and "\x80" for string field.

Required actions: If code has previously relied on \ in default value being passed through verbatim, it must now be changed to \\.

Nanopb-0.3.8 (2017-03-05)

Fully drain substreams before closing

Rationale: If the substream functions were called directly and the caller did not completely empty the substring before closing it, the parent stream would be put into an incorrect state.

Changes: pb_close_string_substream can now error and returns a boolean.

Required actions: Add error checking onto any call to pb_close_string_substream.

Change oneof format in .pb.c files

Rationale: Previously two oneofs in a single message would be erroneously handled as part of the same union.

Changes: Oneofs fields now use special PB_DATAOFFSET_UNION offset type in generated .pb.c files to distinguish whether they are the first or following field inside an union.

Required actions: Regenerate .pb.c/.pb.h files with new nanopb version if oneofs are used.

Nanopb-0.3.5 (2016-02-13)

Add support for platforms without uint8_t

Rationale: Some platforms cannot access 8-bit sized values directly, and do not define uint8_t. Nanopb previously didn't support these platforms.

Changes: References to uint8_t were replaced with several alternatives, one of them being a new pb_byte_t typedef. This in turn uses uint_least8_t which means the smallest available type.

Required actions: If your platform does not have a standards-compliant stdint.h, it may lack the definition for [u]int_least8_t. This must be added manually, example can be found in extra/pb_syshdr.h.

Error indications: Compiler error: "unknown type name 'uint_least8_t'".

Nanopb-0.3.2 (2015-01-24)

Add support for OneOfs

Rationale: Previously nanopb did not support the oneof construct in .proto files. Those fields were generated as regular optional fields.

Changes: OneOfs are now generated as C unions. Callback fields are not supported inside oneof and generator gives an error.

Required actions: The generator option no_unions can be used to restore old behaviour and to allow callbacks to be used. To use unions, one change is needed: use which_xxxx field to detect which field is present, instead of has_xxxx. Compare the value against MyStruct_myfield_tag.

Error indications: Generator error: "Callback fields inside of oneof are not supported". Compiler error: "Message" has no member named "has_xxxx".

Nanopb-0.3.0 (2014-08-26)

Separate field iterator logic to pb_common.c

Rationale: Originally, the field iteration logic was simple enough to be duplicated in pb_decode.c and pb_encode.c. New field types have made the logic more complex, which required the creation of a new file to contain the common functionality.

Changes: There is a new file, pb_common.c, which must be included in builds.

Required actions: Add pb_common.c to build rules. This file is always required. Either pb_decode.c or pb_encode.c can still be left out if some functionality is not needed.

Error indications: Linker error: undefined reference to pb_field_iter_begin, pb_field_iter_next or similar.

Change data type of field counts to pb_size_t

Rationale: Often nanopb is used with small arrays, such as 255 items or less. Using a full size_t field to store the array count wastes memory if there are many arrays. There already exists parameters PB_FIELD_16BIT and PB_FIELD_32BIT which tell nanopb what is the maximum size of arrays in use.

Changes: Generator will now use pb_size_t for the array _count fields. The size of the type will be controlled by the PB_FIELD_16BIT and PB_FIELD_32BIT compilation time options.

Required actions: Regenerate all .pb.h files. In some cases casts to the pb_size_t type may need to be added in the user code when accessing the _count fields.

Error indications: Incorrect data at runtime, crashes. But note that other changes in the same version already require regenerating the files and have better indications of errors, so this is only an issue for development versions.

Renamed some macros and identifiers

Rationale: Some names in nanopb core were badly chosen and conflicted with ISO C99 reserved names or lacked a prefix. While they haven't caused trouble so far, it is reasonable to switch to non-conflicting names as these are rarely used from user code.

Changes: The following identifier names have changed:

• Macros:
  • STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
  • UNUSED(x) -> PB_UNUSED(x)
• Include guards:
  • PB_filename -> PB_filename_INCLUDED
• Structure forward declaration tags:
  • _pb_field_t -> pb_field_s
  • _pb_bytes_array_t -> pb_bytes_array_s
  • _pb_callback_t -> pb_callback_s
  • _pb_extension_type_t -> pb_extension_type_s
  • _pb_extension_t -> pb_extension_s
  • _pb_istream_t -> pb_istream_s
  • _pb_ostream_t -> pb_ostream_s

Required actions: Regenerate all .pb.c files. If you use any of the above identifiers in your application code, perform search-replace to the new name.

Error indications: Compiler errors on lines with the macro/type names.

Nanopb-0.2.9 (2014-08-09)

Change semantics of generator -e option

Rationale: Some compilers do not accept filenames with two dots (like in default extension .pb.c). The -e option to the generator allowed changing the extension, but not skipping the extra dot.

Changes: The -e option in generator will no longer add the prepending dot. The default value has been adjusted accordingly to .pb.c to keep the default behaviour the same as before.

Required actions: Only if using the generator -e option. Add dot before the parameter value on the command line.

Error indications: File not found when trying to compile generated files.

Nanopb-0.2.7 (2014-04-07)

Changed pointer-type bytes field datatype

Rationale: In the initial pointer encoding support since nanopb-0.2.5, the bytes type used a separate pb_bytes_ptr_t type to represent bytes fields. This made it easy to encode data from a separate, user-allocated buffer. However, it made the internal logic more complex and was inconsistent with the other types.

Changes: Dynamically allocated bytes fields now have the pb_bytes_array_t type, just like statically allocated ones.

Required actions: Only if using pointer-type fields with the bytes datatype. Change any access to msg->field.size to msg->field->size. Change any allocation to reserve space of amount PB_BYTES_ARRAY_T_ALLOCSIZE(n). If the data pointer was begin assigned from external source, implement the field using a callback function instead.

Error indications: Compiler error: unknown type name pb_bytes_ptr_t.

Nanopb-0.2.4 (2013-11-07)

Remove the NANOPB_INTERNALS compilation option

Rationale: Having the option in the headers required the functions to be non-static, even if the option is not used. This caused errors on some static analysis tools.

Changes: The \#ifdef and associated functions were removed from the header.

Required actions: Only if the NANOPB_INTERNALS option was previously used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.

Error indications: Compiler warning: implicit declaration of function pb_dec_string, pb_enc_string, or similar.

Nanopb-0.2.1 (2013-04-14)

Callback function signature

Rationale: Previously the auxiliary data to field callbacks was passed as void*. This allowed passing of any data, but made it unnecessarily complex to return a pointer from callback.

Changes: The callback function parameter was changed to void**.

Required actions: You can continue using the old callback style by defining PB_OLD_CALLBACK_STYLE. Recommended action is to:

• Change the callback signatures to contain void** for decoders and void * const * for encoders.
• Change the callback function body to use **arginstead ofarg`.

Error indications: Compiler warning: assignment from incompatible pointer type, when initializing funcs.encode or funcs.decode.

Nanopb-0.2.0 (2013-03-02)

Reformatted generated .pb.c file using macros

Rationale: Previously the generator made a list of C pb_field_t initializers in the .pb.c file. This led to a need to regenerate all .pb.c files after even small changes to the pb_field_t definition.

Changes: Macros were added to pb.h which allow for cleaner definition of the .pb.c contents. By changing the macro definitions, changes to the field structure are possible without breaking compatibility with old .pb.c files.

Required actions: Regenerate all .pb.c files from the .proto sources.

Error indications: Compiler warning: implicit declaration of function pb_delta_end.

Changed pb_type_t definitions

Rationale: The pb_type_t was previously an enumeration type. This caused warnings on some compilers when using bitwise operations to set flags inside the values.

Changes: The pb_type_t was changed to typedef uint8_t. The values were changed to #define. Some value names were changed for consistency.

Required actions: Only if you directly access the pb_field_t contents in your own code, something which is not usually done. Needed changes:

• Change PB_HTYPE_ARRAY to PB_HTYPE_REPEATED.
• Change PB_HTYPE_CALLBACK to PB_ATYPE() and PB_ATYPE_CALLBACK.

Error indications: Compiler error: PB_HTYPE_ARRAY or PB_HTYPE_CALLBACK undeclared.

Nanopb-0.1.6 (2012-09-02)

Refactored field decoder interface

Rationale: Similarly to field encoders in nanopb-0.1.3.

Changes: New functions with names pb_decode_* were added.

Required actions: By defining NANOPB_INTERNALS, you can still keep using the old functions. Recommended action is to replace any calls with the newer pb_decode_* equivalents.

Error indications: Compiler warning: implicit declaration of function pb_dec_string, pb_dec_varint, pb_dec_submessage or similar.

Nanopb-0.1.3 (2012-06-12)

Refactored field encoder interface

Rationale: The old pb_enc_* functions were designed mostly for the internal use by the core. Because they are internally accessed through function pointers, their signatures had to be common. This led to a confusing interface for external users.

Changes: New functions with names pb_encode_* were added. These have easier to use interfaces. The old functions are now only thin wrappers for the new interface.

Required actions: By defining NANOPB_INTERNALS, you can still keep using the old functions. Recommended action is to replace any calls with the newer pb_encode_* equivalents.

Error indications: Compiler warning: implicit declaration of function pb_enc_string, *pb_enc_varint,pb_enc_submessage\ or similar.