pw_hdlc#

High-Level Data Link Control (HDLC) is a data link layer protocol intended for serial communication between devices. HDLC is standardized as ISO/IEC 13239:2002.

The pw_hdlc module provides a simple, robust frame-oriented transport that uses a subset of the HDLC protocol. pw_hdlc supports sending between embedded devices or the host. It can be used with pw_rpc to enable remote procedure calls (RPCs) on embedded on devices.

Why use the pw_hdlc module?

Enables the transmission of RPCs and other data between devices over serial.

Detects corruption and data loss.

Light-weight, simple, and easy to use.

Supports streaming to transport without buffering, since the length is not encoded.

Try it out!

For an example of how to use HDLC with pw_rpc, see the RPC over HDLC example project.

Protocol Description#

Frames#

The HDLC implementation in pw_hdlc supports only HDLC unnumbered information frames. These frames are encoded as follows:

_________________________________________
| | | |                          |    | |...
| | | |                          |    | |... [More frames]
|_|_|_|__________________________|____|_|...
 F A C       Payload              FCS  F

 F = flag byte (0x7e, the ~ character)
 A = address field
 C = control field
 FCS = frame check sequence (CRC-32)

Encoding and sending data#

This module first writes an initial frame delimiter byte (0x7E) to indicate the beginning of the frame. Before sending any of the payload data through serial, the special bytes are escaped:

Unescaped Special Bytes

Escaped Special Bytes

7E

7D 5E

7D

7D 5D

Unescaped Special Bytes	Escaped Special Bytes
7E	7D 5E
7D	7D 5D

The bytes of the payload are escaped and written in a single pass. The frame check sequence is calculated, escaped, and written after. After this, a final frame delimiter byte (0x7E) is written to mark the end of the frame.

Decoding received bytes#

Frames may be received in multiple parts, so we need to store the received data in a buffer until the ending frame delimiter (0x7E) is read. When the pw_hdlc decoder receives data, it unescapes it and adds it to a buffer. When the frame is complete, it calculates and verifies the frame check sequence and does the following:

If correctly verified, the decoder returns the decoded frame.
If the checksum verification fails, the frame is discarded and an error is reported.

API Usage#

There are two primary functions of the pw_hdlc module:

Encoding data by constructing a frame with the escaped payload bytes and frame check sequence.

Decoding data by unescaping the received bytes, verifying the frame check sequence, and returning successfully decoded frames.

Encoder#

The Encoder API provides a single function that encodes data as an HDLC unnumbered information frame.

C++#

Status hdlc::WriteUIFrame(uint64_t address, ConstByteSpan data, stream::Writer &writer)#: Writes a span of data to a pw::stream::Writer and returns the status. This implementation uses the pw_checksum module to compute the CRC-32 frame check sequence.

#include "pw_hdlc/encoder.h"
#include "pw_hdlc/sys_io_stream.h"

int main() {
  pw::stream::SysIoWriter serial_writer;
  Status status = WriteUIFrame(123 /* address */,
                               data,
                               serial_writer);
  if (!status.ok()) {
    PW_LOG_INFO("Writing frame failed! %s", status.str());
  }
}

Python#

The encode module supports encoding HDLC frames.

pw_hdlc.encode.ui_frame(address: int, data: bytes) → bytes#: Encodes an HDLC UI-frame with a CRC-32 frame check sequence.

import serial
from pw_hdlc import encode

ser = serial.Serial()
address = 123
ser.write(encode.ui_frame(address, b'your data here!'))

Typescript#

Encoder#

The Encoder class provides a way to build complete, escaped HDLC UI frames.

Encoder.uiFrame(address, data)#

Arguments

address (number()) – frame address.
data (Uint8Array()) – frame data.

Returns

Uint8Array containing a complete HDLC frame.

Decoder#

The decoder class unescapes received bytes and adds them to a buffer. Complete, valid HDLC frames are yielded as they are received.

Decoder.process(bytes)#

Arguments

bytes (Uint8Array()) – bytes received from the medium.

Yields

Frame complete frames.

C++#

class pw::hdlc::Decoder#

pw::Result<Frame> Process(std::byte b)#

Parses a single byte of an HDLC stream. Returns a Result with the complete frame if the byte completes a frame. The status is the following:

OK - A frame was successfully decoded. The Result contains the Frame, which is invalidated by the next Process call.

UNAVAILABLE - No frame is available.

RESOURCE_EXHAUSTED - A frame completed, but it was too large to fit in the decoder’s buffer.

DATA_LOSS - A frame completed, but it was invalid. The frame was incomplete or the frame check sequence verification failed.

void Process(pw::ConstByteSpan data, F &&callback, Args&&... args)#: Processes a span of data and calls the provided callback with each frame or error.

This example demonstrates reading individual bytes from pw::sys_io and decoding HDLC frames:

#include "pw_hdlc/decoder.h"
#include "pw_sys_io/sys_io.h"

int main() {
  std::byte data;
  while (true) {
    if (!pw::sys_io::ReadByte(&data).ok()) {
      // Log serial reading error
    }
    Result<Frame> decoded_frame = decoder.Process(data);

    if (decoded_frame.ok()) {
      // Handle the decoded frame
    }
  }
}

Python#

class pw_hdlc.decode.FrameDecoder#

Decodes one or more HDLC frames from a stream of data.

__init__() → None#

process(data: bytes) → Iterable[Frame]#

Decodes and yields HDLC frames, including corrupt frames.

The ok() method on Frame indicates whether it is valid or represents a frame parsing error.

Yields: Frames, which may be valid (frame.ok()) or corrupt (!frame.ok())

process_byte(byte: int) → Optional[Frame]#: Processes a single byte and returns a frame if one was completed.

process_valid_frames(data: bytes) → Iterable[Frame]#: Decodes and yields valid HDLC frames, logging any errors.

Below is an example using the decoder class to decode data read from serial:

import serial
from pw_hdlc import decode

ser = serial.Serial()
decoder = decode.FrameDecoder()

while True:
    for frame in decoder.process_valid_frames(ser.read()):
        # Handle the decoded frame

Typescript#

Decodes one or more HDLC frames from a stream of data.

process(data)#

Arguments

data (Uint8Array()) – bytes to be decoded.

Yields

HDLC frames, including corrupt frames. The Frame.ok() method whether the frame is valid.

processValidFrames(data)#

Arguments

data (Uint8Array()) – bytes to be decoded.

Yields

Valid HDLC frames, logging any errors.

Allocating buffers#

Since HDLC’s encoding overhead changes with payload size and what data is being encoded, this module provides helper functions that are useful for determining the size of buffers by providing worst-case sizes of frames given a certain payload size and vice-versa.

#include "pw_assert/check.h"
#include "pw_bytes/span.h"
#include "pw_hdlc/encoder"
#include "pw_hdlc/encoded_size.h"
#include "pw_status/status.h"

// The max on-the-wire size in bytes of a single HDLC frame after encoding.
constexpr size_t kMtu = 512;
constexpr size_t kRpcEncodeBufferSize = pw::hdlc::MaxSafePayloadSize(kMtu);
std::array<std::byte, kRpcEncodeBufferSize> rpc_encode_buffer;

// Any data encoded to this buffer is guaranteed to fit in the MTU after
// HDLC encoding.
pw::ConstByteSpan GetRpcEncodeBuffer() {
  return rpc_encode_buffer;
}

The HDLC Decoder has its own helper for allocating a buffer since it doesn’t need the entire escaped frame in-memory to decode, and therefore has slightly lower overhead.

#include "pw_hdlc/decoder.h"

// The max on-the-wire size in bytes of a single HDLC frame after encoding.
constexpr size_t kMtu = 512;

// Create a decoder given the MTU constraint.
constexpr size_t kDecoderBufferSize =
    pw::hdlc::Decoder::RequiredBufferSizeForFrameSize(kMtu);
pw::hdlc::DecoderBuffer<kDecoderBufferSize> decoder;

Additional features#

Interleaving unstructured data with HDLC#

It is possible to decode HDLC frames from a stream using different protocols or unstructured data. This is not recommended, but may be necessary when introducing HDLC to an existing system.

The FrameAndNonFrameDecoder Python class supports working with raw data and HDLC frames in the same stream.

class pw_hdlc.decode.FrameAndNonFrameDecoder(non_frame_data_handler: Callable[[bytes], Any], *, mtu: Optional[int] = None, timeout_s: Optional[float] = None, handle_shared_flags: bool = True)#

Processes both HDLC frames and non-frame data in a stream.

__init__(non_frame_data_handler: Callable[[bytes], Any], *, mtu: Optional[int] = None, timeout_s: Optional[float] = None, handle_shared_flags: bool = True) → None#

Yields valid HDLC frames and passes non-frame data to callback.

Parameters

mtu – Maximum bytes to receive before flushing raw data. If a valid HDLC frame contains more than MTU bytes, the valid frame will be emitted, but part of the frame will be included in the raw data.
timeout_s – How long to wait before automatically flushing raw data. If a timeout occurs partway through a valid frame, the frame will be emitted, but part of the frame will be included in the raw data.
handle_shared_flags – Whether to permit HDLC frames to share a single flag byte between frames. If False, partial HDLC frames may be emitted as raw data when HDLC frames share a flag byte, but raw data won’t have to wait for a timeout or full MTU to be flushed.

flush_non_frame_data() → None#

Flushes any data in the buffer as non-frame data.

If a valid HDLC frame was flushed partway, the data for the first part of the frame will be included both in the raw data and in the frame.

process(data: bytes) → Iterable[Frame]#

Processes a stream of mixed HDLC and unstructured data.

Yields OK frames and calls non_frame_data_handler with non-HDLC data.

RpcChannelOutput#

The RpcChannelOutput implements pw_rpc’s pw::rpc::ChannelOutput interface, simplifying the process of creating an RPC channel over HDLC. A pw::stream::Writer must be provided as the underlying transport implementation.

If your HDLC routing path has a Maximum Transmission Unit (MTU) limitation, using the FixedMtuChannelOutput is strongly recommended to verify that the currently configured max RPC payload size (dictated by pw_rpc’s static encode buffer) will always fit safely within the limits of the fixed HDLC MTU after HDLC encoding.

HdlcRpcClient#

class pw_hdlc.rpc.HdlcRpcClient(read: ~typing.Callable[[], bytes], paths_or_modules: ~typing.Union[~typing.Iterable[~typing.Union[str, ~pathlib.Path, module]], ~pw_protobuf_compiler.python_protos.Library], channels: ~typing.Iterable[~pw_rpc.descriptors.Channel], output: ~typing.Callable[[bytes], ~typing.Any] = <function write_to_file>, client_impl: ~typing.Optional[~pw_rpc.client.ClientImpl] = None, *, _incoming_packet_filter_for_testing: ~typing.Optional[~pw_rpc.descriptors.ChannelManipulator] = None)#

An RPC client configured to run over HDLC.

__init__(read: ~typing.Callable[[], bytes], paths_or_modules: ~typing.Union[~typing.Iterable[~typing.Union[str, ~pathlib.Path, module]], ~pw_protobuf_compiler.python_protos.Library], channels: ~typing.Iterable[~pw_rpc.descriptors.Channel], output: ~typing.Callable[[bytes], ~typing.Any] = <function write_to_file>, client_impl: ~typing.Optional[~pw_rpc.client.ClientImpl] = None, *, _incoming_packet_filter_for_testing: ~typing.Optional[~pw_rpc.descriptors.ChannelManipulator] = None)#

Creates an RPC client configured to communicate using HDLC.

Parameters

read – Function that reads bytes; e.g serial_device.read.
paths_or_modules – paths to .proto files or proto modules
channel – RPC channels to use for output
output – where to write “stdout” output from the device

rpcs(channel_id: Optional[int] = None) → Any#

Returns object for accessing services on the specified channel.

This skips some intermediate layers to make it simpler to invoke RPCs from an HdlcRpcClient. If only one channel is in use, the channel ID is not necessary.

class pw_hdlc.rpc.HdlcRpcLocalServerAndClient(server_command: Sequence, port: int, protos: Union[Iterable[Union[str, Path, module]], Library], *, incoming_processor: Optional[ChannelManipulator] = None, outgoing_processor: Optional[ChannelManipulator] = None)#

Runs an RPC server in a subprocess and connects to it over a socket.

This can be used to run a local RPC server in an integration test.

__init__(server_command: Sequence, port: int, protos: Union[Iterable[Union[str, Path, module]], Library], *, incoming_processor: Optional[ChannelManipulator] = None, outgoing_processor: Optional[ChannelManipulator] = None) → None#: Creates a new HdlcRpcLocalServerAndClient.

Example pw::rpc::system_server backend#

This module includes an example implementation of pw_rpc’s system_server facade. This implementation sends HDLC encoded RPC packets via pw_sys_io, and has blocking sends/reads, so it is hardly performance-oriented and unsuitable for performance-sensitive applications. This mostly servers as a simplistic example for quickly bringing up RPC over HDLC on bare-metal targets.

Size report#

The HDLC module currently optimizes for robustness and flexibility instead of binary size or performance.

There are two size reports: the first shows the cost of everything needed to use HDLC, including the dependencies on common modules like CRC32 from pw_checksum and variable-length integer handling from pw_varint. The other is the cost if your application is already linking those functions. pw_varint is commonly used since it’s necessary for protocol buffer handling, so is often already present.

Label

Segment

Delta

HDLC encode and decode

+2,528

HDLC encode and decode, ignoring CRC and varint

+1,336

Roadmap#

Expanded protocol support - pw_hdlc currently only supports unnumbered information frames. Support for different frame types and extended control fields may be added in the future.

Compatibility#

C++17

Zephyr#

To enable pw_hdlc.pw_rpc for Zephyr add CONFIG_PIGWEED_HDLC_RPC=y to the project’s configuration.