erts: support endian-aware packet framing by humdrum00001010 · Pull Request #11441 · erlang/otp

humdrum00001010 · 2026-08-09T17:12:55Z

open_port/2 only accepts big-endian packet headers, so callers speaking a little-endian framed protocol have to drop to raw mode and do the framing themselves.

Allow port packet framing to select byte order with {packet, {Width, Endian}} for two-, three- and four-byte headers, where Endian is big, little or native. Existing {packet, N} behavior is unchanged and {packet, {N, big}} is equivalent to it. The inet driver, gen_tcp, ssl and erlang:decode_packet/3 accept the same forms.

Two behavior changes are worth calling out. Scalar {packet, 3} is now accepted, which widens an existing whitelist rather than only adding a tuple form. And native is resolved at the option boundary, so reading the option back reports the concrete byte order rather than native.

CLAassistant · 2026-08-09T17:13:01Z

All committers have signed the CLA.

github-actions · 2026-08-09T17:13:45Z

CT Test Results

No tests were run for this PR. This is either because the build failed, or the PR is based on a branch without GH actions tests configured.

Results for commit 7dea5d4

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

No CT logs found
No HTML docs found
No Windows Installer found

// Erlang/OTP Github Action Bot

humdrum00001010 · 2026-09-07T09:15:20Z

@jhogberg Please let me know if anything needs cleaning up!

jhogberg · 2026-09-07T09:58:17Z

Thanks for your patience, and sorry for the radio silence on this. We had related changes in fixing the CVE GHSA-8m6r-2pj2-25pm and had to wait for that to be published before moving on with this.

Can you rebase this to latest master, resolve any merge conflicts, and check that it doesn't run into the above issue?

humdrum00001010 · 2026-09-08T11:53:06Z

Thanks for your patience, and sorry for the radio silence on this. We had related changes in fixing the CVE GHSA-8m6r-2pj2-25pm and had to wait for that to be published before moving on with this.

Can you rebase this to latest master, resolve any merge conflicts, and check that it doesn't run into the above issue?

I looked into the CVE and was surprised by it.

Using GPT-blue Max, I confirmed that the CVE appears to be properly patched for now. While testing, I did find a few edge cases around TCP send and possible overflow scenario of the port header size, in local Port. They don't appear to be exploitable, but I've included fixes here as separate commits.

For the Windows cached fd case, also in a separate commit, I reproduced the behavior under Windows 11 ARM in QEMU and confirmed that it is a possible scenario. (fa28a00)

jhogberg · 2026-09-09T11:31:41Z

+        case TCP_PB_RAW:        return 0;
+        case TCP_PB_RM:         hlen = 4; break;
+        case TCP_PB_ASN1:       hlen = 2; break;
+        case TCP_PB_SSL_TLS:    hlen = 5; break;
+        case TCP_PB_CDR:        hlen = 12; break;
+        case TCP_PB_FCGI:       hlen = sizeof(struct fcgi_head); break;
+        case TCP_PB_TPKT:       hlen = 4; break;


None of these are variable, you can slap these straight into the definition (TCP_PB_CDR = PACKET_PARSE_TYPE(6, 12), etc).

yes but I kept RAW, to preserve behavior, if that should be included here, let me know!

open_port/2 only accepted scalar packet widths, which fixed the byte order to big-endian. Accept {Width, Endian} for two-, three-, and four-byte headers. Resolve native during parsing. Preserve scalar forms and reject malformed tuples before starting the port. Let stream clear the byte order along with the width, so replacing a packet option cannot leave a stale order behind. Add packet_options/1 to cover parser acceptance and rejection without depending on a system driver's framing implementation.

Encode and decode packet headers using the concrete byte order selected by open_port/2. Cover spawned ports in both wire directions and the fd driver outputv callback. Also cover three-byte widths, a later packet setting replacing both width and byte order, and fragmented input.

Encode and decode packet headers using the concrete byte order selected by open_port/2. Spawned and newly allocated fd ports share the output and ready_input callbacks, so storing the selected byte order in DriverData covers both. The output callback also gains the three-byte width and its bound, which had no check before. Run the shared raw-wire and stream-replacement tests on Windows. Cached standard-fd reuse has a separate lifecycle and is covered by a follow-up commit.

The packet option must have the same meaning in decode_packet/3 and in both selectable TCP backends. Add three-byte headers and explicit big, little, and native byte order without changing existing packet identifiers. Resolve native at the option boundary, keep parser types concrete, and test exact wire bytes for both backends. Numeric packet headers must represent the complete payload length. Reject oversized payloads after selecting the header width in both tcp_sendv() and tcp_send(), instead of truncating the header while sending the full body. Leave raw and parser-only modes unchanged, and use a Uint64 shift so the four-byte bound is defined on every build. Send the first value that does not fit a two-byte little-endian header and require emsgsize, matching the socket backend contract.

SSL performs packet framing above the transport, so the TCP changes do not apply automatically to TLS sockets. Accept the same packet forms in initial and runtime SSL options and encode their headers in SSL. Verify exact bytes with a raw TLS peer.

The legacy packet option does not state its byte order, and the ports reference still excludes three-byte headers. Document scalar three-byte framing and {Width, Endian} forms for open_port/2. State that scalar forms remain big-endian and tuple forms select the wire order in both directions.

The inet driver caches the remaining packet length in i_remain. A packet type change invalidated that cache only in active mode, so a pending passive receive could still complete at the old boundary. Reset the cache for every connected stream packet change. If a passive receive is already pending, continue it through tcp_recv() so buffered bytes are parsed immediately and invalid reparses use the existing emsgsize path. The focused packet_change_pending_recv test leaves data pending under packet 2, changes to packet 1, and requires the same receive to finish without new socket data. It also checks an invalid reparse.

Non-SCTP builds compile out the only users of spec_size, leaving two unused declarations that fail builds using -Werror. Guard the declarations with the same HAVE_SCTP condition as their consumers. SCTP and runtime behavior are unchanged.

The Windows fd driver retains DriverData for standard fd pairs 0/1 and 2/2 after a logical port closes. A pending read can therefore complete after the descriptors reopen with different packet options. Refresh the cached width and byte order on reuse, and reset totalNeeded only when framing changes. Keep bytes already read by the pending operation, including when the reopened port is a stream, so they are delivered with newly completed input instead of being stranded. Extend the fd framing test to leave a partial packet in the cached reader, reopen with another width and byte order, and verify that the retained bytes are parsed under the new framing.

Packet-4 input lengths can exceed the signed state used by both platform fd drivers. On Unix, the decoded length is unsigned while the stored buffer size and remaining count are int. Reject an unrepresentable length before allocation and narrowing. On Windows, reject a length that would overflow totalNeeded when the header width is added. Rearm cached standard-fd readers before failing the logical port because their DriverData survives the failure. Send the first unrepresentable little-endian packet-4 header, require an einval exit, then reopen the cached fd as a stream and verify that input still flows.

rickard-green added the team:VM Assigned to OTP team VM label Aug 10, 2026

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 38d5264 to 122cfae Compare August 12, 2026 16:10

humdrum00001010 changed the title ~~erts: support endian-aware port framing~~ erts: support endian-aware packet framing Aug 12, 2026

humdrum00001010 marked this pull request as draft August 12, 2026 16:51

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 122cfae to 3d73e92 Compare August 12, 2026 17:50

humdrum00001010 marked this pull request as ready for review August 12, 2026 18:04

humdrum00001010 marked this pull request as draft August 12, 2026 20:09

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 3d73e92 to b1c8539 Compare August 12, 2026 21:04

jhogberg reviewed Aug 13, 2026

View reviewed changes

Comment thread erts/emulator/beam/erl_bif_port.c Outdated

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from b1c8539 to 6a9c7c4 Compare August 13, 2026 09:23

jhogberg reviewed Aug 13, 2026

View reviewed changes

Comment thread erts/emulator/beam/erl_bif_port.c Outdated

Comment thread erts/emulator/beam/erl_sys_driver.h Outdated

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 6a9c7c4 to e1d9c9c Compare August 13, 2026 13:24

humdrum00001010 marked this pull request as ready for review August 13, 2026 13:31

humdrum00001010 force-pushed the erts/open-port-packet-endian branch 2 times, most recently from 65bc1b9 to 7dea5d4 Compare August 13, 2026 15:11

jhogberg reviewed Aug 14, 2026

View reviewed changes

humdrum00001010 commented Aug 14, 2026

View reviewed changes

Comment thread erts/emulator/drivers/common/inet_drv.c

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 7dea5d4 to d2a619b Compare August 16, 2026 06:36

humdrum00001010 requested a review from jhogberg August 16, 2026 06:48

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from d2a619b to 59c38c0 Compare August 16, 2026 06:49

jhogberg self-assigned this Aug 17, 2026

humdrum00001010 force-pushed the erts/open-port-packet-endian branch 2 times, most recently from 4727eca to 41ba4cf Compare September 8, 2026 11:46

jhogberg reviewed Sep 9, 2026

View reviewed changes

Comment thread erts/emulator/sys/win32/sys.c Outdated

Comment thread erts/emulator/drivers/common/inet_drv.c Outdated

Comment thread erts/emulator/beam/packet_parser.h Outdated

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 41ba4cf to dc15d55 Compare September 9, 2026 11:19

jhogberg reviewed Sep 9, 2026

View reviewed changes

humdrum00001010 added 3 commits September 9, 2026 21:08

humdrum00001010 force-pushed the erts/open-port-packet-endian branch 2 times, most recently from 7020d87 to 8a17d3c Compare September 9, 2026 12:44

humdrum00001010 added 6 commits September 9, 2026 23:56

ssl: Add endian-aware packet framing

6ef31d7

SSL performs packet framing above the transport, so the TCP changes do not apply automatically to TLS sockets. Accept the same packet forms in initial and runtime SSL options and encode their headers in SSL. Verify exact bytes with a raw TLS peer.

erts: Guard SCTP-only variables

d9e461a

Non-SCTP builds compile out the only users of spec_size, leaving two unused declarations that fail builds using -Werror. Guard the declarations with the same HAVE_SCTP condition as their consumers. SCTP and runtime behavior are unchanged.

humdrum00001010 commented Sep 9, 2026

View reviewed changes

Comment thread erts/emulator/drivers/common/inet_drv.c Outdated

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 8a17d3c to 57608ea Compare September 9, 2026 14:59

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 57608ea to ff9568d Compare September 10, 2026 09:37

Uh oh!

Conversation

humdrum00001010 commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CT Test Results

Artifacts

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

humdrum00001010 commented Sep 7, 2026

Uh oh!

jhogberg commented Sep 7, 2026

Uh oh!

humdrum00001010 commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhogberg Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

humdrum00001010 Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

humdrum00001010 commented Aug 9, 2026 •

edited

Loading

CLAassistant commented Aug 9, 2026 •

edited

Loading

github-actions Bot commented Aug 9, 2026 •

edited

Loading

humdrum00001010 commented Sep 8, 2026 •

edited

Loading

humdrum00001010 Sep 9, 2026 •

edited

Loading