RFC: Proposed protocol for insuring bitswap is more reliable

[hannahhoward]

Goals

We have been dealing with ongoing issues in go-bitswap with remote peers not sending wanted blocks because for any number of reasons, the want list on the remote peer becomes out of sync with the local peer. In addition the remote peer may send the block and think it's no longer wanted, but for whatever reason it's not received. We're currently dealing with all this by periodically re-sending the whole wantlist, but this is an imperfect fix. Fundamentally, the issue is that apart from sending the full wantlist, the bitswap protocol lacks any form of error correction that insures that all messages are received and processed in order.

Implementation

The proposed change adds session ids and frame numbers to the protocol, to make it more reliable. It does so in a way that is backwards compatible when communicating with older peers.

When a new peer connects, the connection is assigned a local unique session id (likely a 16 byte uuid) and a starting frame (should be >0 which is the default value for protobuf ints). Each wantlist change or block send increments the frame counter. This includes a full wantlist send. Frames can be joined together using the following rules:

• wantlist delta frames are joined by simply applying the deltas successively
• wantlist full frames are joined by overwriting all previous changes
• block frames are joined by concatenation
• if the session id changes everything is overwritten and reset

When sending a message, one or more frames are joined together, and sent with the local session id, and the starting and ending frame.

When a message is received, the local copy of the other peers wantlist is rewound to the start frame as necessary, then deltas are applied. (block sends are less important to rewind cause there's no need re-absorb blocks). The change is not applied if the last received frame is less than the starting frame of the incoming message (it may be kept around temporarily in case the missing in between message is received).

A peer also attaches an acknowledgement of the last received remote session id, and the last received remote frame whenever it sends a message. A peer should never send a frame that is less than the lack acknowledged frame (or there's no point in doing so). The lack of any acknowledgement in a message is an indication a peer does not support error correction and the protocol should be downgraded to BS 1.1. The lack of the acknowledged frame advancing means there has been a lost message and old changes should be aggregated and resent.

[hannahhoward]

Tagged a few people cause I wasn't sure who was the appropriate reviewer. Feel free to untag yourself.

[Stebalien]

I'm a bit concerned with the amount of state we'd need to keep with this protocol. We'd basically be implementing our own TCP on top of messages on top of reliable streams.

There are three things I believe we can take advantage of:

1. We can actually know if the other side has successfully received a message (I think). We can do this by:
2. Sending the message update.
3. Closing the stream.
4. Waiting for an EOF.
5. Wantlists are small. We can resend the entire wantlist if something goes wrong.
6. We can detect connection issues through stream resets. When we detect that our control/want stream is reset, we can just resend the entire wantlist. Note: we can still close the want stream and reopen it without having to do any of this.

So, I believe all we really need are:

1. Sequence numbers so we can order messages coming in on different streams. We at least need epoch numbers, incremented once per stream, but sequence numbers are probably simpler.
2. A rule that says that we must send a full update whenever a want stream gets reset (but not when we intentionally close it.
3. Some logic that waits for an EOF when sending blocks and, if the block send fails (reset and not an EOF) we re-queue the task in the task queue (maybe mark it as failed once and backoff?).

[daviddias]

Bitswap 1.2 as in "go package version" or as in "new wire protocol"? Are these spec'ed out somewhere?

[hsanjuan]

I guess this has been outdated given latest changes to bitswap? I'm closing. Reopen if needed (cc. @dirkmc ).