In theory, we do support stop sending, but none of the unit tests or demo programs uses it. Which means there probably are bugs lurking in places.

Draft-08 is not published yet, but it does incorporate a very big change: PR 877. The coding of integers is unified throughout the specification, using a "self describing" format that can encode numbers in the 0..2^62 - 1 range on 1, 2, 4 or 8 octets. This will affect:

1. Packet numbers in the packet header;
2. ACK Frame ID (fixed at 0x0e instead of range) and STREAM Frame ID;
3. New transport parameter for ACK Delay Exponent;
4. Encoding of Stream ID in stream related frames
5. Encoding of Stream Offset in STREAM and Reset frames, and MAX STREAM Data frames
6. Encoding of reason phrase length in error frames (but NOT the reason code)
7. Many fields in the ACK frame and STREAM frame

From a spec point of view, this changes only the maximum number of streams, the maximum number of bytes per stream and per connection, and the maximum number of packets per connection. From the code point of view, this is a very big deal.

See the discussion of invariants in QUIC WG.

And this will require a couple of tweaks in the code and in the API...

That's what was sent by mvfst. May want to check that the http09 server supports that syntax.

Changes the format of Connection Close, Application Close, Stop Sending.
Also, distinguish between Application errors and transport errors.

Will be tested for Interop 3...

Most link MTU are in the1400-1500 bytes range. The default MTU is 1200 bytes. Negotiating the larger MTU would pay off in 5-6 packets, i.e. when sending about 8KB. Better do it...

The current code does not detect spurious retransmissions. This would happen if a packet that is believed lost actually gets acknowledged. Currently, if a packet is believed lost, all traces of it are eliminated. The proposal is no move them from the "retransmit" queue to a new "retransmitted" queue. The life cycle would be:

• On transmission, packet is added to retransmit queue;
• Packets in the retransmit queue may be retransmitted if at some point they are believed lost.
• On retransmission, packet is moved from retransmit queue to retransmitted queue;
• On ACK, packets in the retransmit queue get processed for ACK-of-ACK or ACK-of-STREAM and recycled.
• On ACK, packets in the retransmitted queue also get processed for ACK-of-ACK or ACK-of-STREAM and recycled; in addition, they trigger the spurious retransmission logic.
• On long timer, packets in the retransmitted queue get recycled.

Spurious retransmission indicate a defect in the retransmission logic, caused either by poor timer management or poor estimates of out of sequence deliveries. Some adaptive logic might be needed...

.. after draft 07.

Per draft-07

Observed in the client traces when testing MTU discovery. The ACK range included both the clear text handshake packet and the MTU probe.

The QUIC spec suggests doing pacing of sends, to avoid overloading the network. The current version of Picoquic does not do that -- it implement RENO-like congestion control, meaning that data is sent as soon as the congestion window opens. In theory, this may create peaks of traffic, local congestion, and losses. In practice, it is not clear that it matters all that much.

The implementation of pacing should be part of the performance work. Before we do anything like that, we must instrument the application to detect anomalies like starvation, congestion, packet losses, etc. Then, we have to tune the congestion control and pacing algorithm for performance.

Interop with Winquic failed because the TLS code choked on the incoming resume token.

Kazuho suggests looking at code in picotls key/cli.c for examples.

The current code distinguished "client" and "server" clear text packet types. This is a legacy of the early QUIC draft. Since draft 08, these two types were unified as a single "handshake" type. Handling a single packet type instead of two would probably make the code simpler.

Currently only supports draft-07, but this should not be the case. Could test with the picoquic specific version code. Need to specify version, salt and algorithm in a table.

That's a big change, should be version dependent for now.

A server can, and especially in the case of ipv6 does have multiple addresses. picoquic will have to keep track of which address it is serving data from and reply using that address. I hacked up something quickly for testing v4, but the basic idea is to use cmsg pktinfo to get/set the address

dbavatar@f1db0bd

If for some reason a stream frame cannot be encoded, the server crashes. Yet it can be as simple as not having flow control permission to send on the stream.

During testing of MTU discovery, the client and the server send a first ping packet to test the plausible MTU. I observe that the acknowledgement of that packet arrives very late, causing a spurious retransmission.

Hi, currently the code is not consistent formatted. Would you be open for a pull request that formats all the code in an uniform way? Maybe with clang-format.

Move the code to the end, per draft 07.

Need to find a way to also support the draft 05 format as long as we support that version.

Found that during the interop with Minq -- packet number jumps by a large value, and this throws the ack logic out of the window.

The current test app is limited. We don't want to block on the UI, but we also want to have a single
threaded app. On Unix, this can be handled by also including FD 0 in the read socket list. But Windows
is different. And it would be nice if we could have some kind of interactive usage.

Many Windows applications solve the problem by using a local UDP socket for communication. But that
will not work for UWP apps, because the local firewall prevents UWP apps from binding to localhost.

What seems lausible is to use a combination of WSAEventSelect and WSAWaitForMultipleEvents.

WSAEventSelect tells Windows to trigger a specific event when an action (e.g., FD_READ) occurs on a
specific socket. The event is associated with the socket, so we can be arbitrarily complex: one event if
data arrives on any socket, or a specific event per socket. Note that there is at most one event per
socket, thus there cannot be different events for read and write.

WSAWaitForMultipleEvents can be used for waiting on multiple events and a timer. One of those event
could be "UI message", allowing for the required wait on console.

The simplest implementation would be to have one UI event and one network event. Set the events and
the FD_masks at the same time. Wait for multiple event for the duration of the select timeout (ms). Then
either return a UI event, or perform the select code to read data, etc.

In our programming paradigm, we use only UDP, and thus do not need to wait on FD_WRITE. This could
provide a simpler model: one event per socket. Set the events, then wait for multiple events. On return
from an event, perform the recvfrom operation on the socket corresponding to the event.

All this supposes getting some kind of background thread reading the console. This is a slight departure
from the single thread model, but the protocol execution would remain single threaded.

Current code does not support Application Close frame. It should.

Found a bug during live tests and fixed it, but there may be more.

The current code has started managing "ACK of STREAM". On reception of packet ACK, it marks the stream bytes in the packet as received. This points to a revision of the stream data retransmission logic.

Currently, retransmission is triggered by packet loss. If the packet carried a stream frame, that stream frame will be retransmitted. This is slightly incorrect, because it does not account for possible redundant transmissions. The minimal fix would be to check whether the stream data was acknowledged before deciding to retransmit it.

There is a more ambitious fix, in which the stream data is retransmitted directly from the stream context. This should be considered later, as part of a "zero copy" effort.

There were several cases of failed interop with clients that did not present P256 initial shares in the client hello. The main issue there is the lack of negotiation, tagged in issue #59. But that would cost one round trip. It would be preferable to actually support the Chacha20 + Poly1305 suite in addition to P256/AES.

Moving the version field in front of the sequence number. This breaks the previous "invariant" guarantee. Big change, will be hard to maintain backward compatibility.

The current code is a bit confused when it comes to protocol errors. It needs a simple function to mark an error, setting and error reason and moving the connection to a "disconnecting" state. There also need to be a clearer separation between protocol error and internal errors.

Draft-08 brings PR #899, which further specifies the way connections are closed. It defines a "closing period" period, during which nodes can retransmit a closing frame, and a "draining period", during which nodes abstain from sending anything, including a stateless reset. While specific to draft-08, the behavior is compatible with previous versions.

ACK of ACK code is complicated and some of it may be wrong. Need to write an unit test to check that the code behaves as expected.

Which will require some changes to the sender logic, and the maintenance of a pong queue.

Received a complaint from Lars that some connections were refused by the picoquic server. The only
apparent difference is that the Quant client is not setting values for the max bidir stream and max unidir streams parameters.

Using NULL encryption now. That's really just a place holder. There are several plausible strategies. It might be good to provide hooks so different servers can deploy different logic.

This was just added to the spec by the editors...

So compilation there fails now

Will be tested with Interop 3, should be implemented.

Need to flush the server log at the end of each connection, other wise cannot answer
questions. Also, should add initial connection ID to server log messages.

Per draft 06, but not really version dependent.

The quic client code should have a "-n" option to trigger version negotiation. It would cause the client to propose one of the reserved version numbers, forcing negotiation.

Stop Sending was introduced in draft 05. It is not required for interop yet, but it should still be implemented.

From a security stand point, it is not desirable to use the crypto random generator for "non crypto" fields, because that may leak the state of the random generator, and be used as an attack vector to get the actual secrets. For things like filling random data in stateless reset, random sequence numbers, or even random connection ID, it would be better to use a separate generator.

Doing client tests without sending anything, observe that the server repeats ACK frames, adding another one after each successive repeat.

Failure mode observed with "mozquic": the command "GET doc " is sent in 3 frames. The server does not like that.

Basically, repeat one of the connection tests for each supported version.

Interop with minq show a failed retry by minq because the retry was from the same address and the packets were associated with the wrong context

There were several cases of interop failure during the Melbourne interop. The client would announce its support for multiple encryption protocols, but only provide an initial share for a protocol that Picoquic did not support.

The expected reaction is that Picoquic replies with an HRR message, forcing negotiation.

The current reaction is that Picoquic simply fails the connection, and that's wong.

If a new ticket is received for a given SNI and ALPN, it makes sense to remove the old one. Especially with 0-RTT, it is better to use tickets just once...

Should test that all frame types are properly skipped, using a variety of plausible encodings.