It's also true that for many RF media, the performance of the forward path can be impacted by the rate of acknowledgment packets when the link bandwidth is shared.

9.4. Connection Migration
.... " it sends or it can simply send an IMMEDIATE_ACK frame"

• What is the intended difference between the words "it sends" or "it can simply send"?

Per Section 6.1.1 of RFC 9002, the recommended value of packet reordering threshold is 3. That means that with vanilla QUIC v1, the sender would notice a single packet loss without any ack delay being introduced by the receiver. That's because when the receiver notices a gap, it sends an ACK immediately, then after receiving two more packets, it sends another ACK immediately.

With the ack-frequency extension, this behavior changes.

When the receiver notices a gap, one ACK is sent immediately. But after that, the next ACK will not be sent until the expiry of one of the thresholds being declared by the ACK_FREQUENCY frame.

To give an example, let's say that Ack-Eliciting Threshold is set to 1/8 of CWND, and Request Max Ack Delay is set to 1/8 of RTT.

Then, when there is one packet loss (or one range of packets being lost), the sender cannot notice that packet loss based on the packet reordering threshold until 9/8 RTT after the packet(s) was being sent.

Is this going to be an issue for QUIC performance? I ask this because, compared to TCP, many have highlighted the improved efficiency of recovery in QUIC as one of the benefits.

I think it'll be easier to make both parameters required. Not doing so saves 1 to 2 bytes, which is really not much in a frame I expect will be sent a few times a connection. It also means the frame only needs one type value.

Also, I think 0 should be an invalid value for both params, and 1 should be invalid for the number of packets, though I can see someone arguing that point.

Section 4 explains what all the fields are mean first and then explains the exception cases. It seems like this was done the opposite way for Ignore order.

I propose we change the current text:

An 8-bit field representing a boolean truth value. This field MUST have the value 0x00 (representing false) or 0x01 (representing true). This field can be set to true by an endpoint that does not wish to receive an immediate acknowledgement when the peer observes reordering (Section 7.1). Receipt of any other value MUST be treated as a connection error of type FRAME_ENCODING_ERROR.

To:

An 8-bit field representing a boolean truth value. This field can be set to true by an endpoint that does not wish to receive an immediate acknowledgement when the peer observes reordering (Section 7.1). The value of this field MUST be 0x00 (representing false) or 0x01 (representing true). Receipt of any other value MUST be treated as a connection error of type FRAME_ENCODING_ERROR.

Does it make sense to specify any maximum values for any of the ACK frequency frame fields, such as for Packet Tolerance? Does a value of 4 billion make sense? Can I use a uint8_t in my implementation to store this locally? Is there any expectation around the sizes of these fields? If there is no explicit max, what if I implicitly apply one? I'm working on implementing this and I'm trying to figure out how small (byte size) I can make the variables I use to store the values on the connection.

Commit 8a87dbd seem to have been lost with the rename.

From Mirja: "For ECN, you don't need to send an immediate ACK for each CE. Immediate ACKs are most important when the codepoint switches to CE, but then, if multiple CEs in a row are received, you can bundle the ACK information. See also https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn#section-3.2.2.5.1
(Note that we are still working on that section for AccECN but I think any changes are only relevant for specifics of TCP)"

To me it seems that the draft can be interpreted in the following two ways. It would be great if there was a clarification regarding which is correct:

• min_ack_delay TP is a declaration that the endpoint will recognize ACK_FREQUENCY frames. The receiver of the TP can send ACK_FREQUENCY frames regardless of it sending min_ack_delay TP or not.
• Use of delayed acks are negotiated by both endpoints sending min_ack_delay TP. ACK_FREQUENCY frames can be used only when the both endpoints send the TP.

I assume that the intent to be the latter, but I'm not sure.

There's nothing odd about an immediate ACK! Perhaps the text should start by saying that the spec allows a receiver to do this anyway?

However, the current text doesn't explain why the Immediate ACK might be useful.

I could see three places that might benefit from an IMMEDIATE_ACK:

Two cases relevant to SCTP [RFC7053] are: When the rate of transmission was very low and packets might be lost and not detected (something not uncommon in some SCTP uses); as an alternative to detect connectivity failure and trigger path failover. I'm not sure how important these specific cases are for QUIC, because the discussion at the time in TSVWG appeared to centre around use for signalling traffic?

There could also be real opportunity to use this to get more precise timing/congestion information. A set of per-packet ACKs might be useful while a CC is probing to confirm available capacity, etc.

Have I missed a killer application for the method - it would be helpful to me to understand /some example for how the WG think this could be used.

(aside: I am actually uncertain whether this would really be a good change to TCP, especially with the heterogeniety of TCP middleboxes and senders,... that's another story, but I'd think it might be nice to not rely on draft-gomez-tcpm-ack-pull as a reference. I could suggest RFC7053, where people might recall there was quite some discussion in TSVWG about whether SCTP would benefit from being able to trigger an immediate response.)

I found an interesting regression when implementing the delayed ack extension. The extension allows the sender of the data to control the max ack delay and the ack gap. In general that's a good idea, but not for the 2 RTT of the connection. Once the client obtains the 1RTT key, it will start receiving data sent by the server in "false start". At that point the client has assessed the RTT of the connection, but the server has not. The "max ack delay" sent by the server is a guess based on statistics. It will typically be longer than recommended values like RTT/4. The effect is that the first ACKs are sent too late, which has a cascading effect with some congestion control algorithms.

My fix was to use the values locally computed by the client until the session is confirmed and the server can send accurate "ack frequency" frames. Not sure whether we should have text about that in the draft.

When sending a PTO packet, a sender wants to induce an immediate ACK. The only trick I know to induce such an ACK is intentionally skip the packet number.

However, the problem of that approach is that it relies on the receiver not ignoring reorders.

When an ACK_FREQUENCY frame previously sent had set the ignore_order bit, then it become impossible for a sender to induce an immediate ACK when sending a PTO packet.

At the moment, the only solution is to bundle an ACK_FREQUENCY packet in each PTO packet, but that seems like a bit of unnecessary overhead to me. Should we have a dedicated frame for inducing immediate ACKs instead?

If I understand correctly, the maximum value of the min_ack_delay transport parameter is supposed to ensure that the value is smaller or equal than the max_ack_delay transport parameter.

According to QUIC-TRANSPORT Section 18.1 the maximum value of max_ack_delay is (2^14 - 1) ms = 16,383,000 μs. However, the maximum value of min_ack_delay is (2^24 - 1) μs = 16,777,215 μs.

I'm surprised that Requested Max Ack Delay is in microseconds. I was really expecting it to the be same unit as what's in the Transport Parameter. Of course, this allows max_ack_delay to be really short (< 1ms). Since we now have IMMEDIATE_ACK, is this really necessary? It would simplify the specification (and perhaps it would prevent certain bugs) if the unit of requested max ack delay was the same as max ack delay.

As discussed with @janaiyengar a couple days ago, I think the draft should make it clear that implementations are allowed to send less ACKs than the requested "packet tolerance".

6.3. Batch Processing of Packets
For performance reasons, an endpoint can receive incoming packets
from the underlying platform in a batch of multiple packets. This
batch can contain enough packets to cause multiple acknowledgements
to be sent.

To avoid sending multiple acknowledgements in rapid succession, an
endpoint MAY process all packets in a batch before determining
whether a threshold has been met and an acknowledgement is to be sent
in response.

The last paragraph is a bit unclear since "threshold" is undefined. I think we should just say that:

1. it MAY process all packets in a batch (already there)
2. it MAY send less ACKs when it has finished processing the batch

We currently use PNs for ordering, but that runs afoul of requirements at the receiver specified in https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-13.3-5.

If it doesn't change it, we should say so. If it allows it to be reduced, then we need to understand what the limits are(ie: it can't be lowered to 1us presumably?).

Admittedly, the way we're using max_ack_delay should be safe to assuming a 0 value, since the SRTT will accrue the excess.

Endpoints expect the connection to encounter reordering might want to indicate that the receiver should not send immediate acks on receiving a reordered packet.

"Generally, when using max_ack_delay for PTO computations, endpoints MUST use the maximum of the current value and all those in flight."

• I think the words generally and MUST do not sit well together, and it would be better to say what is the requirement.

I think it'll be easier to make both parameters required. Not doing so saves 1 to 2 bytes, which is really not much in a frame I expect will be sent a few times a connection.

Also, I think 0 should be an invalid value for both params, and 1 should be invalid for the number of packets, though I can see someone arguing that point.

The definition that I use internally is "the number of packets you can receive without sending an immediate acknowledgment". This is different than saying "please acknowledge every N packets".

The main benefit of this definition is that there are no invalid values for the number. I recommend making this adjustment in the frame definition: it will be more efficient (by a tiny amount), but it also removes the need to validate the value.

In #86, I noted that the text around a maximum value is a little squishy. It says:

If an endpoint receives an ACK-Eliciting Threshold value that is larger than the maximum value it can represent, the endpoint MUST use the largest representable value instead.

The problem here is not about representing values. We have to assume that 62-bit values can be represented easily by anything that can do QUIC. The real problem is that there is a value beyond which performance tanks badly. Endpoints might want to set a limit for this to prevent that from happening. Why not instead just say that endpoints can set a limit for this threshold?

That means that we might need to provide some advice around setting the limit. For instance, maybe we might suggest that a value that exceeds the current congestion window (measured in MTUs) might be too high.

Although I understand what min_ack_delay represents it's not clear to me what an implementation should do with that value. Is the purpose to make sure that stacks sufficiently wait enough time before triggering PTO? There's no guidance on this in the draft. Perhaps there should be.

"Starting a connection up quickly without inducing much queue is important for latency reduction, for both short and long flows. The sender often needs frequent acknowledgments during this phase; see slow start and hystart."

There's no description of them in this doc. Slow start is in the recovery draft, but hystart is not, so we'd have to reference a non-QUIC RFC?

While reading the latest ack-frequency draft, I am confused with the wording that was added in #58
Ack-Eliciting Threshold: A variable-length integer representing the maximum number of ack-eliciting packets the recipient of this frame can receive before sending an immediate acknowledgment. A value of 0 will result in an immediate acknowledgement whenever an ack-eliciting packet received.

If I have ack-eliciting threshold = 1, to me it means that I can receive 1 ack-eliciting packet before sending an ACK, i.e. I have to send ACK after receiving 1 packet. But looking at the example of 0, this interpretation looks wrong.

@martinthomson suggested, maybe the "before" needs to be "without" and an additional sentence added: "An immediate acknowledgement is sent when more than this number of packets have been received."

I agree with his suggestion.

I've re-read the draft a few times, but I still want to ask you to verify my understanding, just to be sure:

1. Each side informs the other of its own minimum and maximum ACK delay values using the (new) min_ack_delay and max_ack_delay transport parameters.

2. The Update Max Ack Delay value changes the peer's maximum ACK delay. It must not be smaller than the min_ack_delay advertised by the peer.

In 9.3. Window-based Congestion Controllers

I thought that unless there is loss reported, a QUIC ACK Frame releases sending window. In a similar manner as the accurate byte counting style in TCP, a QUIC sender solely operates on the basis of bytes acked, not the number of ACK frames received.

So while a delayed ACK could delay a round of growth when the ACK Ratio is larger, it is only delayed by the time to ACK the set of received packets. In looking at various QUIC CC over medium and longer path RTTs this effect was quite small, and the default ACK delay was not unreasonable; for shorter path RTTs this might be different

With this extension, an immediate ACK is expected if more than the "Packet Tolerance" ack-eliciting packets are outstanding. Given that, it may be a worthwhile optimization to allow the max_ack_delay to not be included in the PTO computation in some cases. If we're not going to allow that, I think we should explain why.

https://github.com/janaiyengar/ack-frequency/blob/master/draft-iyengar-quic-delayed-ack.md#computation-of-probe-timeout-period

This split off #34, but I think the question of whether max_ack_delay always needs to be included in PTO should be separate from whether QUIC would benefit from an ACK-pull mechanism of some sort. That being said, if a mechanism did exist, we add details about when ACK-pull allowed max_ack_delay to not be included.

Values of 0 and 2^24 or greater are invalid

I think min_ack_delay=0 should be allowed? It means there is no delay sending ack, which is a current behavior.

“As discussed in Section 9 however, there are undesirable consequences to congestion control and loss recovery if a receiver uniltaerally reduces the acknowledgment frequency.“

• This is a harsh statement. I don’t think it is true for smaller changes. There is much experience of using TCP where stretch ACKs cover more than two packets. This is a matter of scale. For instance: I’m confident that stretched TCP ACKs covering 4 segments (maybe more ... I’d love to say up to IW) would not break TCP CC, so I don’t see why the editors want to suggest it could break QUIC CC! This really does need a less obfuscated explanation.

Do we really need to allow for sub-millisecond min_ack_delays? IMO, it would simplify my code if both min and max had the same units of milliseconds. Just set the value to zero if you really want to allow that small of values and then cap it internally if you receive the frame.

Quotes from the relevant texts:

max_ack_delay (0x0b): The maximum acknowledgment delay is an integer
value indicating the maximum amount of time in milliseconds by
which the endpoint will delay sending acknowledgments.

and

min_ack_delay (0xff02de1a): A variable-length integer representing
the minimum amount of time in microseconds by which the endpoint
can delay an acknowledgement.

Issue #34 points out that if ignore_order is true, there's no way to get an immediate ACK(for PTO or otherwise).

This proposal is very similar to that discussed in #18, but I believe I can specify both how to implement it and how it could be used better than I was able to previously.

I believe there's a straightforward use case every implementation can benefit from, which is communicating one's packet threshold for loss detection.

I previously presented data at maprg which showed about half of reordering is twiddles(reorder of 1), so having some tolerance for it can reduce the number of unnecessary immediate acks substantially. In order for this to not delay loss detection in some cases, the ack_delay field needs to be no larger than the time threshold for loss detection, which could be a limitation depending upon min_ack_delay, but I would typically expect to be fine.

The field could also be used for other use cases where packets aren't always sent in order, but that doesn't need to be explained in this draft.

For implementation, on every ack-eliciting packet:

• If it's smaller than the largest acked in the last sent ack frame, send an immediate ACK because it fills a hole.
• If it's larger than the last sent largest acked, only send an immediate ACK if there are missing packets greater than the last sent largest acked that are at least packet_threshold away from the current largest received packet.

This speeds up loss detection very slightly vs the core transport draft if the packet reordering threshold isn't an odd number, because the ACK is sent exactly when the sender will be able to use it to declare a loss.

The draft doesn't mention anything about ACK frequency and connection migration. What should we do, for example, when a client has migrated and now sends a PING to force the server to use a new path? Should it send PING+IMMEDIATE_ACK?

Section 9.1 - There are CC considerations with respect to how long it is reasonable for a flow to hold-off detecting congestion and responding. This needs to be discussed.

When an endpoint detects persistent congestion, it MUST promptly reduce the rate of transmission when it receive or detects an indication of congestion (e.g., loss or ECN marking) [RFC2914], the Ignore Order value of true (0x01) in this ID allows a sender to extend that period, postponing detection of loss. That might be reasonable, but excessive delay can be dangerous - and therefore the impact really needs to be discussed: delaying a fraction of a RTT is in my mind safe, intentionally delaying by an RTT is arguable. A delay of many RTTs is endangers other flows, and we need to at least say that in some way ([RFC8084] took an alternate view of how long might be safe).

See Table in Section 12.4 of the transport draft.

As a natural opposite to #34 adding an IMMEDIATE_ACK frame, I want to know if the WG thinks there's sufficient interest in this. I know there are use cases like WebRTC where fine control over the ACKs is desirable.

@nibanks may be interested in this, since a similar topic came up on quicwg/datagram#42

I don't fully understand when would someone want to set ignore-order = true, i.e., what is a valid use case for this?

Lets say for whatever reason the sender sets this to True, this would if there is actual packet loss and receiver receives OOO packets, it won't send an immediate ACK. Meaning the sender won't enter loss recovery immediately and will continue sending packets at the same rate. This is not good, as it directly increases queuing on the network node and for a tail drop queue (most widespread deployment), it will also affect other flows. Meaning, other flows will experience packet loss due to non-responsiveness of this flow for certain period of time.

I probably missed past conversations about ignore-order field, so my apologies if this has already been discussed. I read the draft and didn't see any explanation about where would this be useful and what are the fallbacks of setting this to true.

Use ACK_FREQUENCY instead of ACK-FREQUENCY following naming convention in the transport draft.

A sender might use timers to detect loss of PMTUD probe packets. A sender SHOULD bundle an IMMEDIATE_ACK frame with any PTMUD probes to avoid triggering such timers.

• This seems also valuable for DPLPMTUD.

Chrome has some logic to delay sending of an immediate acknowledgement for reordered packets. This reduces the number of ACKs sent, since, if I remember @ianswett's presentation in mapgr correctly, most of the packet reorderings are just one packet number apart from each other.

Do we need something like this?

When reviewing the draft I got caught on this part:

As specified in Section 13.2.1 of [QUIC-TRANSPORT], endpoints are expected to send an acknowledgement immediately on receiving a reordered ack-eliciting packet. This extension modifies this behavior.

There is an issue here in how this draft uses "reordering". Section 13.2.1 of RFC 9000 does say that ACKs should be sent immediately if:

"In order to assist loss detection at the sender, an endpoint SHOULD generate and send an ACK frame without delay when it receives an ack-eliciting packet either:

• when the received packet has a packet number less than another ack-eliciting packet that has been received, or

• when the packet has a packet number larger than the highest-numbered ack-eliciting packet that has been received and there are missing packets between that packet and this packet."

Although the first bullet clearly is reordering, the second could be caused by three different mechanisms: Reordering of packets, loss of packets, or sender intentional gap (as discussed in #34)
Thus, I think this section needs to be clear if actually means both of the above bullets for when to send ACK or only one of them?

QUIC Transport specifies a maximum value of 2^14 ms (roughly 16s) for the max_ack_delay transport parameter. Does this limit apply to values sent in the Update Max Ack Delay field of the ACK_FREQUENCY frame, or is any value up to 2^62 microseconds allowed here?
Either way, the draft might benefit from clarifying the rules here.

To me, ACK Clocking is a separate point that applies to any type of CC when the cwnd limits the sender.

I agree there could be a concern is around whether a less frequent ACK policy can induce a cwnd-limited sender to send bursts of packets that could induce loss or disrupt sharing of the path with other flows. The QUIC specification already permits an initial window of 10 packets, and motivates the need for pacing. I think it is important this text refers to the QUIC transport section on pacing to mitigate bursts.

Maybe it would be more consistent if this was a field on the ACK_FREQUENCY frame?

They currently use ascii art.

Will write this up once #64 is merged.

By itself, it's not obvious(at least to me), what the field indicates without reading the description.

An alternate suggestion is "Packet threshold", since recovery calls it "Packet threshold loss detection".

This is pretty clearly editorial.

As discussed on 3304 in the base draft, we have some confidence in a particular heuristic (default tolerance of 2 for the first 100 packets, followed by a tolerance of 10) which we know to work fairly well in multiple deployments serving "typical" Internet resources to typical users. Do we want to make that recommendation explicit, or otherwise give guidance in this extension?

The transport I-D 31 states:

   The definition of a new transport parameter (Section 7.4.2) MUST
   specify whether storing the transport parameter for 0-RTT is
   mandatory, optional, or prohibited.  A client need not store a
   transport parameter it cannot process.

This information is missing from draft-iyengar-quic-delayed-ack-01.

There's a concrete example of one way you can arrange ACKs that has benefit in:
https://datatracker.ietf.org/doc/draft-fairhurst-quic-ack-scaling/
This likely helps in many common scenarios.
We could make a pull-request based on summarising some of this into an annexe?

When thinking about #48, it occurred to me that there may be use cases for not changing the acknowledgement behavior in the face of ECN.

The most obvious example is an ECN scheme where the bit means faster/slower, and so at equilibrium, every other packet should be marked. This type of marking would trigger the most pathological ACK behavior of either the current draft(ie: RFC9000) or #68.