inband-oam / ietf Goto Github PK

Recap on the HbH

There a lot of options how the In situ header will encode inside the HbH header .
For example I put 4 options . However, we can have more

a) In-situ first + padding
Next Header Length Padding padding
In- situ header-start

b)In-situ first no padding
Next Header Length In-situ header-start

c)Other HbH TLV before in-situ with padding
Next Header Length Padding at the beginning
T L V
In-situ header-start

d))Other HbH TLV begore in-situ no padding
Next Header Length Type Length
TLV data /padding In-situ header-start
Padding at the end

It is important to recommended that we have to be the first HbH option since we are pushing data and with padding at the beginning (Option 1)
Other options are possible however will not be HW friendly .

2 use-cases for timestamps exist:
(a) calculate delay: should be ns,
(b) wall-clock time for audit purposes etc.

Changes: 2 timestamp options (short & wide):

1. Short: 32 bit option: relative time
2. Wide: 64 bit option: absolute time
   Precision would be a per domain configurable option, i.e. th 32/64 bit fields are assumed to be their own namespace. Assumption is that relative time would be of ns accuracy for delay computation

In draft-brockners-inband-oam-data:

I believe this is a typo, right? Suggest fix below.

OLD:
Variable-length field. The
format of which is determined by the OAM Type representing the
n-th Node data in the Node data List.

NEW:
Variable-length field. The
format of which is determined by the iOAM Trace Type.

Like any OAM domain, IOAM domains can be nested, even within the same transport.
NSH or other tunneling protocols could be use-cases.
Example: When using NSH, all the nodes which are NSH hops could update IOAM, but at the same time, also transit nodes between NSH nodes that could look deep into the packet might update IOAM in NSH type 2 MD.
In the drafts we should have a discussion of the different nesting approaches, i.e.

• IOAM nesting with different protocols (IOAM with IPv6 over IOAM with NSH)
• IOAM nesting in the same protocol (IOAM with NSH - updated by NSH nodes and intermediate transit nodes which can look deep into the packet (i.e. undelay nodes)).

Jen Linkova
0) "n=max number of nodes to be allocated"
Shall it be "maximum number of nodes updating OAM data"?
BTW I can not see 'n' used in the formulas later...

"Based on PMTU advertised in the domain" - I do not think PMTU is
advertized. MTU on a given link might be advertized somehow.
But end2end PMTU is not advertized. Probably it should be re-phrased
like 'calculated based on the packet size and the minimal MTU on all
links within the OAM domain'

2. "Let k = number of node data that can be allocated by this node" -
   IMHO "number of node data" sounds a bit confusing...at elast to me,
   but it might be just me...

3. Procedures to insert/update IPv6 headers keep referring to
   Segments_Left field, while there is no such field in HbH header. Such
   field does exist in Routing Header, but not in HbH.
   In addition to that, in all those procedures one more step needs to be
   added: modifying "next header" value in the basic IPv6 header and in
   HbH header (after it is inserted).

Hemant Singh:
It's good you have configuration on ingress and egress nodes to add/strip the EH. Maybe add some text to say if the original packet is close to 1280 bytes in size (MTU for IPv6 on Ethernet) an OAM EH can't be added and an error is reported. Likewise if an ICMPv6 forwarding error occurs between ingress and egress routers, the router generating the ICMPv6 error should strip the OAM EH before sending the ICMPv6 message to the source. See if any other failure conditions makes sense to document in your draft.

Interface ID: 16 bits appropriate for some, but potentially not for all deployments (e.g. broadband or mobile gateways have a large number of interfaces).

Changes: 2 interface ID options (short and wide):

1. Short: 16 bit option
2. Wide: 32 bit option
   Choice would be per domain. Interface IDs (ingress/egress) have their own namespace, i.e. structure / allocation is domain specific.

Evolve name of in-band OAM, i.e. "in situ OAM" - to better express the nature of the OAM method used, because it is neither "passive" nor "active" as per RFC 7799.
Also add text, which properly classifies the OAM method per RFC 7799 (probably "hybrid - type 2").

Tal Mizrahi [mailto:[email protected]]
To summarize my take on this thread:
The proposed mechanism has two significant vulnerabilities that (in my understanding) are currently not addressed:

1.   A man-in-the-middle can replace the POT of packet A with the POT of packet B.
   

2.   It is possible to replay POTs within a certain time window, whose length is determined by the timestamp resolution.
   

Add threat model
List out the threats that are addressed by this mechanism

In 4.1.3. IOAM node data fields and associated formats:
length field is defined in bytes resolution.
Recommend to define it as 4B units, as anyway it must be 4B aligned.

A node ID of 24 bits appropriate for single domain, but might be considered too short for some inter-domain deployments.

Changes: Support 2 node ID options (short and wide):

1. Short: 24 bit option
2. Wide: 56 bit option
   Choice would be per domain. Node ID has its own namespace, i.e. structure / allocation is domain specific.

Currently there are nested TLVs, unwind and add each of the ioam option as next header field.

In draft-brockners-inband-oam-data:

The draft says that:

If a node is
updating its data record is not capable of populating the value of a
field set in the iOAM-trace-type, the field value MUST be filled
with zeroes.

However, what if the real value of the field is zero? The timestamp nanoseconds / the ingress_if_id and egress_if_id / the queue depth may indeed be zero.
How about a value of all 1's to denote an error?

For the GRE transport should we recommend checksum to be turned off i.e. C bit 0? Since the IOAM header changes enroute updating checksum at every transit node will be challenging in terms of performance of implementations.

8-bits in octets gives a max of 255 octets, or 63 node data fields. If all (wide where possible) IOAM-Trace-Type bits are set, each node adds 40 octets plus the size of the variable length Opaque State Snapshot. This seems too tight.

One possibility for increasing the size is to use multiples of 4-octets rather than just octets.

https://mailarchive.ietf.org/arch/msg/ippm/52mjEvJIjY82N2Fo3zdttnNvwPk

In transit delay, according to the below definition:

transit delay: 4-octet unsigned integer in the range 0 to 2^30-1.
It is the time in nanoseconds the packet spent in the transit
node. This can serve as an indication of the queuing delay at the
node. If the transit delay exceeds 2^30-1 nanoseconds then the
top bit ’O’ is set to indicate overflow. When this field is part
of the data field but a node populating the field is not able to
fill it, the field position in the field must be filled with value
0xFFFFFFFF to mean not populated.

1. why limit to 30 bits and not to 31 bits?
2. How would end-node differentiate between overflow vs. not-populated, where both have 31 bit set

Incremental extension of the packet (like done in P4 INT) is more friendly to hardware implementations, pre-allocated array is beneficial to software implementations (performance reasons: Packet extension requires ~30 extra cycles in e.g. VPP implementation).
Changes:

• Document both options in the draft (pre-allocated array and incremental array extension)
• State that a domain has to implement either of the approaches, but not both at the same time. This way a mixed HW / SW dataplane could implement incremental extension, whereas a software only deployment could benefit from a pure pre-allocated array implementation. At the same time complexities of a hybrid approach are avoided.
• For implementations which use incremental extension, the ingress node needs to “cheat” on PMTU discovery – i.e. taking the max. size that a packet can grow to into account.

From Jen Linkova:

1. You are saying:
   "In-band mechanisms also don't suffer from implementations, where
   probe traffic is handled differently (and potentially forwarded
   differently) by a router than regular data traffic"
   It is known that in most cases packets with HbH header are processed
   differently (CPU/slow path) so the proposed
   solution still might suffer from that problem...Do you think it worth
   mentioning?

2. The Section 4.2. MTU and Packet Size
   is a bit confusing.
   "Based on the transport protocol used MTU is discovered as a
   configuration parameter or Path MTU (PMTU) is discovered dynamically."
   PMTU can be discovered by a host which sends packet. The intermediate
   routers (ingress routers of OAM domain) can not discover PMTU as it
   would be reported to the host which sends the packet.
   "Example: IPv6 recommends PMTU discovery before data packets are sent
   to prevent packet fragmentation."
   You can not do PMTUD before sending packets. I assume you were going
   to say that IPv6 nodes are strongly recommended to support PMTUD -
   however they can just use MTU of 1280 instead and do not perform
   PMTUD.
   The next statement:
   "It specifies 1280 octets as the default PDU to be carried in a IPv6
   datagram. " is also inaccurate. It is not a default. It is minimal
   MTU each IPv6 device must support (so any node could assume that
   1280-bytes packet is small enough to be delivered).
   If a node supports PMTUD it would use the interface MTU. If it does
   not it may use 1280.
   IMHO that text should be deleted and the section should say that if
   OAM data is inserted at the edge of the domain (by intermediate
   routers) than a) MTU on all interfaces with the domain (MTU_INT) MUST
   be >= the maximum MTU on any "external" facing interfaces (MTU_EXT)
   b) The total size of data to be recorded MUST be <= (MTU_INT - MTU_EXT).

3)" REQ-G5: MTU size: With in-band OAM information added, packets should
not become larger than the path MTU."
IMHO s/should/MUST/
Obviously MTU and data insertion/removal seem to be the biggest issue.
It would be nice to clarify what happens in the following situations:
a) a packet containing OAM data can not be delivered for some reasons
and ICMP needs to be sent back. If we expect the data to be removed -
then it should be a requirement to support such data removal.
b) what if - despite all effort - the ingress router can not add OAM
data because of the resulting packet size exceeding MTU (it might be a
misconfiguration on a router, for example or smth else. I understand
that it should not happen with the administrative domain normally
but....). Shall it send ICMP "packet too big" back?

4. REQ-H3 - do you consider a case when you need to record both
   logical and physical interfaces? (Looking at OAM-trace-type in another
   draft it seems that only one interface ID can be recorded...)

Ignas Bagdonas:
Some assorted comments on your document from the operational perspective.
Overlay and underlay correlation - the information about this
correlation is important to the operator itself, and arguable more
important than to the customer. Ability to find out which underlay
elements are shared by overlays and ability to know which overlays are
mapped to the same underlay elements is needed for capacity and
protection planning. The recorded path information does not directly
answer those two questions, it would require actual overlay traffic to
be present on all possible overlay paths, and even if present, it will
provide reactive feedback. We can argue that this steps into the
pre-deployment plannign aspects, however, it is a function of OAM
mechanism to provide possible indications of how different topologies
corrrelate and map.
Layering awareness should be more than just for overlay and underlay.
Underlay using a form of tunneling dataplanes for protection or traffic
engineering is used in actual deployments. IOAM should be aware of
different layers and be able to record selectively only for the layer of
interest.
SLA verification would be available for the normal conditions of working
path, it would not be able to quantify the standby protection path.
MTU and PMTUD operation on topologies that involve protection - it might
be that the actual MTU of the link changes due to the protection or some
other topology change event, and the MTU negotiated initially is no
longer valid, therefore this would change the ability to transport the
same amount of OAM information as on the previous topology.
Identifiers used for IOAM and "traditional" (for a lack of better word)
OAM should be the same ideally, equivalent if not possible otherwise.
From practical deployment perspective IOAM will be used together with
other OAM and instrumentation mechanisms, and having the same set of
identifiers is definitely a strong requirement.
Security considerations - while it is fashionable to leave it fir future
versions of the documents ;-) it is in fact has severe importance. There
must be a mechanism to control the flooding of IOAM-enabled packets
within the device that is disjoint from the topology control (ie, if
topology changes, it should not be assumed that IOAM enabled packets are
allowed to egress on the new interface by default). Mechanisms must be
in place to prevent injection of IOAM enabled packets into the domain.
Overall this is a useful mechanism, it is not a replacement for out of
band OAM mechanisms, nor should it be positioned as one. What should be
ensured is the equivalence of IOAM and OOB OAM mechanisms protocol
mechanics. It takes substantial amount of time for network operations to
gain competence with new network instrumentation mechanisms, and if it
reuses the majority of concepts and protocol mechanics, the adoption
will see less resistance.

In draft-brockners-inband-oam-data:

The units of the field should be defined.
I suggest to add the sentence:

The queue depth field specifies the number of buffers in the queue.

In draft-brockners-inband-oam-data:

In most of the document (an in general in IETF documents) bit 0 is the most significant bit. So we should align this accordingly, right?

"Overflow" (O-bit) (least significant bit)

Maximum Length:  7-bit unsigned integer.  This field specifies the
maximum length of the node data list in multiples of 4-octets.
Given that the sender knows the minimum path MTU, the sender can
set the maximum length according to the number of node data bytes 
allowed before exceeding the MTU.  Thus, a simple comparison
between "Opt data Len" and "Max Length" allows to decide whether
or not data could be added."

• There is no mention of “opt data len” anyway else in the oam-data-07. “opt data len” is used in oam-transport-07 for the definition of IPv6 hop-by-hop option. Does it mean one way or the other, whether IPv6 HBH or GRE/VXLAN encapsulations, the overall node data list must never exceed “Max Length”?

Raised by - jianl.liu at nephosinc.com

currently the makefile does not generate .txt files with version in their respective file name. this was working before but broken once restructured the directory need to fix this.

We should include an "Implementation Status" section in the draft, as per RFC 7942 https://tools.ietf.org/html/rfc7942.

This will greatly help when advancing the document in considering running code.

Since we have about 8 bits of options ( 7 trace types and 1bit for long/short ) it gave about 128 combination.
I think we should state that control plan will decide on reenable amount of possible combination that need to be limited with upper limit
On addition we should agree of format of not supported trace option (first bit ,0Xff etc)

Loopback mode needs more text:

• The looped-back traffic is not in-situ anymore... It is 'just' OAM.
• The assumption is that there is a reverse path, and that the transit node needs to change the packet in the 'transport' layer to be sent through the reverse path.
• On its way back to the encapsulating node, the packet is processed by transit nodes like a regular iOAM, right?
• On its way back the looped back packet may include a larger number of data elements than the maximum (the number of hops on an almost-round-trip is twice as many hops).
• Whether we expect the loopback bit is set in all packets or only in a fraction of them.

Efficient trace-route use case: Add a section to the draft that discusses a loopback option as it exists in SONET/SDH to provide for efficient traceroute. Requires a target destination node to send the recorded data back to the ingress node.

On 4.1.1. Pre-allocated Trace Option:
IOAM-Trace-Type.bit[0] is defined as (Most significant bit).
Also documented in 4.1.2.

The examples in 4.1.4. Examples of IOAM node data, are not consistent with that.
See below example
0x002B: IOAM-Trace-Type is 0x2B then the format of node data is:

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   Hop_Lim     |              node_id                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     ingress_if_id             |         egress_if_id          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  timestamp nanoseconds                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            app_data                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Once we have some stability (perhaps right after adoption), we should write up an IANA Section with a new set of registries. I will start that.

(reported by Hemant Singh): If section 3.1.2 related to ingress, we have text that says "insertion of the in-band OAM header is enabled at the required edge nodes by means of configuration." However, the same text does not exist in the egress section in 3.1.4. Why not add common text for ingress/egress before any of these two sections start?

This field specifies the length of data added by each node in multiples of 4-octets. For example, if 3 IOAM-Trace-Type bits are set and none of them is wide, then the Node Data Length would be 3. If 3 IOAM-Trace-Type bits are set and 2 of them are wide, then the Node Data Length would be 5.

There are several possibilities for where to put this field:

1. Squeeze the IOAM Trace Header by specifying Maximum Length as 7-bits in multiples of 4-octets, add 4-bits of Node Data Length, and squeeze flags down to 5 bits.
2. Increase the size of the fixed part of the IOAM Trace Header to 8 octets.
3. Use a reserved field in the IOAM Transport Header just before the IOAM Trace Header.

In support of UDP-pinger, add a "loopback" IOAM trace type.

Throughout the four drafts iOAM is used with a lowercase 'i'. In some presentations IOAM with an uppercase 'I' was used. What is the correct form?
Let's be consistent and let's add it to the acronym list at the beginning of each document.

In 4.1.1. Pre-allocated Trace Option:
Flags.bit[0] is defined as (most significant bit)

while in 4.1.2. Incremental Trace Option:
Flags.bit[0] is defined as (least significant bit)

The TLV format needs clarifying in the document.

  In-situ OAM data is
  defined as options in Type-Length-Value (TLV) format. The TLV format for
  each of the three different types of in-situ OAM data is defined in this
  document.

The text mentions a TLV format, but I haven't found a place in the document that actually defines a TLV format. The 'three types' are not completely clear either.

More specifically, it is not clear where in the packet the POT option resides, and where the E2E option resides. We should probably add a figure that specifies the packet format of a packet that includes trace information, POT, and E2E.

https://mailarchive.ietf.org/arch/msg/ippm/ZtOiCINFAJkYeP_UCP5jDqs9vJk

NodeLen described as the total length of data needed to be added by each node.
It contains 4bits in 4B resolution which means 4B up to 60B.

The issues we see with this are:
1.
There is no reference to bit[7] which has variable length.

Regardless bit[7], when all bits are set where some are short and some are wide, the current total is 14 where we have spare of 4 bits (12-15). If they will be used this field will not be able to show the total node length

Even of all the above are solved, checks needs to be defined in case the NodeLen is not matching the number of bits in the trace type.
Suggest to add text that trace type has precedence and if a node decides to use the NodeLen it is required to do checks on the trace type.

According to spec, see below.

queue depth: 4-octet unsigned integer field. This field indicates
the current length of the egress interface queue of the interface
from where the packet is forwarded out. The queue depth is
expressed as the current number of memory buffers used by the
queue (a packet may consume one or more memory buffers, depending
on its size). When this field is part of the data field but a
node populating the field is not able to fill it, the field
position in the field must be filled with value 0xFFFF to mean not
populated.

should be 0xFFFFFFFF (32 bits of 1) and not 0xFFFF (16 bits of 1)
I think it is a typo and means 32 bits of 1. Please ack.

In draft-brockners-inband-oam-data:

I believe the following terms should be defined (preferably in the terminology section at the beginning):
data type
data record
data format
node-data elements
node data list
node data
node-data
trace type
trace element

We should probably have a search for these terms and align the terminology + eliminate the redundant terms.

I suggest to remove this sentence:
"In the event that both options are utilized at the same time, the Incremental Trace Option MUST be placed before the Pre-allocated Trace Option."

I am not sure I understand how the two trace options can be supported at the same time for a given packet. The IOAM header has a different format in each of the trace options. I suggest to remove this sentence.

Rather than using NSH Type 2 metadata TLVs, encapsulating iOAM in NSH, use an encapsulation of iOAM over NSH, allocating new values for Next Protocol in the NSH Base Header. This would be similar to the approach taken for VXLAN-GPE and GRE. In order to make iOAM transport encapsulations more hardware friendly, whenever possible prefer encaps of iOAM over another protocol rather than in another protocol, especially when that protocol uses nested TLVs internally.

Pre-allocated Trace Option
a) We need to have the over flow bit as well the same as in the incremental

In his review (https://mailarchive.ietf.org/arch/msg/ippm/ZtOiCINFAJkYeP_UCP5jDqs9vJk) Al makes the case for a new IOAM node type (complementing encap/decap/transit nodes). Once the discussion on the IPPM mailing list settles this might need to be reflected in the data draft.

Jen Linkova:
1)
"In-band OAM data is added to a packet on entering the in-band
OAM-domain and is removed from the packet when exiting the domain. " -
it might be just me but I found the terminology a bit confusing. When
I read it for the first time I assumed that "data" means "data in OAM
header. While it looks like by "data" you mean "the data container".
Shall the container and the data be called differently?

2. IMHO calling a node "in-band OAM encapsulating" is a bit confusing
   as at least for some cases there is no encapsulation. Maybe smth like
   'pop/push' MPLS terminology? Or would it confuse people even more?

3. (sorry for nitpicking)
   "Hop_Lim: 1 octet Hop limit that is set to the TTL value in the
   packet at the node that records this data."
   s/TTL/TTL or Hop Limit/ - that field is called Hop Limit in IPv6, not TTL.

Deepak Kumar:
I believe just I-oam header should be placed flexibly, if network is l2/l3 leaf and traffic is within one data center then current format is fine.
If traffic is going over L2 and L3 vxlan seperated, or Border Leaf, Data Center Interconnect I-oam header can come fixed byte inside data and configuration policy can determine this behavior if required.
Now How to indicate if common OAM header is starting immediately or after Fixed byte, I will left to community.
Using single "O" bit is also fine for active OAM as we can look at two places if we use optional Fixed byte.
As I-OAM is for real data it will be issue and they might not support that mode or getting extra bit in Datapath header will solve this issue.
L2/L3 distributed scenario is currently deployed by operator.
In-band OAM header in VXLAN GPE header:

 In-band OAM header in VXLAN GPE header:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Outer Ethernet Header                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Outer IP Header                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Outer UDP Header                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
   |R|R|Ver|I|P|R|O|          Reserved             | NP = i.b.OAM  |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ GPE
   |     Virtual Network Identifier (VNI)          | Reserved      |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
            Fixed Byte of Inner Packet Real Data (Optional)
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
   | Type =i.b.OAM | i.b.OAM HDR len |  Reserved     | NP = IP/Eth |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+iOAM
   |                     in-band OAM options                       |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
   |                                                               |
   |                                                               |
   |                     Payload + Padding (L2/L3/ESP/...)         |
   |                                                               |
   |                                                               |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Tal:

OLD:

  Bit 0    When set indicates presence of node_id in the Node data.

  Bit 1    When set indicates presence of ingress_if_id in the Node
           data.

  Bit 2    When set indicates presence of egress_if_id in the Node
           data.

  Bit 3    When set indicates presence of timestamp in the Node
           data.

  Bit 4    When set indicates presence of app_data in the Node data.

  Bit 5-7  Undefined in this document.

NEW:

  Bit 0    When set indicates presence of Hop_Lim and node_id in the Node data.

  Bit 1    When set indicates presence of ingress_if_id and egress_if_id in the Node
           data.

  Bit 2    When set indicates presence of short timestamp data.

  Bit 3    When set indicates presence of long timestamp in the Node
           data.

  Bit 4    When set indicates presence of app_data in the Node data.

  Bit 5-7  Undefined in this document.

OLD:
timestamp: 4 octet timestamp when packet has been processed by the node.

NEW:
short timestamp: a 4-octet timestamp that specifies the time at which the packet was received by the node. [TBD] Need to specify the semantics of this field.

long timestamp: an 8-octet timestamp that specifies the time at which the packet was received by the node. The format of this field is identical to the 64 least significant bits of the IEEE 1588-2008 Precision Time Protocol timestamp format [IEEE1588v2]. This truncated format consists of a 32-bit seconds field followed by a 32-bit nanoseconds field. As defined in [IEEE1588v2], the timestamp specifies the number of seconds elapsed since 1 January 1970 00:00:00 according to the International Atomic Time (TAI).

Since no longer define the Trace, POT and E2E as TLVs how about calling them

s/In-situ OAM Tracing Options/In-situ OAM Tracing Records
s/Pre-allocated Trace Option/Pre-allocated Trace Record
s/Incremental Trace Option/Incremental Trace Record
s/IOAM E2EOption/ IOAM E2E Record
and so on

This would make it easier to embed these trace records into an option format or in any other container that can have a length field.

There is no easy answer to the question: "Will a packet which is added in-situ OAM data be treated the same as if it would not be added in-situ OAM data?", because it'll likely depend on implementation details as well as the encapsulation used. Example: There are implementations which forward packets differently in case an IPv6 extension header is present in the packet - while not the case for all implementations, some have that behavior. Now if IOAM would be added as an extension header and it would be the only extension header present in the packet, some implementations might indeed change their forwarding behavior. Hence it is important to make sure that for the IOAM domain you indeed have nodes which conform to the assumption that forwarding behavior will not change when IOAM data is added to the packet.
What this leads to is probably another requirement we need to take into consideration - and that we should add to the requirements draft: E.g. something like "The addition of IOAM data-records should not change the way packets are forwarded within the IOAM domain."

It stated in the draft : “The index 

contained in "Elements-left" identifies the current active Node
data to be populated.”
This is not clear after we add the user defined option . Should be the pointer in bytes for the last place the active node should be write into

Hop_Lim:" 1-octet unsigned integer. It is set to the Hop Limit
value in the packet at the node that records this data. Hop
Limit information is used to identify the location of the node
in the communication path. This is copied from the lower layer
for e.g. TTL value in IPv4 header or hop limit field from IPv6
header of the packet"

Should be defined from where it is taken (In the ingress , after the router )

Make it explicit in the draft that all data records have to be 4 byte aligned.