currently there are a lot of different CIS running,
Ultimately All the CIs should be converged into one, so that we can get a consolidated result

These are the problems:

• Tests are not segregated based on supported topology
• Users cannot run the test just by passing test file path to do.sh (they have to use the actual test name to run it, which requires effort)

Config

• Two back-to-back connected test ports
  • One for tx
  • Another for rx
• One Flow with,
  • Fixed frame size of 128B
  • Rate of 50 PPS
  • Packet count 100
  • Packet headers ethernet, ipv4 and tcp
• TCP header with,
  • Fixed source port
  • Fixed destination port

Validation

• Push configuration
• Start capture on rx port
• Start transmit on tx port
• Wait until following is as expected,
  • Frames tx/rx, bytes tx/rx and transmit state in flow metrics
• Fetch capture and ensure following is as expected,
  • ethernet.ether_type
  • ipv4.protocol, ipv4.total_length
  • tcp.src_port, tcp.dst_port
  • size of each frame
  • (best effort) checksum of all headers

Currently CI doesn't fail upon failing deployment or test

Config

• Two back-to-back connected test ports
  • One for tx
  • Another for rx
• One Flow with,
  • Fixed frame size of 256B
  • Rate of 50 PPS
  • Packet count 100
  • Packet headers ethernet, ipv6, udp, vxlan, ethernet, ipv4, tcp
• VxLAN header with,
  • List of VxLAN IDs

Validation

• Push configuration
• Start capture on rx port
• Start transmit on tx port
• Wait until following is as expected,
  • Frames tx/rx, bytes tx/rx and transmit state in flow metrics
• Fetch capture and ensure following is as expected,
  • outer and inner ethernet.ether_type
  • outer ipv6.protocol, ipv6.total_length
  • outer udp.src_port, udp.dst_port, udp.length
  • inner ipv4.protocol, ipv4.total_length
  • inner tcp.src_port, tcp.dst_port
  • size of each frame
  • (best effort) checksum of all headers

• The test should configure LAG (and LACP) with one member port (min link = 1)
• The test should configure a unidirectional UDP flow with LAG ports as endpoints
• The test should fetch LACP metrics to ensure partner system ID is as expected
• The test should fetch LAG metrics to ensure oper status is up and tx/rx frames are as expected

Should be a Python equivalent of #66

The snippets on for both snappi and gosnappi are outdated and not CI tested.

Moreover, currently there's no quick way for snappi/gosnappi users to try out the tests.

While running the test on ixia-c and Ixnetwork, faced the below issue:
Test name: https://github.com/open-traffic-generator/conformance/blob/main/devices/bgp/test_ebgpv_route_install.py

The below attributes are not getting set in the Ixnetwork while creating ipv6 prefix pools on BGP:
next_hop_ipv6_address
next_hop_address_type
next_hop_mode

Linked issue in snappi-ixnetwork: open-traffic-generator/snappi-ixnetwork#559

The test should:

• Configure LLDP on both the ports with advertisement interval of 5 seconds
• Configure chassis ID as MAC address
• Poll for LLDP metrics to validate at least 2 PDUs have been Tx/Rx on each port
• Poll for LLDP neighbors to fetch 2 entries, each with correct chassis ID

The test should have these scenarios

• configure two interface
• configure flow from one interface to another
• check for mac resolution
• start traffic
• verify frames rx and tx

https://github.com/open-traffic-generator/conformance/actions/runs/3928629942

IS-IS test has been skipped for now because of this

Config

• Two back-to-back connected test ports
  • One for Tx
  • Another for Rx
• One Flow with,
  • Fixed frame size of 128B
  • Rate of 50 PPS
  • Packet count 100
  • Packet headers ethernet, ipv6, and UDP
• UDP header with,
  • Source port value list
  • Destination port value list

Validation

• Push configuration
• Start capture on Rx port
• Start transmitting on the Tx port
• Wait until the following is as expected,
  • Frames Tx/Rx, bytes Tx/Rx and transmit state in flow metrics
• Fetch capture and ensure the following is as expected,
  • ethernet.ether_type
  • ipv6.protocol, ipv6.total_length
  • udp.src_port, udp.dst_port
  • size of each frame

Describe the bug
TCP data offset validation is not present in TCP capture tests

Expected behavior
All TCP capture tests should validate the TCP data offset field

use black to lint and auto-format python code: https://github.com/open-traffic-generator/openapiart/blob/main/do.py#L144 for reference

use golangci-lint and gofmt to lint and format Go code: https://github.com/open-traffic-generator/openapiart/blob/main/do.py#L247 for reference

users should be able to run ./do.sh lint in order to format and lint all the tests. It should optionally accept an argument to specify path to be linted / format.

NOTE: When run by user, it should automatically fix any bad formatting and just raise linter error if there's any. But in CI, also raise an error if there's a need to fix bad formatting.

Currently, license-server check is not done, post container deployment. This may lead to intermittent failures.

As of now support is only available for byte level offset validation from packet capture, needs support for bit level validation as well.

Blocked due to open-traffic-generator/openapiart#363

Currently there's no CI tested hello-snappi test

Turns out, it's an issue with protocol-engine build for scenarios where 2 cores were present.

Describe the bug
Tx and Rx count for some flows in the UDP header mesh flows test do not match towards the end of the test when 4096 flows are configured for each port in b2b.

To Reproduce

git clone https://github.com/open-traffic-generator/conformance.git
cd conformance && git checkout c2c306b

./do.sh topo new cpdp
./do.sh pregotest

OTG_ITERATIONS=2 ./do.sh gotest -tags="perf" -run "^TestUdpHeaderMeshFlowsPerf$"

• Test Link: https://github.com/open-traffic-generator/conformance/blob/c2c306b4e219cb6606cd2a3d1a6eeba1b56f04c5/performance/flows/udp_header_mesh_flows_perf_test.go
• Failure logs: https://github.com/open-traffic-generator/conformance/actions/runs/3959361872/jobs/6782085950
• Diagnostics: https://github.com/open-traffic-generator/conformance/suites/10466195810/artifacts/518327292

Expected behavior
The packet count for tx and rx should be equal for all flows.

Versions (please complete the following information):

• OS: Ubuntu 22.04
• Go: 1.18
• Ixia-C release: 0.0.1-3722
• snappi release: 0.10.4

If possible, need to add Ipv6 Protocol Header Validation for udp_header_ipv6_valuelist_ports test : both python & go.
related to issue #13

Currently BGP prefixes and ISIS LSPs are not validated in tests, and hence the validation in tests is incomplete.

Currently CI duration has exceeded 20mins. Need to breakdown things further

Python test should be equivalent to #68

The device configuration is needed currently to start protocols and fetch protocol states. It ideally should not be needed and should be fixed in upcoming release of ixia-c.

PCAPNG capture load support is not available with pcapfile library, need to look for one more library which supports loading of pcap and pcapng format.

kisom/pypcapfile#15

• add func for generating Go SDK from openconfig repo
• add gnmi helpers for get, delete and replace

Add a new CI job to deploy port-dut-port topology using KNE

Config

• Two back-to-back connected test ports
  • One for tx
  • Another for rx
• One Flow with,
  • Fixed frame size of 128B
  • Rate of 50 PPS
  • Packet count 100
  • Packet headers ethernet, ipv4 and udp
• UDP header with,
  • Fixed source port
  • Fixed destination port

Validation

• Push configuration
• Start capture on rx port
• Start transmit on tx port
• Wait until following is as expected,
  • Frames tx/rx, bytes tx/rx and transmit state in flow metrics
• Fetch capture and ensure following is as expected,
  • ethernet.ether_type
  • ipv4.protocol, ipv4.total_length
  • udp.src_port, udp.dst_port, udp.length
  • size of each frame
  • (best effort) checksum of all headers

Currently each run is taking about 20 mins just to build

Config

• Two back-to-back connected test ports
  • One for tx
  • Another for rx
• One Flow with,
  • Fixed frame size of 256B
  • Rate of 50 PPS
  • Packet count 100
  • Packet headers ethernet, ipv4, udp, vxlan, ethernet, ipv6, tcp
• VxLAN header with,
  • List of VxLAN IDs

Validation

• Push configuration
• Start capture on rx port
• Start transmit on tx port
• Wait until following is as expected,
  • Frames tx/rx, bytes tx/rx and transmit state in flow metrics
• Fetch capture and ensure following is as expected,
  • outer and inner ethernet.ether_type
  • outer ipv4.protocol, ipv4.total_length
  • outer udp.src_port, udp.dst_port, udp.length
  • inner ipv6.protocol, ipv6.total_length
  • inner tcp.src_port, tcp.dst_port
  • size of each frame
  • (best effort) checksum of all headers

• Add a helper for pushing SSH config
• Add helpers for setting and un-setting config easily

I have added check for unexpected packets - so those can be conditionally ignored.
This should avoid intermittent failures during batch run.

If we detect any unexpected packets (due to ICMP and what not), those will still be printed as warnings. Even though we’re ignoring such packets, we’re still ensuring that packet count is correct - so if packet sent out from the flow itself turns out to be incorrect, it will still be detected and fail the test.

Hi Team,

I tried using following sample for packet capture but is is throwing Error.

Sample Code : https://github.com/open-traffic-generator/conformance/blob/main/features/flows/headers/tcp/test_tcp_header.py#L48

My code :

def init(self):
    # Configure a new API instance where the location points to controller
    if self.mode == 'ixnet':
        self.api = snappi.api(location=self.exec_params['controller'], verify=False, ext='ixnetwork')
    elif self.mode == 'trex':
        self.api = snappi.api(location=self.exec_params['controller'], verify=False, ext='trex')
    else:
        self.api = snappi.api(location=self.exec_params['controller'], verify=False)
    logging.info("%s Starting connection to controller... " % time.strftime("%s"))
     self.configuration = self.api.config()

    # Configure two ports where the location points to the port location:

    for pid, port in enumerate(self.exec_params['port_groups']):
        print("pid and port is : {} {}".format(pid,port))
        location = port.get('location', f"localhost:{BASE_TENGINE_PORT+pid}")
        print("location, port is {}, {}".format(location,port[port['init']]))
        self.configuration.ports.port(name=port[port['init']], location=location)

    self.configuration.options.port_options.location_preemption = True


    ca = self.configuration.captures.add(name="ca", port_names=[p.name for p in self.configuration.ports])
    ca.format = ca.PCAP


In script file:
==========
def test_bgp_as_number_with_packet_capture(self):
    self.dp.stop_protocols()
    self.dp.start_capture()
    time.sleep(5)
    self.dp.start_protocols()
    time.sleep(5)
    self.dp.stop_capture()

    captured_packets = self.dp.api.get_capture(self.dp.configuration.ports[0].name)

Error:

  captured_packets = self.dp.api.get_capture(self.dp.configuration.ports[0].name)

test_hero_8_24k_ips_b2b.py:299:

self = <snappi_ixnetwork.snappi_api.Api object at 0x7fbb950482b0>, request = 'VNET-1'

def get_capture(self, request):
    """Gets capture file and returns it as a byte stream"""
    try:
        if (
            isinstance(request, (type(self._capture_request), str))
            is False
        ):
            raise TypeError(
                "The content must be of type Union[CaptureRequest, str]"
            )
        if isinstance(request, str) is True:
            request = self._capture_request.deserialize(request)
        self._connect()
    except Exception as err:

      raise SnappiIxnException(err)

E snappi_ixnetwork.exceptions.SnappiIxnException: File "/usr/local/lib/python3.7/dist-packages/snappi_ixnetwork/snappi_api.py", line 446, in get_capture
E request = self._capture_request.deserialize(request)
E File "/usr/local/lib/python3.7/dist-packages/snappi/snappi.py", line 244, in deserialize
E self._decode(serialized_object)
E File "/usr/local/lib/python3.7/dist-packages/snappi/snappi.py", line 532, in _decode
E for property_name, property_value in obj.items():
E 'str' object has no attribute 'items'

/usr/local/lib/python3.7/dist-packages/snappi_ixnetwork/snappi_api.py:449: SnappiIxnException

Currently the TCP data offset validates something else at index 65 data offset for eth->ipv4->tcp should be at the index 46