If I run my application for the second time, i will always receive a SIGSEGV about ofp HOT 18 OPEN

'default_event_dispatcher' calls ofp_init_local() indeed. The question is: do you have other threads that are using ofp API ? Those threads should be created with odph_thread_create() and should call ofp_init_local() at the beginning.

Note: odph_thread_create() calls underneath odp_init_local() and odp_term_local(). This is why you should use this API for those threads.

from ofp.

I have no idea about wsl....

Else, what's with:

if (m_instance) {
  	odp_term_global(m_instance);
  }

The instance is a number, actually a pid (getpid())

typedef uint64_t odp_instance_t;

from ofp.

the m_instance part is not my problem. as I told you for the first time, everything is OK.

My concern is this part:

I was also thinking that maybe after first run, some processes or FD or something else left on my system. but there is no process left as I saw at htop.

I want to know if there are any leftovers left on the system after an application finishes.

from ofp.

from ofp.

Are you able to run any of the examples or tests for a second time?

I couldn't even run webserver2 for the first time.

Maybe this is not related to the problem at hand, but you are reserving 2M pages, but mounting 1G pages.

Do you mean I should put a number here: /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages ?

Actually, I put 11 there and I still have this issue.

from ofp.

from ofp.

Just found something maybe related to my problem. after I called: odp_init_global(), odp_init_local(), odph_thread_create() I never called: odph_thread_join()

My application still continue to run until in some parts of my code, it tries to create another thread like this:

	pthread_attr_t tattr;
	if (pthread_attr_init(&tattr)) {
		throw Exception("Awww");
	}

and after: pthread_attr_init(&tattr) call, application will crash.

Maybe this is the reason of crash?

from ofp.

I have tried 'webserver2' with @JereLeppanen 's fix and more changes but I don't reproduce this error on my "Ubuntu 22.04 LTS".
ofp_term_global(); odp_term_local(); odp_term_global(instance);
and 'linux_sigaction', etc.

Maybe is related to wsl or application.

On, pthread_attr_init(), remember to pthread_attr_destroy() and not to pthread_attr_init() twice on the same object.

Else, out of curiosity, what exactly are you trying to do (what use case)?

You may also have a look at this more advanced implementation: https://github.com/NetInoSoftware/nfp

from ofp.

I don't talk about webserver2. my use case is very simple. I have a huge application that I try to integrate ofp, odp into it. I ran into crashes in certain cases. Let me clarify the flow of the crash:

At the beginning of the application before doing anything i setup ofp/odp like this:

odp_init_global();
odp_init_local();
ofp_init_global()
ofp_init_local()
odph_thread_create()
...
...

• Notice that i didn't call odph_thread_join() after above function calls.

...
...
...

my applications start to run and continue ...
...
...
as I told in, somewhere in our application we call pthread_attr_init() like this:

pthread_attr_t tattr;
if (pthread_attr_init(&tattr)) {
throw Exception("Awww");
}

And exactly this place is where my application will crash.

My questions are clear:

1. should i call odph_thread_join() after odph_thread_create()?
2. what is the relation of my crash and pthread_attr_init()? (i want to know internally why this happens. why ofp and odp cause this problem? since if i remove ofp and odp from my application i never see this crash.)

from ofp.

from ofp.

Are you not calling ofp_init_local() in every ofp thread at start? If not, anything can happen.

from ofp.

how can i know how many ofp thread i have?

from ofp.

With "ofp thread" I meant a thread that you create and in which you call ofp. IOW, if you create a thread and intend to call ofp functions in it, the first ofp function you call in the thread must be ofp_init_local().

from ofp.

Oh, that's so hard.

What about setting this parameters: thr_params.start = default_event_dispatcher;? it won't do the same thing?

from ofp.

@bogdanPricope aweaome tips. thank you. just one thing. i'm using process instead of pthreads for thread_model:
https://github.com/OpenDataPlane/odp/blob/master/helper/include/odp/helper/threads.h#L166

still i should follow your approach? i mean i should call
odph_thread_create() and ofp_init_local() at the beginning of each thread?

from ofp.

For processes you should use odph_linux_process_fork() (or odph_linux_process_fork_n()). It calls underneath odp_init_local() on child process. That means, you should call ofp_init_local() when child process starts and ofp_term_local() / odp_term_local() when child ends.

from ofp.

I am using the same thr_common.instance that I used for odp_init_global(), for my other thread also. but I get this error:

[New Thread 0xd07fdb40 (LWP 21150)]
E 21 4136856064 thread.cpp:123] SUCCESS: odph_thread_create()
ERR: odp_init.c:611:odp_init_local(): Bad instance.
threads.c:56:run_thread(): Local init failed
[Thread 0xd17ffb40 (LWP 21148) exited]
ERR: odp_init.c:611:odp_init_local(): Bad instance.
threads.c:56:run_thread(): Local init failed
[Thread 0xd0ffeb40 (LWP 21149) exited]
[New Thread 0xcfbffb40 (LWP 21151)]
[New Thread 0xcfbffb40 (LWP 21151)]
E 27 4136856064 thread.cpp:123] SUCCESS: odph_thread_create()

And this is my code for another thread:

	if (0 == ofp_init_local()) {
		odph_thread_t thread_tbl[MAX_WORKERS];
		odph_thread_param_t thr_params;
		odph_thread_common_param_t thr_common;
		memset(thread_tbl, 0, sizeof(thread_tbl));
		/* Start dataplane dispatcher worker threads */
		odph_thread_param_init(&thr_params);
		thr_params.start = default_event_dispatcher;
		thr_params.arg = (void*)ofp_eth_vlan_processing;
		thr_params.thr_type = ODP_THREAD_WORKER;
		odph_thread_common_param_init(&thr_common);
		thr_common.instance = MySingletoonClass::getInstance()->ofpInstance();
		thr_common.cpumask = &cpumask;
		thr_common.share_param = 1;

		if (num_workers == odph_thread_create(thread_tbl, &thr_common, &thr_params, num_workers)) {
			OFP_ERR("SUCCESS: odph_thread_create() .\n");
	
		}
		else {
			OFP_ERR("Error: odph_thread_create() failed.\n");
		}
	}

This is my code for the main thread(beginning of my application):

	if (0 == odp_init_global(&m_instance, NULL, NULL)) {
		if (0 == odp_init_local(m_instance, ODP_THREAD_CONTROL)) {
			ofp_global_param_t app_init_params;
			ofp_init_global_param(&app_init_params);
			int num_workers = 1;
			char cpumaskstr[64];
			odp_cpumask_t cpumask;
			num_workers = odp_cpumask_default_worker(&cpumask, num_workers);
			if (odp_cpumask_to_str(&cpumask, cpumaskstr, sizeof(cpumaskstr)) < 0) {
				OFP_ERR("Error: Too small buffer provided to odp_cpumask_to_str");
			}
			OFP_INFO("Num worker threads: %i", num_workers);
			OFP_INFO("First CPU:          %i", odp_cpumask_first(&cpumask));
			OFP_INFO("CPU mask:           %s", cpumaskstr);

			char interface[25];
			strncpy(interface, "eth0", sizeof(interface)-1);

			char* interfaces[] = {interface};
			app_init_params.if_count = 1;
			app_init_params.if_names = interfaces;

			if (app_init_params.pktin_mode != ODP_PKTIN_MODE_SCHED) {
				app_init_params.pktin_mode = ODP_PKTIN_MODE_SCHED;
			}
			switch (app_init_params.sched_sync) {
			case ODP_SCHED_SYNC_PARALLEL:
				OFP_WARN("Warning: Packet order is not preserved with parallel RX queues\n");
				break;
			case ODP_SCHED_SYNC_ATOMIC:
				break;
			case ODP_SCHED_SYNC_ORDERED:
				if (app_init_params.pktout_mode != ODP_PKTOUT_MODE_QUEUE) {
					OFP_WARN("Warning: Packet order is not preserved with ordered RX queues and direct TX queues.\n");
				}
				break;
			default:
				OFP_WARN("Warning: Unknown scheduling synchronization mode. Forcing atomic mode.\n");
				app_init_params.sched_sync = ODP_SCHED_SYNC_ATOMIC;
				break;
			}

			app_init_params.pkt_hook[OFP_HOOK_LOCAL] = fastpath_local_hook;

			if (0 == ofp_init_global(m_instance, &app_init_params)) {
				if (0 == ofp_init_local()) {
					odph_thread_t thread_tbl[MAX_WORKERS];
					odph_thread_param_t thr_params;
					odph_thread_common_param_t thr_common;
					memset(thread_tbl, 0, sizeof(thread_tbl));
					/* Start dataplane dispatcher worker threads */
					odph_thread_param_init(&thr_params);
					thr_params.start = default_event_dispatcher;
					thr_params.arg = (void*)ofp_eth_vlan_processing;
					thr_params.thr_type = ODP_THREAD_WORKER;
					odph_thread_common_param_init(&thr_common);
					thr_common.instance = m_instance;
					thr_common.cpumask = &cpumask;
					thr_common.share_param = 1;
					//thr_common.sync = 1;
					thr_common.thread_model = 1;

					if (num_workers == odph_thread_create(thread_tbl, &thr_common, &thr_params, num_workers)) {
	                                       // some internal process
					}
					else {
						OFP_ERR("Error: odph_thread_create() failed.\n");
					}
				}
				else {
					OFP_ERR("Error: OFP local init failed.");
				}
			}
			else {
				OFP_ERR("Error: OFP global init failed.");
			}
		}
		else {
			OFP_ERR("Error: ODP local init failed.");
		}
	}
	else {
		OFP_ERR("Error: ODP global init failed.");
	}

from ofp.

Recap:

• the main thread initializes odp, ofp and creates num_workers processes. Is that this part alone (without 'another thread') working well?
• the 'another thread' starts a new worker.

Question:
How is started this 'another thread'? Is it a thread or a process and if is a process when it was forked and with what API?

`int odp_init_local(odp_instance_t instance, odp_thread_type_t thr_type)
{
enum init_stage stage = NO_INIT;

if (instance != (odp_instance_t)odp_global_ro.main_pid) {
	ODP_ERR("Bad instance.\n");
	goto init_fail;
}

.......
`
Either 'odp_global_ro.main_pid' is not initialized or 'instance' is invalid
You may try to print that MySingletoonClass::getInstance()->ofpInstance() (cast it to pid_t or int) and see if it is valid.

from ofp.