Comments (6)
aabc
commented on May 22, 2024
При чём тут peak flows? [...] Текст паники не полностью, что до NMI backtrace?
from ipt-netflow.
aabc
commented on May 22, 2024
А, вы о том что 2000001 больше чем 2000000... Превышение счетчика возможно, на кол-во процессоров в системе, это не критический счетчик. Вам, кстати, нужно бы увеличить maxflows, судя по количеству дропов из-за него (9657973).
from ipt-netflow.
mort1k
commented on May 22, 2024
Feb 29 19:25:37 r2 kernel: [6571717.011269] INFO: rcu_sched self-detected stall on CPU
Feb 29 19:25:37 r2 kernel: [6571717.051277] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=6004 jiffies, g=119659536, c=119659535, q=16496)
Feb 29 19:25:37 r2 kernel: [6571717.051285] NMI backtrace for cpu 0
Feb 29 19:25:37 r2 kernel: [6571717.051288] CPU: 0 PID: 2116 Comm: kworker/0:1 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.051289] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.051293] Workqueue: events netflow_work_fn [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051295] task: ffff88007c1f3100 ti: ffff88005527c000 task.ti: ffff88005527c000
Feb 29 19:25:37 r2 kernel: [6571717.051296] RIP: 0010:[<ffffffff8128de4d>] [<ffffffff8128de4d>] io_serial_in+0xc/0x10
Feb 29 19:25:37 r2 kernel: [6571717.051300] RSP: 0018:ffff88007f403768 EFLAGS: 00000002
Feb 29 19:25:37 r2 kernel: [6571717.051301] RAX: 0034c6e682f2dd00 RBX: ffffffff817b7c90 RCX: 0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051302] RDX: 00000000000003fd RSI: 00000000000003fd RDI: ffffffff817b7c90
Feb 29 19:25:37 r2 kernel: [6571717.051303] RBP: 000000000000256c R08: 000000000000807e R09: 0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051304] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000020
Feb 29 19:25:37 r2 kernel: [6571717.051304] R13: ffffffff8175b3c8 R14: ffffffff8128e3f2 R15: 0000000000000082
Feb 29 19:25:37 r2 kernel: [6571717.051306] FS: 0000000000000000(0000) GS:ffff88007f400000(0000) knlGS:0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051307] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 29 19:25:37 r2 kernel: [6571717.051308] CR2: ffffffffff600400 CR3: 000000005335a000 CR4: 00000000000007f0
Feb 29 19:25:37 r2 kernel: [6571717.051308] Stack:
Feb 29 19:25:37 r2 kernel: [6571717.051309] ffffffff8128e394 ffffffff817b7c90 0000000000000072 000000000000003b
Feb 29 19:25:37 r2 kernel: [6571717.051310] ffffffff8128e407 0000000000000000 ffffffff8175b3b0 ffffffff817b7c90
Feb 29 19:25:37 r2 kernel: [6571717.051312] ffffffff8128abc1 ffffffff817b7c90 0000000000000000 0000000000000001
Feb 29 19:25:37 r2 kernel: [6571717.051314] Call Trace:
Feb 29 19:25:37 r2 kernel: [6571717.051315] <IRQ> [<ffffffff8128e394>] ? wait_for_xmitr+0x1a/0x78
Feb 29 19:25:37 r2 kernel: [6571717.051318] [<ffffffff8128e407>] ? serial8250_console_putchar+0x15/0x25
Feb 29 19:25:37 r2 kernel: [6571717.051320] [<ffffffff8128abc1>] ? uart_console_write+0x39/0x4c
Feb 29 19:25:37 r2 kernel: [6571717.051323] [<ffffffff8128fe7c>] ? serial8250_console_write+0xb3/0x109
Feb 29 19:25:37 r2 kernel: [6571717.051326] [<ffffffff8107b394>] ? T.998+0xaa/0xb8
Feb 29 19:25:37 r2 kernel: [6571717.051329] [<ffffffff8107b4cd>] ? console_unlock+0x12b/0x340
Feb 29 19:25:37 r2 kernel: [6571717.051331] [<ffffffff8107bb7c>] ? vprintk_emit+0x36c/0x39d
Feb 29 19:25:37 r2 kernel: [6571717.051332] [<ffffffff810871b1>] ? __getnstimeofday+0x28/0x6d
Feb 29 19:25:37 r2 kernel: [6571717.051335] [<ffffffff813e84a6>] ? printk+0x4c/0x51
Feb 29 19:25:37 r2 kernel: [6571717.051340] [<ffffffffa0395b3e>] ? hash_ip4_test+0xfe/0x10f [ip_set_hash_ip]
Feb 29 19:25:37 r2 kernel: [6571717.051342] [<ffffffff81085f39>] ? rcu_check_callbacks+0x154/0x54f
Feb 29 19:25:37 r2 kernel: [6571717.051344] [<ffffffff81088087>] ? do_timer+0x3cc/0x3eb
Feb 29 19:25:37 r2 kernel: [6571717.051346] [<ffffffff8108db58>] ? tick_nohz_handler+0xce/0xce
Feb 29 19:25:37 r2 kernel: [6571717.051348] [<ffffffff810534d7>] ? update_process_times+0x31/0x56
Feb 29 19:25:37 r2 kernel: [6571717.051350] [<ffffffff8108dbcc>] ? tick_sched_timer+0x74/0x90
Feb 29 19:25:37 r2 kernel: [6571717.051352] [<ffffffff81066c93>] ? __run_hrtimer+0x92/0x11c
Feb 29 19:25:37 r2 kernel: [6571717.051354] [<ffffffff81066f7a>] ? hrtimer_interrupt+0xde/0x1ec
Feb 29 19:25:37 r2 kernel: [6571717.051356] [<ffffffffa03879e6>] ? ip_set_test+0xba/0x142 [ip_set]
Feb 29 19:25:37 r2 kernel: [6571717.051359] [<ffffffff81034548>] ? smp_apic_timer_interrupt+0x1d/0x2d
Feb 29 19:25:37 r2 kernel: [6571717.051362] [<ffffffff813ecc5d>] ? apic_timer_interrupt+0x6d/0x80
Feb 29 19:25:37 r2 kernel: [6571717.051365] [<ffffffff813b6242>] ? check_leaf+0x11a/0x135
Feb 29 19:25:37 r2 kernel: [6571717.051368] [<ffffffff813b667b>] ? fib_table_lookup+0xf3/0x246
Feb 29 19:25:37 r2 kernel: [6571717.051370] [<ffffffff8137e034>] ? fib_lookup+0x53/0x90
Feb 29 19:25:37 r2 kernel: [6571717.051373] [<ffffffff8137f6cd>] ? ip_route_input_noref+0x38a/0x9c8
Feb 29 19:25:37 r2 kernel: [6571717.051375] [<ffffffff813835b0>] ? ip_check_defrag+0x13a/0x13a
Feb 29 19:25:37 r2 kernel: [6571717.051377] [<ffffffff81381d3a>] ? ip_rcv_finish+0x7e/0x2b9
Feb 29 19:25:37 r2 kernel: [6571717.051379] [<ffffffff813585e2>] ? __netif_receive_skb_core+0x4c5/0x4fd
Feb 29 19:25:37 r2 kernel: [6571717.051382] [<ffffffff81014f15>] ? read_tsc+0x5/0x16
Feb 29 19:25:37 r2 kernel: [6571717.051385] [<ffffffff81086ebd>] ? T.823+0xd/0x31
Feb 29 19:25:37 r2 kernel: [6571717.051386] [<ffffffff813588bc>] ? netif_receive_skb+0x81/0x87
Feb 29 19:25:37 r2 kernel: [6571717.051388] [<ffffffff81359254>] ? napi_gro_receive+0xa7/0xe5
Feb 29 19:25:37 r2 kernel: [6571717.051399] [<ffffffffa0064005>] ? ixgbe_clean_rx_irq+0x751/0x7f7 [ixgbe]
Feb 29 19:25:37 r2 kernel: [6571717.051405] [<ffffffffa0064683>] ? ixgbe_poll+0x4ea/0x655 [ixgbe]
Feb 29 19:25:37 r2 kernel: [6571717.051410] [<ffffffff813588bc>] ? netif_receive_skb+0x81/0x87
Feb 29 19:25:37 r2 kernel: [6571717.051412] [<ffffffff81358dfa>] ? net_rx_action+0xa8/0x22e
Feb 29 19:25:37 r2 kernel: [6571717.051414] [<ffffffff8104d563>] ? __do_softirq+0x100/0x244
Feb 29 19:25:37 r2 kernel: [6571717.051416] [<ffffffff813ed91c>] ? do_softirq_own_stack+0x1c/0x30
Feb 29 19:25:37 r2 kernel: [6571717.051419] <EOI> [<ffffffff8104d2fa>] ? do_softirq+0x3a/0x4b
Feb 29 19:25:37 r2 kernel: [6571717.051421] [<ffffffff8104d3b3>] ? _local_bh_enable_ip+0x6c/0x76
Feb 29 19:25:37 r2 kernel: [6571717.051422] [<ffffffff813869d7>] ? ip_finish_output2+0x298/0x306
Feb 29 19:25:37 r2 kernel: [6571717.051425] [<ffffffff81386d14>] ? ip_send_skb+0xc/0x2f
Feb 29 19:25:37 r2 kernel: [6571717.051426] [<ffffffff813a5dc1>] ? udp_send_skb+0x187/0x1e6
Feb 29 19:25:37 r2 kernel: [6571717.051429] [<ffffffff813a661a>] ? udp_sendmsg+0x71d/0x739
Feb 29 19:25:37 r2 kernel: [6571717.051431] [<ffffffff81385daf>] ? ip_append_page+0x4b4/0x4b4
Feb 29 19:25:37 r2 kernel: [6571717.051433] [<ffffffff8134299d>] ? sock_sendmsg+0x4e/0x66
Feb 29 19:25:37 r2 kernel: [6571717.051436] [<ffffffff811d39d9>] ? string+0x43/0xa2
Feb 29 19:25:37 r2 kernel: [6571717.051438] [<ffffffff813eb43a>] ? _raw_spin_unlock+0x5/0x6
Feb 29 19:25:37 r2 kernel: [6571717.051440] [<ffffffff8110e1f9>] ? unfreeze_partials+0xcf/0xf6
Feb 29 19:25:37 r2 kernel: [6571717.051443] [<ffffffff81342f85>] ? kernel_sendmsg+0x31/0x3c
Feb 29 19:25:37 r2 kernel: [6571717.051445] [<ffffffffa03f5f4d>] ? netflow_sendmsg+0xe9/0x2ab [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051447] [<ffffffff810871b1>] ? __getnstimeofday+0x28/0x6d
Feb 29 19:25:37 r2 kernel: [6571717.051449] [<ffffffffa03f619c>] ? netflow_export_pdu_ipfix+0x8d/0xee [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051451] [<ffffffffa03f11f1>] ? pdu_alloc_fail_export+0x1e/0x2b [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051453] [<ffffffffa03f250d>] ? alloc_record_key+0x66/0x315 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051455] [<ffffffff8110ed62>] ? kmem_cache_free+0x81/0xb9
Feb 29 19:25:37 r2 kernel: [6571717.051457] [<ffffffffa03f3fcb>] ? netflow_export_flow_tpl+0x18e/0x6e1 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051459] [<ffffffff81014f15>] ? read_tsc+0x5/0x16
Feb 29 19:25:37 r2 kernel: [6571717.051461] [<ffffffff81086ebd>] ? T.823+0xd/0x31
Feb 29 19:25:37 r2 kernel: [6571717.051462] [<ffffffff810871b1>] ? __getnstimeofday+0x28/0x6d
Feb 29 19:25:37 r2 kernel: [6571717.051464] [<ffffffffa03f32fe>] ? netflow_scan_and_export+0x4b7/0x530 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051466] [<ffffffffa03f33bc>] ? netflow_work_fn+0x45/0x64 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.051468] [<ffffffff81060867>] ? process_one_work+0x1fb/0x302
Feb 29 19:25:37 r2 kernel: [6571717.051470] [<ffffffff81060acd>] ? worker_thread+0x15f/0x26c
Feb 29 19:25:37 r2 kernel: [6571717.051472] [<ffffffff8106096e>] ? process_one_work+0x302/0x302
Feb 29 19:25:37 r2 kernel: [6571717.051474] [<ffffffff8106096e>] ? process_one_work+0x302/0x302
Feb 29 19:25:37 r2 kernel: [6571717.051475] [<ffffffff810640b5>] ? kthread+0xc3/0xcb
Feb 29 19:25:37 r2 kernel: [6571717.051477] [<ffffffff81063ff2>] ? kthread_freezable_should_stop+0x51/0x51
Feb 29 19:25:37 r2 kernel: [6571717.051479] [<ffffffff813ebfcc>] ? ret_from_fork+0x7c/0xb0
Feb 29 19:25:37 r2 kernel: [6571717.051480] [<ffffffff81063ff2>] ? kthread_freezable_should_stop+0x51/0x51
Feb 29 19:25:37 r2 kernel: [6571717.051482] Code: b6 4f 61 d3 e6 48 63 f6 48 03 77 10 89 16 c3 0f b6 4f 61 d3 e6 48 63 f6 48 03 77 10 8b 06 c3 0f b6 4f 61 d3 e6 03 77 08 89 f2 ec <0f> b6 c0 c3 0f b6 4f 61 89 d0 d3 e6 03 77 08 89 f2 ee c3 8a 47
Feb 29 19:25:37 r2 kernel: [6571717.051500] NMI backtrace for cpu 1
Feb 29 19:25:37 r2 kernel: [6571717.051502] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.051502] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.051504] task: ffff88007cb93f00 ti: ffff88007cbb8000 task.ti: ffff88007cbb8000
Feb 29 19:25:37 r2 kernel: [6571717.051505] RIP: 0010:[<ffffffff81014ed5>] [<ffffffff81014ed5>] native_read_tsc+0x2/0x11
Feb 29 19:25:37 r2 kernel: [6571717.051508] RSP: 0018:ffff88007f423d98 EFLAGS: 00000002
Feb 29 19:25:37 r2 kernel: [6571717.051509] RAX: 0000000082f65af9 RBX: 0000000082f65ad5 RCX: 0000000082f65ad5
Feb 29 19:25:37 r2 kernel: [6571717.051510] RDX: 000000000034c6e6 RSI: 0000000000000040 RDI: 0000000000037328
Feb 29 19:25:37 r2 kernel: [6571717.051511] RBP: 0000000000000001 R08: 0000000000000000 R09: ffffffff8161d2e8
Feb 29 19:25:37 r2 kernel: [6571717.051512] R10: 000000000721dc0f R11: 0000000000000000 R12: 0000000000000002
Feb 29 19:25:37 r2 kernel: [6571717.051513] R13: 0000000000037328 R14: ffffffff8161d2e8 R15: 0000000000000002
Feb 29 19:25:37 r2 kernel: [6571717.051514] FS: 0000000000000000(0000) GS:ffff88007f420000(0000) knlGS:0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051515] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 29 19:25:37 r2 kernel: [6571717.051516] CR2: 00007f89897b3000 CR3: 000000005335a000 CR4: 00000000000007e0
Feb 29 19:25:37 r2 kernel: [6571717.051516] Stack:
Feb 29 19:25:37 r2 kernel: [6571717.051517] ffffffff811d5d49 0000000000000000 0000000000001000 0000000000000002
Feb 29 19:25:37 r2 kernel: [6571717.051519] 0000000000000400 ffffffff810348e9 ffffffff8161d2e0 0000000000000096
Feb 29 19:25:37 r2 kernel: [6571717.051520] 000000000000b024 ffffffff81034ced ffff88007f420004 ffff88007f423e48
Feb 29 19:25:37 r2 kernel: [6571717.051522] Call Trace:
Feb 29 19:25:37 r2 kernel: [6571717.051523] <IRQ> [<ffffffff811d5d49>] ? delay_tsc+0x2b/0x72
Feb 29 19:25:37 r2 kernel: [6571717.051526] [<ffffffff810348e9>] ? native_safe_apic_wait_icr_idle+0x36/0x48
Feb 29 19:25:37 r2 kernel: [6571717.051529] [<ffffffff81034ced>] ? default_send_IPI_mask_sequence_phys+0x5c/0xc7
Feb 29 19:25:37 r2 kernel: [6571717.051531] [<ffffffff81034d9c>] ? arch_trigger_all_cpu_backtrace+0x44/0x6c
Feb 29 19:25:37 r2 kernel: [6571717.051533] [<ffffffff81086247>] ? rcu_check_callbacks+0x462/0x54f
Feb 29 19:25:37 r2 kernel: [6571717.051535] [<ffffffff8108db58>] ? tick_nohz_handler+0xce/0xce
Feb 29 19:25:37 r2 kernel: [6571717.051536] [<ffffffff810534d7>] ? update_process_times+0x31/0x56
Feb 29 19:25:37 r2 kernel: [6571717.051538] [<ffffffff8108dbcc>] ? tick_sched_timer+0x74/0x90
Feb 29 19:25:37 r2 kernel: [6571717.051540] [<ffffffff81066c93>] ? __run_hrtimer+0x92/0x11c
Feb 29 19:25:37 r2 kernel: [6571717.051542] [<ffffffff81066f7a>] ? hrtimer_interrupt+0xde/0x1ec
Feb 29 19:25:37 r2 kernel: [6571717.051544] [<ffffffff81034548>] ? smp_apic_timer_interrupt+0x1d/0x2d
Feb 29 19:25:37 r2 kernel: [6571717.051546] [<ffffffff813ecc5d>] ? apic_timer_interrupt+0x6d/0x80
Feb 29 19:25:37 r2 kernel: [6571717.051548] <EOI> [<ffffffff81066a61>] ? __hrtimer_start_range_ns+0x26e/0x281
Feb 29 19:25:37 r2 kernel: [6571717.051550] [<ffffffff81337a69>] ? cpuidle_enter_state+0x3d/0xa8
Feb 29 19:25:37 r2 kernel: [6571717.051553] [<ffffffff81337a62>] ? cpuidle_enter_state+0x36/0xa8
Feb 29 19:25:37 r2 kernel: [6571717.051554] [<ffffffff81337bf1>] ? cpuidle_idle_call+0xd6/0x129
Feb 29 19:25:37 r2 kernel: [6571717.051556] [<ffffffff81015e64>] ? arch_cpu_idle+0x9/0x20
Feb 29 19:25:37 r2 kernel: [6571717.051558] [<ffffffff8107d151>] ? cpu_startup_entry+0x132/0x1a9
Feb 29 19:25:37 r2 kernel: [6571717.051560] [<ffffffff8103244a>] ? start_secondary+0x25e/0x263
Feb 29 19:25:37 r2 kernel: [6571717.051562] Code: f8 81 e7 ff 03 00 00 48 c1 e8 0a 48 0f af 3c 11 48 0f af 04 11 48 03 04 31 48 c1 ef 0a 48 01 f8 c3 e8 ed 08 00 00 66 90 c3 0f 31 <89> c1 48 89 d0 48 c1 e0 20 89 c9 48 09 c8 c3 8b 05 7a 65 60 00
Feb 29 19:25:37 r2 kernel: [6571717.051648] NMI backtrace for cpu 2
Feb 29 19:25:37 r2 kernel: [6571717.051651] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.051651] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.051653] task: ffff88007cb94600 ti: ffff88007cbba000 task.ti: ffff88007cbba000
Feb 29 19:25:37 r2 kernel: [6571717.051654] RIP: 0010:[<ffffffff81218e52>] [<ffffffff81218e52>] intel_idle+0xc6/0xe8
Feb 29 19:25:37 r2 kernel: [6571717.051658] RSP: 0018:ffff88007cbbbe28 EFLAGS: 00000046
Feb 29 19:25:37 r2 kernel: [6571717.051659] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Feb 29 19:25:37 r2 kernel: [6571717.051660] RDX: 0000000000000000 RSI: ffff88007cbbbfd8 RDI: 0000000000000092
Feb 29 19:25:37 r2 kernel: [6571717.051661] RBP: 0000000000000004 R08: 00175791e8168c8a R09: 00000000000016f1
Feb 29 19:25:37 r2 kernel: [6571717.051662] R10: ffff88007f44db00 R11: ffff88007f44e601 R12: 0000000000000020
Feb 29 19:25:37 r2 kernel: [6571717.051662] R13: 0000000000000004 R14: ffff88007cbba010 R15: ffff88007cbba010
Feb 29 19:25:37 r2 kernel: [6571717.051664] FS: 0000000000000000(0000) GS:ffff88007f440000(0000) knlGS:0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051665] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 29 19:25:37 r2 kernel: [6571717.051666] CR2: ffffffffff600400 CR3: 000000000159c000 CR4: 00000000000007e0
Feb 29 19:25:37 r2 kernel: [6571717.051666] Stack:
Feb 29 19:25:37 r2 kernel: [6571717.051667] 00000000006440e0 0000000281086f24 ffff88007f45a000 ffffffff815df8a0
Feb 29 19:25:37 r2 kernel: [6571717.051669] 00175791e8166d8a ffffffff81337a5b 0000000000000000 0000000000000004
Feb 29 19:25:37 r2 kernel: [6571717.051670] 0000000000000000 ffff88007f45a000 ffffffff815df8a0 ffff88007cbba000
Feb 29 19:25:37 r2 kernel: [6571717.051672] Call Trace:
Feb 29 19:25:37 r2 kernel: [6571717.051673] [<ffffffff81337a5b>] ? cpuidle_enter_state+0x2f/0xa8
Feb 29 19:25:37 r2 kernel: [6571717.051675] [<ffffffff81337bf1>] ? cpuidle_idle_call+0xd6/0x129
Feb 29 19:25:37 r2 kernel: [6571717.051676] [<ffffffff81015e64>] ? arch_cpu_idle+0x9/0x20
Feb 29 19:25:37 r2 kernel: [6571717.051679] [<ffffffff8107d151>] ? cpu_startup_entry+0x132/0x1a9
Feb 29 19:25:37 r2 kernel: [6571717.051681] [<ffffffff8103244a>] ? start_secondary+0x25e/0x263
Feb 29 19:25:37 r2 kernel: [6571717.051684] Code: 48 8b 34 25 08 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <85> 1d 00 6e 3c 00 75 0f 48 8d 74 24 0c bf 05 00 00 00 e8 15 2e
Feb 29 19:25:37 r2 kernel: [6571717.051702] NMI backtrace for cpu 3
Feb 29 19:25:37 r2 kernel: [6571717.051704] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.051705] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.051706] task: ffff88007cb94d00 ti: ffff88007cbbc000 task.ti: ffff88007cbbc000
Feb 29 19:25:37 r2 kernel: [6571717.051707] RIP: 0010:[<ffffffff81218e52>] [<ffffffff81218e52>] intel_idle+0xc6/0xe8
Feb 29 19:25:37 r2 kernel: [6571717.051711] RSP: 0018:ffff88007cbbde28 EFLAGS: 00000046
Feb 29 19:25:37 r2 kernel: [6571717.051711] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
Feb 29 19:25:37 r2 kernel: [6571717.051712] RDX: 0000000000000000 RSI: ffff88007cbbdfd8 RDI: 0000000000000003
Feb 29 19:25:37 r2 kernel: [6571717.051713] RBP: 0000000000000001 R08: 0000000000000406 R09: 0000000000009c62
Feb 29 19:25:37 r2 kernel: [6571717.051714] R10: 0140000000000000 R11: 0000000000000206 R12: 0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051715] R13: 0000000000000001 R14: ffff88007cbbc010 R15: ffff88007cbbc010
Feb 29 19:25:37 r2 kernel: [6571717.051716] FS: 0000000000000000(0000) GS:ffff88007f460000(0000) knlGS:0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.051717] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 29 19:25:37 r2 kernel: [6571717.051718] CR2: ffffffffff600400 CR3: 00000000712ae000 CR4: 00000000000007e0
Feb 29 19:25:37 r2 kernel: [6571717.051718] Stack:
Feb 29 19:25:37 r2 kernel: [6571717.051719] 00000000006440e0 0000000381086f24 ffff88007f47a000 ffffffff815df8a0
Feb 29 19:25:37 r2 kernel: [6571717.051721] 00175791e9e03cf7 ffffffff81337a5b 0000000000000000 0000000000000001
Feb 29 19:25:37 r2 kernel: [6571717.051722] 0000000000000000 ffff88007f47a000 ffffffff815df8a0 ffff88007cbbc000
Feb 29 19:25:37 r2 kernel: [6571717.051724] Call Trace:
Feb 29 19:25:37 r2 kernel: [6571717.051725] [<ffffffff81337a5b>] ? cpuidle_enter_state+0x2f/0xa8
Feb 29 19:25:37 r2 kernel: [6571717.051727] [<ffffffff81337bf1>] ? cpuidle_idle_call+0xd6/0x129
Feb 29 19:25:37 r2 kernel: [6571717.051729] [<ffffffff81015e64>] ? arch_cpu_idle+0x9/0x20
Feb 29 19:25:37 r2 kernel: [6571717.051731] [<ffffffff8107d151>] ? cpu_startup_entry+0x132/0x1a9
Feb 29 19:25:37 r2 kernel: [6571717.051733] [<ffffffff8103244a>] ? start_secondary+0x25e/0x263
Feb 29 19:25:37 r2 kernel: [6571717.051735] Code: 48 8b 34 25 08 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <85> 1d 00 6e 3c 00 75 0f 48 8d 74 24 0c bf 05 00 00 00 e8 15 2e
Feb 29 19:25:37 r2 kernel: [6571717.154498] { 0} (t=6008 jiffies g=119659536 c=119659535 q=16496)
Feb 29 19:25:37 r2 kernel: [6571717.177132] NMI backtrace for cpu 0
Feb 29 19:25:37 r2 kernel: [6571717.177135] CPU: 0 PID: 2116 Comm: kworker/0:1 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.177136] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.177139] Workqueue: events netflow_work_fn [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177141] task: ffff88007c1f3100 ti: ffff88005527c000 task.ti: ffff88005527c000
Feb 29 19:25:37 r2 kernel: [6571717.177143] RIP: 0010:[<ffffffff81014ed3>] [<ffffffff81014ed3>] native_read_tsc+0x0/0x11
Feb 29 19:25:37 r2 kernel: [6571717.177146] RSP: 0018:ffff88007f4038b0 EFLAGS: 00000002
Feb 29 19:25:37 r2 kernel: [6571717.177148] RAX: 0034c6e693e7b7a8 RBX: 0000000093e7b7a8 RCX: 0000000093e7b7a8
Feb 29 19:25:37 r2 kernel: [6571717.177150] RDX: 000000000034c6e6 RSI: 0000000000000040 RDI: 0000000000037328
Feb 29 19:25:37 r2 kernel: [6571717.177151] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff8161d2e8
Feb 29 19:25:37 r2 kernel: [6571717.177153] R10: 0000000000004070 R11: 0000000000000000 R12: 0000000000000002
Feb 29 19:25:37 r2 kernel: [6571717.177155] R13: 0000000000037328 R14: ffffffff8161d2e8 R15: 0000000000000001
Feb 29 19:25:37 r2 kernel: [6571717.177157] FS: 0000000000000000(0000) GS:ffff88007f400000(0000) knlGS:0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.177159] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 29 19:25:37 r2 kernel: [6571717.177160] CR2: ffffffffff600400 CR3: 000000005335a000 CR4: 00000000000007f0
Feb 29 19:25:37 r2 kernel: [6571717.177162] Stack:
Feb 29 19:25:37 r2 kernel: [6571717.177163] ffffffff811d5d49 0000000000000000 0000000000001000 0000000000000002
Feb 29 19:25:37 r2 kernel: [6571717.177165] 0000000000000400 ffffffff810348e9 ffffffff8161d2e0 0000000000000086
Feb 29 19:25:37 r2 kernel: [6571717.177168] 000000000000b024 ffffffff81034ced ffff88007f400002 ffff88007f403960
Feb 29 19:25:37 r2 kernel: [6571717.177171] Call Trace:
Feb 29 19:25:37 r2 kernel: [6571717.177172] <IRQ>
Feb 29 19:25:37 r2 kernel: [6571717.177173] [<ffffffff811d5d49>] ? delay_tsc+0x2b/0x72
Feb 29 19:25:37 r2 kernel: [6571717.177178] [<ffffffff810348e9>] ? native_safe_apic_wait_icr_idle+0x36/0x48
Feb 29 19:25:37 r2 kernel: [6571717.177181] [<ffffffff81034ced>] ? default_send_IPI_mask_sequence_phys+0x5c/0xc7
Feb 29 19:25:37 r2 kernel: [6571717.177184] [<ffffffff81034d9c>] ? arch_trigger_all_cpu_backtrace+0x44/0x6c
Feb 29 19:25:37 r2 kernel: [6571717.177186] [<ffffffff81085fdd>] ? rcu_check_callbacks+0x1f8/0x54f
Feb 29 19:25:37 r2 kernel: [6571717.177189] [<ffffffff81088087>] ? do_timer+0x3cc/0x3eb
Feb 29 19:25:37 r2 kernel: [6571717.177191] [<ffffffff8108db58>] ? tick_nohz_handler+0xce/0xce
Feb 29 19:25:37 r2 kernel: [6571717.177194] [<ffffffff810534d7>] ? update_process_times+0x31/0x56
Feb 29 19:25:37 r2 kernel: [6571717.177196] [<ffffffff8108dbcc>] ? tick_sched_timer+0x74/0x90
Feb 29 19:25:37 r2 kernel: [6571717.177199] [<ffffffff81066c93>] ? __run_hrtimer+0x92/0x11c
Feb 29 19:25:37 r2 kernel: [6571717.177201] [<ffffffff81066f7a>] ? hrtimer_interrupt+0xde/0x1ec
Feb 29 19:25:37 r2 kernel: [6571717.177205] [<ffffffffa03879e6>] ? ip_set_test+0xba/0x142 [ip_set]
Feb 29 19:25:37 r2 kernel: [6571717.177208] [<ffffffff81034548>] ? smp_apic_timer_interrupt+0x1d/0x2d
Feb 29 19:25:37 r2 kernel: [6571717.177211] [<ffffffff813ecc5d>] ? apic_timer_interrupt+0x6d/0x80
Feb 29 19:25:37 r2 kernel: [6571717.177214] [<ffffffff813b6242>] ? check_leaf+0x11a/0x135
Feb 29 19:25:37 r2 kernel: [6571717.177217] [<ffffffff813b667b>] ? fib_table_lookup+0xf3/0x246
Feb 29 19:25:37 r2 kernel: [6571717.177219] [<ffffffff8137e034>] ? fib_lookup+0x53/0x90
Feb 29 19:25:37 r2 kernel: [6571717.177222] [<ffffffff8137f6cd>] ? ip_route_input_noref+0x38a/0x9c8
Feb 29 19:25:37 r2 kernel: [6571717.177224] [<ffffffff813835b0>] ? ip_check_defrag+0x13a/0x13a
Feb 29 19:25:37 r2 kernel: [6571717.177227] [<ffffffff81381d3a>] ? ip_rcv_finish+0x7e/0x2b9
Feb 29 19:25:37 r2 kernel: [6571717.177230] [<ffffffff813585e2>] ? __netif_receive_skb_core+0x4c5/0x4fd
Feb 29 19:25:37 r2 kernel: [6571717.177233] [<ffffffff81014f15>] ? read_tsc+0x5/0x16
Feb 29 19:25:37 r2 kernel: [6571717.177235] [<ffffffff81086ebd>] ? T.823+0xd/0x31
Feb 29 19:25:37 r2 kernel: [6571717.177238] [<ffffffff813588bc>] ? netif_receive_skb+0x81/0x87
Feb 29 19:25:37 r2 kernel: [6571717.177241] [<ffffffff81359254>] ? napi_gro_receive+0xa7/0xe5
Feb 29 19:25:37 r2 kernel: [6571717.177247] [<ffffffffa0064005>] ? ixgbe_clean_rx_irq+0x751/0x7f7 [ixgbe]
Feb 29 19:25:37 r2 kernel: [6571717.177254] [<ffffffffa0064683>] ? ixgbe_poll+0x4ea/0x655 [ixgbe]
Feb 29 19:25:37 r2 kernel: [6571717.177257] [<ffffffff813588bc>] ? netif_receive_skb+0x81/0x87
Feb 29 19:25:37 r2 kernel: [6571717.177260] [<ffffffff81358dfa>] ? net_rx_action+0xa8/0x22e
Feb 29 19:25:37 r2 kernel: [6571717.177262] [<ffffffff8104d563>] ? __do_softirq+0x100/0x244
Feb 29 19:25:37 r2 kernel: [6571717.177265] [<ffffffff813ed91c>] ? do_softirq_own_stack+0x1c/0x30
Feb 29 19:25:37 r2 kernel: [6571717.177267] <EOI>
Feb 29 19:25:37 r2 kernel: [6571717.177267] [<ffffffff8104d2fa>] ? do_softirq+0x3a/0x4b
Feb 29 19:25:37 r2 kernel: [6571717.177272] [<ffffffff8104d3b3>] ? _local_bh_enable_ip+0x6c/0x76
Feb 29 19:25:37 r2 kernel: [6571717.177275] [<ffffffff813869d7>] ? ip_finish_output2+0x298/0x306
Feb 29 19:25:37 r2 kernel: [6571717.177277] [<ffffffff81386d14>] ? ip_send_skb+0xc/0x2f
Feb 29 19:25:37 r2 kernel: [6571717.177280] [<ffffffff813a5dc1>] ? udp_send_skb+0x187/0x1e6
Feb 29 19:25:37 r2 kernel: [6571717.177282] [<ffffffff813a661a>] ? udp_sendmsg+0x71d/0x739
Feb 29 19:25:37 r2 kernel: [6571717.177285] [<ffffffff81385daf>] ? ip_append_page+0x4b4/0x4b4
Feb 29 19:25:37 r2 kernel: [6571717.177288] [<ffffffff8134299d>] ? sock_sendmsg+0x4e/0x66
Feb 29 19:25:37 r2 kernel: [6571717.177291] [<ffffffff811d39d9>] ? string+0x43/0xa2
Feb 29 19:25:37 r2 kernel: [6571717.177294] [<ffffffff813eb43a>] ? _raw_spin_unlock+0x5/0x6
Feb 29 19:25:37 r2 kernel: [6571717.177296] [<ffffffff8110e1f9>] ? unfreeze_partials+0xcf/0xf6
Feb 29 19:25:37 r2 kernel: [6571717.177299] [<ffffffff81342f85>] ? kernel_sendmsg+0x31/0x3c
Feb 29 19:25:37 r2 kernel: [6571717.177302] [<ffffffffa03f5f4d>] ? netflow_sendmsg+0xe9/0x2ab [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177305] [<ffffffff810871b1>] ? __getnstimeofday+0x28/0x6d
Feb 29 19:25:37 r2 kernel: [6571717.177309] [<ffffffffa03f619c>] ? netflow_export_pdu_ipfix+0x8d/0xee [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177312] [<ffffffffa03f11f1>] ? pdu_alloc_fail_export+0x1e/0x2b [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177315] [<ffffffffa03f250d>] ? alloc_record_key+0x66/0x315 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177317] [<ffffffff8110ed62>] ? kmem_cache_free+0x81/0xb9
Feb 29 19:25:37 r2 kernel: [6571717.177320] [<ffffffffa03f3fcb>] ? netflow_export_flow_tpl+0x18e/0x6e1 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177323] [<ffffffff81014f15>] ? read_tsc+0x5/0x16
Feb 29 19:25:37 r2 kernel: [6571717.177325] [<ffffffff81086ebd>] ? T.823+0xd/0x31
Feb 29 19:25:37 r2 kernel: [6571717.177328] [<ffffffff810871b1>] ? __getnstimeofday+0x28/0x6d
Feb 29 19:25:37 r2 kernel: [6571717.177331] [<ffffffffa03f32fe>] ? netflow_scan_and_export+0x4b7/0x530 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177334] [<ffffffffa03f33bc>] ? netflow_work_fn+0x45/0x64 [ipt_NETFLOW]
Feb 29 19:25:37 r2 kernel: [6571717.177337] [<ffffffff81060867>] ? process_one_work+0x1fb/0x302
Feb 29 19:25:37 r2 kernel: [6571717.177340] [<ffffffff81060acd>] ? worker_thread+0x15f/0x26c
Feb 29 19:25:37 r2 kernel: [6571717.177342] [<ffffffff8106096e>] ? process_one_work+0x302/0x302
Feb 29 19:25:37 r2 kernel: [6571717.177345] [<ffffffff8106096e>] ? process_one_work+0x302/0x302
Feb 29 19:25:37 r2 kernel: [6571717.177347] [<ffffffff810640b5>] ? kthread+0xc3/0xcb
Feb 29 19:25:37 r2 kernel: [6571717.177350] [<ffffffff81063ff2>] ? kthread_freezable_should_stop+0x51/0x51
Feb 29 19:25:37 r2 kernel: [6571717.177353] [<ffffffff813ebfcc>] ? ret_from_fork+0x7c/0xb0
Feb 29 19:25:37 r2 kernel: [6571717.177355] [<ffffffff81063ff2>] ? kthread_freezable_should_stop+0x51/0x51
Feb 29 19:25:37 r2 kernel: [6571717.177356] Code: 48 89 f8 81 e7 ff 03 00 00 48 c1 e8 0a 48 0f af 3c 11 48 0f af 04 11 48 03 04 31 48 c1 ef 0a 48 01 f8 c3 e8 ed 08 00 00 66 90 c3 <0f> 31 89 c1 48 89 d0 48 c1 e0 20 89 c9 48 09 c8 c3 8b 05 7a 65
Feb 29 19:25:37 r2 kernel: [6571717.177439] NMI backtrace for cpu 1
Feb 29 19:25:37 r2 kernel: [6571717.177443] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.177445] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.177447] task: ffff88007cb93f00 ti: ffff88007cbb8000 task.ti: ffff88007cbb8000
Feb 29 19:25:37 r2 kernel: [6571717.177449] RIP: 0010:[<ffffffff81218e52>] [<ffffffff81218e52>] intel_idle+0xc6/0xe8
Feb 29 19:25:37 r2 kernel: [6571717.177454] RSP: 0018:ffff88007cbb9e28 EFLAGS: 00000046
Feb 29 19:25:37 r2 kernel: [6571717.177455] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Feb 29 19:25:37 r2 kernel: [6571717.177457] RDX: 0000000000000000 RSI: ffff88007cbb9fd8 RDI: 0000000000000092
Feb 29 19:25:37 r2 kernel: [6571717.177459] RBP: 0000000000000004 R08: 00175791ea78908e R09: 0000000000002185
Feb 29 19:25:37 r2 kernel: [6571717.177460] R10: 0140000000000000 R11: 0000000000000206 R12: 0000000000000020
Feb 29 19:25:37 r2 kernel: [6571717.177462] R13: 0000000000000004 R14: ffff88007cbb8010 R15: ffff88007cbb8010
Feb 29 19:25:37 r2 kernel: [6571717.177464] FS: 0000000000000000(0000) GS:ffff88007f420000(0000) knlGS:0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.177466] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 29 19:25:37 r2 kernel: [6571717.177468] CR2: 00007f89897b3000 CR3: 000000000159c000 CR4: 00000000000007e0
Feb 29 19:25:37 r2 kernel: [6571717.177469] Stack:
Feb 29 19:25:37 r2 kernel: [6571717.177470] 00000000006440e0 0000000181086f24 ffff88007f43a000 ffffffff815df8a0
Feb 29 19:25:37 r2 kernel: [6571717.177473] 00175791ea78c664 ffffffff81337a5b 0000000000000000 0000000000000004
Feb 29 19:25:37 r2 kernel: [6571717.177476] 0000000000000000 ffff88007f43a000 ffffffff815df8a0 ffff88007cbb8000
Feb 29 19:25:37 r2 kernel: [6571717.177479] Call Trace:
Feb 29 19:25:37 r2 kernel: [6571717.177482] [<ffffffff81337a5b>] ? cpuidle_enter_state+0x2f/0xa8
Feb 29 19:25:37 r2 kernel: [6571717.177485] [<ffffffff81337bf1>] ? cpuidle_idle_call+0xd6/0x129
Feb 29 19:25:37 r2 kernel: [6571717.177488] [<ffffffff81015e64>] ? arch_cpu_idle+0x9/0x20
Feb 29 19:25:37 r2 kernel: [6571717.177491] [<ffffffff8107d151>] ? cpu_startup_entry+0x132/0x1a9
Feb 29 19:25:37 r2 kernel: [6571717.177494] [<ffffffff8103244a>] ? start_secondary+0x25e/0x263
Feb 29 19:25:37 r2 kernel: [6571717.177496] Code: 48 8b 34 25 08 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <85> 1d 00 6e 3c 00 75 0f 48 8d 74 24 0c bf 05 00 00 00 e8 15 2e
Feb 29 19:25:37 r2 kernel: [6571717.177519] NMI backtrace for cpu 2
Feb 29 19:25:37 r2 kernel: [6571717.177522] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.177524] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.177526] task: ffff88007cb94600 ti: ffff88007cbba000 task.ti: ffff88007cbba000
Feb 29 19:25:37 r2 kernel: [6571717.177527] RIP: 0010:[<ffffffff8107d11b>] [<ffffffff8107d11b>] cpu_startup_entry+0xfc/0x1a9
Feb 29 19:25:37 r2 kernel: [6571717.177531] RSP: 0018:ffff88007cbbbee8 EFLAGS: 00000246
Feb 29 19:25:37 r2 kernel: [6571717.177533] RAX: 0000000400000000 RBX: ffff88007cbba010 RCX: 000000000721dc11
Feb 29 19:25:37 r2 kernel: [6571717.177534] RDX: 0000000000000021 RSI: 0000000000000002 RDI: 0000000000000000
Feb 29 19:25:37 r2 kernel: [6571717.177536] RBP: ffff88007cbba000 R08: ffff88007f445e5c R09: ffff88007cbba000
Feb 29 19:25:37 r2 kernel: [6571717.177538] R10: 0140000000000000 R11: ffff88007cbba010 R12: ffff88007cbba000
Feb 29 19:25:37 r2 kernel: [6571717.177540] R13: ffff88007cbba010 R14: ffff88007cbba010 R15: ffff88007cbba010
Feb 29 19:25:37 r2 kernel: [6571717.177542] FS: 0000000000000000(0000) GS:ffff88007f440000(0000) knlGS:0000000000000000
да, очень смуттило что счетчик вылез на единицу. Обычно кол-во сессий до 1млн.
from ipt-netflow.
aabc
commented on May 22, 2024
Feb 29 19:25:37 r2 kernel: [6571717.051318] [<ffffffff8128e407>] ? serial8250_console_putchar+0x15/0x25
Feb 29 19:25:37 r2 kernel: [6571717.051320] [<ffffffff8128abc1>] ? uart_console_write+0x39/0x4c
Опять запись в консоль, как в #50. Я там советовал поднять скорость до 115200.
from ipt-netflow.
mort1k
commented on May 22, 2024
очень странно но в консоль ничего не выводилось
from ipt-netflow.
aabc
commented on May 22, 2024
А, значит не в этом дело. Насколько я понял, происходит некое подвисание (stall) во время вызова модулем функции ядра kernel_sendmsg (функция, которая просто шлёт UDP пакет). Зависание, таким образом, в самом ядре (не в модуле). С чем это связано мне не понятно. Может ведь быть и просто баг ядра, или VyOS, или ipset. Самое высокое по стеку это вызов каких-то ipset функций. В какой именно функции зависание по трейсу не видно. Вот разбор трейса как я его вижу:
Feb 29 19:25:37 r2 kernel: [6571717.011269] INFO: rcu_sched self-detected stall on CPU
Feb 29 19:25:37 r2 kernel: [6571717.051277] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=6004 jiffies, g=119659536, c=119659535, q=16496)
Feb 29 19:25:37 r2 kernel: [6571717.051285] NMI backtrace for cpu 0
Feb 29 19:25:37 r2 kernel: [6571717.051288] CPU: 0 PID: 2116 Comm: kworker/0:1 Tainted: G O 3.13.11-1-amd64-vyos #1
Feb 29 19:25:37 r2 kernel: [6571717.051289] Hardware name: SGI.COM 99-01-003431/Rack-TY1 , BIOS 1.20 12/01/2009
Feb 29 19:25:37 r2 kernel: [6571717.051293] Workqueue: events netflow_work_fn [ipt_NETFLOW]
Ядро обнаружило stall и послало NMI IRQ на все ядраб, чтоб выполнился backstrace. В это время на cpu 0 был запущен процесс kworker/0 и в нем, до IRQ, самая верхняя функция, это netflow_work_fn. Далее трейс стека IRQ:
[6572239.876571] <IRQ>
[6572239.876572] [<ffffffff811d5d49>] ? delay_tsc+0x2b/0x72
[6572239.876577] [<ffffffff810348e9>] ? native_safe_apic_wait_icr_idle+0x36/0x48
[6572239.876580] [<ffffffff81034ced>] ? default_send_IPI_mask_sequence_phys+0x5c/0xc7
[6572239.876582] [<ffffffff81034d9c>] ? arch_trigger_all_cpu_backtrace+0x44/0x6c
[6572239.876585] [<ffffffff81085fdd>] ? rcu_check_callbacks+0x1f8/0x54f
[6572239.876587] [<ffffffff81088087>] ? do_timer+0x3cc/0x3eb
[6572239.876590] [<ffffffff8108db58>] ? tick_nohz_handler+0xce/0xce
[6572239.876592] [<ffffffff810534d7>] ? update_process_times+0x31/0x56
[6572239.876595] [<ffffffff8108dbcc>] ? tick_sched_timer+0x74/0x90
[6572239.876597] [<ffffffff81066c93>] ? __run_hrtimer+0x92/0x11c
[6572239.876600] [<ffffffff81066f7a>] ? hrtimer_interrupt+0xde/0x1ec
[6572239.876603] [<ffffffff81034548>] ? smp_apic_timer_interrupt+0x1d/0x2d
[6572239.876606] [<ffffffff813ecc5d>] ? apic_timer_interrupt+0x6d/0x80
Выше что-то с таймером (может как раз stall детектор), ниже ip_set_hash_ip (т.е. обработка ipset):
[6572239.876609] [<ffffffffa0395a40>] ? hash_ip6_test+0x11e/0x11e [ip_set_hash_ip]
[6572239.876613] [<ffffffffa0395a9c>] ? hash_ip4_test+0x5c/0x10f [ip_set_hash_ip]
[6572239.876616] [<ffffffffa0395a7c>] ? hash_ip4_test+0x3c/0x10f [ip_set_hash_ip]
[6572239.876620] [<ffffffffa03951b1>] ? hash_ip4_kadt+0x9c/0xa2 [ip_set_hash_ip]
[6572239.876623] [<ffffffffa03879db>] ? ip_set_test+0xaf/0x142 [ip_set]
[6572239.876626] [<ffffffffa03b266c>] ? set_match_v3+0x6d/0x10a [xt_set]
Выше ipset, ниже прогон пакета по iptables.
[6572239.876629] [<ffffffff813c3a56>] ? ipt_do_table+0x2af/0x6dd
[6572239.876631] [<ffffffff813c3e47>] ? ipt_do_table+0x6a0/0x6dd
[6572239.876635] [<ffffffff8137a8b7>] ? nf_iterate+0x5b/0x9a
[6572239.876638] [<ffffffff813835b0>] ? ip_check_defrag+0x13a/0x13a
[6572239.876640] [<ffffffff8137aa85>] ? nf_hook_slow+0x72/0x107
[6572239.876643] [<ffffffff813835b0>] ? ip_check_defrag+0x13a/0x13a
[6572239.876646] [<ffffffff813839ad>] ? ip_forward+0x2bb/0x372
[6572239.876648] [<ffffffff81381d3a>] ? ip_rcv_finish+0x7e/0x2b9
Выше iptables, ниже прием пакета с карты.
[6572239.876651] [<ffffffff813585e2>] ? __netif_receive_skb_core+0x4c5/0x4fd
[6572239.876654] [<ffffffff81014f15>] ? read_tsc+0x5/0x16
[6572239.876656] [<ffffffff81086ebd>] ? T.823+0xd/0x31
[6572239.876659] [<ffffffff813588bc>] ? netif_receive_skb+0x81/0x87
[6572239.876662] [<ffffffff81359254>] ? napi_gro_receive+0xa7/0xe5
[6572239.876668] [<ffffffffa0064005>] ? ixgbe_clean_rx_irq+0x751/0x7f7 [ixgbe]
[6572239.876675] [<ffffffffa0064683>] ? ixgbe_poll+0x4ea/0x655 [ixgbe]
[6572239.876678] [<ffffffff813588bc>] ? netif_receive_skb+0x81/0x87
[6572239.876681] [<ffffffff81358dfa>] ? net_rx_action+0xa8/0x22e
[6572239.876683] [<ffffffff813eb43a>] ? _raw_spin_unlock+0x5/0x6
[6572239.876686] [<ffffffff8104d563>] ? __do_softirq+0x100/0x244
[6572239.876689] [<ffffffff813ed91c>] ? do_softirq_own_stack+0x1c/0x30
[6572239.876690] <EOI>
Стек soft irq закончился, далее стек worker/0, внутри которого вызов sendmsg и правил OUTPUT chain:
[6572239.876691] [<ffffffff8104d2fa>] ? do_softirq+0x3a/0x4b
[6572239.876696] [<ffffffff8104d3b3>] ? _local_bh_enable_ip+0x6c/0x76
[6572239.876698] [<ffffffff813c3e47>] ? ipt_do_table+0x6a0/0x6dd
[6572239.876701] [<ffffffff8134b449>] ? __alloc_skb+0x9d/0x19a
[6572239.876704] [<ffffffff81346887>] ? sock_alloc_send_pskb+0x33b/0x35d
[6572239.876707] [<ffffffff8137a8b7>] ? nf_iterate+0x5b/0x9a
[6572239.876710] [<ffffffff8138493c>] ? ip_options_echo+0x2f0/0x2f0
[6572239.876713] [<ffffffff8137aa85>] ? nf_hook_slow+0x72/0x107
[6572239.876715] [<ffffffff8138493c>] ? ip_options_echo+0x2f0/0x2f0
[6572239.876718] [<ffffffff81386b03>] ? T.1229+0x39/0x3e
[6572239.876721] [<ffffffff81386cf8>] ? ip_local_out+0x9/0x19
[6572239.876724] [<ffffffff81386d14>] ? ip_send_skb+0xc/0x2f
[6572239.876726] [<ffffffff813a5dc1>] ? udp_send_skb+0x187/0x1e6
[6572239.876729] [<ffffffff813a661a>] ? udp_sendmsg+0x71d/0x739
[6572239.876731] [<ffffffff81385daf>] ? ip_append_page+0x4b4/0x4b4
[6572239.876734] [<ffffffff813aeec6>] ? inet_autobind+0x4d/0x4d
[6572239.876737] [<ffffffff8134299d>] ? sock_sendmsg+0x4e/0x66
[6572239.876740] [<ffffffff811d17eb>] ? sha_transform+0x3db/0x1248
[6572239.876743] [<ffffffff813eb43a>] ? _raw_spin_unlock+0x5/0x6
[6572239.876746] [<ffffffff8110e1f9>] ? unfreeze_partials+0xcf/0xf6
Причём отсылка через lo. Возможно, если бы в destination была указана отсылка не через 127.0.0.1, а через адрес за eth интерфейсом, то эта проблема бы не возникала. Или убрать ipset правила с lo или из OUTPUT chain. Ну, и ниже модуль netflow вызывает sendmsg.
[6572239.876749] [<ffffffff81342f85>] ? kernel_sendmsg+0x31/0x3c
[6572239.876752] [<ffffffffa03f5f4d>] ? netflow_sendmsg+0xe9/0x2ab [ipt_NETFLOW]
[6572239.876755] [<ffffffff810871b1>] ? __getnstimeofday+0x28/0x6d
[6572239.876758] [<ffffffffa03f619c>] ? netflow_export_pdu_ipfix+0x8d/0xee [ipt_NETFLOW]
[6572239.876762] [<ffffffffa03f11f1>] ? pdu_alloc_fail_export+0x1e/0x2b [ipt_NETFLOW]
[6572239.876765] [<ffffffffa03f277e>] ? alloc_record_key+0x2d7/0x315 [ipt_NETFLOW]
[6572239.876767] [<ffffffff8110ed62>] ? kmem_cache_free+0x81/0xb9
[6572239.876770] [<ffffffffa03f3fcb>] ? netflow_export_flow_tpl+0x18e/0x6e1 [ipt_NETFLOW]
[6572239.876774] [<ffffffffa03f32fe>] ? netflow_scan_and_export+0x4b7/0x530 [ipt_NETFLOW]
[6572239.876777] [<ffffffffa03f33bc>] ? netflow_work_fn+0x45/0x64 [ipt_NETFLOW]
[6572239.876780] [<ffffffff81060867>] ? process_one_work+0x1fb/0x302
[6572239.876782] [<ffffffff81060acd>] ? worker_thread+0x15f/0x26c
[6572239.876785] [<ffffffff8106096e>] ? process_one_work+0x302/0x302
[6572239.876787] [<ffffffff8106096e>] ? process_one_work+0x302/0x302
[6572239.876790] [<ffffffff810640b5>] ? kthread+0xc3/0xcb
[6572239.876793] [<ffffffff81063ff2>] ? kthread_freezable_should_stop+0x51/0x51
[6572239.876795] [<ffffffff813ebfcc>] ? ret_from_fork+0x7c/0xb0
[6572239.876798] [<ffffffff81063ff2>] ? kthread_freezable_should_stop+0x51/0x51
Если во VyOS есть debug ядро, то можете попробовать погонять модуль на нём. Если есть какая-то проблема с локами, то оно может её автоматически идентифицировать. Но, как минус, оно может сильно грузить проц и не потянуть большой трафик из-за проверок.
from ipt-netflow.
Related Issues (20)
- specify source port
- natevents sending flows only if "conntrack -E" is running HOT 5
- Compile error on kernel 4.15.0, Ubuntu 18.04.6 LTS HOT 2
- wk_cpu = __smp_processor_id(); HOT 4
- no data flow HOT 1
- Error in 5.15.86 ct_event HOT 17
- periodic will reconnect HOT 5
- Warning on module unload: "BUG: using smp_processor_id() in preemptible" HOT 1
- Missing logs HOT 1
- compiling error on ubuntu 18.04 HOT 2
- disable traffic flow
- Cross-compiling master HOT 2
- Use diferent protocol versions for different destinations
- git does not build on arch - 6.4.1-arch2-1 implicit declaration of function ‘register_sysctl_paths’; did you mean ‘register_sysctl_table’? [-Werror=implicit-function-declaration] HOT 9
- Couldn't load target `NETFLOW' debian 11.7 kernel 5.10.0-22 HOT 1
- Unusual CPU usage spike after certain pps count? HOT 1
- How to correctly set the time of sending netflow
- natevent not working (Debian 12/6.1.0-10-amd64) HOT 1
- ipt_netflow Issue on Ubuntu 18: Nat Events Missing HOT 1
- No longer builds Arch HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ipt-netflow.