openrisc / linux Goto Github PK
View Code? Open in Web Editor NEWThis project forked from skristiansson/linux
Linux kernel source tree
License: Other
linux's People
Stargazers
Watchers
Forkers
benpayne erikandren silviumusa vidyagovindan jm2000 euryecetelecom bandvig thaolt daveshah1 janweinstock pgali93 wuhuaroubj loesterfranco tablebearlinux's Issues
Failure when catting /proc/xxx/stack
Seen with 5.7-rc2
Example:
/ # cat /proc/690/stack
[108997.050000] Unable to handle kernel access
[108997.050000] at virtual address 0x7fc60f58
[108997.050000]
[108997.050000] Oops#: 0000
[108997.060000] CPU #: 0
[108997.060000] PC: c00097fc SR: 0000807f SP: d6f09b9c
[108997.060000] GPR00: 00000000 GPR01: d6f09b9c GPR02: d6f09bb8 GPR03: d6f09bc4
[108997.060000] GPR04: 7fc60f5c GPR05: c00099b4 GPR06: 00000000 GPR07: d6f09ba3
[108997.060000] GPR08: ffffff00 GPR09: c0009804 GPR10: d6f08000 GPR11: 00000000
[108997.060000] GPR12: ffffe000 GPR13: dbb86000 GPR14: 00000001 GPR15: dbb86250
[108997.060000] GPR16: 7fc60f63 GPR17: 00000f5c GPR18: d6f09bc4 GPR19: 00000000
[108997.060000] GPR20: c00099b4 GPR21: ffffffc0 GPR22: 00000000 GPR23: 00000000
[108997.060000] GPR24: 00000001 GPR25: 000002c6 GPR26: d78b6850 GPR27: 00000001
[108997.060000] GPR28: 00000000 GPR29: dbb86000 GPR30: ffffffff GPR31: dbb862fc
[108997.060000] RES: 00000000 oGPR11: ffffffff
[108997.060000] Process cat (pid: 702, stackpage=d79d6000)
[108997.060000]
[108997.060000] Stack:
[108997.060000] Call trace:
[108997.060000] [<598977f2>] save_stack_trace_tsk+0x40/0x74
[108997.060000] [<95063f0e>] stack_trace_save_tsk+0x44/0x58
[108997.060000] [<b557bfdd>] proc_pid_stack+0xd0/0x13c
[108997.060000] [<a2df8eda>] proc_single_show+0x6c/0xf0
[108997.060000] [<e5a737b7>] seq_read+0x1b4/0x688
[108997.060000] [<2d6c7480>] do_iter_read+0x208/0x248
[108997.060000] [<2182a2fb>] vfs_readv+0x64/0x90
[108997.060000] [<3ff02271>] ? iov_iter_get_pages_alloc+0x3d4/0x670
[108997.060000] [<bab8ec1a>] ? slob_alloc.isra.0+0xfc/0x348
[108997.070000] [<d58f9294>] ? __wake_up_common_lock+0x90/0xcc
[108997.070000] [<49544e36>] default_file_splice_read+0x1cc/0x3b0
[108997.070000] [<ab9ebd9e>] ? kmem_cache_alloc+0x48/0x104
[108997.070000] [<6dd300f2>] do_splice_to+0xa8/0x100
[108997.070000] [<caed44c0>] splice_direct_to_actor+0xf8/0x33c
[108997.070000] [<b2a5b2b1>] ? direct_splice_actor+0x0/0x70
[108997.070000] [<886bec27>] do_splice_direct+0xa4/0x100
[108997.070000] [<d3c7c436>] do_sendfile+0x2bc/0x4f0
[108997.070000] [<bd6b01e4>] sys_sendfile64+0x130/0x138
[108997.070000] [<7f032d7a>] ? _syscall_return+0x0/0x4
NETDEV WATCHDOG: eth0 (ethoc): transmit queue 0 timed out
When running in qemu with latest 5.5 kernel with networking turned on.
We get transmit queue timeouts.
Trace:
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.700000] ------------[ cut here ]------------
[ 259.700000] WARNING: CPU: 0 PID: 7 at net/sched/sch_generic.c:442 dev_watchdog+0x2dc/0x2e4
[ 259.700000] NETDEV WATCHDOG: eth0 (ethoc): transmit queue 0 timed out
[ 259.700000] Modules linked in:
[ 259.700000] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 5.5.0-simple-smp-00006-g9737e2c5f0bc #90
[ 259.700000] Call trace:
[ 259.700000] [<(ptrval)>] show_stack+0x48/0x58
[ 259.710000] [<(ptrval)>] dump_stack+0x28/0x38
[ 259.710000] [<(ptrval)>] __warn+0xf0/0x120
[ 259.710000] [<(ptrval)>] ? dev_watchdog+0x2dc/0x2e4
[ 259.710000] [<(ptrval)>] warn_slowpath_fmt+0x7c/0x94
[ 259.710000] [<(ptrval)>] dev_watchdog+0x2dc/0x2e4
[ 259.710000] [<(ptrval)>] ? dev_watchdog+0x0/0x2e4
[ 259.710000] [<(ptrval)>] call_timer_fn.isra.0+0x24/0xa0
[ 259.710000] [<(ptrval)>] run_timer_softirq+0x238/0x544
[ 259.710000] [<(ptrval)>] ? dev_watchdog+0x0/0x2e4
[ 259.710000] [<(ptrval)>] ? mempool_kfree+0x14/0x24
[ 259.710000] [<(ptrval)>] ? mempool_free+0x54/0xec
[ 259.710000] [<(ptrval)>] ? rcu_process_callbacks+0x9c/0x144
[ 259.710000] [<(ptrval)>] __do_softirq+0x128/0x320
[ 259.710000] [<(ptrval)>] ? smpboot_thread_fn+0x0/0x26c
[ 259.710000] [<(ptrval)>] run_ksoftirqd+0x50/0x70
[ 259.710000] [<(ptrval)>] smpboot_thread_fn+0x1b8/0x26c
[ 259.710000] [<(ptrval)>] ? __kthread_parkme+0x68/0xa8
[ 259.710000] [<(ptrval)>] kthread+0x120/0x150
[ 259.710000] [<(ptrval)>] ? kthread+0x0/0x150
[ 259.710000] [<(ptrval)>] ret_from_fork+0x1c/0x9c
[ 259.720000] ---[ end trace 2e12edb0824fd0ae ]---
[ 260.660000] nfs: server 10.0.0.27 OK
[ 260.660000] nfs: server 10.0.0.27 OK
[ 260.700000] nfs: server 10.0.0.27 OK
[ 260.740000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
Lockups when running network stack
When just running linux with every once in a while we get complete lockups with no errors.
Running in qemu in the monitor I can see:
- PC is stuck in the same place (don't trust as QEMU only updates PC at certain times)
- LR (r9) is stuck in the same place
c00075e0 - SP (r1) the stack is just going crazy
See dumps below.
Note, the LR points to :
c0007550 <_external_irq_handler>:
c0007550: d4 01 10 08 l.sw 8(r1),r2
c0007554: d4 01 18 0c l.sw 12(r1),r3
c0007558: d4 01 28 14 l.sw 20(r1),r5
...
c00075d8: 48 00 40 00 l.jalr r8
c00075dc: 15 00 00 00 l.nop 0x0
c00075e0: 00 00 03 5a l.j c0008348 <_ret_from_exception>
c00075e4: 15 00 00 00 l.nop 0x0
c039bfd8 <do_IRQ>:
c039bfd8: 9c 21 ff f8 l.addi r1,r1,-8
c039bfdc: d4 01 10 00 l.sw 0(r1),r2
c039bfe0: d4 01 48 04 l.sw 4(r1),r9
c039bfe4: 1a 20 c0 8f l.movhi r17,0xc08f
c039bfe8: 86 31 cb fc l.lwz r17,-13316(r17)
c039bfec: 48 00 88 00 l.jalr r17
c039bff0: 9c 41 00 08 l.addi r2,r1,8
c039bff4: 85 21 00 04 l.lwz r9,4(r1)
c039bff8: 84 41 00 00 l.lwz r2,0(r1)
c039bffc: 44 00 48 00 l.jr r9
c039c000: 9c 21 00 08 l.addi r1,r1,8
c039c004 <__irqentry_text_end>:
c039c004: 00 00 00 00 l.j c039c004 <__irqentry_text_end>
PC=c039c004
R00=00000000 R01=34d08aec R02=c15b1804 R03=00000000
R04=ffffffff R05=00000000 R06=00000012 R07=0000000e
R08=0000001a R09=c00075e0 R10=c15b0000 R11=00000000
R12=0000001b R13=00000010 R14=c1076840 R15=00000005
R16=c1076840 R17=00000000 R18=c1f69758 R19=00000000
R20=00ca4800 R21=00000000 R22=00000428 R23=00000003
R24=01511e00 R25=ce63f7b2 R26=0000042c R27=22bdcb7b
R28=00000005 R29=3416c000 R30=ffffffff R31=c62b0000
A second later.
(qemu) info registers
PC=c039c004
R00=00000000 R01=b96c226c R02=c15b1804 R03=00000000
R04=ffffffff R05=00000000 R06=00000012 R07=0000000e
R08=0000001a R09=c00075e0 R10=c15b0000 R11=00000000
R12=0000001b R13=00000010 R14=c1076840 R15=00000005
R16=c1076840 R17=00000000 R18=c1f69758 R19=00000000
R20=00ca4800 R21=00000000 R22=00000428 R23=00000003
R24=01511e00 R25=ce63f7b2 R26=0000042c R27=22bdcb7b
R28=00000005 R29=3416c000 R30=ffffffff R31=c62b0000
watchdog: BUG: soft lockup - CPU#0 stuck for 187s! [ksoftirqd/0:7]
While running linux 5.5 in qemu with full network stack connected to NFS and SSH server running.
Trace:
[ 1020.840000] watchdog: BUG: soft lockup - CPU#0 stuck for 187s! [ksoftirqd/0:7]
[ 1020.840000] Modules linked in:
[ 1020.840000] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 5.5.0-simple-smp #91
[ 1020.840000] CPU #: 0
[ 1020.840000] PC: c03aa6e0 SR: 00008a7f SP: c1025c68
[ 1020.840000] GPR00: 00000000 GPR01: c1025c68 GPR02: c1025c8c GPR03: 00000000
[ 1020.840000] GPR04: eb893541 GPR05: 00000000 GPR06: c8000000 GPR07: 14090179
[ 1020.840000] GPR08: 000003ca GPR09: c006c1f4 GPR10: c1024000 GPR11: fffffffc
[ 1020.840000] GPR12: c133e319 GPR13: 00000000 GPR14: 01726000 GPR15: 00000000
[ 1020.840000] GPR16: c0910c50 GPR17: eb893541 GPR18: a0000000 GPR19: 00000001
[ 1020.840000] GPR20: 8f6237da GPR21: 00000003 GPR22: 000000bf GPR23: 0000014c
[ 1020.840000] GPR24: 0000001a GPR25: 00000010 GPR26: 00027f10 GPR27: 0000000e
[ 1020.840000] GPR28: 007c85fd GPR29: 00000012 GPR30: 00000200 GPR31: c1d9e870
[ 1020.840000] RES: fffffffc oGPR11: ffffffff
[ 1020.840000] Process ksoftirqd/0 (pid: 7, stackpage=c101b400)
[ 1020.840000]
[ 1020.840000] Stack:
[ 1020.840000] Call trace:
[ 1020.840000] [<5344f851>] ktime_get+0x70/0x1bc
[ 1020.840000] [<88164d97>] tcp_mstamp_refresh+0x30/0x190
[ 1020.840000] [<497cf51c>] tcp_rcv_established+0x38/0x798
[ 1020.840000] [<1447d5c1>] tcp_v4_do_rcv+0x1e4/0x238
[ 1020.840000] [<21cdc523>] tcp_v4_rcv+0xd1c/0xdd0
[ 1020.840000] [<f65863ef>] ? __inet_lookup_established+0x6c/0x248
[ 1020.840000] [<278b79da>] ip_protocol_deliver_rcu+0x44/0x1e8
[ 1020.840000] [<7bb669e4>] ip_local_deliver+0xa4/0xf8
[ 1020.840000] [<3e703180>] ip_rcv+0x6c/0xa0
[ 1020.840000] [<cf154ebe>] __netif_receive_skb_one_core+0x50/0x68
[ 1020.840000] [<7a0f7052>] __netif_receive_skb+0x38/0x98
[ 1020.840000] [<72f95335>] netif_receive_skb+0x38/0x90
[ 1020.840000] [<9b4bb2ad>] ethoc_poll+0x280/0x900
[ 1020.840000] [<917ada6e>] net_rx_action+0x11c/0x3cc
[ 1020.840000] [<6554ac93>] ? mempool_kfree+0x14/0x24
[ 1020.840000] [<e4702a67>] __do_softirq+0x128/0x320
[ 1020.840000] [<0edefd55>] ? smpboot_thread_fn+0x0/0x26c
[ 1020.840000] [<ddaf11a2>] run_ksoftirqd+0x50/0x70
[ 1020.840000] [<6abd7f59>] smpboot_thread_fn+0x1b8/0x26c
[ 1020.840000] [<e436cf21>] ? __kthread_parkme+0x68/0xa8
[ 1020.840000] [<410cfcab>] kthread+0x120/0x150
[ 1020.840000] [<12695f1f>] ? kthread+0x0/0x150
[ 1020.840000] [<161c3108>] ret_from_fork+0x1c/0x9c
[ 1020.840000]
[ 1020.840000] Code:
watchdog: BUG: soft lockup - CPU#0 stuck for 186s
Even after increasing memory to 512mb on qemu, I still occasionally see this.
/ # [90216.140000] watchdog: BUG: soft lockup - CPU#0 stuck for 186s! [kworker/u3:3:2073]
[90216.140000] Modules linked in:
[90216.140000] CPU: 0 PID: 2073 Comm: kworker/u3:3 Tainted: G W 5.7.0-rc2-simple-smp-00001-g6bd140e14d9a-dirty #109
[90216.140000] Workqueue: xprtiod xs_stream_data_receive_workfn
[90216.140000] CPU #: 0
[90216.140000] PC: c03c1694 SR: 00008a7f SP: d89a3c90
[90216.140000] GPR00: 00000000 GPR01: d89a3c90 GPR02: d89a3cb4 GPR03: 00000000
[90216.140000] GPR04: eb895d10 GPR05: 00000000 GPR06: c8000000 GPR07: 00000000
[90216.140000] GPR08: c029c738 GPR09: c006b898 GPR10: d89a2000 GPR11: fffffe5c
[90216.140000] GPR12: 1a69ac61 GPR13: 00000000 GPR14: df4f5800 GPR15: 00000000
[90216.140000] GPR16: c08c2c20 GPR17: eb895d10 GPR18: 20000000 GPR19: 00000001
[90216.140000] GPR20: 098a4d3e GPR21: 000001a3 GPR22: 000051df GPR23: 00000000
[90216.140000] GPR24: 0000001a GPR25: 00000000 GPR26: 01125cf2 GPR27: 00000000
[90216.140000] GPR28: 002c771e GPR29: c04c4fa1 GPR30: df4f5b9c GPR31: c04b63c0
[90216.140000] RES: fffffe5c oGPR11: ffffffff
[90216.140000] Process kworker/u3:3 (pid: 2073, stackpage=d8850000)
[90216.140000]
[90216.140000] Stack:
[90216.140000] Call trace:
[90216.140000] [<8558de7f>] ktime_get+0x70/0x1bc
[90216.140000] [<8d9c6218>] tcp_mstamp_refresh+0x30/0x190
[90216.140000] [<d9699fc9>] tcp_rcv_space_adjust+0x38/0x300
[90216.140000] [<93cbbb5f>] tcp_recvmsg+0x558/0xebc
[90216.140000] [<a2442133>] ? __switch_to+0x50/0x7c
[90216.140000] [<e17f0602>] ? __enqueue_entity.constprop.0+0xc4/0x104
[90216.140000] [<85044956>] inet_recvmsg+0x3c/0x64
[90216.140000] [<96e0a9e6>] sock_recvmsg+0x24/0x34
[90216.140000] [<a02d272c>] xs_sock_recvmsg.constprop.0+0x44/0x88
[90216.140000] [<4c68e7cc>] xs_stream_data_receive_workfn+0x110/0xaac
[90216.140000] [<e17f0602>] ? __enqueue_entity.constprop.0+0xc4/0x104
[90216.140000] [<ee8f26ff>] process_one_work+0x254/0x538
[90216.140000] [<a246f55d>] worker_thread+0x64/0x694
[90216.140000] [<3bf47ca9>] ? worker_thread+0x0/0x694
[90216.140000] [<818fceb2>] kthread+0x120/0x150
[90216.140000] [<0db975eb>] ? kthread+0x0/0x150
[90216.140000] [<336cbdd5>] ret_from_fork+0x1c/0x9c
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.