Describe the bug mopidy crashes with <code class="n

My theory so far is that it's related to <a href="https://gitlab.freedesktop.org/gstre

This is likely a duplicate of <a class="issue-link js-issue-link" data-error-text="Fai

With Ubuntu you can set the env variable described at <a href="https://ubuntu.com/serv

SIGILL on Arch on RaspberryPi 5,about mopidy/mopidy

Comments (13)

kingosticks commented on September 23, 2024 1

My theory so far is that it's related to making the current volume an int64 and then passing that into volume_orc_process_int16 which expects an "int". But I have no idea what this orc stuff is about (or if is enabled in your build).... I think we should raise this with the gstreamer folks given that it's now easily reproducible outside of Mopidy. They'll have a much, much better idea.

from mopidy.

kingosticks commented on September 23, 2024

Is that the back trace of the correct thread ? Can you please get debugging symbols ?

from mopidy.

kingosticks commented on September 23, 2024

This is likely a duplicate of #2147

from mopidy.

kingosticks commented on September 23, 2024

This is also missing all version information, please provide what we asked for.

Your config (output of sudo mopidyctl config)

Software versions (output of sudo mopidyctl deps)

from mopidy.

blacklight commented on September 23, 2024

Is that the back trace of the correct thread ? Can you please get debugging symbols ?

It looks like the right stack to me, as both the libc report and the stack head on gdb seem to point to the same address:

libc stack report:

Stack trace of thread 16257:
#0  0x0000007f84010150 n/a (n/a + 0x0)
#1  0x0000007f7d692bd0 n/a (libgstvolume.so + 0x2bd0)
#2  0x0000007ef804aa70 n/a (n/a + 0x0)
#3  0x0000007ef804aa70 n/a (n/a + 0x0)
ELF object binary architecture: AARCH64

gdb:

(gdb) i threads
  Id   Target Id                       Frame
* 1    Thread 0x7efcf9f120 (LWP 16257) 0x0000007f84010150 in ?? ()
  2    Thread 0x7efe7cf120 (LWP 16254) 0x0000007f5c5bfc54 in ?? () from /usr/lib/libmpg123.so.0
  3    Thread 0x7efd7af120 (LWP 16256) 0x0000007f84b96424 in syscall () from /usr/lib/libc.so.6
  4    Thread 0x7f7f85f120 (LWP 16074) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  5    Thread 0x7f7f04f120 (LWP 16075) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  6    Thread 0x7f1d7af120 (LWP 16244) 0x0000007f84b2d2dc in ?? () from /usr/lib/libc.so.6
  7    Thread 0x7f7dedf120 (LWP 16077) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  8    Thread 0x7f7ce7f120 (LWP 16079) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  9    Thread 0x7efdfbf120 (LWP 16255) 0x0000007f84b96428 in syscall () from /usr/lib/libc.so.6
  10   Thread 0x7f7e77f120 (LWP 16076) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  11   Thread 0x7f5f7ef120 (LWP 16081) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  12   Thread 0x7f3f7ef120 (LWP 16088) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  13   Thread 0x7f5e7cf120 (LWP 16083) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  14   Thread 0x7f3e7cf120 (LWP 16090) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  15   Thread 0x7efefdf120 (LWP 16253) 0x0000007f84b96424 in syscall () from /usr/lib/libc.so.6
  16   Thread 0x7f8006f120 (LWP 16073) 0x0000007f84b904d8 in poll () from /usr/lib/libc.so.6
  17   Thread 0x7f3dfbf120 (LWP 16091) 0x0000007f84b9a77c in epoll_pwait () from /usr/lib/libc.so.6
  18   Thread 0x7f1dfbf120 (LWP 16098) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  19   Thread 0x7f1ffff120 (LWP 16094) 0x0000007f84b64864 in clock_nanosleep () from /usr/lib/libc.so.6
  20   Thread 0x7f1cf9f120 (LWP 16246) 0x0000007f84b96424 in syscall () from /usr/lib/libc.so.6
  21   Thread 0x7f3ffff120 (LWP 16087) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  22   Thread 0x7f3d7af120 (LWP 16092) 0x0000007f84b8c250 in read () from /usr/lib/libc.so.6
  23   Thread 0x7f852ce020 (LWP 16070) 0x0000007f84b904d8 in poll () from /usr/lib/libc.so.6
  24   Thread 0x7efffff120 (LWP 16245) 0x0000007f84b96424 in syscall () from /usr/lib/libc.so.6
  25   Thread 0x7f7d68f120 (LWP 16078) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  26   Thread 0x7f5ffff120 (LWP 16080) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  27   Thread 0x7eff7ef120 (LWP 16252) 0x0000007f84b904d8 in poll () from /usr/lib/libc.so.6
  28   Thread 0x7f1f7ef120 (LWP 16095) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  29   Thread 0x7f5cf9f120 (LWP 16086) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  30   Thread 0x7f5efdf120 (LWP 16082) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  31   Thread 0x7f1e7cf120 (LWP 16097) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  32   Thread 0x7f5d7af120 (LWP 16085) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  33   Thread 0x7f5dfbf120 (LWP 16084) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  34   Thread 0x7f3efdf120 (LWP 16089) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6
  35   Thread 0x7f3cf9f120 (LWP 16093) 0x0000007f84b9a77c in epoll_pwait () from /usr/lib/libc.so.6
  36   Thread 0x7f1efdf120 (LWP 16096) 0x0000007f84b2d2d8 in ?? () from /usr/lib/libc.so.6

(gdb) bt
#0  0x0000007f84010150 in ??? ()
#1  0x0000007ef804aa70 in ??? ()
(gdb) thread 2
[Switching to thread 2 (Thread 0x7efe7cf120 (LWP 16254))]
#0  0x0000007f5c5bfc54 in ?? () from /usr/lib/libmpg123.so.0
(gdb) bt
#0  0x0000007f5c5bfc54 in ??? () at /usr/lib/libmpg123.so.0
#1  0x0000007f5c5c2fc4 in ??? () at /usr/lib/libmpg123.so.0
#2  0x0000007f5c5b6dfc in ??? () at /usr/lib/libmpg123.so.0
#3  0x0000007f5c5b85f0 in mpg123_decode_frame64 () at /usr/lib/libmpg123.so.0
#4  0x0000007f5c4f4248 in ??? () at /usr/lib/gstreamer-1.0/libgstmpg123.so
#5  0x0000007f81af44c0 in ??? () at /usr/lib/libgstaudio-1.0.so.0
#6  0x0000007f81af48cc in ??? () at /usr/lib/libgstaudio-1.0.so.0
#7  0x0000007f81af5b88 in ??? () at /usr/lib/libgstaudio-1.0.so.0
#8  0x0000007f820e5604 in ??? () at /usr/lib/libgstreamer-1.0.so.0
#9  0x0000007f820e7cc8 in ??? () at /usr/lib/libgstreamer-1.0.so.0
#10 0x0000007f820f0a68 in gst_pad_push () at /usr/lib/libgstreamer-1.0.so.0
#11 0x0000007f81c5d648 in gst_base_parse_push_frame () at /usr/lib/libgstbase-1.0.so.0
#12 0x0000007f7c2a3ab8 in ??? () at /usr/lib/gstreamer-1.0/libgstaudioparsers.so
#13 0x0000007f81c584c0 in ??? () at /usr/lib/libgstbase-1.0.so.0
#14 0x0000007f81c58c80 in ??? () at /usr/lib/libgstbase-1.0.so.0
#15 0x0000007f81c5c010 in ??? () at /usr/lib/libgstbase-1.0.so.0
#16 0x0000007f8212c2dc in ??? () at /usr/lib/libgstreamer-1.0.so.0
#17 0x0000007f8316ecf8 in ??? () at /usr/lib/libglib-2.0.so.0
#18 0x0000007f8316deac in ??? () at /usr/lib/libglib-2.0.so.0
#19 0x0000007f84b30aec in ??? () at /usr/lib/libc.so.6
#20 0x0000007f84b9a5dc in ??? () at /usr/lib/libc.so.6

It looks like the stack got corrupted before it got dumped though.

This is likely a duplicate of #2147

I don't think so. It's a SIGSEGV on armv7l in that case (and things work while playing local files), while it's a SIGILL on aarch64 here (and playback is broken also for local files).

Btw I think I may have found the culprit. The clue was in this stack pointer:

#1  0x0000007f85792bd0 n/a (libgstvolume.so + 0x2bd0)

My configuration has the following lines:

[audio]
mixer_volume = 50

Once mixer_volume is removed, playback works. However, it breaks again as soon as I try to set the volume. So it looks like an issue with volume control on gst side rather than playback in general.

The output of mopidy deps is here.

from mopidy.

kingosticks commented on September 23, 2024

It looks like the stack got corrupted before it got dumped though.

Why do you think that? GDB can report that erroneously.

Try, they are slightly different. But both are arch linux reports on arm with gstreamer 1.22.9 (released just last week) within a few days. Maybe it's just the new release. In terms of this one, all I can find is https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5063. Could do with the full debug symbols, can you use Debuginfod?

from mopidy.

blacklight commented on September 23, 2024

Why do you think that? GDB can report that erroneously.

Because the libc crash report gives me this stack:

#0  0x0000007f84010150 n/a (n/a + 0x0)
#1  0x0000007f7d692bd0 n/a (libgstvolume.so + 0x2bd0)
#2  0x0000007ef804aa70 n/a (n/a + 0x0)
#3  0x0000007ef804aa70 n/a (n/a + 0x0)

While gdb gives me this for the same thread - the first frame is the same, but the second is at a different address, while apparently the crash occurs in libgstvolume.so, and no further frames are available:

(gdb) bt
#0  0x0000007f84010150 in ??? ()
#1  0x0000007ef804aa70 in ??? ()

Which seems to be compatible with the scenario of some SIGILL I've seen before. Could it be some binary incompatibility on ARM64?

Could do with the full debug symbols, can you use Debuginfod?

Since the crash apparently occurs in the GStreamer stack what's the best way to replicate it with debugging symbols, other than building GStreamer with -g? I haven't used Debuginfod before.

from mopidy.

kingosticks commented on September 23, 2024

With Ubuntu you can set the env variable described at https://ubuntu.com/server/docs/service-debuginfod and gdb downloads the debug symbols when it starts, sometimes it works... It looks like Arch has something similar described at https://wiki.archlinux.org/title/Debuginfod

from mopidy.

blacklight commented on September 23, 2024

I've run a gdb session for mopidy with DEBUGINFOD but I still couldn't get any debugging symbols in the impacted thread:

Thread 46 "queue1:src" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7f51fbf120 (LWP 2653)]
0x0000007ff69f0150 in ?? ()
(gdb) bt
#0  0x0000007ff69f0150 in ??? ()
#1  0x0000007f702c11f0 in ??? ()

Which seems to confirm the stack corruption hypothesis.

Any more ideas that could help debugging? I'm also attaching a recent core dump for reference.

Btw after a bit of testing I can confirm that the GStreamer crash only occurs when changing the volume. Other Gst events (like skipping, pausing etc.) work just fine.

from mopidy.

kingosticks commented on September 23, 2024

I think the best way would be to rebuild with debug symbols.

You could also try:

Muting/un-muting Mopidy intead. Same issue?
gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! fakesink silent=TRUE
gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! <whatever your sink is from missing mopidy config>
Reproduce the issue in Mopidy with GST_DEBUG=3,volume:5 set (try this with mixer_volume = 50 set so it crashes quickly and the log isn't enormous). I don't get much from the volume element on my system:

0:01:19.648093527  9673 0x7f4c44048de0 DEBUG                 volume gstvolume.c:239:volume_update_volume:<volume0> configure mute 0, volume 0.500000
0:01:19.648128233  9673 0x7f4c44048de0 DEBUG                 volume gstvolume.c:273:volume_update_volume:<volume0> set passthrough 0
0:01:19.651841558  9673 0x7f4c44048de0 DEBUG                 volume gstvolume.c:698:volume_before_transform:<volume0> sync to 0:00:00.000000000
0:01:19.652007894  9673 0x7f4c44048de0 DEBUG                 volume gstvolume.c:698:volume_before_transform:<volume0> sync to 0:00:00.013061224
0:01:19.652039073  9673 0x7f4c44048de0 DEBUG                 volume gstvolume.c:698:volume_before_transform:<volume0> sync to 0:00:00.036281179
<snip>

from mopidy.

blacklight commented on September 23, 2024

gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! fakesink silent=TRUE

gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! <whatever your sink is from missing mopidy config>

Actually both these commands could reproduce the issue on the system:

gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! autoaudiosink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got message #12 from element "autoaudiosink0-actual-sink-pulse" (state-changed): GstMessageStateChanged, old-state=(GstState)null, new-state=(GstState)ready, pending-state=(GstState)void-pending;
Got message #13 from element "autoaudiosink0" (state-changed): GstMessageStateChanged, old-state=(GstState)null, new-state=(GstState)ready, pending-state=(GstState)void-pending;
Got message #14 from element "volume0" (state-changed): GstMessageStateChanged, old-state=(GstState)null, new-state=(GstState)ready, pending-state=(GstState)void-pending;
Got message #15 from element "audiotestsrc0" (state-changed): GstMessageStateChanged, old-state=(GstState)null, new-state=(GstState)ready, pending-state=(GstState)void-pending;
Got message #16 from element "pipeline0" (state-changed): GstMessageStateChanged, old-state=(GstState)null, new-state=(GstState)ready, pending-state=(GstState)paused;
Got message #20 from element "volume0" (state-changed): GstMessageStateChanged, old-state=(GstState)ready, new-state=(GstState)paused, pending-state=(GstState)void-pending;
Got message #23 from pad "audiotestsrc0:src" (stream-status): GstMessageStreamStatus, type=(GstStreamStatusType)create, owner=(GstElement)"\(GstAudioTestSrc\)\ audiotestsrc0", object=(GstTask)"\(GstTask\)\ audiotestsrc0:src";
Got message #24 from element "audiotestsrc0" (state-changed): GstMessageStateChanged, old-state=(GstState)ready, new-state=(GstState)paused, pending-state=(GstState)void-pending;
Got message #25 from pad "audiotestsrc0:src" (stream-status): GstMessageStreamStatus, type=(GstStreamStatusType)enter, owner=(GstElement)"\(GstAudioTestSrc\)\ audiotestsrc0", object=(GstTask)"\(GstTask\)\ audiotestsrc0:src";
Got message #26 from element "pipeline0" (stream-start): GstMessageStreamStart, group-id=(uint)1;
Got message #31 from pad "audiotestsrc0:src" (property-notify): GstMessagePropertyNotify, property-name=(string)caps, property-value=(GstCaps)"audio/x-raw\,\ format\=\(string\)S16LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)1";
/GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1
Got message #33 from pad "volume0:src" (property-notify): GstMessagePropertyNotify, property-name=(string)caps, property-value=(GstCaps)"audio/x-raw\,\ format\=\(string\)S16LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)1";
/GstPipeline:pipeline0/GstVolume:volume0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1
Got message #34 from pad "sink:proxypad0" (property-notify): GstMessagePropertyNotify, property-name=(string)caps, property-value=(GstCaps)"audio/x-raw\,\ format\=\(string\)S16LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)1";
/GstPipeline:pipeline0/GstAutoAudioSink:autoaudiosink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1
Got message #35 from element "autoaudiosink0-actual-sink-pulse" (latency): no message details
Redistribute latency...
Got message #36 from pad "autoaudiosink0-actual-sink-pulse:sink" (property-notify): GstMessagePropertyNotify, property-name=(string)caps, property-value=(GstCaps)"audio/x-raw\,\ format\=\(string\)S16LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)1";
/GstPipeline:pipeline0/GstAutoAudioSink:autoaudiosink0/GstPulseSink:autoaudiosink0-actual-sink-pulse.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1
Got message #37 from pad "autoaudiosink0:sink" (property-notify): GstMessagePropertyNotify, property-name=(string)caps, property-value=(GstCaps)"audio/x-raw\,\ format\=\(string\)S16LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)1";
/GstPipeline:pipeline0/GstAutoAudioSink:autoaudiosink0.GstGhostPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1
Got message #38 from pad "volume0:sink" (property-notify): GstMessagePropertyNotify, property-name=(string)caps, property-value=(GstCaps)"audio/x-raw\,\ format\=\(string\)S16LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)1";
/GstPipeline:pipeline0/GstVolume:volume0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1
Got message #43 from element "autoaudiosink0-actual-sink-pulse" (tag): GstMessageTag, taglist=(taglist)"taglist\,\ description\=\(string\)\"audiotest\\\ wave\"\;";
[1]    14557 illegal hardware instruction (core dumped)  gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! autoaudiosink

Which points in the direction of a GStreamer bug. But debugging, even with DEBUGINFOD_URLS properly set, doesn't yield much luck yet:

Core was generated by `gst-launch-1.0 -v -m audiotestsrc ! volume volume=0.5 ! fakesink silent=TRUE'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x0000007f9d691000 in ?? ()

Maybe gstreamer symbols haven't been included yet in Arch's debuginfod repos, or the stack is really messed up?

Anyway, the core is quite small and it should be quite straightforward in this case, so I'll attach it just in case you get more luck: core.tar.gz

I'd rather avoid rebuilding the whole gstreamer package with -g, it's like >400MB of sources and it'll take forever on the RPi. Let me know if the information from the gst-launch-1.0 or the core dump suffices, otherwise I may either try to get more information from gst-shark or downgrade gstreamer.

--EDIT--

Nevermind, downgrading GStreamer isn't really an easy option, since it comes tied to a lot of other system dependencies on Arch. I've also done another test replacing Pulseaudio with Pipewire, but still same result (it just crashes on a different sink).

from mopidy.

blacklight commented on September 23, 2024

I've tried to disable both the HDMI and built-in jack audio on the RPi, just in case, still the same error.

New logs generated with GST_DEBUG=9 (only for the gst-launch command): log.tar.gz

from mopidy.

blacklight commented on September 23, 2024

Ok, thanks for the directions so far! FYI I've opened an issue on gstreamer, I'll close this one as it's clearly not mopidy-related.

from mopidy.

SIGILL on Arch on RaspberryPi 5 about mopidy HOT 13 CLOSED

Comments (13)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent