crash-utility / crash Goto Github PK

I have installed crash using dnf in Fedora Server 31, everything works well but dis -s.

The output likes below:

$ sudo crash
crash 7.2.6-2.fc31
...
GNU gdb (GDB) 7.6
...
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [240MB]: patching 107037 gdb minimal_symbol values

  KERNEL: /usr/lib/debug/lib/modules/5.3.7-301.fc31.x86_64/vmlinux
DUMPFILE: /dev/crash**
    CPUS: 8
    DATE: Sat Nov 30 14:14:44 2019
  UPTIME: 00:15:34

LOAD AVERAGE: 0.37, 0.18, 0.11
TASKS: 515
NODENAME: z.me
RELEASE: 5.3.7-301.fc31.x86_64
VERSION: #1 SMP Mon Oct 21 19:18:58 UTC 2019
MACHINE: x86_64 (2592 Mhz)
MEMORY: 8 GB
PID: 2525
COMMAND: "crash"
TASK: ffff8c23c755bf80 [THREAD_INFO: ffff8c23c755bf80]
CPU: 2
STATE: TASK_RUNNING (ACTIVE)

crash> dis -s printk
FILE: (unknown)
LINE: (unknown)
dis: printk: source code is not available

crash> dis printk
0xffffffff9014727e : nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffff90147283 <printk+5>: push %rbp
0xffffffff90147284 <printk+6>: mov %rsp,%rbp
0xffffffff90147287 <printk+9>: sub $0x50,%rsp
0xffffffff9014728b <printk+13>: mov %rsi,0x28(%rsp)
0xffffffff90147290 <printk+18>: mov %rsp,%rsi
0xffffffff90147293 <printk+21>: mov %rdx,0x30(%rsp)
0xffffffff90147298 <printk+26>: mov %rcx,0x38(%rsp)
0xffffffff9014729d <printk+31>: mov %r8,0x40(%rsp)
0xffffffff901472a2 <printk+36>: mov %r9,0x48(%rsp)
...

However, the l command in gdb works well:
$ gdb /usr/lib/debug/usr/lib/modules/5.3.7-301.fc31.x86_64/vmlinux
(gdb) l printk
2036 * printf(3)
2037 *
2038 * See the vsnprintf() documentation for format string extensions over C99.
2039 */
2040 asmlinkage __visible int printk(const char *fmt, ...)
2041 {
2042 va_list args;
2043 int r;
2044
2045 va_start(args, fmt);

The dis -s command also can't work when I give the path to crash explicitly (like gdb above) in command line.

I also can't compile crash from source code. when I run make, the terminal shows:
cc1:all warnings being treated as errors.
Maybe there is something wrong with the function list_source_code.

Infos below may be helpful:

$ uname -a
Linux z.me 5.3.7-301.fc31.x86_64 #1 SMP Mon Oct 21 19:18:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ gcc --version
gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)

$ gdb --version
GNU gdb (GDB) Fedora 8.3.50.20190824-25.fc31

I am working with a target device having an i386 CPU and Gentoo-based GNU/Linux OS.
I have enabled kdump/kexec to attempt capture of an issue customers are encountering but cannot reproduce in-house. In anticipation of collecting dump files, I would like to be able to utilize the crash utility. I was able to "cross" compile the utility to run on our 64-bit Ubuntu 18.04 workstations but read the 32-bit files from our device.

However, when attempting to run the utility, I encounter this error.
Complete output:

$ ~/Downloads/crash-7.2.7_X86/crash vmlinux vmcore-201909301633_SOFTLOCKUP 

crash 7.2.7
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=i686-pc-linux-gnu"...

WARNING: kernel version inconsistency between vmlinux and dumpfile

please wait... (gathering module symbol data)   
crash: v2 module syment space malloc (-1037442748 symbols): Cannot allocate memory

This message is printed from symbols.c::store_modules_symbols_v2()

        if ((st->ext_module_symtable = (struct syment *)
             calloc(total, sizeof(struct syment))) == NULL)
                error(FATAL, "v2 module syment space malloc (%ld symbols): %s\n",
			total, strerror(errno));

I will attempt to do some additional investigation, but am hoping someone can identify something I did incorrectly in my build process to get past this or a fix that is not yet available in the repo.

Thanks!
.Tim

Hi:
I use crash 7.2.6-3 to parse vmcore. The vmcore was generated by kernel 4.19 aarch64。

When I read the global variables in the module, the values ​​returned by the p command and the rd command are different.

#crash /boot/vmlinux vmcore
crash 7.2.6-3
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu"...

WARNING: cannot find NT_PRSTATUS note for cpu: 78
KERNEL: /boot/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 96
DATE: Wed Feb 12 16:23:47 2020
UPTIME: 17 days, 13:05:44
LOAD AVERAGE: 5253.54, 5244.11, 5221.62
TASKS: 11580
NODENAME: 121-6
RELEASE: 4.19.aarch64
VERSION: #1 SMP Mon Jul 22 00:00:00 UTC 2019
MACHINE: aarch64 (unknown Mhz)
MEMORY: 96 GB
PANIC: "kernel BUG at /xxx/upi_cache.c:120!"
PID: 29229
COMMAND: "Jpool"
TASK: ffff8022be10be00 [THREAD_INFO: ffff8022be10be00]
CPU: 18
STATE: TASK_RUNNING (PANIC)

crash> mod -s snas_ds ./modules/snas_ds.ko
MODULE NAME SIZE OBJECT FILE
ffff000003ed0900 snas_ds 2887680 ./modules/snas_ds.ko
crash> p g_bCheckMetaCap
g_bCheckMetaCap = $1 = 2432712771
crash>
crash> rd g_bCheckMetaCap
ffff000003ececc0: 0000000000000001 ........
crash>
crash> set debug 31
debug: 31
crash> set debug 31
debug: 31
text hit rate: 0% (0 of 1)
crash> rd g_bCheckMetaCap
<addr: ffff000003ececc0 count: 1 flag: 490 (KVADDR)>
<readmem: ffff000003ececc0, KVADDR, "64-bit KVADDR", 8, (FOE), ffffc570ddb0>
<read_diskdump: addr: ffff000003ececc0 paddr: 202d7e8cecc0 cnt: 8>
read_diskdump: paddr/pfn: 202d7e8cecc0/202d7e8ce -> physical page is cached: 202d7e8ce000
ffff000003ececc0: 0000000000000001 ........
text hit rate: 0% (0 of 1)
crash> p g_bCheckMetaCap
p: per_cpu_symbol_search(g_bCheckMetaCap): NULL
g_bCheckMetaCap = GETBUF(328 -> 0)
$2 = 2432712771
FREEBUF(0)
text hit rate: 50% (1 of 2)

When attempting to debug a kernel panic on ubuntu 16.04 LTS with hwe the stack is unavailable.

Having manually built 7.2.7 I got a little further but still hit a dead end with it failing with:
bt: cannot transition from exception stack to current process stack:

./crash /usr/lib/debug/boot/vmlinux-4.15.0-1041-gcp /var/crash/201911112306/dump.201911112306 

crash 7.2.7
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [918MB]: patching 99648 gdb minimal_symbol values

      KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-1041-gcp              
    DUMPFILE: /var/crash/201911112306/dump.201911112306  [PARTIAL DUMP]
        CPUS: 8
        DATE: Mon Nov 11 23:06:47 2019
      UPTIME: 02:06:18
LOAD AVERAGE: 1.49, 1.69, 1.72
       TASKS: 396
    NODENAME: XXXXX
     RELEASE: 4.15.0-1041-gcp
     VERSION: #43-Ubuntu SMP Wed Aug 21 09:04:51 UTC 2019
     MACHINE: x86_64  (2300 Mhz)
      MEMORY: 30 GB
       PANIC: "BUG: unable to handle kernel paging request at ffffffffba678770"
         PID: 13869
     COMMAND: "PoolThread 6"
        TASK: ffffa0245a690000  [THREAD_INFO: ffffa0245a690000]
         CPU: 1
       STATE: TASK_RUNNING (PANIC)

crash> bt 13869
PID: 13869  TASK: ffffa0245a690000  CPU: 1   COMMAND: "PoolThread 6"
 #0 [fffffe0000033d60] machine_kexec at ffffffffba6669ce
 #1 [fffffe0000033dc0] __crash_kexec at ffffffffba732bd9
 #2 [fffffe0000033e88] panic at ffffffffba691a45
 #3 [fffffe0000033f10] df_debug at ffffffffba66ae0d
 #4 [fffffe0000033f28] do_double_fault at ffffffffba62f49a
 #5 [fffffe0000033f50] double_fault at ffffffffbb000fe3
    [exception RIP: __sprint_symbol+69]
    RIP: ffffffffba731165  RSP: fffffe0000032fe8  RFLAGS: 00010046
    RAX: 0000000000000000  RBX: ffffffffba678770  RCX: fffffe0000032fe8
    RDX: fffffe0000032ff0  RSI: fffffe0000032ff8  RDI: ffffffffba678770
    RBP: fffffe0000033030   R8: fffffe0000033051   R9: fffffe0000033320
    R10: fffffe0000033388  R11: ffffffffbbd5e80d  R12: fffffe0000033051
    R13: 0000000000000000  R14: 0000000000000001  R15: ffffffffbb6a51b0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <DOUBLEFAULT exception stack> ---
 #6 [fffffe0000032fe8] __sprint_symbol at ffffffffba731165
bt: cannot transition from exception stack to current process stack:
    exception stack pointer: fffffe0000033d60
      process stack pointer: fffffe0000033038
         current stack base: ffffb2d48cfdc000

Last two entries from the log which may be relevant:

[29413.763776] unable to execute userspace code (SMEP?) (uid: 2000)
[29413.769982] BUG: unable to handle kernel paging request at ffffffffba678770

Any ideas on how to identify what user space stack caused the kernel to panic?

This is something that happens on a semi regular basis with this app.

Build fails with LTO on aarch64 due to multiple definition of tdesc_aarch64:

[  467s] /usr/lib64/gcc/aarch64-suse-linux/10/../../../../aarch64-suse-linux/bin/ld: aarch64-linux-nat.o (symbol from plugin): in function `tdesc_aarch64':
[  467s] (.text+0x0): multiple definition of `tdesc_aarch64'; aarch64-tdep.o (symbol from plugin):(.text+0x0): first defined here

crash can't switch frame and look up the local variable of stack like gdb
p local_var_a
p local_var_b
f 1
f 2
f 3
is there any plan support those operation?

I'm building and running crash-7.1.9. But the tool itself crashes because of memory overflow. The core dump point is at free time, which makes it a little tricky to debug. The error message looks like this:

crash 7.1.9
Copyright (C) 2002-2016 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

*** Error in `/tmp/crash': free(): invalid pointer: 0x0000000001022270 ***
Aborted (core dumped)

The following patch can fix the problem:

`--- /tmp/crash/crash-7.1.9/filesys.c 2017-04-21 02:54:33.000000000 +0800
+++ crash-7.1.9/filesys.c 2017-08-31 06:18:32.000040000 +0800
@@ -308,6 +308,7 @@
#define CREATE 1
#define DESTROY 0
#define DEFAULT_SEARCHDIRS 5
+#define EXTRA_SEARCHDIRS 5

static char **
build_searchdirs(int create, int *preferred)
@@ -340,11 +341,11 @@
*preferred = 0;

    /*

•    *  Allow, at a minimum, the defaults plus an extra three directories
  

•    *  Allow, at a minimum, the defaults plus EXTRA_SEARCHDIRS directories
     *  for the two possible /usr/src/redhat/BUILD/kernel-xxx locations
     *  plus the Red Hat debug directory.
     */
  

•   cnt = DEFAULT_SEARCHDIRS + 3;
  

•   cnt = DEFAULT_SEARCHDIRS + EXTRA_SEARCHDIRS;
  
     if ((dirp = opendir("/usr/src"))) {
             for (dp = readdir(dirp); dp != NULL; dp = readdir(dirp))
  

`

hi，
I used crash to analyze vmcore under arm64 architecture, and found that when crash calculates the bitmap and pfn, an integer overflow occurs, causing the crash parsing to fail.

[root@along crash]# crash vmlinux vmcore
......
reason: variable overflow causes a logic error in crash.
crash: page excluded: kernel virtual address: ffff0000089c9100 type: "kernel_config_data"
WARNING: cannot read kernel_config_data
crash: page excluded: kernel virtual address: ffff00000911b938 type: "possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffff00000911b8b8 type: "present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffff00000911b838 type: "online"
WARNING: cannot read cpu_online_map
crash: page excluded: kernel virtual address: ffff00000911b9b8 type: "active"
WARNING: cannot read cpu_active_map
crash: page excluded: kernel virtual address: ffff0000093ec9d0 type: "shadow_timekeeper xtime_sec"
crash: page excluded: kernel virtual address: ffff000009124d2c type: "init_uts_ns"
crash: vmlinux and vmcore do not match!

/proc/iomem info:
2e69267000-2fffffffff : System RAM
......
602770ecf000-6027ffffffff : System RAM

1. calculate bitmap_len overflow in function read_dump_header()
   int block_size=(int)sysconf(_SC_PAGESIZE);
   off_t bitmap_len;
   unsigned int bitmap_blocks;
   ...
   bitmap_len = block_size * header->bitmap_blocks;

here block_size = 4096, header->bitmap_blocks = 0x180a00, so bitmap_len = 0x180a00000
but block_size is int type, bitmap_blocks is unsigned int type, so block_size * header->bitmap_blocks > MAX(unsigned int)
bitmap_len overflow.

2. calculate data_offset overflow in function read_dump_header()
   info in makedumpfile:
   info->offset_bitmap1=0x15000 info->len_bitmap=0x182000000 bit2_offset=0xc1015000

in read_dump_header():
info->len_bitmap = header->bitmap_blocks * header->block_size = 0x182000000 is greater than the range represented by integers.

The members of the following expression are all int type, but the calculation result exceeds MAX(int),
dd->data_offset = (1 + header->sub_hdr_size + header->bitmap_blocks) * header->block_size = 0x82015000,
The correct value is 0x182015000.

3. byte overflow in function get_bit()
   static inline int
   get_bit(char *map, int byte, int bit)
   {
   return map[byte] & (1<<bit);
   }

static inline int
page_is_ram(unsigned long nr)
{
return get_bit(dd->bitmap, nr >> 3, nr & 7);
}
if nr=0x6027fff4f, 0x6027fff4f >> 3 => ‭0xC04FFFE9‬ > MAX(int),
so byte parameter overflow when call get_bit.

The following is my patch, please review. Thanks.

Signed-off-by: Jialong Chen <[email protected]>
---
 diskdump.c | 6 ++--
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/diskdump.c b/diskdump.c
index c3e343b..1a2a5ce 100644
--- a/diskdump.c
+++ b/diskdump.c
@@ -233,7 +233,7 @@ clean_diskdump_data(void)
 }

 static inline int
-get_bit(char *map, int byte, int bit)
+get_bit(char *map, unsigned long byte, int bit)
 {
        return map[byte] & (1<<bit);
 }
@@ -674,7 +674,7 @@ restart:
                dd->max_mapnr = header->max_mapnr;

        /* read memory bitmap */
-       bitmap_len = block_size * header->bitmap_blocks;
+       bitmap_len = (off_t)block_size * header->bitmap_blocks;
        dd->bitmap_len = bitmap_len;

        offset = (off_t)block_size * (1 + header->sub_hdr_size);
@@ -724,7 +724,7 @@ restart:
                memcpy(dd->dumpable_bitmap, dd->bitmap, bitmap_len);

        dd->data_offset
-               = (1 + header->sub_hdr_size + header->bitmap_blocks)
+               = (1UL + header->sub_hdr_size + header->bitmap_blocks)
                * header->block_size;

        dd->header = header;

crash 7.2.8 fails to load on ARM64 live kernel on 5.4.

# uname -a
Linux com7 5.4.42-xxxxxxxx-standard #1 SMP Wed Jun 3 05:17:19 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
# crash -s /proc/kcore vmlinux-stable-master-dev54
crash: /proc/kcore: cannot read vabits_actual

Is this a known issue? Currently when reading of vabits_actual symbol value from /proc/kallsyms isn't successful crash returns error (FATAL), instead should it continue to read tcr_el1_t1sz from VMCOREINFO to determine vabits_actual? Please share your thoughts.

Made a small source change to calculate the vabits_actual from vmcoreinfo's tcr_el1_t1sz, with the change I can load crash on live kernel. Tried few sample crash commands, they all seem to work.

# git diff arm64.c
diff --git a/arm64.c b/arm64.c
index 7662d71..d8515b2 100644
--- a/arm64.c
+++ b/arm64.c
@@ -3869,10 +3869,11 @@ arm64_calc_VA_BITS(void)
                                machdep->machspec->VA_BITS = value;
                                machdep->machspec->VA_START = _VA_START(machdep->machspec->VA_BITS_ACTUAL);
                        } else
-                               error(FATAL, "/proc/kcore: cannot read vabits_actual\n");
+                               error(WARNING, "/proc/kcore: cannot read vabits_actual\n");
                } else if (ACTIVE())
                        error(FATAL, "cannot determine VA_BITS_ACTUAL: please use /proc/kcore\n");
-               else {
+
+               if (!machdep->machspec->VA_BITS_ACTUAL) {
                        if ((string = pc->read_vmcoreinfo("NUMBER(tcr_el1_t1sz)"))) {
                                /* See ARMv8 ARM for the description of
                                 * TCR_EL1.T1SZ and how it can be used

com5:/tmp# /tmp/crash -s /proc/kcore /tmp/vmlinux
WARNING: /proc/kcore: cannot read vabits_actual
crash> sys
      KERNEL: /tmp/vmlinux
    DUMPFILE: /proc/kcore
        CPUS: 8
        DATE: Mon Jun 22 23:24:53 2020
      UPTIME: 00:28:21
LOAD AVERAGE: 0.14, 0.05, 0.01
       TASKS: 138
    NODENAME: com5
     RELEASE: 5.4.44-xxxxxxxx-standard
     VERSION: #1 SMP Sat Jun 20 00:55:50 UTC 2020
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 7.8 GB
crash>

I know i'm able to define macros in ~/.gdbinit (just not interactively using define from the crash prompt)

hoping to have a string comparison in such a macro using $_streq as per https://sourceware.org/gdb/onlinedocs/gdb/Convenience-Funs.html

When I have the test macro

define dotest
  if $_streq("1", "1")
    echo ok\n
  end
end

I observe different results between gdb and crash

(gdb) dotest
ok

vs

crash> dotest
You can't do that without a process to debug.
gdb: gdb request failed: dotest

Hi all, when I make crash, the following appear:

| In file included from amd64-linux-nat.c:61:0:
| gdb_proc_service.h:162:9: error: unknown type name 'gdb_fpregset_t'
|typedef gdb_fpregset_t gdb_prfpregset_t;
| ^~~~~~~~~~~~~~
| make[4]: *** [amd64-linux-nat.o] Error 1
| make[4]: *** Waiting for unfinished jobs....
| make[4]: *** [linux-thread-db.o] Error 1
| In file included from proc-service.c:28:0:
| gdb_proc_service.h:162:9: error: unknown type name 'gdb_fpregset_t'
| typedef gdb_fpregset_t gdb_prfpregset_t;
| ^~~~~~~~~~~~~~

crash 7.2.8

Currently it's not easy to distinguish which timezone the output of DATE uses.

Actual:

crash> sys |grep -i date
        DATE: Thu Jul  2 16:54:08 2020

Expected:
Thu Jul 2 16:54:08 UTC 2020

Same format of the date command.

And maybe it would be nice if it uses /etc/localtime of the crashed server instead of the server used to analyze the vmcore.
Right now we can do so by using the TZ environment variable:
TZ=/usr/share/zoneinfo/America/Danmarkshavn crash vmlinux vmcore

p -lz -lm -llzma ../libiberty/libiberty.a build-gnulib/import/libgnu.a -ldl -Wl,--dynamic-list=./proc-service.list -lz -ldl -rdynamic
c-exp.o: In function main': /root/crash-7.1.9/gdb-7.6/gdb/c-exp.c:1: multiple definition of main'
../../crashlib.a(main.o):/root/crash-7.1.9/main.c:81: first defined here
cp-name-parser.o: In function main': /root/crash-7.1.9/gdb-7.6/gdb/cp-name-parser.c:1: multiple definition of main'
../../crashlib.a(main.o):/root/crash-7.1.9/main.c:81: first defined here
ada-exp.o: In function main': /root/crash-7.1.9/gdb-7.6/gdb/ada-exp.c:1: multiple definition of main'
../../crashlib.a(main.o):/root/crash-7.1.9/main.c:81: first defined here
f-exp.o: In function main': /root/crash-7.1.9/gdb-7.6/gdb/f-exp.c:1: multiple definition of main'
../../crashlib.a(main.o):/root/crash-7.1.9/main.c:81: first defined here
p-exp.o: In function main': /root/crash-7.1.9/gdb-7.6/gdb/p-exp.c:1: multiple definition of main'
../../crashlib.a(main.o):/root/crash-7.1.9/main.c:81: first defined here
go-exp.o: In function parse_string_or_char': /root/crash-7.1.9/gdb-7.6/gdb/go-exp.y:943: undefined reference to c_parse_escape'
macroexp.o: In function get_character_constant': /root/crash-7.1.9/gdb-7.6/gdb/macroexp.c:364: undefined reference to c_parse_escape'
macroexp.o: In function `get_string_literal':

Loading crash module on s390x these messages are seen:

[ 129.794298] crash: loading out-of-tree module taints kernel.
[ 129.795196] crash: Unknown symbol unxlate_dev_mem_ptr (err 0)
[ 129.795379] crash: Unknown symbol xlate_dev_mem_ptr (err 0)

On most architectures xlate_dev_mem_ptr is #defined to __va but on s390x it is implemented as actual function.

I try to introduce crash tool into ALT Linux, but, there is a problem - if vmlinux.debug is eu-elfcompressed (no matter gabi or gnu compressed) - crash is unable to load it with:

4.19.127-rt-alt2.rt54:~# crash

crash 7.2.8.0.21.gc4862e1-alt1
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
gdb called without error_hook: Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/usr/lib/debug/lib/modules/4.19.127-rt-alt2.rt54/vmlinux.debug]
Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/usr/lib/debug/lib/modules/4.19.127-rt-alt2.rt54/vmlinux.debug]

crash: /usr/lib/debug/lib/modules/4.19.127-rt-alt2.rt54/vmlinux: no debugging data available

Is it possible to have support for elfcompressed debug into crash? Would it be hard to add?

Is there any support for PPC64 big endian target?

Thank you.

Hi,
We are getting uptime values based off of jiffies values in crash. Unfortunately, on arm64, the CPU frequency is assumed to be 100Hz and this causes unrealistic uptimes to be reported.

Is it possible to follow the kernel approach to computing uptime from, get_monotonic_boottime(.), with something like:
uptime in nano-seconds = (tk_core->timekeeper->tkr_mono->base + tk_core->timekeeper->offs_boot)
From:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/time/timekeeping.c?h=v4.12#n782

Cheers,
Steve

Hi, I have a problem with the latest kernel and the latest crash.
My kernel is built on the commit f9893351acaecf0a414baf9942b48d5bb5c688c6,
and the crash is built on the commit 17e6a44b406cbf07ab8d82cbaf4c52404ec46ba7.

When I ran the crash crash ./vmlinux ./vmcore, I got the following error message.
crash: cannot determine length of symbol: log_end..
The vmcore file is generated with QEMU and its arch is x86_64.

It worked fine with a kernel commit generated about a month ago (I don't remember the exact commit),
so I suspect recent changes of the linux kernel somehow breaks the crash.

I want to do my own research on it, but I don't know where to start.
It seems a similar problem was fixed in the old crash version 6.0.8.
Do you have any advice to mitigate this problem?

I am trying to open a vmcore generated by kernel 4.19.21, crash tool fails and I get the following error:
crash: seek error: kernel virtual address: ffff888627a14d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627a54d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627a94d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627ad4d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627b14d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627b54d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627b94d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627bd4d00 type: "current_task (per_cpu)"
crash: seek error: kernel virtual address: ffff888627a05024 type: "tss_struct ist array"

This patch seems to fix the issue, but it is obviously only checked on a specific config and version of kernel.

git diff x86_64.c
diff --git a/x86_64.c b/x86_64.c
index d57b602..601350f 100644
--- a/x86_64.c
+++ b/x86_64.c
@@ -382,7 +382,8 @@ x86_64_init(int when)
 
        case POST_GDB:
                if (!(machdep->flags & RANDOMIZED) &&
-                   ((THIS_KERNEL_VERSION >= LINUX(4,20,0)) || 
+                   ((THIS_KERNEL_VERSION >= LINUX(4,20,0)) ||
+                         (THIS_KERNEL_VERSION == LINUX(4,19,21)) ||
                    ((THIS_KERNEL_VERSION >= LINUX(4,14,84)) && 
                     (THIS_KERNEL_VERSION < LINUX(4,15,0))))) {
                        machdep->machspec->page_offset = machdep->flags & VM_5LEVEL ?

Hi, I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host for live debug. Is there any method to do this? I used make target=ARM64 but it still can only run in x86. Thanks.

Hi,

I built crash 7.1.9 for the Yocto project and make some experiments on a BeagleBone Black and it looks like it runs out of memory now and it's unusable.

crash 7.1.9
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi"...

crash: vmlinux: Cannot allocate memory
      KERNEL: vmlinux
    DUMPFILE: /dev/mem
        CPUS: 1
        DATE: Mon May 29 12:03:54 2017
      UPTIME: 00:05:17
LOAD AVERAGE: 1.30, 1.17, 0.56
       TASKS: 94
    NODENAME: multi-v7-ml
     RELEASE: 4.9.36-custom-student-custom-ml-debug
     VERSION: #1 SMP Fri Jul 21 17:49:10 CEST 2017
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 510 MB
         PID: 631
     COMMAND: "crash"
        TASK: d9ed1000  [THREAD_INFO: d9c66000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
crash> p jiffies
crash: fork system call failed: Cannot allocate memorycrash: cannot open pipe
crash> 

Am I doing something wrong?

Is this a known issue?

Last login: Mon May 29 11:59:59 2017
root@multi-v7-ml:~# cat /proc/meminfo
MemTotal:         486028 kB
MemFree:            3564 kB
MemAvailable:     150560 kB
Buffers:             184 kB
Cached:           145820 kB
SwapCached:            0 kB
Active:           334496 kB
Inactive:         120340 kB
Active(anon):     308900 kB
Inactive(anon):       92 kB
Active(file):      25596 kB
Inactive(file):   120248 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         486028 kB
LowFree:            3564 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        308832 kB
Mapped:            16036 kB
Shmem:               164 kB
Slab:              21068 kB
SReclaimable:      11992 kB
SUnreclaim:         9076 kB
KernelStack:         760 kB
PageTables:         1300 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      243012 kB
Committed_AS:     331640 kB
VmallocTotal:     516096 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:          65536 kB
CmaFree:               0 kB
root@multi-v7-ml:~# 

root@multi-v7-ml:/tmp# cp /proc/config.gz .
root@multi-v7-ml:/tmp# gunzip config.gz 
root@multi-v7-ml:/tmp# cat config | grep DEVMEM
CONFIG_DEVMEM=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
root@multi-v7-ml:/tmp# 

the error like this ：
crash64> extend /home/huangzaiyang/WorkSpack/crash_ext_64/gcore.so
/home/huangzaiyang/WorkSpack/crash_ext_64/gcore.so: shared object already loaded

crash64> bt
PID: 545 TASK: ffffffc0620cf000 CPU: 3 COMMAND: "ftmdaemon"
bt: WARNING: cannot determine starting stack frame for task ffffffc0620cf000

crash64> gcore 545
Failed.

can any body help me with this？

hi

I met below warning when to use bt in version 7.2.8 to analyze offline ramdump for ARM.

bt: WARNING: cannot determine starting stack frame for task

i already checked related issue and code, i added debug code, and found no crash_notes symbol, note that arm_get_crash_notes():

 544 static void
 545 arm_get_crash_notes(void)
 546 {
 547         struct machine_specific *ms = machdep->machspec;
 548         ulong crash_notes;
 549         Elf32_Nhdr *note;
 550         ulong offset;
 551         char *buf, *p;
 552         ulong *notes_ptrs;
 553         ulong i, found;
 554
 555         if (!symbol_exists("crash_notes"))
 556                 return;

only arm_init() will call arm_get_crash_notes().

		if (!ACTIVE())
			arm_get_crash_notes();

And the "crash_notes' is from kexec, rt?

139#ifdef CONFIG_KEXEC
140#include <linux/kexec.h>
141
142static ssize_t show_crash_notes(struct device *dev, struct device_attribute *attr,
143				char *buf)

so, kernel must configurate CONFIG_KEXEC=y to use bt command to analyze dumpfile for ARM/ARM64?

btw: seems lots of android kernel do not enable the CONFIG_KEXEC.

Thx.,

As I am able to evaluate two values (casting pointers to uints) - I should be able to figure out the distance between said pointers (note - don't want pointer math)

As can be seen I can evaluate each cated-pointer as can be seen in $39 & $40
but when i try to evaluate/print the byte-delta between those crash reports an error from gdb, but if I force it into gdb as can be seen in the last line it just works. So looks like something in the crash command-line processing is eating up the - (minus)

crash> p ((uint)((struct sk_buff*)0xe47d9900)->tail)
$39 = 3778076956
crash> p ((uint)((struct sk_buff*)0xe47d9900)->data)
$40 = 3778076872
crash> p (((uint)((struct sk_buff*)0xe47d9900)->tail) - ((uint)((struct sk_buff*)0xe47d9900)->data))
You can't do that without a process to debug.
p: gdb request failed: p (((uint)((struct sk_buff*)0xe47d9900)->tail) ((uint)((struct sk_buff*)0xe47d9900)->data))
crash> gdb p (((uint)((struct sk_buff*)0xe47d9900)->tail) - ((uint)((struct sk_buff*)0xe47d9900)->data))
$41 = 84

Hi
I found Trace32 can show inline calling,but crash-utility couldn't do it,i use same ramdump and O2 optimization do the test,it's very useful feature,but i don't understand why trace32 can parse inline call,does any expert understand this technology or method?

kconfig:
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y

Trace32 stackframe:
1 -000|queued_write_lock_slowpath(?)
2 -001|queued_write_lock(inline)
3 -001|__raw_write_lock_irq(inline)
4 -001|raw_write_lock_irq(?)
5 -002|mhi_pm_m0_transition(mhi_cntrl = 0xFFFFFFF9AFA86C00)
6 -003|mhi_pm_fast_resume(mhi_cntrl = 0xFFFFFFF9AFA86C00, ?)
7 -004|mhi_runtime_resume(?)
8 -005|pci_pm_runtime_resume(dev = 0xFFFFFFF9903C6098)
9 -006|__rpm_callback(cb = 0xFFFFFF99C4390608, dev = 0xFFFFFFF9903C6098)
10 -007|memalloc_noio_restore(inline)
11 -007|rpm_callback(inline)
12 -007|rpm_resume(dev = 0xFFFFFFF9903C6098, ?)
13 -008|pm_runtime_work(work = 0xFFFFFFF9903C61E0)
14 -009|__read_once_size(inline)
15 -009|static_key_count(inline)
16 -009|static_key_false(inline)
17 -009|trace_workqueue_execute_end(inline)
18 -009|process_one_work(worker = 0xFFFFFFF9027FD700, work = 0xFFFFFFF9903C61E0)
19 -010|__read_once_size(inline)
20 -010|list_empty(inline)
21 -010|worker_thread(__worker = 0xFFFFFFF9027FD700)
22 -011|kthread(_create = 0xFFFFFFF84719A780)
23 -012|ret_from_fork(asm)
24 ---|end of frame

crash> bt fffffff9025c0000
PID: 14537 TASK: fffffff9025c0000 CPU: 2 COMMAND: "kworker/2:3"
#0 [ffffff8025983b90] queued_write_lock_slowpath at ffffff99c3f400e8
#1 [ffffff8025983ba0] _raw_write_lock_irq at ffffff99c524b9f8
#2 [ffffff8025983be0] mhi_pm_m0_transition at ffffff99c4337c84
#3 [ffffff8025983c10] mhi_pm_fast_resume at ffffff99c433b2bc
#4 [ffffff8025983c40] mhi_runtime_resume at ffffff99c433fe80
#5 [ffffff8025983c70] pci_pm_runtime_resume at ffffff99c43906a8
#6 [ffffff8025983cd0] __rpm_callback at ffffff99c45aaeac
#7 [ffffff8025983d70] rpm_resume at ffffff99c45a98f0
#8 [ffffff8025983d90] pm_runtime_work at ffffff99c45aa638
#9 [ffffff8025983de0] process_one_work at ffffff99c3ed8f08
#10 [ffffff8025983e50] worker_thread at ffffff99c3ed9460
#11 [ffffff8025983eb0] kthread at ffffff99c3edf204

fp for example is only for x86_64 but from this webpage it looks like arch independent

http://people.redhat.com/~anderson/extensions.html

I try:
tar -xf crash-7.2.5.tar.gz --> cd crash-7.2.5 --> make
to make a crash tool to analyze my vmcore. But it fails with the following error msg at the end:

make  all-recursive
make[10]: Nothing to be done for 'all-am'.
make[3]: Nothing to be done for 'all-target'.

crash build failed

Makefile:229: recipe for target 'gdb_merge' failed
make[1]: *** [gdb_merge] Error 1
Makefile:224: recipe for target 'all' failed
make: *** [all] Error 2

How does this issue occurs? Because there is no more any other error during compiling.
By the way, because my gcc version is larger than 4.1, I add "--disable-werror" when building gdb config to avoid the error shown below:

bfd.h:537:65: error: right-hand operand of comma expression has no effect [-Werror=unused-value]
 #define bfd_set_cacheable(abfd,bool) (((abfd)->cacheable = bool), TRUE)
                                                                 ^
opncls.c:234:5: note: in expansion of macro 'bfd_set_cacheable'
     bfd_set_cacheable (nbfd, TRUE);
     ^

crash version: 7.2.8
kernel version: 4.19.91 with CONFIG_STRICT_DEVMEM=y
when use crash in live system without any parameters, it will get error like this:
crash

crash 7.2.8-2
Copyright (C) 2002-2020 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu"...

crash: read error: kernel virtual address: ffff000008988cf8 type: "kernel_config_data"
WARNING: cannot access vmalloc'd module memory

  KERNEL: /usr/lib/debug/lib/modules/4.19.91/vmlinux
DUMPFILE: /proc/kcore
    CPUS: 128
    DATE: Tue Aug 25 18:49:40 2020
  UPTIME: 19:26:37

LOAD AVERAGE: 1.17, 1.08, 1.02
TASKS: 1377
NODENAME: j66e03465.sqa.eu95
RELEASE: 4.19.91
VERSION: #1 SMP Mon Aug 24 22:37:05 CST 2020
MACHINE: aarch64 (unknown Mhz)
MEMORY: 1024 GB
PID: 28923
COMMAND: "crash"
TASK: ffffa07ecbccc100 [THREAD_INFO: ffffa07ecbccc100]
CPU: 87
STATE: TASK_RUNNING (ACTIVE)

WARNING: could not find MAGIC_START!

the result of read symbol is not correct:
*crash> p cpufreq_global_kobject
cpufreq_global_kobject = $1 = (struct kobject ) 0x0
WARNING: could not find MAGIC_START!

as crash man files says:
If a MEMORY-IMAGE argument is not entered, the session will be invoked on the live system,
which typically requires root privileges because of the device file used to access system
RAM. By default, /dev/crash will be used if it exists. If it does not exist, then /dev/mem
will be used; but if the kernel has been configured with CONFIG_STRICT_DEVMEM, then
/proc/kcore will be used. It is permissible to explicitly enter /dev/crash, /dev/mem or
/proc/kcore.

when CONFIG_STRICT_DEVMEM is set, will use /proc/kcore, but the fact is not.
when use "crash /dev/mem" it will get error like "crash", when use "crash /proc/kcore", it perform ok.
when use "crash -d 4", the output shows:

<readmem: ffff000008988cf8, KVADDR, "kernel_config_data", 32768, (ROE), 36e931f0>
<read_dev_mem: addr: ffff000008988cf8 paddr: 988cf8 cnt: 776>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 988cf8, 776): 4294967295 (ffffffff)
crash: read error: kernel virtual address: ffff000008988cf8 type: "kernel_config_data"
<readmem: ffff000009137980, KVADDR, "devmem_is_allowed - jiffies", 8, (ROE|Q|NDS), ffffe06afe00>
<read_dev_mem: addr: ffff000009137980 paddr: 1137980 cnt: 8>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 1137980, 8): 4294967295 (ffffffff)
crash: read error: kernel virtual address: ffff000009137980 type: "devmem_is_allowed - jiffies"
crash: this kernel may be configured with CONFIG_STRICT_DEVMEM, which
renders /dev/mem unusable as a live memory source.
crash: trying /proc/kcore as an alternative to /dev/mem

[ 180.501732] RIP: 0033:0x7f8593edf337
[ 180.501819] Code: 25 00 00 41 00 3d 00 00 41 00 74 47 64 8b 04 25 18 00 00 00 85 c0 75 6b 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 95 00 00 00 48 8b 4c 24 28 64 48 33 0c 25
[ 180.502033] RSP: 002b:00007ffc9c915b80 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 180.502179] RAX: ffffffffffffffda RBX: 000055d574b0f2d0 RCX: 00007f8593edf337
[ 180.502270] RDX: 0000000000080880 RSI: 00007ffc9c917c2f RDI: 00000000ffffff9c
[ 180.502361] RBP: 00007ffc9c917c2f R08: 0000000000000000 R09: 00007ffc9c9156c0
[ 180.502451] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000080880
[ 180.502542] R13: 00007ffc9c915c30 R14: 0000000000000014 R15: 00007ffc9c917c2f
[ 180.502633] Modules linked in: vfat fat nfsv3 nfs_acl nfs lockd grace sunrpc scsi_transport_iscsi fscache af_packet br_netfilter bridge stp llc iscsi_ibft iscsi_boot_sysfs ip6t_REJECT xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip6table_filter ip6_tables x_tables snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss dmi_sysfs msr snd_hda_codec_idt snd_hda_codec_generic coretemp ledtrig_audio kvm_intel snd_hda_intel ppdev snd_hda_codec gpio_ich iTCO_wdt iTCO_vendor_support kvm snd_hda_core snd_hwdep irqbypass snd_pcm i2c_i801 mei_me snd_timer pcspkr e1000e button snd parport_pc mei parport soundcore lpc_ich mfd_core acpi_cpufreq ext4 crc16 mbcache jbd2 sd_mod firewire_ohci firewire_core serio_raw crc_itu_t ahci libahci ehci_pci uhci_hcd ehci_hcd usbcore sr_mod cdrom thermal ata_generic pata_marvell libata sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod
[ 180.503460] Supported: No, Unreleased kernel
[ 180.503548] CR2: 0000000000000038
[ 180.503656] ---[ end trace c1d770387093adf2 ]---
[ 180.503751] RIP: 0010:cdrom_release+0x19/0x1f0 [cdrom]
[ 180.503842] Code: 24 02 f7 d3 83 e3 01 eb e1 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 41 89 f5 55 53 48 89 fb 48 83 ec 48 <4c> 8b 27 48 c7 c7 50 34 51 c0 65 48 8b 04 25 28 00 00 00 48 89 44
[ 180.504059] RSP: 0018:ffffa93140abfb08 EFLAGS: 00010292
[ 180.504148] RAX: 0000000000000000 RBX: 0000000000000038 RCX: 0000000000000000
[ 180.504253] RDX: ffff92a8a5d80000 RSI: 00000000080000dd RDI: 0000000000000038
[ 180.504344] RBP: 00000000080000dd R08: 00000000000003b9 R09: 0000000000000037
[ 180.504436] R10: 0000000000000001 R11: ffffa93140abfab0 R12: ffff92a8a1c67418
[ 180.504527] R13: 00000000080000dd R14: ffff92a8a1c67400 R15: 00000000fffffffa
[ 180.504619] FS: 00007f85934f0dc0(0000) GS:ffff92a8a7a00000(0000) knlGS:0000000000000000
[ 180.504773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 180.504863] CR2: 0000000000000038 CR3: 0000000126ae0000 CR4: 00000000000006f0

dis cdrom_release
0xffffffffc050d7c0 <cdrom_release>: data32 data32 data32 xchg %ax,%ax [FTRACE NOP]
0xffffffffc050d7c5 <cdrom_release+5>: push %r15
0xffffffffc050d7c7 <cdrom_release+7>: push %r14
0xffffffffc050d7c9 <cdrom_release+9>: push %r13
0xffffffffc050d7cb <cdrom_release+11>: push %r12
0xffffffffc050d7cd <cdrom_release+13>: mov %esi,%r13d
0xffffffffc050d7d0 <cdrom_release+16>: push %rbp
0xffffffffc050d7d1 <cdrom_release+17>: push %rbx
0xffffffffc050d7d2 <cdrom_release+18>: mov %rdi,%rbx
0xffffffffc050d7d5 <cdrom_release+21>: sub $0x48,%rsp
0xffffffffc050d7d9 <cdrom_release+25>: mov (%rdi),%r12
0xffffffffc050d7dc <cdrom_release+28>: mov $0xffffffffc0513450,%rdi
0xffffffffc050d7e3 <cdrom_release+35>: mov %gs:0x28,%rax
0xffffffffc050d7ec <cdrom_release+44>: mov %rax,0x40(%rsp)

No offset 0x19 there

Hello,

1. boot Ubuntu 16.04.1 LTS with Linux kernel 4.4.0-47-generic (buildd@lcy01-03) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2) ) #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016

2. load kdump using kexec --load-panic --reuse-cmdline --initrd=/boot/initrd.img-4.8.11-grsec /boot/vmlinuz-4.8.11-grsec

3. trigger crash echo c > /proc/sysrq-trigger

4. expected: system to load 4.8.11-grsec custom kernel
   reality: got 'out of memory' kernel oops, whole trace ->> http://pastebin.com/zjMVeekZ

5. did VM dump using virsh dump instance-## ubuntu1604-grsec-4811_2.dump

6. tried to load a dump using latest crash tool, but never succeeded. Please find the logs here http://pastebin.com/QfrGUd1W

Please let me know if there is anything I can do to load the dump?

Thanks in advance!

This is in crash 7.2.1-2 and from compiling the source from 'master' branch (pulled today)

I get the 'crash: cannot allocate any more memory!' error. I have 8GB and 2 CPUs dedicated to my Fedora VM, I thought that would be plenty of memory as when I start up crash I don't see 8GB being used up in system monitor.

I searched around with Google and it seems like when this error shows up its associated with a bugzilla entry, so maybe this is a real problem with crash? I certainly cannot find any suggested command-line startups to use to try and work around this.

All I am doing is typing 'crash' on command-line and hitting enter. I was trying to use:

'crash vmlinux vmcore'

in which the vmcore is 214MB but I get that same error regardless if I type 'crash' on the command-line or try and load a vmcore file.

I've got
crash-7.2.5: page excluded: kernel virtual address: ffff880825d47000 type: "memory section"

7.2.1 crash works fine with the same coredump/vmlinux. Both versions were builded with the same gcc/gdb.
build_target: X86_64
build_version: 7.2.5
compiler version: gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17)
SLES kernel is 4.12.14-25.22 x86_64 SMP
I've checked 7.2.2 and it doesn't work also with the same error msg.

Hi,

i was unable to build with make but i install with apt install crash on ubuntu 16.10

uname -a
Linux raminfp 4.10.8-041008-generic #201703310531 SMP Fri Mar 31 09:33:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

kernel version 4.10.8-041008-generic

root@raminfp /h/r/D/crash# crash

crash 7.1.5
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
crash: cannot find booted kernel -- please enter namelist argument


Usage:

  crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]	(dumpfile form)
  crash [OPTION]... [NAMELIST]             		(live system form)

Enter "crash -h" for details.

i get error crash: cannot find booted kernel -- please enter namelist argument

you say on docs :

If the kernel file is stored in /boot, /, /boot/efi, or in any /usr/src
or /usr/lib/debug/lib/modules subdirectory, then no command line arguments
are required -- the first kernel found that matches /proc/version will be
used as the namelist.

root@raminfp /h/r/D/crash# cat /proc/version 
Linux version 4.10.8-041008-generic (kernel@gomeisa) (gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12) ) #201703310531 SMP Fri Mar 31 09:33:56 UTC 2017

root@raminfp /h/r/D/crash# ls /boot/
vmlinuz-4.10.8-041008-generic

crash not work on ubuntu 16.10, if i am mistake please help for resolve it,

Cheers,
Ramin

On full slab is tracked if SLAB_STORE_USER is not enabled. So cann't get full slabs via the node 'full' list. And by now crash just print '(not tracked)' for those full slabs.

But, checking full slab objects sometimes is usefull. And we can still find them by looking up all pages with PG_slab set and the page::slab_cache matches current slab cache. Then just print all objects in them.

Hi,

VM OMU is stuck somewhere.
Then dump the VM core with the command "virsh dump OMU OMU.dump"
crash can be used to read the file OMU.dump

Thanks

A vmcore core generated with makedumpfile -d1 ( or -d0 ") can not be processed.
It fails with :

crash: seek error: physical address: 83640e000 type: "pud page"

Adding debug to crash ( -d 8 ) shows this sequence of messages :

<readmem: 83640e000, PHYSADDR, "pud page", 4096, (FOE), 2a74f20>
<read_diskdump: addr: 83640e000 paddr: 83640e000 cnt: 4096>
xen_kdump_p2m: paddr/pfn: 83640e000/83640e: read mfn_frame: 1049d27000
<readmem: 1049d27000, PHYSADDR, "xen_kdump_p2m mfn frame", 4096, (ROE), 3a06e80>
<read_diskdump: addr: 1049d27000 paddr: 1049d27000 cnt: 4096>
read_diskdump: xen_kdump_p2m(1049d27000): 1049d27000
read_diskdump: paddr/pfn: 1049d27000/1049d27 -> cache physical page: 1049d27000
xen_kdump_p2m(83640e000): mfn_idx: 16818 frame_idx: 14 mfn_frame: 1049d27 mfn: ffffffffffffffff => fffffffffffff000
read_diskdump: xen_kdump_p2m(83640e000): fffffffffffff000
read_diskdump: SEEK_ERROR: paddr/pfn: fffffffffffff000/fffffffffffff max_mapnr: 1080000
crash: seek error: physical address: 83640e000 type: "pud page"

It appears to me

mfn: ffffffffffffffff => fffffffffffff000

Is not valid.

I am not certain if this is a kernel issue or crash issue.

--

makefiledump messages:

makedumpfile -X -d 0 --message-level 31 -D --work-dir=/tmp /proc/vmcore /kdumproot/home//crash/127.0.0.1-2021-04-06-12:45:10/vmcore-incomplete
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
phys_start phys_end virt_start virt_end
LOAD[ 0] 0 9b000 ffff880000000000 ffff88000009b000
LOAD[ 1] 100000 57000000 ffff880000100000 ffff880057000000
LOAD[ 2] 77000000 77928000 ffff880077000000 ffff880077928000
LOAD[ 3] 7bd4d000 7bd58000 ffff88007bd4d000 ffff88007bd58000
LOAD[ 4] 7bd59000 7bd5c000 ffff88007bd59000 ffff88007bd5c000
LOAD[ 5] 7bd5d000 7bd5e000 ffff88007bd5d000 ffff88007bd5e000
LOAD[ 6] 7bde4000 7c000000 ffff88007bde4000 ffff88007c000000
LOAD[ 7] 100000000 1080000000 ffff880100000000 ffff881080000000
Xen kdump
page_size : 4096

SYMBOL(dom_xen): ffff82d080740350
SYMBOL(dom_io): ffff82d080740348
SYMBOL(domain_list): ffff82d0805413a8
SYMBOL(xen_heap_start): 0
SYMBOL(frame_table): ffff82d08038b018
SYMBOL(alloc_bitmap): 0
SYMBOL(max_page): ffff82d080740338
SYMBOL(pgd_l2): 0
SYMBOL(pgd_l3): 0
SYMBOL(pgd_l4): ffff82d0803c0000
SYMBOL(xenheap_phys_end): 0
SYMBOL(xen_pstart): 0
SYMBOL(frametable_pg_dir): 0
SIZE(page_info): 32
OFFSET(page_info.count_info): 8
OFFSET(page_info._domain): 24
SIZE(domain): 3712
OFFSET(domain.domain_id): 0
OFFSET(domain.next_in_list): 112

xen_major_version: 4
xen_minor_version: 4
xen_phys_start: 77000000
frame_table_vaddr: ffff82e000000000
xen_heap_start: 0
xen_heap_end:0
alloc_bitmap: 0
max_page: 0
num_domain: 3
0: 839b07: ffff830839b07000
32754: 839d99: ffff830839d99000
32753: 83ff89: ffff83083ff89000

max_mapnr : 1080000

max_mapnr : 1080000
mem_map pfn_start pfn_end
mem_map[ 0] 0 0 0
mmap() is available on the kernel.
Checking for memory holes : [100.0 %] | [Checking for memory holes ] : 0.098263 seconds
Copying data : [100.0 %] - output line

Thank you.

When I use the kernel 5.10, and upgrade the crash version to 7.2.9, the crash tool do not work, and same issue with the version 7.2.8 on CentOS 8.2. It reports "crash: cannot determine length of symbol: log_end".

Hi,
I'm Intel employee, having a MIPS Platform running Linux OS.
I enabled the kexec\kdump flow in Linux version we use and now I have vmcore memory dump, and now I'm getting into troubles....
I'm running the crash utility compiled with "target=MIPS" and getting the output:

crash: read error: kernel virtual address: xxxx30fc type: "possible"
WARNING: cannot read cpu_possible_map
crash: read error: kernel virtual address: xxxx30f4 type: "present"
WARNING: cannot read cpu_present_map
crash: read error: kernel virtual address: xxxx30f8 type: "online"
WARNING: cannot read cpu_online_map
crash: read error: kernel virtual address: xxxx30f0 type: "active"
WARNING: cannot read cpu_active_map
crash: read error: kernel virtual address:xxxx0a48 type: "shadow_timekeeper xtime_sec"
crash: read error: kernel virtual address: xxxxa444 type: "init_uts_ns"
crash: linux/vmlinux and vmcore do not match!

As you see the tool claims that the vmlinux doesn't match to the vmcore although the vmcore captured from the given vmlinux.
When I open the vmcore with gdb from our toolchain provided by MIPS, it's successfully opened and I see that the init_uts_ns is in offset xxxxa440 instead of xxxxa444 it looks for in the crash utility.

I thought that maybe the issue is we use a propriety toolchain provided by Codescape (the next link) and tried to replace the gdb packet in the Makefile - but then the tool wasn't compile, it fails in the gdb_merge step:
http://codescape-mips-sdk.imgtec.com/components/toolchain/2017.10-05/downloads.html

Can you help us to understand how to overcome this issue? is there any easy way to replace the gdb binary the tool uses?

we'll very appreciate your help.

Hi,
I'm enabling core dump flow on my Linux, using Linux kexec mechanism. This flow ends with vmcore elf image reflecting the full memory situation at the crash moment. The vmcore driver in kernel that outputs the vmcore elf image has vmcore_add_device_dump API that's used for pushing some extra hardware memory sections into the ELF.

However, After vmcore file is generated, I want to see this data, but don't know how to do that. I'm using crash-utility for the core analysis, so I wonder - maybe there is an option to see also this extra data using crash-utility?

i am using Linux 5.4 kernel, and found that there is a issue on crash.

root@ubuntu:~# crash /mnt/dump.202003310331 /usr/src/linux/vmlinux

crash 7.2.8
Copyright (C) 2002-2020 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

crash: cannot determine VA_BITS_ACTUAL

root@ubuntu:~#

i enable the debug message:
root@ubuntu:~# crash /mnt/dump.202003310331 -d 12 /usr/src/linux/vmlinux

crash 7.2.8
Copyright (C) 2002-2020 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

compressed kdump: header->utsname.machine: aarch64
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: /mnt/dump.202003310331
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 183 (EM_AARCH64)

        header: aaaafcd11150
       signature: "KDUMP   "
  header_version: 6
         utsname:
           sysname: Linux
          nodename: ubuntu
           release: 5.4.0+
           version: #14 SMP Sun Mar 29 02:05:34 PDT 2020
           machine: aarch64
        domainname: (none)
       timestamp:
            tv_sec: 5e82b92c
           tv_usec: 0
          status: 1 (DUMP_DH_COMPRESSED_ZLIB)
      block_size: 4096
    sub_hdr_size: 1
   bitmap_blocks: 28
       max_mapnr: 458752
total_ram_blocks: 0
   device_blocks: 0
  written_blocks: 0
     current_cpu: 0
         nr_cpus: 4
  tasks[nr_cpus]: 0
                  0
                  0
                  0

    sub_header: 0 (n/a)

sub_header_kdump: aaaafcd12160
phys_base: 40000000
dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 5872 (0x16f0)
size_vmcoreinfo: 1875 (0x753)
OSRELEASE=5.4.0+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffff8000116a3368
SYMBOL(node_online_map)=ffff80001169ac78
SYMBOL(swapper_pg_dir)=ffff80001156f000
SYMBOL(_stext)=ffff800010081000
SYMBOL(vmap_area_list)=ffff8000116e1300
SYMBOL(mem_section)=ffff00002bdf1f00
LENGTH(mem_section)=2048
SIZE(mem_section)=32
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=4352
SIZE(zone)=1664
SIZE(free_area)=104
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._refcount)=52
OFFSET(page.mapping)=24
OFFSET(page.lru)=8
OFFSET(page._mapcount)=48
OFFSET(page.private)=40
OFFSET(page.compound_dtor)=16
OFFSET(page.compound_order)=17
OFFSET(page.compound_head)=8
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=3616
OFFSET(pglist_data.node_start_pfn)=3624
OFFSET(pglist_data.node_spanned_pages)=3640
OFFSET(pglist_data.node_id)=3648
OFFSET(zone.free_area)=192
OFFSET(zone.vm_stat)=1472
OFFSET(zone.spanned_pages)=96
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=40
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffff8000116bb240
SYMBOL(log_buf_len)=ffff8000116bb248
SYMBOL(log_first_idx)=ffff800011794408
SYMBOL(clear_idx)=ffff800011794440
SYMBOL(log_next_idx)=ffff800011794418
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=6
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=4
NUMBER(PG_private)=13
NUMBER(PG_swapcache)=10
NUMBER(PG_swapbacked)=19
NUMBER(PG_slab)=9
NUMBER(PG_hwpoison)=22
NUMBER(PG_head_mask)=65536
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
NUMBER(HUGETLB_PAGE_DTOR)=2
NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
NUMBER(VA_BITS)=48
NUMBER(kimage_voffset)=0xffff7fffd0000000
NUMBER(PHYS_OFFSET)=0x40000000
KERNELOFFSET=0
CRASHTIME=1585625388
offset_note: 4200 (0x1068)
size_note: 3548 (0xddc)
notes_buf: aaaafcd4b190
num_vmcoredd_notes: 0
num_prstatus_notes: 4
notes[0]: aaaafcd4b190 (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 0 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
X0: 00000000000000e0 X1: 4c8d7d360216aa00 X2: 4c8d7d360216aa00
X3: ffff00002bd7e6a0 X4: 0000000000000000 X5: 0000000000000000
X6: 0000000000000000 X7: 0000000000000000 X8: ffff8000116a4bc0
X9: ffff800011693de0 X10: 00000000000009e0 X11: 0000000000000000
X12: 0000000000000000 X13: 0000000000000000 X14: 0000000000000000
X15: 0000000000000000 X16: 0000000000000000 X17: 0000000000000000
X18: 0000000000000000 X19: ffff800011607000 X20: 0000000000000000
X21: 0000000048000000 X22: 0000000000000000 X23: 0000000000000000
X24: 0000000000000000 X25: 0000000000000000 X26: 0000000000000000
X27: 0000000000000000 X28: 0000000041570018 X29: ffff800011693e50
LR: ffff80001008f5d0 SP: ffff800011693e50 PC: ffff80001008f644
PSTATE: 60000005 FPVALID: 00000000
notes[1]: aaaafcd4b32c (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 0 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
X0: 00000000000000e0 X1: 1784968832808700 X2: 1784968832808700
X3: ffff00002bda06a0 X4: 0000000000000000 X5: ffff800010262cac
X6: 0000000000000000 X7: 0000000000000000 X8: ffff00002a5a5040
X9: ffff00002a5b7e40 X10: 00000000000009e0 X11: 0000000000000000
X12: 00001e8480000000 X13: 003d088b9f27daac X14: 000000005b8048cf
X15: 0000000000000000 X16: 0000000000000000 X17: 0000000000000000
X18: 0000000000000000 X19: 0000000000000000 X20: 0000000000000000
X21: 0000000000000000 X22: 0000000000000000 X23: 0000000000000000
X24: 0000000000000000 X25: 0000000000000000 X26: 0000000000000000
X27: 0000000000000000 X28: 0000000000000000 X29: ffff00002a5b7eb0
LR: ffff80001008f5d0 SP: ffff00002a5b7eb0 PC: ffff80001008f644
PSTATE: 60000005 FPVALID: 00000000
notes[2]: aaaafcd4b4c8 (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 0 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
X0: 00000000000000e0 X1: 6092186060fd2600 X2: 6092186060fd2600
X3: ffff00002bdc26a0 X4: 0000000000000000 X5: ffff800010262cac
X6: 0000000000000000 X7: 0000000000000000 X8: ffff00002a5a6c40
X9: ffff00002a5bbe40 X10: 00000000000009e0 X11: 0000000000000000
X12: 0000000000000000 X13: 0000000000000000 X14: 0000000000000000
X15: 0000000000000000 X16: 0000000000000000 X17: 0000000000000000
X18: 0000000000000000 X19: 0000000000000000 X20: 0000000000000000
X21: 0000000000000000 X22: 0000000000000000 X23: 0000000000000000
X24: 0000000000000000 X25: 0000000000000000 X26: 0000000000000000
X27: 0000000000000000 X28: 0000000000000000 X29: ffff00002a5bbeb0
LR: ffff80001008f5d0 SP: ffff00002a5bbeb0 PC: ffff80001008f644
PSTATE: 60000005 FPVALID: 00000000
notes[3]: aaaafcd4b664 (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 611 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
X0: 0000000000000000 X1: 0000000000000000 X2: ffff0000218b7938
X3: 0000000000000000 X4: ffff8000116c62b8 X5: 0000000000000000
X6: ffff00002a435f00 X7: 0720072007200720 X8: 0720072007200720
X9: 0720072007200720 X10: 0720072007200720 X11: 0720072007200720
X12: 0720072007200720 X13: 0720072007200720 X14: 0720072007200720
X15: 0000000000000000 X16: 0000000000000000 X17: 0000000000000000
X18: 0000000000000000 X19: 0000000000000000 X20: ffff80001a761000
X21: 00000000ffffffff X22: 0000ffffb51997e0 X23: 0000000020001000
X24: 0000000000000015 X25: 0000000056000000 X26: 0000000000000000
X27: 0000000000000000 X28: ffff000021934600 X29: ffff0000218b78e0
LR: ffff8000102b3b44 SP: ffff0000218b78e0 PC: ffff8000102b1af0
PSTATE: 60000085 FPVALID: 00000000
snapshot_task: 0
num_qemu_notes: 0
NOTE offsets: 1068 (NT_PRSTATUS)
1204 (NT_PRSTATUS)
13a0 (NT_PRSTATUS)
153c (NT_PRSTATUS)
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 458752 (0x70000)

   data_offset: 1e000
    block_size: 4096
   block_shift: 12
        bitmap: aaaafcd13170
    bitmap_len: 114688
     max_mapnr: 458752 (0x70000)

dumpable_bitmap: aaaafcd2f180
byte: 0
bit: 0
compressed_page: aaaafcd74560
curbufptr: 0

page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd64550
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd65550
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd66550
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd67550
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd68550
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd69550
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd6a550
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd6b550
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd6c550
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd6d550
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd6e550
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd6f550
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd70550
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd71550
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd72550
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: aaaafcd73550
pg_hit_count: 0

page_cache_buf: aaaafcd64550
   evict_index: 0
     evictions: 0
      accesses: 0
  cached_reads: 0 
   valid_pages: aaaafcd641c0

readmem: read_diskdump()
crash: pv_ops exists: ARCH_PVOPS

crash: cannot determine VA_BITS_ACTUAL

root@ubuntu:~#

Hi Dave, I'd like to ask you about the cadence of tagging in the project.

What triggers this question is that we wanna bump Ubuntu's version, and for that we'd like to bump Debian version first (the process works this way). But I've checked the upstream (which I believe corresponds to this repo), and we are at 7.2.7 for 4 months with about ~30 commits on top of it.

Given the repo history, you seem to be bumping versions within a 4-month cycle - if that's true and a bump to 7.2.8 comes soon, I'd prefer to wait for it and ask debian to bump to 7.2.8 directly.

Thanks in advance!

Hi.

I am able to build default target on my host (x86_64 arch) on ubuntu 16.04. But I need to build crash for ARM target.
So I run command : make target=ARM as suggested in the documentation.

Result :

configure: WARNING: no enhanced curses library found; disabling TUI
checking for library containing tgetent... no
configure: error: no termcap library found
Makefile:8235: recipe for target 'configure-gdb' failed
make[3]: *** [configure-gdb] Error 1
Makefile:834: recipe for target 'all' failed
make[2]: *** [all] Error 2

crash build failed

Makefile:229: recipe for target 'gdb_merge' failed
make[1]: *** [gdb_merge] Error 1
Makefile:224: recipe for target 'all' failed
make: *** [all] Error 2

I check and libncurses is already installed :

[arthur * (HEAD detached at 7.2.1)] sudo apt-get install libncurses5
[sudo] password for arthur: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libncurses5 is already the newest version (6.0+20160213-1ubuntu1).

I tried to build it with last master and last tag (7.2.1) without more success.
Do I miss something ?

Thanks,
Eva.

when trying to get stack frame with the command such as "bt -f" then, I could see a warning message like below
"cannot determine starting stack frame for task".

and still could not get the correct callstack with its registers even though message was just a warning.

could anyone solve this problem??

Thanks,

Hi, I have a problem with kernel 5.4 on arm.
crash is invoked like this:
./crash vmlinux system.map dump

I get the following result:

kernel 5.4: DBG: WARNING: invalid linux_banner pointer: 756e694c
DBG: crash: system.map and dump do not match!

Here are my observations:

cat system.map | grep linux_banner
c0c00084 D linux_banner

kernel.c:

	if (!(sp = symbol_search("linux_banner")))  // 1. struct syment sp gets filled here
		error(FATAL, "linux_banner symbol does not exist?\n");
	else if ((sp->type == 'R') || (sp->type == 'r') ||
		 (machine_type("ARM") && sp->type == 'T') ||
		 (machine_type("ARM64")))
		linux_banner = symbol_value("linux_banner");   // 2. we don't enter here because sp->type == 'D'
	else                         
		get_symbol_data("linux_banner", sizeof(ulong), &linux_banner); // 3. This doesn't work and gives a wrong address		

With gdb debugging I can see that the struct syment is fine address-wise:

(gdb) print sp->type
$2 = 68 'D'
(gdb) print sp->name
$3 = 0xf73de297 "linux_banner"
(gdb) print sp->value
$4 = 3233808516 (note: c0c00084 hex)

sp->type is equal to 'D' this prevents us to enter the else if even though calling symbol_value("linux_banner"); would be fine:

ulong
symbol_value(char *symbol)
{
        struct syment *sp;

        if (!(sp = symbol_search(symbol))) // calls symbol_search again
                error(FATAL, "cannot resolve \"%s\"\n", symbol);

        return(sp->value);  // correct value (see gdb output)
}

I fixed this locally by adding sp->type == 'D' in the else if condition.

What does 'D' mean in struct syment type ? Is this a bug ? If yes I can submit a patch.

I'm using crash for parsing sym -l output but it crashes with segment fault.
So I tried to enable -fsanitize=address for crash tool, but it reports below double-free during crash startup.

So is it possible for you to test crash with AddressSanitizer enabled?

thanks

==13580==ERROR: AddressSanitizer: attempting double-free on 0x6020012fb630 in thread T0:
#0 0x7f9e6dc54187 in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x107187)
#1 0x563f8540f7ea in free_ikconfig /home/jiangenj/workspace/github/crash/kernel.c:10188
#2 0x563f854109e0 in read_in_kernel_config /home/jiangenj/workspace/github/crash/kernel.c:10394
#3 0x563f8546d40b in restore_sanity /home/jiangenj/workspace/github/crash/cmdline.c:1223
#4 0x563f85470c82 in process_command_line /home/jiangenj/workspace/github/crash/cmdline.c:63
#5 0x563f8533acb4 in main_loop /home/jiangenj/workspace/github/crash/main.c:825
#6 0x563f8558edc2 in captured_command_loop /home/jiangenj/workspace/github/crash/gdb-7.6/gdb/main.c:258
#7 0x563f8558d4c9 in catch_errors /home/jiangenj/workspace/github/crash/gdb-7.6/gdb/exceptions.c:557
#8 0x563f8558fe85 in captured_main /home/jiangenj/workspace/github/crash/gdb-7.6/gdb/main.c:1064
#9 0x563f8558d4c9 in catch_errors /home/jiangenj/workspace/github/crash/gdb-7.6/gdb/exceptions.c:557
#10 0x563f85590220 in gdb_main /home/jiangenj/workspace/github/crash/gdb-7.6/gdb/main.c:1079
#11 0x563f85590220 in gdb_main_entry /home/jiangenj/workspace/github/crash/gdb-7.6/gdb/main.c:1099
#12 0x563f85339296 in main /home/jiangenj/workspace/github/crash/main.c:707
#13 0x7f9e6d7c7bba in __libc_start_main ../csu/libc-start.c:308
#14 0x563f8533a2e9 in _start (/home/jiangenj/bin/crash-extensions/crash64+0x10c2e9)

0x6020012fb630 is located 0 bytes inside of 3-byte region [0x6020012fb630,0x6020012fb633)
freed by thread T0 here:
#0 0x7f9e6dc54187 in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x107187)
#1 0x563f8543328c in arm64_get_section_size_bits /home/jiangenj/workspace/github/crash/arm64.c:1078
#2 0x563f8543328c in arm64_init /home/jiangenj/workspace/github/crash/arm64.c:378

previously allocated by thread T0 here:
#0 0x7f9e6dbdf0b5 in strdup (/lib/x86_64-linux-gnu/libasan.so.5+0x920b5)
#1 0x563f8540f231 in add_ikconfig_entry /home/jiangenj/workspace/github/crash/kernel.c:10140
#2 0x563f8540f49e in setup_ikconfig /home/jiangenj/workspace/github/crash/kernel.c:10160
#3 0x563f85410779 in read_in_kernel_config /home/jiangenj/workspace/github/crash/kernel.c:10380
#4 0x563f8540f9a4 in get_kernel_config /home/jiangenj/workspace/github/crash/kernel.c:10204
#5 0x563f85433ce1 in arm64_get_section_size_bits /home/jiangenj/workspace/github/crash/arm64.c:1075
#6 0x563f85433ce1 in arm64_init /home/jiangenj/workspace/github/crash/arm64.c:378

SUMMARY: AddressSanitizer: double-free (/lib/x86_64-linux-gnu/libasan.so.5+0x107187) in __interceptor_free
==13580==ABORTING

Currently if you calls ipcs on new kernels, you got:

crash> ipcs -m
SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS

ipcs: invalid structure member offset: idr_top
FILE: ipcs.c LINE: 628 FUNCTION: idr_find()

[/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488

ipcs: invalid structure member offset: idr_top
FILE: ipcs.c LINE: 628 FUNCTION: idr_find()

crash> ipcs -M

ipcs: invalid structure member offset: idr_top
FILE: ipcs.c LINE: 628 FUNCTION: idr_find()

[/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488

ipcs: invalid structure member offset: idr_top
FILE: ipcs.c LINE: 628 FUNCTION: idr_find()

This was caused after the Linux code decided to implement IDR using a radix tree, which is not how 'crash' is reading it.

commit 0a835c4f090af2c76fc2932c539c3b32fd21fbbb
Author: Matthew Wilcox <[email protected]>
Date:   Tue Dec 20 10:27:56 2016 -0500

Reimplement IDR and IDA using the radix tree

The IDR is very similar to the radix tree.  It has some functionality that
the radix tree did not have (alloc next free, cyclic allocation, a
callback-based for_each, destroy tree), which is readily implementable on
top of the radix tree.  A few small changes were needed in order to use a
tag to represent nodes with free space below them.  More extensive
changes were needed to support storing NULL as a valid entry in an IDR.
Plain radix trees still interpret NULL as a not-present entry.

The IDA is reimplemented as a client of the newly enhanced radix tree.  As
in the current implementation, it uses a bitmap at the last level of the
tree.

I start the domU with linux 4.12.13(bzImage) and xen 4.9.0, and I manually dump the guest memory with sudo xl dump-core myvm dump-res.

Then I try to analyze it in crash using sudo crash bzImage dump-res, but it tells me that

crash: bzImage: not a supported file format
I have tried to add -f option using sudo crash bzImage dump-res, but got the same error.

I then replace the bzImage with corresponding vmlinux, this time it gives me the following message

WARNING: cannot read linux_banner string
crash: src/linux-4.12.13/vmlinux and dump-res do not match!

How can I make them match? Xen does not support directly boot a uncompressed vmlinux file(in HVM mode) so I have keep using bzImage.

Thanks

I was trying to load a crash but got the following error
crash: cannot malloc kernel task array (1852408451 tasks)

It is Red Hat Enterprise Linux Server release 7.9 (Maipo).

Hi, I got result below when using bt command。

crash> bt
PID: 25723 TASK: ffffffd095bde400 CPU: 0 COMMAND: "bash"
bt: WARNING: corrupt prstatus? pstate=0x20000000, but no user frame found
bt: WARNING: cannot determine starting stack frame for task ffffffd095bde

the crash version is 7.1.5 and the vmcore is produced by a kernel panic injection. I wrote a panic in a module init function and insmod this module to make system panic.

It seems like some thing wrong with arm64_get_dumpfile_stackframe in arm64. When the panic happend on the kernel stack of a user space task.

    if (user_mode(ptregs)) {
            frame->sp = user_stack_pointer(ptregs);
            frame->fp = user_frame_pointer(ptregs);
            if (is_kernel_text(frame->pc) ||
                !in_user_stack(bt->tc->task, frame->sp)) {
                    error(WARNING,
                        "corrupt prstatus? pstate=0x%lx, but no user frame found\n",
                            ptregs->pstate);
                    bt->flags |= BT_REGS_NOT_FOUND;
                    return FALSE;
            }
            bt->flags |= BT_USER_SPACE;
    } else {
            frame->sp = ptregs->sp;
            frame->fp = ptregs->regs[29];
    }

I replace this part with implementation in crash-7.1.3 below ,and got the right result.

    frame->pc = ptregs->pc;
    frame->fp = ptregs->regs[29];

    if (!is_kernel_text(frame->pc) &&
        in_user_stack(bt->tc->task, frame->sp))
            bt->flags |= BT_USER_SPACE;


    if (arm64_in_kdump_text(bt, frame))
            bt->flags |= BT_KDUMP_ADJUST;