ksymoops reads a Linux kernel Oops file and makes the best stab at
converting the code to instructions and mapping addresses to kernel
symbols.
This Quick-way-to-HowTo is based upon "Oopss ! An introduction to kernel debugging" lecture.
Ok,
your kernel have oopsed on your development platform, something
went wrong and you wonder why/where.
You intent to clear this out
on another, stable environnement. Please, just fellow the guide :
0. Get your kernel in debug mode
If not already done, turn your kernel debug on (
CONFIG_DEBUG_INFO=y on 2.6)
and check when compiling the
-g flag is being used.
in order to check this out, you could start a more verbose display :
cd /usr/src/linux
make V=1
1. OOPS Collect
Sometimes OOPS can be fatal and your system is messed up, sometime you still
can collect current kernel symbols on
/proc/ksyms. If you can issue a :
cat /proc/ksyms >file
Then send this file or cut/paste from it to your stable
env. you'll get more infos. If you can't get ksyms, hope you'll have
enough infos without (usually true).
Nevertheless best is to cut and paste from a terminal the OOPS message itself
to a oops file on your stable environnement.
Cut from the first line
'Unable to handle..." down to the end line "Code : 0xXX 0xXX .. ".
2. Depack Ksymsoops
You'll find a prepared package on my site Programs > Ksymsoops.tar.gz.
3. Building ksymoops for cross compile and debugging
Ksymoops INSTALL File stats :
"When you are building a kernel using a cross compiler, it may be useful to
build a dedicated version of ksymoops for this cross compile environment.
It is not necessary to do so, you can always use the native ksymoops with
suitable options (e.g. the e, t and a flags), together with environment
variables KSYMOOPS_NM and KSYMOOPS_OBJDUMP. It may be more convenient to
build a special version of ksymoops and install it along with the other cross
compile tools."
Yes it is more convinient to modify makefile to match your needs
Ksymoops INSTALL File also stats :
"At build time you can specify the CROSS, BFD_PREFIX, DEF_TARGET and DEF_ARCH
options to make. For example,"
make BFD_PREFIX=/usr/mips64-linux \
DEF_TARGET='\"elf64-bigmips\"' \
DEF_ARCH='\"mips:8000\"' \
CROSS=mips64-linux-
IMHO drop this, have your custom Makefile to be over with. The plus
is that you'll get a dedicated ksymoops to deploy in the same cross compile
binaires directory just by doing make install.
For example :
- using bfd files from /usr/mips64-linux/include and /usr/mips64-linux/lib
(BFD_PREFIX)
- the default value for -t will be "elf64-bigmips" (DEF_TARGET)
- the default value for -a will be "mips:8000" (DEF_ARCH)
- the result will be installed as /usr/bin/mips64-linux-ksymoops (CROSS)
- it will default to using /usr/bin/mips64-linux-{nm,objdump} (INSTALL_PREFIX,
/bin and CROSS).
Any variable starting with DEF_ takes a string value. These variables go
through two levels of expansion, shell (use '...' to avoid shell expansion),
and make commands (prefix " with \ to preserve " characters).
4. Make use of the debug.sh script
This script preparams a call to ksymsoops, it is simple to customize and to use.
#sh debug.sh MMC_OOPS
(!) trig no dynamic kernel symbols.
Remember if oops isn't fatal that you can retrieve it in /proc/ksyms !
ksymoops 2.4.9 on i686 2.4.20-8. Options used
-v /usr/src/linux-2.6.10/vmlinux (specified)
-K (specified)
-L (specified)
-O (specified)
-m /usr/src/linux-2.6.10/System.map (specified)
-t elf32-littlearm -a armv5te
Unable to handle kernel paging request at virtual address 10001244
Internal error: Oops: 805 [#1]
CPU: 0
pc : [] lr : [<00000001>] Not tainted
sp : c0337dc8 ip : 60000013 fp : c0337e04
r10: 00000002 r9 : c0223854 r8 : 00000001
r7 : 00000000 r6 : c0337e80 r5 : c0337e80 r4 : c2c5c900
r3 : 10001000 r2 : 00000091 r1 : c2e8aed8 r0 : 00000027
Flags: nzcv IRQs on FIQs on Mode SVC_32 Segment kernel
Control: 5317F Table: C0004000 DAC: 00000017
[...]
Code: e1a02a22 e3c33eff e3c3300f e1a02142 (e7831102)
>>PC; c0177a28 <sdhc_request+310/5dc> <=====
>>r9; c0223854 <imx21_sdhc1_device+8/b4>
Code; c0177a18 <sdhc_request+300/5dc>
00000000 <_PC>:
Code; c0177a18 <sdhc_request+300/5dc>
0: e1a02a22 mov r2, r2, lsr #20
Code; c0177a1c <sdhc_request+304/5dc>
4: e3c33eff bic r3, r3, #4080 ; 0xff0
Code; c0177a20 <sdhc_request+308/5dc>
8: e3c3300f bic r3, r3, #15 ; 0xf
Code; c0177a24 <sdhc_request+30c/5dc>
c: e1a02142 mov r2, r2, asr #2
Code; c0177a28 <sdhc_request+310/5dc>
10: e7831102 str r1, [r3, r2, lsl #2]
5. Pin pointing in the dessambly with the source
In the example, the program counter (PC or EIP) is set on
sdhc_request at offset 310.
Go find where this symbol/function is coded, in what object file. Once retreived you could make use of
objdump -g -d object.o to disassemble this function code. I advice you to use
-S option to mix ASM disassembly with C source, and even to use
-l option to get C source line numbering.
#objdump -S -g -l -d imx21sdhc.o
imx21sdhc.o: file format elf32-littlearm
Disassembly of section .text:
00000000 :
sdhc_exit():
drivers/mmc/imx21sdhc.c:132
0: e1a0c00d mov ip, sp
sdhc_stop_clock():
4: e92dd800 stmdb sp!, {fp, ip, lr, pc}
8: e24cb004 sub fp, ip, #4 ; 0x4
sdhc_exit():
drivers/mmc/imx21sdhc.c:140
c: e59fe048 ldr lr, [pc, #72] ; 5c <.text+0x5c>
drivers/mmc/imx21sdhc.c:135
10: e5902008 ldr r2, [r0, #8]
drivers/mmc/imx21sdhc.c:136
14: e3a0c000 mov ip, #0 ; 0x0
drivers/mmc/imx21sdhc.c:135
18: e1d230b0 ldrh r3, [r2]
sdhc_init():
1c: e3833001 orr r3, r3, #1 ; 0x1
drivers/mmc/imx21sdhc.c:931
20: e1c230b0 strh r3, [r2]
drivers/mmc/imx21sdhc.c:137
24: e5902008 ldr r2, [r0, #8]
28: e1a0100c mov r1, ip
2c: e1d230b4 ldrh r3, [r2, #4]
[...]
00000660 :
sdhc_request():
drivers/mmc/imx21sdhc.c:460
660: e1a0c00d mov ip,
664: e92ddff0 stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
668: e24cb004 sub fp, ip, #4 ; 0x4
66c: e24dd014 sub sp, sp, #20 ; 0x14
drivers/mmc/imx21sdhc.c:222
94c: e1a03302 mov r3, r2, lsl #6
950: e2833d42 add r3, r3, #4224 ; 0x1080
954: e2833241 add r3, r3, #268435460 ; 0x10000004
958: e1a02a03 mov r2, r3, lsl #20
95c: e5911008 ldr r1, [r1, #8]
960: e1a02a22 mov r2, r2, lsr #20
964: e3c33eff bic r3, r3, #4080 ; 0xff0
968: e3c3300f bic r3, r3, #15 ; 0xf
96c: e1a02142 mov r2, r2, asr #2
970: e7831102 str r1, [r3, r2, lsl #2]
drivers/mmc/imx21sdhc.c:223
974: e5943014 ldr r3, [r4, #20]
978: e594100c ldr r1, [r4, #12]
97c: e1a03303 mov r3, r3, lsl #6
980: e2833201 add r3, r3, #268435456 ; 0x10000000
984: e2833d42 add r3, r3, #4224 ; 0x1080
988: ea000270 b 9c8
Note that as
sdhc_request() starts at offset 660h then
660h+310h is 970h.
The faulty code is
str r1, [r3, r2, lsl #2] in line 223 of drivers/mmc/imx21sdhc.c.
However note also, offset 660h matches line 460 whereas 970h matches line 222.
This is due to inner-function
function calls, in the said case,
sdhc_request() (l.460) calls
sdhc_setup_data() (l.222).
Here is a extract of
lines 126 to 227 when sdhc_setup_data() has been called :
static void sdhc_setup_data(Sdhc *sdhc, struct mmc_data *data)
{
[...]
dprintk("%s from %08x to %08x \n",
sdhc->dma_dir == DMA_FROM_DEVICE ? "device read" : "device write",
sdhc->dma_dir == DMA_FROM_DEVICE ? SDHC_BUFFER_ACCESS_PHYS(sdhc) : sg_dma_address(&data->sg[0]),
sdhc->dma_dir == DMA_FROM_DEVICE ? sg_dma_address(&data->sg[0]) : SDHC_BUFFER_ACCESS_PHYS(sdhc) );
if (data->flags & MMC_DATA_READ) {
DMA_DAR(sdhc->dma) = sg_dma_address(&data->sg[0]);
DMA_SAR(sdhc->dma) = SDHC_BUFFER_ACCESS_PHYS(sdhc);
} else {
DMA_SAR(sdhc->dma) = sg_dma_address(&data->sg[0]);
DMA_DAR(sdhc->dma) = SDHC_BUFFER_ACCESS_PHYS(sdhc);
}
[...]
}
Finally I come to discover it was a bad macro substitution of DMA_SAR & DMA_DAR.