<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>Prex Kernel Debugging Tips and Tricks</title>
<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type">
<meta name="keywords" content="Prex, embedded, real-time, operating system, RTOS, open source, free">
<meta name="author" content="Kohsuke Ohtani">
<link rel="stylesheet" type="text/css" href="../default.css" media="screen">
<link rel="stylesheet" type="text/css" href="../print.css" media="print">
</head>
<body>
<div id="top">
</div>
<div id="middle">
<table id="content" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td id="header" colspan="2" valign="top">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td id="logo">
<a href="http://prex.sourceforge.net/">
<img alt="Prex logo" src="../img/logo.gif" border="0"
style="width: 281px; height: 56px;"></a>
</td>
<td id="brief" align="right" valign="bottom">
An Open Source, Royalty-free,<br>
Real-time Operating System
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="directory" style="vertical-align: top;">
<a href="http://prex.sourceforge.net/">Prex Home</a> >
<a href="index.html">Document Index</a> >
Kernel Debugging Tips and Tricks
</tr>
<tr><td class="pad" colspan="2" style="vertical-align: top;"></td></tr>
<tr>
<td id="main" style="vertical-align: top;">
<h1>Prex Kernel Debugging Tips and Tricks</h1>
<i>Version 1.0, 2005/09/05</i><br>
<h3>Table of Contents</h3>
<ul>
<li><a href="#dbg">Building a Debugging Kernel</a></li>
<li><a href="#print">Kernel Print</a></li>
<li><a href="#panic">Kernel Panic</a></li>
<li><a href="#assert">Assertion Check</a></li>
<li><a href="#trace">Function Trace</a></li>
<li><a href="#dump">Kernel Dump</a></li>
<li><a href="#gdb">Remote debugging with GDB</a></li>
</ul>
<br>
<br>
<h2 id="dbg">Building a Debugging Kernel</h2>
<p>
The Prex kernel is compiled with debugging version by default.
The debugging version will have the following extra functions
compared with the release version.
<ul>
<li>printk() to output the debug message.</li>
<li>panic() log for the fatal error.</li>
<li>ASSERT() for the assertion check.</li>
<li>Kernel function trace.</li>
<li>Dump of the kernel objects.</li>
</ul>
<p>
However, you should care about the following two points.
</p>
<ul>
<li>The size of executable image will become larger than
the release version.</li>
<li>The execution speed becomes slower than the release version.
</ul>
<p>
In order to build a release version, you must set the shell variable
named NDEBUG to 1 before compiling the kernel.
</p>
<pre class="terminal">
$ export NDEBUG=1
</pre>
<h2 id="print">Kernel Print</h2>
<p>
The most powerful tool for kernel debugging is printk(). You can check the
code path, or get the data value by putting the printk() anywhere in
the kernel code.
</p>
<p>
printk() is the subset of printf() in standard C library. You can use
only following identifiers for the output string.
</p>
<ul>
<li>%d - Decimal signed integer</li>
<li>%x - Hex integer</li>
<li>%u - Unsigned integer</li>
<li>%c - Character</li>
<li>%s - String</li>
</ul>
<h2 id="panic">Kernel Panic</h2>
<h3>What is Kernel Panic?</h3>
<p>
Panic is used to detect unrecoverable error in the kernel.
The kernel code can call panic() routine anytime if it
encounters a fatal error. panic() function will stop the system and
display the message if output device is available. If the kernel is not
compiled with debug mode, panic() will just reset the system without
any message. When GDB is connect to the kernel, the control will be
dropped to the debugger in panic().
</p>
<h3>Panic Screen</h3>
The following is a sample screen of panic() on i386.
<pre class="terminal">
=====================================================
Kernel panic!
Failed to create interrupt thread
=====================================================
============================
Trap 3: Breakpoint
============================
Trap frame 80002f28 error 0
eax 00000001 ebx 800110ac ecx ffffffff edx 800192b6 esi 00000000 edi 00000000
eip 80014289 esp 80002f28 ebp 80002f78 eflags 00000002
cs 00000010 ss 00000018 ds 00000018 es 00000018 esp0 00002fff
>> interrupt is disabled
Stack trace:
80011103
8001130c
80010115
</pre>
<h2 id="assert">Assertion Check</h2>
<h3>What is Assertion Check?</h3>
<p>
The assert macro puts error messages into the kernel.
If the specified expression in ASSERT() is false, it displays
the message to the output device and stops the system.
</p>
<p>
For example, the following ASSERT() macro is put to the entry of the
irq_attach() routine.
This means the caller of irq_attach() must provide a valid
function pointer for ISR (interrupt service routine).
</p>
<pre>
int irq_attach(int vector, int prio, int shared, int (*isr)(int), void (*ist)(int))
{
ASSERT(isr != NULL);
...
</pre>
If the kernel detect the invalid argument for isr, it will cause
a panic() with the following message.
<pre class="terminal">
Assertion failed: irq.c line:120 in irq_attach() 'isr != NULL'
</pre>
<p>
The important point is ASSERT() macro is only enabled with the kernel
debug version. If the assertion is caused by an application bug
or h/w bug, it should not be covered by ASSERT(). Such fault should be handled
as an "error" properly.
</p>
<h3>IRQ Assertion</h3>
<p>
There are some kernel routines that can not be called during
an interrupt context. For example, the following functions can not
be called by ISR.
</p>
<ul>
<li>IRQ control code. (irq_attach() or irq_detach).</li>
<li>The system call code that requires an user mode context.</li>
<li>The kernel code that assumes it can synchronize by scheduling lock.
(kmem).</li>
</ul>
<p>
In order to detect the invalid call of these functions, the kernel defines
a ASSERT_IRQ() macro. The kernel developer can put this macro in the
head of the kernel routine which can not be called by ISR.
When ISR call such routine, the kernel will report the problem with panic()
message.
</p>
<h2 id="trace">Function Trace</h2>
The kernel trace is used to log the entry/exit of all functions at
runtime.
It is useful to debug the timing critical issue related to
an interrupt handler.
<h3>Installing the kernel trace</h3>
<p>
You must set the shell variable named KTRACE to 1 before compiling the kernel.
</p>
<pre class="terminal">
$ export KTRACE=1
</pre>
<p>
The kernel function trace is disabled by default. If you get the error
from gcc, you should check the function with "extern inline" in the header
file. Such function must be replaced as "static inline" routine to
enable the function trace.
</p>
<h3>Using the kernel trace</h3>
<p>
The following functions are available for the kernel trace.
</p>
<ul>
<li>trace_on: Enable a function trace.</li>
<li>trace_off: Disable a function trace.</li>
<li>trace_dump: Dump the latest function log.</li>
</ul>
<p>
The trace code is using a ring buffer to store the log data.
So, older information will be disposed when newer log is filled in the buffer.
</p>
<h2 id="dump">Kernel Dump</h2>
<h3>Installing Kernel Dump</h3>
<p>
The kernel dump is used to dump the current status of each kernel
objects.
To install the kernel dump function, you must uncomment the following line
in config.h.
</p>
<pre>#define CONFIG_KDUMP 1</pre>
<p>
If you assign magic keys for the kernel dump, you can get the
kernel dump as far as the keyboard IRQ is enabled.
The key assignment of the kernel dump depends on each platform.
</p>
<h3>Dump Sample</h3>
<p>
The following dump screen was got with Prex-i386pc.
</p>
<pre class="terminal">
Kernel dump usage:
F1=help F2=thread F3=task F4=object F5=timer F6=irq F7=dev F8=mem
thread_dump:
mod thread task stat pol prio base lock qntm total susp sleep event
--- -------- -------- ---- ---- ---- ---- ---- ---- -------- ---- ------------
Knl 8002a014 800191a0 SLP FIFO 022 022 1 50 1 0 interrupt
Knl 80029a54 800191a0 SLP FIFO 020 020 1 50 0 0 interrupt
Knl 80029464 800191a0 SLP RR 015 015 1 50 0 0 timer
Knl 800190c0 800191a0 RUN* FIFO 255 255 1 0 44719 0 -
Usr 8002a6e4 8002a574 SLP RR 200 200 1 50 3 0 kbd
task_dump:
mod task nr_obj nr_thr map susp exc hdlr name
--- --------- ------ ------ -------- ---- -------- ------------
Knl 800191a0* 1 4 8001ac20 0 00000000 kernel
Usr 8002a574 1 1 8002a5e4 0 00000000 kmon
IRQ dump:
vector isr ist ist-thr ist-prio count
------ -------- -------- -------- -------- --------
00 800110c8 00000000 00000000 0 54124
01 80020784 800209f8 80029a54 20 66
06 800212d0 80021360 8002a014 22 1
12 80021aec 00000000 00000000 0 0
device_dump:
device open close read write ioctl event name
-------- -------- -------- -------- -------- -------- -------- ------------
8002a544 00000000 00000000 00000000 80020ecc 00000000 00000000 console
80029fb4 80021b80 80021bb0 80021be0 00000000 00000000 00000000 mouse
80029f44 8002161c 8002163c 80021714 80021800 800218dc 00000000 fd0
800299e4 80020a30 80020a50 80020a70 00000000 00000000 00000000 kbd
800299b4 00000000 00000000 80020584 00000000 00000000 00000000 rtc
80029984 80022808 80022828 00000000 00000000 80022848 00000000 pm
80029954 00000000 00000000 00000000 00000000 8002212c 00000000 cpu
page_dump:
free pages:
start end size
-------- -------- --------
00025000 - 00027000 2000
00034000 - 0009f000 6b000
00100000 - 040ff000 3fff000
used=200K free=65971K total=66171K
kmem_dump: page=2 (8Kbyte) alloc=7056byte unused=1136byte
Allocated block:
Size 48 bytes => 11
Size 64 bytes => 5
Size 112 bytes => 1
Size 224 bytes => 4
Size 1040 bytes => 5
Free block:
Size 32 bytes => 1
Size 1072 bytes => 1
vm_dump:
task=8002a574 map=8002a5e4 name=kmon
region virtual physical size flags
-------- -------- -------- -------- -----
8002a624 08048000 0002c000 1000 R----
8002a654 08049000 0002e000 1000 RW---
8002a6b4 7ffff000 0002f000 1000 RW---
*total=12K bytes
</pre>
<h2 id="gdb">Remote Debugging with GDB</h2>
<h3>Installing the GDB remote stub</h3>
<p>
To install GDB stub into the Prex kernel, you must uncomment the following line
in config.h.
</p>
<pre>#define CONFIG_GDB 1</pre>
<p>
The GDB support is disabled by default.
</p>
<h3>Connecting to the remote machine</h3>
You should specify the path to the symbol file.
<pre class="terminal">(gdb) symbol-file /usr/src/prex/sys/prex.sym
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0</pre>
<h3>Debugging Sample</h3>
<pre class="terminal">$ i386-elf-gdb
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "--host=i686-pc-cygwin --target=i386-elf".
(gdb) symbol-file /usr/src/prex-0.1.2/sys/prex.sym
Reading symbols from /usr/src/prex-0.1.2/sys/prex.sym...done.
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
Remote debugging using /dev/ttyS0
gdb_init () at gdb_glue.c:98
98 }
(gdb) b context_init
Breakpoint 1 at 0x80010d27: file context.c, line 72.
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x80010d27 in context_init at context.c:72
(gdb) c
Continuing.
Breakpoint 1, context_init (ctx=0x8001a16c, kstack=0x8002c414) at context.c:72
72 {
(gdb) bt 3
#0 context_init (ctx=0x8001a16c, kstack=0x8002c414) at context.c:72
#1 0x800137fe in thread_init () at thread.c:469
#2 0x800125e0 in kernel_main () at main.c:76
(More stack frames follow...)
(gdb) list context_init
67 *
68 * @ctx: context id (pointer)
69 * @kstack: kernel stack for the context
70 */
71 void context_init(context_t ctx, void *kstack)
72 {
73 struct kern_regs *k;
74 struct cpu_regs *u;
75
76 ctx->uregs = (struct cpu_regs *)(kstack - sizeof(struct cpu_regs
));
(gdb) up
#1 0x800137fe in thread_init () at thread.c:469
469 context_init(&idle_thread.context, stk + KSTACK_SIZE);
(gdb) list
464 panic("Failed to allocate idle stack");
465 memset(stk, 0, KSTACK_SIZE);
466 idle_thread.kstack = stk;
467 printk("kernel stack size=%d\n", KSTACK_SIZE);
468
469 context_init(&idle_thread.context, stk + KSTACK_SIZE);
470 list_insert(&kern_task.threads, &idle_thread.task_link);
471 }
(gdb) b 465
Breakpoint 2 at 0x800137c6: file thread.c, line 465.
(gdb) c
Continuing.
</pre>
</td>
</tr>
<tr>
<td id="footer" colspan="2" style="vertical-align: top;">
<a href="http://sourceforge.net">
<img src="http://sourceforge.net/sflogo.php?group_id=132028&type=1"
alt="SourceForge.net Logo" border="0" height="31" width="88"></a><br>
Copyright© 2005-2007 Kohsuke Ohtani
</td>
</tr>
</tbody>
</table>
</div>
<div id="bottom"></div>
</body>
</html>
![[BACK]](/icons/back.gif){kind=icon}
Return to debug.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / prex-old / doc / html / doc