GDB debugging

[Christian Prochaska]

Hi Daniel,

I can reproduce the crash and this is what happens:

 > 0x00054b20 in ?? ()
 > (gdb) break Genode::printf
 > Breakpoint 1 at 0x1003574
 > (gdb) c
 > --continuing, test program does not break on printf.

At this point in time ld.lib.so is running, but it hasn't loaded the test application into memory yet. GDB wants to insert a 
software breakpoint at address 0x1003574 which belongs to the application. It starts reading the memory beginning at address 
0x1003574 and receives all zeroes, because the application is not there yet. Then it wants to replace the first two bytes with 'cd 
00' (int $0), which fails for the same reason. Therefore the test application does not break this time.

 > Continuing.
 > ^C
 > Program received signal SIGINT, Interrupt.
 > 0x00054443 in ?? ()

After receiving the interrupt message from the GDB monitor, GDB deactivates the breakpoint by restoring the saved "original" 
instructions (zeroes in this case).

 > (gdb) break Genode::printf
 > Note: breakpoint 1 also set at pc 0x1003574.
 > Breakpoint 2 at 0x1003574
 > (gdb) c

Now GDB wants to reinsert the breakpoint and reads the original instructions (the zeroes it had just restored) to save them again. 
Then it writes the breakpoint instruction again, which succeeds this time.

 > Continuing.
 > Breakpoint 1, 0x01003574 in Genode::printf ()
 > --program this time breaks on breakpoint

The breakpoint triggered and GDB restores the previously saved original instructions (zeroes) again.

 > (gdb) c
 > Continuing.
 > --program reports invalid context capability and stops.

Finally, the zeroes get executed by the CPU, which causes the crash.

We probably need to find some way to get GDB monitor notified when the application has been loaded into memory by ld.lib.so. Until 
then it is necessary to continue the execution for a short time and then manually interrupt (or let the application trigger a 
breakpoint itself) before breakpoints can be set in GDB. So, for the GDB monitor test application, a working command sequence would be:

(gdb) c
^C
(gdb) break Genode::printf
(gdb) c

or, even better:

(gdb) c
^C
(gdb) sharedlibrary
(gdb) break Genode::printf
(gdb) c

The "sharedlibrary" command causes GDB to load the symbol and debug information of all the shared libraries that had not been loaded 
yet at the beginning of the debugging session.

HTH,
Christian

On 21.10.2011 22:37, Daniel Waddington wrote:
> Hi Christian,
>
> OK. I tried this gdb and it did not work.
>
> So, lets now talk about this in the context of a freshly download genode and just using the "make app/gdb_monitor" and "make 
> run/gdb_monitor" commands.
>
> This is what I tried and it failed (log below).
>
> (gdb) break Genode::printf
> (gdb) c
> --continuing, test program does not break on printf.
> (gdb) ctrl-c
> (gdb) break Genode::printf
> (gdb) c
> --program this time breaks on breakpoint
> (gdb) c
> --program reports invalid context capability and stops.
>
>
> Can you try this simple test yourself?  The only other thing I could try is to rebuild the custom genode-gcc compiler.
>
> Thanks
> Daniel
>
> ---
> GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://bugs.launchpad.net/gdb-linaro/>...
> Reading symbols from /home/dwaddington/git/vanilla-genode/build/bin/test-gdb_monitor...done.
> Remote debugging using localhost:5555
> warning: Unable to find dynamic linker breakpoint function.
> GDB will be unable to debug shared library initializers
> and track explicitly loaded dynamic code.
> 0x00054b20 in ?? ()
> (gdb) break Genode::printf
> Breakpoint 1 at 0x1003574
> (gdb) c
> Continuing.
> ^C
> Program received signal SIGINT, Interrupt.
> 0x00054443 in ?? ()
> (gdb) break Genode::printf
> Note: breakpoint 1 also set at pc 0x1003574.
> Breakpoint 2 at 0x1003574
> (gdb) c
> Continuing.
>
> Breakpoint 1, 0x01003574 in Genode::printf ()
> (gdb) c
> Continuing.
>
> ---
> [init -> gdb_monitor] void init_libc_lock_pipe(): init_libc_lock_pipe()
> [init -> gdb_monitor] using the pipe libc plugin
> [init -> gdb_monitor] Starting application ... environ: 82e04
> [init -> gdb_monitor] add_lwp(1, 1, 0)
> [init -> gdb_monitor] Remote debugging using /dev/terminal
> GDB monitor is up, starting GDB in a new terminal
> [init -> gdb_monitor] Memory model: no memory at address 10047b0
> [init -> gdb_monitor] Memory model: no memory at address 10047b1
> [init -> gdb_monitor] Memory model: no memory at address 10047b2
> [init -> gdb_monitor] Memory model: no memory at address 10047b3
> [init -> gdb_monitor] Memory model: no memory at address 10047b0
> [init -> gdb_monitor] Memory model: no memory at address 10047b1
> [init -> gdb_monitor] Memory model: no memory at address 10047b2
> [init -> gdb_monitor] Memory model: no memory at address 10047b3
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003575
> [init -> gdb_monitor] Memory model: no memory at address 1003576
> [init -> gdb_monitor] Memory model: no memory at address 1003577
> [init -> gdb_monitor] Memory model: no memory at address 1003578
> [init -> gdb_monitor] Memory model: no memory at address 1003579
> [init -> gdb_monitor] Memory model: no memory at address 100357a
> [init -> gdb_monitor] Memory model: no memory at address 100357b
> [init -> gdb_monitor] Memory model: no memory at address 100357c
> [init -> gdb_monitor] Memory model: no memory at address 100357d
> [init -> gdb_monitor] Memory model: no memory at address 100357e
> [init -> gdb_monitor] Memory model: no memory at address 100357f
> [init -> gdb_monitor] Memory model: no memory at address 1003580
> [init -> gdb_monitor] Memory model: no memory at address 1003581
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] Memory model: no memory at address 1003575
> [init -> gdb_monitor] Memory model: no memory at address 1003574
> [init -> gdb_monitor] (attempted to write cd)
> [init -> gdb_monitor] Memory model: no memory at address 1003575
> [init -> gdb_monitor] (attempted to write 0)
> [init -> gdb_monitor] linux_resume_one_lwp(step = 0, signal = 0)
> [init -> gdb_monitor] genode_store_registers() - not yet implemented
> [init -> gdb_monitor] genode_wait_for_signal_or_gdb_interrupt
> [init -> gdb_monitor -> test-gdb_monitor] Starting ldso ...
> [init -> gdb_monitor -> test-gdb_monitor] Starting application ... environ: 82e04
> [init -> gdb_monitor] add_lwp(1, 2, 0)
> [init -> gdb_monitor -> test-gdb_monitor] void func2(): GDB monitor test is running...
> [init -> gdb_monitor -> test-gdb_monitor] Test thread is running, cnt=0
> [init -> gdb_monitor -> test-gdb_monitor] void func2(): GDB monitor test is running...
> [init -> gdb_monitor -> test-gdb_monitor] Test thread is running, cnt=1
> [init -> gdb_monitor -> test-gdb_monitor] void func2(): GDB monitor test is running...
> [init -> gdb_monitor -> test-gdb_monitor] Test thread is running, cnt=2
> [init -> gdb_monitor -> test-gdb_monitor] void func2(): GDB monitor test is running...
> [init -> gdb_monitor -> test-gdb_monitor] Test thread is running, cnt=3
> [init -> gdb_monitor] linux_resume_one_lwp(step = 0, signal = 0)
> [init -> gdb_monitor] genode_store_registers() - not yet implemented
> [init -> gdb_monitor] linux_resume_one_lwp(step = 0, signal = 0)
> [init -> gdb_monitor] genode_store_registers() - not yet implemented
> [init -> gdb_monitor] genode_wait_for_signal_or_gdb_interrupt
> [init -> gdb_monitor -> test-gdb_monitor] Test thread is running, cnt=4
> [init -> gdb_monitor] received signal for lwpid 1
> [init -> gdb_monitor] linux_resume_one_lwp(step = 1, signal = 0)
> [init -> gdb_monitor] genode_store_registers() - not yet implemented
> [init -> gdb_monitor] genode_wait_for_signal_or_gdb_interrupt
> no RM attachment (READ pf_addr=c pf_ip=1003574 from 636000)
> invalid signal-context capability
> virtual void Genode::Pager_activation_base::entry(): Could not resolve pf=c ip=1003574
>
>
>
> On 10/21/2011 06:14 AM, Christian Prochaska wrote:
>> Hi Daniel,
>>
>> you're right about the quota of the signal session. It gets upgraded
>> dynamically and the "Quota exceeded!" warning can be ignored in this case.
>>
>> I'm not sure about the cause of the breakpoint problem. Which version of
>> GDB are you using? If it's not the current version 7.3.1, my first
>> suggestion would be to try this version, which is also the version the
>> GDB monitor implementation is based on. You can find the GDB 7.3.1
>> source archive in the 'ports/download' directory and build a GDB
>> specifically for Genode by using the following 'configure' line:
>>
>> configure --prefix=/usr/local/genode-gcc --program-prefix=genode-x86-
>>
>> When starting GDB from a run script, the '[gdb]' function looks for
>> CROSS_DEV_PREFIX-gdb first, so it already prefers the Genode-GDB to the
>> host GDB.
>>
>> There's currently no programmatic API to trigger a breakpoint, but it
>> should work with asm("int $4").
>>
>> Regards,
>> Christian
>>
>> On 10/20/2011 05:39 PM, Daniel Waddington wrote:
>>> Hi Norman,
>>> This worked, but I am still not out of the woods.  First, there seems
>>> to be a signal quota issue (although I think this may be OK as more
>>> quota is requested dynamically?).  Second I have trouble setting break
>>> points.  When I set a break point and continue in gdb, it reports
>>> warning: Error removing breakpoint X.
>>>
>>> Also, is there a programmatic API to trigger a break into GDB
>>> (asm("int3") enters JDB)?
>>>
>>> Regards,
>>> Daniel
>>>
>>> [init ->  gdb_monitor] Remote debugging using /dev/terminal
>>> [init ->  gdb_monitor] Memory model: no memory at address 1009164
>>> [init ->  gdb_monitor] Memory model: no memory at address 1009165
>>> [init ->  gdb_monitor] Memory model: no memory at address 1009166
>>> ... REPEAT MANY TIMES
>>> [init ->  gdb_monitor] Memory model: no memory at address 1000160
>>> [init ->  gdb_monitor] Memory model: no memory at address 1000161
>>> [init ->  gdb_monitor] Memory model: no memory at address 1000162
>>> [init ->  gdb_monitor] Memory model: no memory at address 1000163
>>> [init ->  gdb_monitor] Memory model: no memory at address 1000164
>>> [init ->  gdb_monitor] Memory model: no memory at address 1000161
>>> [init ->  gdb_monitor] (attempted to write 0)
>>> [init ->  gdb_monitor] linux_resume_one_lwp(step = 0, signal = 0)
>>> [init ->  gdb_monitor] genode_store_registers() - not yet implemented
>>> [init ->  gdb_monitor] genode_wait_for_signal_or_gdb_interrupt
>>> [init ->  gdb_monitor ->  thread-migration] Starting ldso ...
>>> [init ->  gdb_monitor] received signal for lwpid 1
>>> [init ->  gdb_monitor] linux_resume_one_lwp(step = 1, signal = 0)
>>> [init ->  gdb_monitor] genode_store_registers() - not yet implemented
>>> [init ->  gdb_monitor] genode_wait_for_signal_or_gdb_interrupt
>>> [init ->  gdb_monitor] received signal for lwpid 1
>>> [init ->  gdb_monitor] linux_resume_one_lwp(step = 0, signal = 0)
>>> [init ->  gdb_monitor] genode_store_registers() - not yet implemented
>>> [init ->  gdb_monitor] genode_wait_for_signal_or_gdb_interrupt
>>> [init ->  gdb_monitor ->  thread-migration] Starting application ...
>>> environ: 85004
>>> [init ->  gdb_monitor ->  thread-migration] ==THREAD MIGRATION
>>> EXAMPLE===============
>>> [init ->  gdb_monitor ->  thread-migration] Creating thread [0]
>>> [init ->  gdb_monitor] add_lwp(1, 2, 0)
>>> [init ->  gdb_monitor ->  thread-migration] Creating thread [1]
>>> [init ->  gdb_monitor] add_lwp(1, 3, 0)
>>> [init ->  gdb_monitor ->  thread-migration] Creating thread [2]
>>> [init ->  gdb_monitor] received signal for lwpid 2
>>> Quota exceeded! amount=4096, size=4096, consumed=4096
>>> ??
>>>
>>> ---- GDB SIDE ---
>>> (gdb) target remote localhost:5555
>>> Remote debugging using localhost:5555
>>> Reading symbols from ld.lib.so...done.
>>> Loaded symbols for ld.lib.so
>>> 0x00055c50 in _start_ldso () from ld.lib.so
>>> (gdb)
>>> (gdb) break OmniOS::sleep
>>> Breakpoint 1 at 0x1000160: file
>>> /home/dwaddington/git/omnios/genode/base/../omnios/include/omnios/sleep.h,
>>> line 45.
>>> (gdb) c
>>> Continuing.
>>> warning: Error removing breakpoint 1
>>>
>>> -- 
>> ------------------------------------------------------------------------------
>> The demand for IT networking professionals continues to grow, and the
>> demand for specialized networking skills is growing even more rapidly.
>> Take a complimentary Learning at ...83... Self-Assessment and learn
>> about Cisco certifications, training, and career opportunities.
>> http://p.sf.net/sfu/cisco-dev2dev
>> _______________________________________________
>> Genode-main mailing list
>> Genode-main at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/genode-main
>>
>
>
> ------------------------------------------------------------------------------
> The demand for IT networking professionals continues to grow, and the
> demand for specialized networking skills is growing even more rapidly.
> Take a complimentary Learning at ...83... Self-Assessment and learn
> about Cisco certifications, training, and career opportunities.
> http://p.sf.net/sfu/cisco-dev2dev
>
>
> _______________________________________________
> Genode-main mailing list
> Genode-main at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/genode-main


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.genode.org/pipermail/users/attachments/20111022/0ab0ad24/attachment.html>