I have an application which run somehow normally.
I try to run it via gdb, using the following run file attached below (which works in the past).
And now I have a problem with quota (see log below), after this message it just hangs, and my attempts to update figures in run file lead to nothing, same err.
I found that in general it try to upgrade session quota as a result of rpc call return, but why? what particular parameter responsible for it?
How I can find the reason? e.g. print current cap used/etc?
Booting from Hard Disk... Boot failed: could not read the boot disk
Booting from DVD/CD... Bender: Hello World. NOVA'
NOVA Microhypervisor v8-98b501b (x86_64): Apr 7 2021 23:34:52 [gcc 8.3.0] [MBI2]
[ 0] TSC:2011222 kHz BUS:1007275 kHz (measured) [ 0] CORE:0:0:0 6:f:b:0 [0] Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz [ 1] CORE:1:0:0 6:f:b:0 [0] Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
Hypervisor NOVA (API v8) core image [0000000000100000,0000000000cef000) binaries region [000000000029d000,0000000000cef000) free for reuse detected physical memory: 0x0000000000000000 - size: 0x000000000009fc00 use physical memory: 0x0000000000000000 - size: 0x000000000009f000 detected physical memory: 0x0000000000100000 - size: 0x00000000242df000 use physical memory: 0x0000000000100000 - size: 0x00000000242df000 reserved memory: 0x237ee000 - size: 0xbf02f0 type=-2 reserved memory: 0x9fc00 - size: 0x400 type=2 reserved memory: 0xf0000 - size: 0x10000 type=2 reserved memory: 0x243df000 - size: 0x21000 type=2 reserved memory: 0xb0000000 - size: 0x10000000 type=2 reserved memory: 0xfed1c000 - size: 0x4000 type=2 reserved memory: 0xfffc0000 - size: 0x40000 type=2 reserved memory: 0xfd000000 - size: 0x300000 type=-5 reserved memory: 0x1000000 - size: 0x800000 type=-1 reserved memory: 0x1800000 - size: 0x1c00000 type=-1 reserved memory: 0x243e2152 - size: 0x0 type=-3 Hypervisor reports 2x1 CPUs Warning: CPU has no invariant TSC. mapping: affinity space -> kernel cpu id - package:core:thread remap (0x0) -> 0 - 0:0:0) boot cpu remap (1x0) -> 1 - 1:0:0) ROM modules: ROM: [000000002398c000,000000002398c5e1) config ROM: [000000000000f000,0000000000013000) core_log ROM: [00000000242f5000,00000000243b88e0) gdb_monitor ROM: [00000000243b9000,00000000243dd0c8) gdbserver_platform.lib.so ROM: [000000002398d000,00000000239ed940) init ROM: [0000000023a0d000,0000000023b0b3a8) ld.lib.so ROM: [0000000023b0c000,0000000023ce88e8) libc.lib.so ROM: [0000000023ce9000,0000000023d23d40) libm.lib.so ROM: [00000000242b4000,00000000242d4a20) pc_uart_drv ROM: [000000000000d000,000000000000e000) platform_info ROM: [0000000023d24000,0000000023d34fb0) posix.lib.so ROM: [0000000023d35000,0000000023eea148) stdcxx.lib.so ROM: [0000000023f87000,00000000242b3ac0) test-go ROM: [00000000239ee000,0000000023a0c240) timer ROM: [0000000023eeb000,0000000023f86628) vfs.lib.so ROM: [00000000242d5000,00000000242f4b18) vfs_pipe.lib.so
36M kernel memory
Genode 20.11-203-g9c8a05c12 <local changes> 530 MiB RAM and 63253 caps assigned to init [init] 0x1000000 .. 0x10ffffff: linker area [init] 0x40000000 .. 0x4fffffff: stack area [init] 0x30000 .. 0x161fff: ld.lib.so [init] parent provides [init] service "ROM" [init] service "LOG" [init] service "RM" [init] service "CPU" [init] service "PD" [init] service "IRQ" [init] service "IO_MEM" [init] service "IO_PORT" [init] child "timer" [init] RAM quota: 1800K [init] cap quota: 66 [init] ELF binary: timer [init] priority: 0 [init] provides service Timer [init] child "pc_uart_drv" [init] RAM quota: 1800K [init] cap quota: 66 [init] ELF binary: pc_uart_drv [init] priority: 0 [init] provides service Uart [init] provides service Terminal [init] child "gdb_monitor" [init] RAM quota: 255752K [init] cap quota: 466 [init] ELF binary: gdb_monitor [init] priority: 0 [init] provides service test-go [init -> timer] 0x1000000 .. 0x10ffffff: linker area [init -> gdb_monitor] 0x1000000 .. 0x10ffffff: linker area [init -> pc_uart_drv] --- UART driver started --- [init -> timer] 0x40000000 .. 0x4fffffff: stack area [init -> timer] 0x30000 .. 0x161fff: ld.lib.so [init] child "pc_uart_drv" announces service "Uart" [init] child "pc_uart_drv" announces service "Terminal" [init] child "timer" announces service "Timer" [init -> gdb_monitor] 0x40000000 .. 0x4fffffff: stack area [init -> gdb_monitor] 0x30000 .. 0x161fff: ld.lib.so [init -> gdb_monitor] 0x10c7000 .. 0x10e9fff: gdbserver_platform.lib.so [init -> gdb_monitor] 0x10e0c000 .. 0x10ffffff: libc.lib.so [init -> gdb_monitor] 0x10d70000 .. 0x10e0bfff: vfs.lib.so [init -> gdb_monitor] 0x10d36000 .. 0x10d6ffff: libm.lib.so [init -> gdb_monitor] 0x10ea000 .. 0x12a1fff: stdcxx.lib.so [init -> pc_uart_drv] open IRQ 3 [init -> pc_uart_drv] [init -> pc_uart_drv] Warning: baud_rate 0 not supported, set to default [init -> pc_uart_drv] [init -> gdb_monitor] 0x10d0f000 .. 0x10d2dfff: vfs_pipe.lib.so
… [init -> gdb_monitor] Warning: unsupported ioctl (request=0x40047477) [init -> gdb_monitor] Warning: unsupported ioctl (request=0x80047476) [init -> gdb_monitor] Process target created; pid = 1 [init -> gdb_monitor] Remote debugging using /dev/terminal [init -> gdb_monitor] Error: fcntl(): command 6 not supported - vfs …
[init -> gdb_monitor] Warning: Pd_session::transfer_quota not implemented [init -> gdb_monitor] upgrading quota donation for PD session (0 bytes, 4 caps) [init] child "gdb_monitor" requests resources: ram_quota=0, cap_quota=4
run file: <config verbose="yes" ld_verbose="yes"> <parent-provides> <service name="ROM" /> <service name="LOG" /> <service name="RM" /> <service name="CPU" /> <service name="PD" /> <service name="IRQ" /> <service name="IO_MEM" /> <service name="IO_PORT" /> </parent-provides> <default-route> <any-service> <parent/> <any-child/> </any-service> </default-route> <default caps="250" /> <start name="timer" caps="100"> <resource name="RAM" quantum="2M"/> <provides> <service name="Timer"/> </provides> </start> <start name="pc_uart_drv" caps="100"> <resource name="RAM" quantum="2M" /> <provides> <service name="Uart"/> <service name="Terminal"/> </provides> <config> <policy label_prefix="gdb_monitor" uart="1" /> </config> </start> <start name="gdb_monitor" caps="500"> <resource name="RAM" quantum="250M" /> <provides> <service name="test-go" /> </provides> <config verbose="yes" ld_verbose="yes"> <target name="test-go" > <config verbose="yes" ld_verbose="yes"> <vfs> <dir name="dev"> <log /> </dir> <dir name="proc"> <dir name="self"> <rom name="exe" label="test-go"/> </dir> </dir> </vfs> <libc stdout="/dev/log" stderr="/dev/log" /> </config> </target> <preserve name="RAM" quantum="50M" /> <vfs> <dir name="dev"> <log/> <terminal/> </dir> <dir name="pipe"> <pipe/> </dir> </vfs> <libc stdout="/dev/log" stderr="/dev/log" pipe="/pipe"/> </config> </start> </config>
Hello,
On 08.04.21 19:42, Alexander Tormasov via users wrote:
I have an application which run somehow normally.
I try to run it via gdb, using the following run file attached below (which works in the past).
And now I have a problem with quota (see log below), after this message it just hangs, and my attempts to update figures in run file lead to nothing, same err.
I found that in general it try to upgrade session quota as a result of rpc call return, but why? what particular parameter responsible for it?
How I can find the reason? e.g. print current cap used/etc?
it could be possible that the 100 caps which gdb_monitor preserves for itself are not enough. Currently, this amount is defined in the source code, although a config option like for the RAM quota would be useful:
https://github.com/genodelabs/genode/blob/2e15ea3c5dab7da6423c7851e0e9ad40ab...
Christian
Hi Alexander,
On 08.04.21 19:42, Alexander Tormasov via users wrote:
I try to run it via gdb, using the following run file attached below (which works in the past).
I wonder, have you considered debugging such issues on base-linux as described here?
https://genodians.org/ssumpf/2019-04-26-java-gdb
At least that is my time-tested work flow when working on any regular component or protocol stack, including low-level code like the dynamic linker, libc, vfs, etc.
By debugging on base-nova, I'm afraid that you make your life harder than it needs to be. Still, even if you insist on using base-nova, I'd recommend first giving the Qemu GDB stub a try (using the argument -s) before going the on-target GDB route as you did.
Norman
I try to start linux version and fail ... Program lib/ld/linux/ld-linux make[1]: Leaving directory '/var/services/homes/admin/gen/21.02/build/x86_64' genode build completed using 'core-linux' as 'core' using 'linux_timer_drv' as 'timer' using 'ld-linux.lib.so' as 'ld.lib.so' spawn ./core Genode 20.11-203-g9c8a05c12 <local changes> 17592186044415 MiB RAM and 8997 caps assigned to init [init] Error: SECCOMP_SET_MODE_FILTER failed -38 [init] Error: Uncaught exception of type 'Genode::Exception' [init] Warning: abort called - thread: ep
9 апр. 2021 г., в 14:45, Norman Feske norman.feske@genode-labs.com написал(а):
Hi Alexander,
On 08.04.21 19:42, Alexander Tormasov via users wrote:
I try to run it via gdb, using the following run file attached below (which works in the past).
I wonder, have you considered debugging such issues on base-linux as described here?
https://genodians.org/ssumpf/2019-04-26-java-gdb
At least that is my time-tested work flow when working on any regular component or protocol stack, including low-level code like the dynamic linker, libc, vfs, etc.
I suspect that I need low level memory handling functions like mmap/vfork/etc
not sure for 100% that linux version support them
Hi Alexander,
On 11.04.21 00:03, Alexander Tormasov via users wrote:
I try to start linux version and fail ... Program lib/ld/linux/ld-linux make[1]: Leaving directory '/var/services/homes/admin/gen/21.02/build/x86_64' genode build completed using 'core-linux' as 'core' using 'linux_timer_drv' as 'timer' using 'ld-linux.lib.so' as 'ld.lib.so' spawn ./core Genode 20.11-203-g9c8a05c12 <local changes> 17592186044415 MiB RAM and 8997 caps assigned to init [init] Error: SECCOMP_SET_MODE_FILTER failed -38
since Genode 20.05, Genode on Linux uses seccomp to sandbox components by default [1]. The error -38 corresponds to ENOSYS. Your Linux kernel apparently lacks seccomp support.
For your debugging purposes, you may try side-stepping the seccomp-based sandboxing by placing an early return at the beginning of the 'binary_ready_hook_for_platform' in 'base-linux/src/lib/base/platform.cc'.
[1] https://genode.org/documentation/release-notes/20.05#Capability-based_securi...
Cheers Norman
Hi Norman,
For your debugging purposes, you may try side-stepping the seccomp-based sandboxing by placing an early return at the beginning of the 'binary_ready_hook_for_platform' in 'base-linux/src/lib/base/platform.cc'.
[1] https://genode.org/documentation/release-notes/20.05#Capability-based_securi...
after this patch it start working - while linux version of mmap allocation fails, can’\t give me requested addresses ranges (I try a couple of them, need to investigate more - probably mmap do not flexible here).
suggested solution with nova inside qemu -s somehow work except that it do not see threads! Now I found that for UP even in the very beginning gdb show only CPU0 single thread (while genode do create at least 2 - main and for signals). for 2 CPU version it show only 2 of them/etc.
the only workable solution with own gdbserver inside qemu with nova in this moment hangs, as described, in some attempt to upgrade quota: [init -> gdb_monitor] upgrading quota donation for PD session (0 bytes, 4 caps) [init] child "gdb_monitor" requests resources: ram_quota=0, cap_quota=4
when i update a bit run file it give me another message before hang (while I give 3500 caps): [init] child "gdb_monitor" requests resources: cap_quota=2
so… still not clear for me where it the best solution to continue
so far qemu -s works in faster way, while it do not see internal nova threads… Can we somehow force qemu to show nova threads?
Alexander
Hi Alexander,
Alexander Tormasov via users users@lists.genode.org writes:
the only workable solution with own gdbserver inside qemu with nova in this moment hangs, as described, in some attempt to upgrade quota: [init -> gdb_monitor] upgrading quota donation for PD session (0 bytes, 4 caps) [init] child "gdb_monitor" requests resources: ram_quota=0, cap_quota=4
when i update a bit run file it give me another message before hang (while I give 3500 caps): [init] child "gdb_monitor" requests resources: cap_quota=2
Is it possible that you missed message from Christian Prochasa he sent on April 9th? Increasing that constant helped me to further with debugging service using gdb_monitor on Nova in qemu.
suggested solution with nova inside qemu -s somehow work except that it do not see threads! Now I found that for UP even in the very beginning gdb show only CPU0 single thread (while genode do create at least 2 - main and for signals). for 2 CPU version it show only 2 of them/etc.
so… still not clear for me where it the best solution to continue
so far qemu -s works in faster way, while it do not see internal nova threads… Can we somehow force qemu to show nova threads?
Currently I stumbled on problem with threads too. I know that creating thread using 'pthread_create' in debugged code does not raise any error but after this call number of thread visible in gdb (using 'info thread') does not change and breakpoint set inside a function assigned for thread is never triggered.
Does anybody know if it is a sign of not creating thread at all or some deficiency in 'gdb_monitor' that fails to update presented list of threads? From the behavior it seems it is a first option.
Another question is if it is supposed to work and it doesn't only because of some error or gdb_monitor lacks this functionality completely?
Regards Tomasz Gajewski
Hi Tomasz,
On 06.05.21 12:42, Tomasz Gajewski wrote:
Currently I stumbled on problem with threads too. I know that creating thread using 'pthread_create' in debugged code does not raise any error but after this call number of thread visible in gdb (using 'info thread') does not change and breakpoint set inside a function assigned for thread is never triggered.
Does anybody know if it is a sign of not creating thread at all or some deficiency in 'gdb_monitor' that fails to update presented list of threads? From the behavior it seems it is a first option.
Another question is if it is supposed to work and it doesn't only because of some error or gdb_monitor lacks this functionality completely?
can you somehow reproduce the problem with the gdb monitor test which is used by gdb_monitor.run? This test also creates a pthread and the breakpoint in 'test_thread_start()' appears to work as well as the appearance in the thread info list.
Christian
Hi Christian,
Christian Prochaska christian.prochaska@genode-labs.com writes:
Hi Tomasz,
On 06.05.21 12:42, Tomasz Gajewski wrote:
Currently I stumbled on problem with threads too. I know that creating thread using 'pthread_create' in debugged code does not raise any error but after this call number of thread visible in gdb (using 'info thread') does not change and breakpoint set inside a function assigned for thread is never triggered.
Does anybody know if it is a sign of not creating thread at all or some deficiency in 'gdb_monitor' that fails to update presented list of threads? From the behavior it seems it is a first option.
Another question is if it is supposed to work and it doesn't only because of some error or gdb_monitor lacks this functionality completely?
can you somehow reproduce the problem with the gdb monitor test which is used by gdb_monitor.run? This test also creates a pthread and the breakpoint in 'test_thread_start()' appears to work as well as the appearance in the thread info list.
I'll experiment in the evening, compare bahavior and let know results. Your answer is encouraging. Thanks.
Hi Christian,
can you somehow reproduce the problem with the gdb monitor test which is used by gdb_monitor.run? This test also creates a pthread and the breakpoint in 'test_thread_start()' appears to work as well as the appearance in the thread info list.
I'll experiment in the evening, compare bahavior and let know results. Your answer is encouraging. Thanks.
I checked and there are two things that led me to wrong conclusions.
First is that executing just 'pthread_create' under debugger does not cause newly created thread to appear in output of 'info thread'. There has to be something that is needed to execute asynchronously but since program is stopped it does not happen.
Second is that I tried to put a breakpoint in function passed to 'pthread_create' and debugger never made it to stop on that breakpoint. I also added some 'Genode::log' in this function but received no output from it. That is why I had doubts about state of threads support in 'gdb_monitor'.
Generally, I'm trying to run 'ssh_terminal' under 'gdb_monitor' and my current state is published in my branch [1].
[1] https://github.com/tomga/genode/tree/gdb_ssh_terminal
After information that threads should not be a problem I checked again and wrote quite some text about that probably I messed up something with service routing that causes 'ssh_terminal' to hang in a call to announce 'Terminal' service in 'main.cc'. Later I removed what I wrote after checking one idea that occured to be true - there are two 'asynchronous' operations that 'gdb_monitor' intercepts quickly one after another:
* pthread create * service announce
and it seems that if they are executed too quickly by debugged program then there is a hang.
Adding a busy loop (for few seconds) before service announcement lets 'ssh_terminal' to go further, so it seems that it is a bug in gdb_monitor.
I pushed additional commit that workarounds the hang problem, so it should be easily testable.
Christian, can you confirm - given your knowledge about 'gdb_monitor' internals - that a bug in it that reveals in such scenerio is possible?
Regards Tomasz
Hi Tomas, thanks to point me to fix (I initially suppose that this is related to run file options), I can move further (while mostly again start to work with printf ;-) ).
About your problem - I can say that for me for NOVA at least with gdb_monitor as a part of run script can support pthread_create, I see changes and data related (up to 8 threads checked in my golang program) in my frontend gdb (use VSCode for this purposes). traps / breaks somehow work (while periodically can disappears completely, despite that VSCode show them, only reliable source is a gdb console direct command like «break …»/«info break»/«info thread» ). You need to follow article about genode gdb, at least to use manually «file» command initially for ld.lib.so, then setup break to the function where it load shared libraries and only after that you allowed to set breaks to your main code/libs - no pending operations works.
The same code do not show threads inside qemu -s, only show single-per-core thread.
I assume that qemu build-in gdb stub (which was recommended to me by Norman) do not able to see new threads for NOVA, probably it requires some special call from inside OS to inside qemu during creation of thread (may be because thread is not "native kind", native processes only has separate set of TLB/etc related to virtual space, - I think that this could be a reason - while I did not check it).
Sincerely, Alexander
Hello,
The same code do not show threads inside qemu -s, only show single-per-core thread.
I assume that qemu build-in gdb stub (which was recommended to me by Norman) do not able to see new threads for NOVA, probably it requires some special call from inside OS to inside qemu during creation of thread (may be because thread is not "native kind", native processes only has separate set of TLB/etc related to virtual space, - I think that this could be a reason - while I did not check it).
for debugging threads, Qemu -s is indeed not very useful per se.
In many debugging situations, however, it is often possible to work around this limitation. In [1] I have documented some life hacks that I find useful. In particular by adding an infinite loop at a point of interest in the code and attaching GDB once this loop is entered, one can observe the state of the thread that is currently spinning. In many situations that's all what's needed to prove or rule out a suspicion.
[1] https://genode.org/documentation/articles/sel4_part_1#A_root_task_for_exerci...
Norman
Followup to my own message.
Tomasz Gajewski tomga@wp.pl writes:
... there are two 'asynchronous' operations that 'gdb_monitor' intercepts quickly one after another:
- pthread create
- service announce
and it seems that if they are executed too quickly by debugged program then there is a hang.
Given that there are issues with 'ssh_terminal' stability (issue #4095 https://github.com/genodelabs/genode/issues/4095) I wasn't sure if observed hangs when executing those commands (creating thread and announcing service) quickly weren't a sign of some problem in 'ssh_terminal' itself. To verify I extended 'test-gdb_monitor' with announcement of simple LOG service (based on test code from os/src/test/init/main.cc).
Using this test I confifmed that:
* creating thread (using pthread_create) and announcing a service executed one after another quickly cause a hang when executed with 'gdb_monitor'
* it happens no matter which is first
* adding busy loop between them eliminates a hang problem (no matter which comes first)
* when executed on Nova announcing a new service causes creation of two threads visible in gdb_monitor - I think that this observation can be really important
I don't know internals well enough but I think that either:
* creating threads quickly may cause problems for gdb_monitor
* creating threads from different worlds (posix thread and some internal Genode service threads which I assume are created for service) may require some special care.
Hopefully those observations and thoughts will be helpful.
Regards Tomasz
Hi Tomasz,
to debug pthreads it might be worth considering running the scenario on base-linux and attach gdb directly [1]. Even though the ssh run-scripts currently don't support base-linux, I'm pretty sure there is nothing substantial that prevents us doing so, as fork/execve are now supported in linux as well. Maybe all you need to do is change nic_drv and rtc_drv with the linux counterparts and add `ld="no"` to their start nodes.
[1] https://genodians.org/ssumpf/2019-04-26-java-gdb
Best, Sid
Hi Sid,
Sid Hussmann sid.hussmann@gapfruit.com writes:
to debug pthreads it might be worth considering running the scenario on base-linux and attach gdb directly [1]. Even though the ssh run-scripts currently don't support base-linux, I'm pretty sure there is nothing substantial that prevents us doing so, as fork/execve are now supported in linux as well. Maybe all you need to do is change nic_drv and rtc_drv with the linux counterparts and add `ld="no"` to their start nodes.
Actually I did that yesterday and in that case even more important for me was article by Christian Helmuth [2].
Currently I tend to believe that diagnosis of Alexander Böttcher [3] about memory corruption is accurate (I can observe that structure with file descriptor numbers is changing during call to 'select' so probably another thread overwrites this memory).
Probably it would be much easier to find root cause if ubsan could be used (on whichever of platforms) but as I mentioned at the end of [4] I think it is not possible currently.
Anyway I'll continue investigation in the evening.
Regards, Tomasz
[2] https://genodians.org/chelmuth/2019-01-31-network-dev [3] https://github.com/genodelabs/genode/issues/4095#issuecomment-829913604 [4] https://github.com/genodelabs/genode/issues/4095#issuecomment-835312180
Gesendet: Donnerstag, 06. Mai 2021 um 12:42 Uhr Von: "Tomasz Gajewski" tomga@wp.pl An: users@lists.genode.org Betreff: Re: how to debug gdb?
Hi Alexander,
Alexander Tormasov via users users@lists.genode.org writes:
the only workable solution with own gdbserver inside qemu with nova in this moment hangs, as described, in some attempt to upgrade quota: [init -> gdb_monitor] upgrading quota donation for PD session (0 bytes, 4 caps) [init] child "gdb_monitor" requests resources: ram_quota=0, cap_quota=4
when i update a bit run file it give me another message before hang (while I give 3500 caps): [init] child "gdb_monitor" requests resources: cap_quota=2
Is it possible that you missed message from Christian Prochasa he sent on April 9th? Increasing that constant helped me to further with debugging service using gdb_monitor on Nova in qemu.
suggested solution with nova inside qemu -s somehow work except that it do not see threads! Now I found that for UP even in the very beginning gdb show only CPU0 single thread (while genode do create at least 2 - main and for signals). for 2 CPU version it show only 2 of them/etc.
so… still not clear for me where it the best solution to continue
so far qemu -s works in faster way, while it do not see internal nova threads… Can we somehow force qemu to show nova threads?
Currently I stumbled on problem with threads too. I know that creating thread using 'pthread_create' in debugged code does not raise any error but after this call number of thread visible in gdb (using 'info thread') does not change and breakpoint set inside a function assigned for thread is never triggered.
Does anybody know if it is a sign of not creating thread at all or some deficiency in 'gdb_monitor' that fails to update presented list of threads? From the behavior it seems it is a first option.
Qemu is not supposed to create threads. All threads in qemu are created at startup. For every (emulated) CPU core one thread, never more. For OS level threads you need a gdb server at OS level.
Another question is if it is supposed to work and it doesn't only because of some error or gdb_monitor lacks this functionality completely?
Regards Tomasz Gajewski
Genode users mailing list users@lists.genode.org https://lists.genode.org/listinfo/users
Uwe geno.de@public-files.de writes:
Qemu is not supposed to create threads. All threads in qemu are created at startup. For every (emulated) CPU core one thread, never more. For OS level threads you need a gdb server at OS level.
Thanks Uwe for clarification although actually it was Alexander which I think mixed application threads with threads in debugger attached to qemu. My fault was that I've chosen bad context from his message to my response.
My problems were entirely with threads not visible in gdb connected to gdb_monitor which controlled running application and now after response from Christian are partially resolved.
Regards Tomasz