users

users@lists.genode.org

2285 discussions

components sequence and libc
by Alexander Tormasov 07 Jul '20

07 Jul '20

I am trying to run application which have libc component and my own golang part. I use nova kernel on x86_64 version. I found that during the process of linker initialization of components I have uninitiated variable: #0 Genode::Stack::size (this=0x401fef28, size=0x40000) at /media/psf/Home/gen/20.05/repos/base/src/lib/base/thread.cc:34 #1 0x00000000000fb094 in Linker::Binary::call_entry_point (env=..., this=0x15e010 <unmanaged_singleton<Linker::Binary, 8, Genode::Env&, Genode::Heap&, Linker::Config const&, char const*>(Genode::Env&, Genode::Heap&, Linker::Config const&, char const*&&)::object_space>) at /media/psf/Home/gen/20.05/repos/base/src/lib/ldso/main.cc:436 #2 Component::construct (env=...) at /media/psf/Home/gen/20.05/repos/base/src/lib/ldso/main.cc:825 #3 0x000000000007cf7f in (anonymous namespace)::Constructor_component::construct (this=0x400fec10) at /media/psf/Home/gen/20.05/repos/base/src/lib/base/entrypoint.cc:319 #4 Genode::Meta::call_member<Genode::Meta::Empty, (anonymous namespace)::Constructor_component, Genode::Meta::Empty> (func=<optimized out>, server=...) at /media/psf/Home/gen/20.05/repos/base/include/util/meta.h:464 #5 (anonymous namespace)::Constructor::Rpc_construct::serve<(anonymous namespace)::Constructor_component, Genode::Meta::Empty> (args=<synthetic pointer>..., server=...) at /media/psf/Home/gen/20.05/repos/base/src/lib/base/entrypoint.cc:293 #6 Genode::Rpc_dispatcher<(anonymous namespace)::Constructor, (anonymous namespace)::Constructor_component>::_do_serve<(anonymous namespace)::Constructor::Rpc_construct> (args=<synthetic pointer>..., this=0x400fec98) at /media/psf/Home/gen/20.05/repos/base/include/base/rpc_server.h:146 #7 Genode::Rpc_dispatcher<(anonymous namespace)::Constructor, (anonymous namespace)::Constructor_component>::_do_dispatch<Genode::Meta::Type_list<(anonymous namespace)::Constructor::Rpc_construct> > (in=..., out=..., opcode=..., this=0x400fec98) at /media/psf/Home/gen/20.05/repos/base/include/base/rpc_server.h:180 #8 Genode::Rpc_dispatcher<(anonymous namespace)::Constructor, (anonymous namespace)::Constructor_component>::dispatch (in=..., out=..., opcode=..., this=0x400fec98) at /media/psf/Home/gen/20.05/repos/base/include/base/rpc_server.h:235 #9 Genode::Rpc_object<(anonymous namespace)::Constructor, (anonymous namespace)::Constructor_component>::dispatch (this=0x400fec10, opcode=..., in=..., out=...) at /media/psf/Home/gen/20.05/repos/base/include/base/rpc_server.h:272 #10 0x000000000009d08d in Genode::Rpc_entrypoint::<lambda(Genode::Rpc_object_base*)>::operator() (obj=0x400fec10, __closure=<synthetic pointer>) at /media/psf/Home/gen/20.05/repos/base-nova/src/lib/base/rpc_entrypoint.cc:177 #11 Genode::Object_pool<Genode::Rpc_object_base>::apply<Genode::Rpc_entrypoint::_activation_entry()::<lambda(Genode::Rpc_object_base*)> > (func=..., capid=<optimized out>, this=0x12e4b0 <Genode::bootstrap_component()::startup+528>) at /media/psf/Home/gen/20.05/repos/base/include/base/object_pool.h:147 #12 Genode::Rpc_entrypoint::_activation_entry () at /media/psf/Home/gen/20.05/repos/base-nova/src/lib/base/rpc_entrypoint.cc:180 #13 0x0000000000000000 in ?? () code #1 contains the following: void call_entry_point(Env &env) { /* apply the component-provided stack size */ if (Elf::Addr addr = lookup_symbol("_ZN9Component10stack_sizeEv")) { /* call 'Component::stack_size()' */ size_t const stack_size = ((size_t(*)())addr)(); /* expand stack according to the component's needs */ Thread::myself()->stack_size(stack_size); } … it found _ZN9Component10stack_sizeEv function for libc and then call the following function #0 from base/src/lib/base/thread.cc: void Stack::size(size_t const size) { /* check if the stack needs to be enhanced */ size_t const stack_size = (addr_t)_stack - _base; if (stack_size >= size) { return; } /* check if the stack enhancement fits the stack region */ enum { UTCB_SIZE = sizeof(Native_utcb), PAGE_SIZE_LOG2 = 12, PAGE_SIZE = (1UL << PAGE_SIZE_LOG2), }; addr_t const stack_slot_base = Stack_allocator::addr_to_base(this); size_t const ds_size = align_addr(size - stack_size, PAGE_SIZE_LOG2); if (_base - ds_size < stack_slot_base) throw Thread::Stack_too_large(); /* allocate and attach backing store for the stack enhancement */ addr_t const ds_addr = _base - ds_size - stack_area_virtual_base(); try { Ram_allocator * const ram = env_stack_area_ram_allocator; Ram_dataspace_capability const ds_cap = ram->alloc(ds_size); … here I found that env_stack_area_ram_allocator is null, not initialized! in core base/src/core/stack_area.cc I found part which probably should be called before previous code: void Genode::init_stack_area() { static Stack_area_region_map rm; env_stack_area_region_map = &rm; static Stack_area_ram_allocator ram; env_stack_area_ram_allocator = &ram; } seems that this initializer should be called before libc component initializer (it belong to different component)? Nesessary initialization probably happens in class constructor of class Genode::Core_env : public Env_deprecated, Noncopyable base/src/core/include/core_env.h seems that something wrong in link sequence. Question: ======== how I can fix the order of different component initialization? I probably need class Genode::Core_ to be initialized first as a part of general code of core-nova.o. may be I missed something like crt*.o in link? crt0.o definitely present in $libs/startup-nova/startup-nova.lib.a… on previous version I found some magical sequence which works after some manipulations with link order, but now with 20.05 it is not working any more for me. Some info about my code: in my code dummy.cc I have function (C style) main and component constructor void Libc::Component::construct(Libc::Env &env) { // Genode::Heap heap(env.ram(), env.rm()); // _verbose = false; static int c = 1; static char * myname = "GolangApp"; int r = 0; Libc::with_libc([&r] () { r = main(c, &myname); }); env.parent().exit(r); } my current link command line: libs=/media/psf/Home/gen/20.05/build/x86_64/var/libcache; genode-x86-gcc -B /media/psf/Home/gen/20.05/build/x86_64/../tool_chain-19.05/x86/gcc/gcc/ -Wl,-melf_x86_64 -Wl,-gc-sections -Wl,-z -Wl,max-page-size=0x1000 -Wl,--dynamic-list=/media/psf/Home/gen/20.05/repos/base/src/ld/genode_dyn.dl -nostdlib -Wl,-nostdlib -Wl,-Ttext=0x01000000 -m64 -mcmodel=large -Wl,--dynamic-linker=ld.lib.so -Wl,--eh-frame-hdr -Wl,-rpath-link=. -Wl,-T -Wl,/media/psf/Home/gen/20.05/repos/base/src/ld/genode_dyn.ld -Wl,--whole-archive -Wl,--start-group dummy.o main.o myacontext.o mycontext.o $libs/base-nova-common/base-nova-common.lib.a $libs/base-nova/base-nova.lib.a $libs/startup-nova/startup-nova.lib.a $libs/timeout/timeout.lib.a ld.lib.so libc.lib.so libm.lib.so vfs.lib.so -Wl,--no-whole-archive -Wl,--end-group /media/psf/Home/gen/20.05/build/x86_64/var/libcache/libc-gen/libc-gen.lib.a /media/psf/Home/gen/20.05/build/x86_64/var/libcache/libgo/libgobegin.a /media/psf/Home/gen/20.05/build/x86_64/var/libcache/libgo/libgolibbegin.a /media/psf/Home/gen/20.05/build/x86_64/var/libcache/libgo/libgo.a /media/psf/Home/gen/20.05/build/x86_64/var/libcache/libc-stdlib/libc-stdlib.lib.a /media/psf/Home/gen/20.05/build/x86_64/var/libcache/cxx/cxx.lib.a /media/psf/Home/gen/20.05/build/x86_64/../tool_chain-19.05/x86/gcc/x86_64-pc-elf/64/libgcc/libgcc_eh.a /media/psf/Home/gen/20.05/build/x86_64/../tool_chain-19.05/x86/gcc/x86_64-pc-elf/64/libgcc/libgcc.a Thanks!

2 3

Re: Undefined references when extending Child in repo "Base"
by Simon Himmelbauer 06 Jul '20

06 Jul '20

Hi Sebastian, > In case you want to call 'Child::test_print()' from outside of the base > library (e.g., a dynamically linked program), you need to add the > mangled 'Child::test_print' symbol to the Genode base library symbol > file [1]. This way the dynamic linker can find and export the function. > > [1] repos/base/lib/symbols/ld Thanks for your help, I could fix the problem. However, I am wondering whether there is an automated way to insert the mangled symbols? It seems cumbersome to add every symbol by hand. Best regards, Simon

2 1

Virtualbox recipe?
by Lonnie Cumberland 06 Jul '20

06 Jul '20

Greetgins all, I am back at it to try and work with Genode and Sulpt OS 20.02 now and am wondering if there is a setting up and running virtualbox in sculpt OS recipe that is known to work and that I can follow to test out things here? Any help would be greatly appreciated. Cheers, Lonnie

2 1

Undefined references when extending Child in repo "Base"
by Simon Himmelbauer 06 Jul '20

06 Jul '20

Hi Genodians, I am attempting to extend the class Child (base/include/base/child.h) with an additional function. However, when compiling my test program, I receive an undefined references error, even though I added the implementation to src/lib/base/child.cc. My observation seems to be that the generated object file does not get linked into the executable but calling any other Child-related function works fine. My child.h looks pretty much like this: class Genode::Child : .... { .... void test_print(); } and in child.cc: void Child::test_print() { log(__PRETTY_FUNCTION__); } Is there anything I am overseeing? Thanks in advance! Cheers, Simon

2 1

Re: Genode 20.05 : how to build kernel nova with debugging symbol ?
by Alexander Boettcher 19 Jun '20

19 Jun '20

On 19.06.20 13:11, Parfait Tok wrote: > Hi Alex, > > > Le ven. 19 juin 2020 à 10:55, Alexander Boettcher < > alexander.boettcher(a)genode-labs.com> a écrit : > >> >> look into build/x86_64/debug directory. There you find the hypervisor >> binary with debugging symbols. >> >> That used to be the case until now. > I've just made a fresh install from the git and there is nowhere I can find > the *kernel* debugging symbols, > although other binaries debugging symbols are right there in the > build/x86_64/debug directory. > This is the list of commands I issued. > git clone https://github.com/genodelabs/genode.git genode20.05 > cd genode20.05/ > ./tool/create_builddir x86_64 > xxx/genode20.05/prepare_port dde_linux nova x86emu dde_ipxe dde_zircon libc > seoul x86emu nova grub2 -j8 > xxx/genode20.05/tool/depot/create > genodelabs/pkg/x86_64/drivers_interactive-pc > genodelabs/bin/x86_64/base-nova genodelabs/bin/x86_64/demo > genodelabs/bin/x86_64/global_keys_handler genodelabs/bin/x86_64/init > genodelabs/bin/x86_64/nit_focus genodelabs/bin/x86_64/nitpicker > genodelabs/bin/x86_64/report_rom genodelabs/bin/x86_64/rom_filter > CROSS_DEV_PREFIX=/usr/local/genode/tool/19.05/bin/genode-x86- > cd build/x86_64/ > make run/demo KERNEL=nova > After build completed, > find -name *hypervisor* > ./var/run/demo/boot/hypervisor > What did I miss? If you invoke a script, using depot packages, you indeed get no binaries with symbols. The packages by default contain only stripped binaries. In this case you have to build the binary once, since you want to debug it, e.g. in your case: make -C build/x86_64 KERNEL=nova kernel to find in build/x86_64/debug the binaries with debugging symbols. Cheers, Alex. -- Alexander Boettcher Genode Labs https://www.genode-labs.com - https://www.genode.org Genode Labs GmbH - Amtsgericht Dresden - HRB 28424 - Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth

1 0

Genode 20.05 : how to build kernel nova with debugging symbol ?
by Parfait Tok 18 Jun '20

18 Jun '20

Hello, With the latest version, I don't know where to find the kernel binary (Nova) with its debugging symbols. Is there any option to pass to the building script in order to get this? Or how to modify the script to generate the kernel debugging symbols? I used to comment the line > exec [cross_dev_prefix]strip [run_dir]/boot/hypervisor > in file > tool/run/boot_dir/nova > but with the 20.05 version, it does not help. Thanks

1 0

Memory bandwidth degradation of Armv8 virtualization
by Chris Hofer 09 Jun '20

09 Jun '20

Hi everyone, I'm currently measuring the memory bandwidth of the Genode Armv8 Linux Virtualization by running tinymembench[1] and lmbench's memory bandwidth[2]. On the NXP I.mx8 the results of both benchmarks showed a degradation of roughly 25 to 35 percent (depending on the exact benchmark) compared to native Linux. I tried to ensure comparability with the native setup by disabling in native Linux all but one core and fixing this core's frequency to 1 GHz (which should be the frequency Genode uses). Both setups use the same Kernel version. To improve the performance I played with the register values controlling caching (clear HCR_EL2.CD - enable S2 Data cacheability; VTCR_EL2.Irgn0 and VTCR_EL2.Orgn0 respectively set to b11 - set Normal memory, Inner/Outer Write-Back Read-allocate No Write-Allocate Cachable). However, this had no significant effect. What's your opinion on that? Based on your experience with virtualization, do these values surprise you or is this an expected degradation? All the best, Chris [1] https://github.com/ssvb/tinymembench [2] http://lmbench.sourceforge.net/

2 1

Synchronous RPC and Session Types
by Sid Hussmann 03 Jun '20

03 Jun '20

Hi Genodians, in our current project, the question of synchronous communications between processes comes up quite often. For background, our scenario processes numerous parallel requests, which requires, among other operations, verifying a signature of a request and signing its response. The signature verification and signing operations are encapsulated in separate processes. That way they do not need to understand the complex message format of requests and responses, but only binary hashes and signatures. Therefore, the component processing the request must send the input data to the verify or sign component and wait for the response before continuing processing. In our current solution we use Genode's basic synchronous RPC mechanism. This approach has the effect of creating new session types when more different functionalities such as verifying or signing are added. This seems to be in conflict with the Genode philosophy of using only few selected session types. On the other hand, building the scenario using the asynchronous report/rom mechanism would lead to significantly more complex code to avoid mixing up the many requests being processed in parallel. Is there any better way to accomplish this? Cheers, Sid -- Sid Hussmann CTO & Founder gapfruit AG Baarerstrasse 135 6300 Zug - Switzerland sid.hussmann(a)gapfruit.com

2 1

Announcement: Genode OS Framework version 20.05 released
by Christian Helmuth 28 May '20

28 May '20

The principal topic of Genode 20.05 is "consolidation", which is a polite synonym for hard work hidden from the user and developer under the hood of our widely stabilized API. The strong motivation for this work stems from the confrontation of our software with increasingly complex real world workloads. Along these lines we generalized of our central component for managing platform I/O resources with a from-the-scratch platform driver for ARM as the first step. Also, we unified the virtualization support for ARMv7/ARMv8 and added support for Qemu virt ARM platforms for 32 and 64 bit. A functional highlight marks the feature completeness of our block-encryption component CBE implemented in ADA/SPARK clearing the way to integrate this feature into Sculpt soon. Also in a joint effort with the community, we improved Genode on Linux from a sheer development tool to a product-ready platform with effective sandboxing of components based on seccomp and object capabilities. So, adopters can employ Genode/Linux and start right away with the application development on the target platform, and at any time migrate transparently to a microkernel. Last but not least, don't miss the annual update of the Genode Foundations book. More highlights of the current release are - Capability-based security via seccomp on Linux - Feature-completeness of our consistent block encrypter - Consolidation of the framework API and feature set - New revision of the Genode Foundations book - Ability to host board support outside the Genode source tree - Platform-driver architecture for the ARM universe - Unified virtual-machine monitor for ARMv7 and ARMv8 - Qemu-virt platform support - MSI-X support on x86 - Improved CPU affinity management on the NOVA hypervisor The complete release documentation for the version 20.05 is available at https://genode.org/documentation/release-notes/20.05 Have fun -- Christian Helmuth Genode Labs https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/ Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth

1 0

Deadlock in combination with pthread, lwip and grpc
by Sid Hussmann 22 May '20

22 May '20

Dear Genodians, during stress testing a gRPC server with parallel connections I witness a race condition where the gRPC server component becomes unresponsive, while being able to respond to Genode heartbeats. Examining the different threads using GDB after the deadlock occurs, I notice one thread waiting at `spinlock_lock` after `Lwip::File_system::write()` threw an `Insufficient_buffer` exception: ```gdb (gdb) t 13 [Switching to thread 13 (LWP 520409)] #0 spinlock_lock (lock_variable=lock_variable@entry=0x501cad40 <Linker::mutex()::_mutex>) at /home/user/genode/repos/base/include/spec/x86/cpu/atomic.h:51 51 return tmp == cmp_val; (gdb) bt #0 spinlock_lock (lock_variable=lock_variable@entry=0x501cad40 <Linker::mutex()::_mutex>) at /home/user/genode/repos/base/include/spec/x86/cpu/atomic.h:51 #1 Genode::Cancelable_lock::lock (this=this@entry=0x501cad40 <Linker::mutex()::_mutex>, myself=...) at /home/user/genode/repos/base/src/lib/base/lock.cc:71 #2 0x000000005004fa02 in Genode::Mutex::acquire (this=this@entry=0x501cad40 <Linker::mutex()::_mutex>) at /home/user/genode/repos/base/src/lib/base/mutex.cc:27 #3 0x00000000500bb050 in Genode::Mutex::Guard::Guard (mutex=..., this=<synthetic pointer>) at /home/user/genode/repos/base/include/base/mutex.h:40 #4 dl_iterate_phdr (callback=callback@entry=0x500a3a40 <_Unwind_IteratePhdrCallback>, data=data@entry=0x41ffa1b0) at /home/user/genode/repos/base/src/lib/ldso/exception.cc:41 #5 0x00000000500a4caf in _Unwind_Find_FDE (pc= 0xaf828f2 <Libc::Vfs_plugin::write(Libc::File_descriptor*, void const*, unsigned long)+722>, bases=bases@entry=0x41ffa418) at /data/genode/contrib/gcc-893d85767720fc2b3c695c56d737d2307b1f2229/src/noux-pkg/gcc/libgcc/unwind-dw2-fde-dip.c:469 #6 0x00000000500a12a3 in uw_frame_state_for (context=context@entry=0x41ffa370, fs=fs@entry=0x41ffa460) at /data/genode/contrib/gcc-893d85767720fc2b3c695c56d737d2307b1f2229/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1257 #7 0x00000000500a2bdb in _Unwind_RaiseException (exc=0x501be8e8 <Genode::init_cxx_heap(Genode::Env&)::initial_block+7528>) at /data/genode/contrib/gcc-893d85767720fc2b3c695c56d737d2307b1f2229/src/noux-pkg/gcc/libgcc/unwind.inc:104 #8 0x00000000500a98c5 in __cxa_throw () at /data/genode/contrib/gcc-893d85767720fc2b3c695c56d737d2307b1f2229/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.h:157 #9 0x000000000ad17404 in non-virtual thunk to Lwip::File_system::write(Vfs::Vfs_handle*, char const*, unsigned long long, unsigned long long&) () from vfs_lwip.lib.so #10 0x000000000af828f3 in Libc::Vfs_plugin::write ( this=0xaff6dc0 <unmanaged_singleton<Libc::Kernel, 8, Genode::Env&, Genode::Allocator&>(Genode::Env&, Genode::Allocator&)::object_space+9272>, fd=0x7ffff7c586b0, buf=0x7ffff7abfd6e, count=3419) at /home/user/genode/repos/libports/src/lib/libc/vfs_plugin.cc:806 #11 0x000000000af711e7 in do_sendto (fd=<optimized out>, buf=0x7ffff7abfd6e, len=3419, dest_addr=<optimized out>, dest_addrlen=<optimized out>, flags=<optimized out>) at /home/user/genode/repos/libports/src/lib/libc/socket_fs_plugin.cc:176 #12 0x000000000af72a28 in Libc::Socket_fs::Plugin::write (this=<optimized out>, fd=<optimized out>, buf=<optimized out>, count=<optimized out>) at /home/user/genode/repos/libports/src/lib/libc/socket_fs_plugin.cc:1099 #13 0x00000000011c9f60 in tcp_send (fd=84, msg=msg@entry=0x41ffb2f0, additional_flags=additional_flags@entry=0) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/tcp_posix.cc:958 #14 0x00000000011ca18a in tcp_flush (tcp=tcp@entry=0x7ffff7d7da80, error=error@entry=0x41fff238) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/tcp_posix.cc:1499 #15 0x00000000011ca56b in tcp_write (ep=0x7ffff7d7da80, buf=<optimized out>, cb=<optimized out>, arg=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/tcp_posix.cc:1636 #16 0x000000000116c6d0 in write_action (gt=0x7ffff7e90e20) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/closure.h:110 #17 write_action_begin_locked (gt=0x7ffff7e90e20) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:956 #18 0x00000000011b25c1 in grpc_combiner_continue_exec_ctx () at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/combiner.cc:252 #19 0x00000000011b9818 in grpc_core::ExecCtx::Flush (this=this@entry=0x41fff3b0) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/exec_ctx.cc:156 #20 0x0000000001217b64 in grpc_core::ExecCtx::~ExecCtx (this=0x41fff3b0, __in_chrg=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/iomgr/exec_ctx.h:124 #21 grpc_call_start_batch (call=<optimized out>, ops=0x41fff470, nops=3, tag=0x41fff8b0, reserved=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/surface/call.cc:1973 #22 0x0000000001006eaa in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpServerSendStatus, grpc::internal::CallNoOp<4>, grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::ContinueFillOpsAfterInterception (this=0x41fff8b0) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/include/grpcpp/impl/codegen/call.h:72 #23 0x00000000010067ec in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpServerSendStatus, grpc::internal::CallNoOp<4>, grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::FillOps (this=0x41fff8b0, call=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/include/grpcpp/impl/codegen/interceptor_common.h:235 #24 0x0000000001008aa4 in grpc::internal::Call::PerformOps (ops=0x41fff8b0, this=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/include/grpcpp/impl/codegen/call.h:69 #25 grpc_impl::internal::RpcMethodHandler<helloworld::Greeter::Service, helloworld::HelloRequest, helloworld::HelloReply>::RunHandler (this=this@entry=0x7ffff7f55300, param=...) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/include/grpcpp/impl/codegen/method_handler_impl.h:90 #26 0x000000000126891a in grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception (this=this@entry=0x7ffff7cdb650) at /home/user/genode/contrib/stdcxx-79ad8821269a443aab21bfc1d0e9b8913332b6d1/include/stdcxx/bits/move.h:182 #27 0x000000000126916c in grpc_impl::Server::SyncRequest::CallData::Run (resources=true, global_callbacks=..., this=0x7ffff7cdb650) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/cpp/server/server_cc.cc:489 #28 grpc_impl::Server::SyncRequestThreadManager::DoWork (this=<optimized out>, tag=<optimized out>, ok=<optimized out>, resources=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/cpp/server/server_cc.cc:900 #29 0x00000000012afa66 in grpc::ThreadManager::MainWorkLoop (this=0x7ffff7f4f460) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/cpp/thread_manager/thread_manager.cc:211 #30 0x00000000012afcb0 in grpc::ThreadManager::WorkerThread::Run (this=0x7ffff7eae6e0) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/cpp/thread_manager/thread_manager.cc:43 #31 0x00000000011ae0ac in grpc_core::(anonymous namespace)::ThreadInternalsPosix::<lambda(void*)>::operator() (__closure=0x0, v=<optimized out>) at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/gprpp/thd_posix.cc:140 #32 grpc_core::(anonymous namespace)::ThreadInternalsPosix::<lambda(void*)>::_FUN(void *) () at /home/user/genode/contrib/protobuf_grpc-3414c16ccc9b43fa990653590ac986adf8780fce/src/lib/grpc/src/core/lib/gprpp/thd_posix.cc:145 #33 0x000000000af66391 in Libc::Pthread::Thread_object::entry (this=0x7ffff7d374f8) at /home/user/genode/repos/libports/src/lib/libc/pthread.cc:71 #34 0x0000000050096295 in Genode::Thread::_thread_start () at /home/user/genode/repos/base/include/base/output.h:54 #35 0x00000000500abab2 in thread_start () at /home/user/genode/repos/base-linux/src/lib/syscall/spec/x86_64/lx_clone.S:62 (gdb) ``` At this point, I also notice many threads being stuck after a `pthread_exit` call: ```gdb (gdb) t 10 [Switching to thread 10 (LWP 520056)] #0 pseudo_end () at /home/user/genode/repos/base-linux/src/lib/syscall/spec/x86_64/lx_syscall.S:29 29 ret /* Return to caller. */ (gdb) bt #0 pseudo_end () at /home/user/genode/repos/base-linux/src/lib/syscall/spec/x86_64/lx_syscall.S:29 #1 0x000000005004f3fd in lx_futex (val=0, op=0, uaddr=<optimized out>) at /home/user/genode/repos/base-linux/src/lib/syscall/linux_syscalls.h:436 #2 thread_stop_myself (myself=<optimized out>) at /home/user/genode/repos/base-linux/src/include/base/internal/lock_helper.h:62 #3 Genode::Cancelable_lock::lock (this=this@entry=0x426ffd60, myself=...) at /home/user/genode/repos/base/src/lib/base/lock.cc:128 #4 0x000000005004f5a3 in Genode::Cancelable_lock::lock (this=this@entry=0x426ffd60) at /home/user/genode/repos/base/src/lib/base/lock.cc:66 #5 0x0000000050075990 in Genode::Lock::lock (this=<optimized out>) at /home/user/genode/repos/base/include/base/lock.h:33 #6 Genode::Blockade::block (this=<optimized out>) at /home/user/genode/repos/base/include/base/blockade.h:31 #7 Genode::sleep_forever () at /home/user/genode/repos/base/src/lib/base/sleep.cc:21 #8 0x000000000af66332 in pthread_exit (value_ptr=0x0) at /home/user/genode/repos/libports/src/lib/libc/pthread.cc:558 #9 0x000000000af663a3 in Libc::Pthread::Thread_object::entry (this=0x7ffff7d7a078) at /home/user/genode/repos/libports/src/lib/libc/pthread.cc:71 #10 0x0000000050096295 in Genode::Thread::_thread_start () at /home/user/genode/repos/base/include/base/output.h:54 #11 0x00000000500abab2 in thread_start () at /home/user/genode/repos/base-linux/src/lib/syscall/spec/x86_64/lx_clone.S:62 (gdb) ``` Despite our best efforts we have failed to pinpoint the problem so far. A few questions: 1. Is the spinlock really a problem here? 2. I was able to reproduce this problem using the newest version of gRPC [1] and the current `genodelabs/staging` branch. Reading [0] makes me wonder if the problem can be seen as "yet another instance where the VFS synchronization has bitten us". 3. Unlike `lwip`, `lxip` seems to give up after just one or two connections with this warning: `_wait_and_dispatch_one_io_signal called from non-entrypoint thread "pthread.5"`. May this issue be related or should we just focus on `lwip`? 4. In the GDB backtrace e.g. of `thread 13` at `#5` I notice a path to `libgcc` that does not belong to my machine. Is this neglegtable or does it mess with my GDB output? We just updated the `grpc` and `protobuf` libraries [1], which we will publish shortly. The problem however persists with both versions. To reproduce this issue, please checkout branch [2] and run `grpc-server-lwip` under Linux. Starting multiple client connections in parallel can be done by running `./stress_parallel_execute.sh 10.0.2.55:8899` using [3]. Note that the problem sometimes takes a few thousand of parallel connections to reproduce. [0] https://github.com/genodelabs/genode/issues/3738#issuecomment-620496337 [1] https://github.com/trimpim/genode-world/tree/xxxx-update_protobuf_grpc_port [2] https://github.com/sidhussmann/genode-world/tree/grpc-server-race-condition [3] https://github.com/sidhussmann/grpc-connection-stress Cheers, Sid -- Sid Hussmann CTO & Founder gapfruit AG Baarerstrasse 135 6300 Zug - Switzerland sid.hussmann(a)gapfruit.com

2 11

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

users