Genode: Thread Stack Allocation

[Frank Kaiser]

Hello

In a recent project (using Genode on top of OKL4 v2.1 on an ARM
platform) I ran into the problem that in one protection domain (PD)
occasionally the exception Stack_alloc_failed was thrown. Tracing this
back it came out that this was a Region_conflict exception thrown by the
RM session client code due to the return value given by core's RM
session server.
In that particular PD the main() function and the first started thread
both created subthreads independently of each other. The exception
occurred because an ATTACH command for the same thread stack address
area was sent to core twice.
The thread stack allocation is done by the constructor
Thread_base::Thread_base() with its 2nd initializer element
_context(_alloc_context(stack_size)). The method
Thread_base::_alloc_context() handles the address assignment of a thread
stack locally within a PD, which is done by the call
_context_allocator()->alloc(this). Beginning from virtual address
0x4000.0000 for each new stack a segment of 1 MB is reserved at which
end the stack area with the requested size is allocated. To find out
whether a stack area is already in use, the so-called Context_allocator
provides the method Thread_base::Context_allocator::_is_in_use().
The thread's context is maintained in structure Thread_base::Context,
which is written at the top of each allocated stack. The class
Thread_base owns the member _context to point to this data. The
Context_allocator itself is a static object instantiated on the first
call to function _context_allocator(). It provides a chained list
(member _threads) of all created threads. The method _is_in_use() walks
through the thread list and determines whether any of the existing
threads points to the context address of the stack area in question
(thread member _context equal to the context address at the top of stack
area under consideration) and returns true, if this is the case. The
method Thread_base::Context_allocator::alloc() iterates through the
stack segments until it finds an unused stack area. It inserts the new
thread into the chained list (call
_threads.insert(&thread_base->_list_element);) and returns the stack's
Context address to the caller Thread_base::_alloc_context().
The problem is that at this point a decision about a new stack
allocation is made which is not yet visible in the chained list of
threads, because the new thread is already in the list, but its member
_context is not yet set. This happens on the return of
Thread_base::_alloc_context() to the constructor, but before that an IPC
is made to core to register the new stack allocation. On OKL4 each IPC
invokes the scheduler, and what happens in the failure case, is that
another process is scheduled which starts instantiating a further new
thread. In this situation the method
Thread_base::Context_allocator::_is_in_use() does not find the stack
area of the previously created stack as occupied, and the caller tries
to allocate the stack area another time. However, core detects the
double allocation and returns a bad result code on the 2nd ATTACH
command.
To fix the problem a lock is required to cover the whole sequence from
beginning the search for a free stack area up to the assignment of
Thread_base::_context. The existing lock
Thread_base::Context_allocator::_threads_lock, used in
Thread_base::Context_allocator::alloc(), is not sufficient for this
purpose. For a quick fix I inserted the following two lines at the
beginning of Thread_base::_alloc_context():
	static Lock alloc_lock;
	Lock::Guard _lock_guard(alloc_lock);
However, this solution is not perfect, since the lock is released before
the assignment to Thread_base::_context, which leaves a gap of a few
machine instructions, where a preemptive scheduling still could trigger
the exception. Practically it worked for my case.

During analyzing this problem I began wondering about the overall design
of the thread stack allocation. Obtaining the stack area is done by the
call env_context_area_rm_session()->attach_at(ds_cap, attach_addr,
ds_size) within the method Thread_base::_alloc_context(). The parameter
attach_addr is not the complete stack address base (for instance
0x400f.c000), but the offset to the PD's stack area base address (for
instance 0xf.c000). On the other hand the function
env_context_area_rm_session() instantiates a PD_wide RM session which
attaches the whole address area of 256 MB at 0x4000.0000 for the PD.
Doesn't that instruct the pager to provide memory on a page fault of any
address between 0x4000.0000 and 0x4fff.ffff, this way making the system
unable to detect any stack over- and underflow? Additionally memory
mappings are created for the stack offset addresses which are not really
used.
Maybe I missed something. If so, please let me know.

Regards
Frank

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.genode.org/pipermail/users/attachments/20101025/cb4f9717/attachment.html>