Thanks for the explanation of virtualized memory, Stefan. I had it in my head there was one page table, not one per pd.
Bob
Sent from my android device.
-----Original Message----- From: Stefan Kalkowski <stefan.kalkowski@...1...> To: Genode OS Framework Mailing List genode-main@lists.sourceforge.net Sent: Mon, 08 Sep 2014 3:14 AM Subject: Re: Translation Table Memory Attribute bits
Hi Bob,
On 09/07/2014 07:09 PM, Bob Stewart wrote:
Stefan, I believe I see where the issue is and it relates to another change I need to make to released kernel/core code to make Genode run on an Am335x-based Beaglebone board.
Here are all the changes I need to make to a release since 13.08:
- The Beaglebone board that I am using uses the TL16C750 UART chip. The
Pandaboard also uses this chip and code exists in base-hw to provide serial line support. This code is hard-coded to use uart3 as the console port which is fine for the Pandaboard. The Beagleboard used uart0, so I alter serial.h accordingly.
- As I've mentioned in this email thread, without modify any of the
translation table memory access bits, the kernel will never initialize. Once the MMU is turned on no further output or apparent cpu activity occurs. This is fixed if the access bits for device memory is set to Tex[0:2] = 0, C = 0, and B =1. As you will notice from the TI code snippet below, device memory is shared and these bits setting are in agreement with the Arm v7 spec as defined in table B3-10. I also change the access control bits to make cacheability agree with that used by the TI code. After those changes everything works fine and the kernel initializes and user programs run ok, until release 14.08
- The Am335x requires a kernel privilege level to write to to sets of
registers that are critical for any embedded application. The first set of registers are in the "Control Module" where register settings enable/disable subsystems and set up various functional clocks. The second set of registers are the pin-mux registers through which the operating modes available and gpio pins can be set. These registers cannot be written to from userland. So I created two kernel calls to handle writing to each of these register sets. That worked fine, until 14.08.
So back to the current issue. Currently the kernel initializes and init starts and the various modules specified in the run script get executed in threads. All the modules start up, but shortly after starting the whole system appears to freeze. When I saw the MMU exceptions from my debugging and they mostly related to the section entry for address 0x81000000, I assumed it was an issue with the bits in the translation table entry. I was also confused by the output which shows a series of MMU exceptions in sequence on that same section table entry. I still don't understand why this occurs when the currently used translation table entry is still valid. For example, the following debug output sequence seems to me to indicated that each thread causes a MMU fault even though the section entry in the translation table should still be valid: void Kernel::Thread::_mmu_exception(): f_addr 0x81000000 f_writes 0x0 f_pd 0x8143d088 f_signal 0x86 label timer void Kernel::Thread::_mmu_exception(): f_addr 0x81000000 f_writes 0x0 f_pd 0x81474088 f_signal 0x88 label gpio_drv void Kernel::Thread::_mmu_exception(): f_addr 0x81000000 f_writes 0x0 f_pd 0x814a9088 f_signal 0x8a label ctl_module_drv void Kernel::Thread::_mmu_exception(): f_addr 0x81000000 f_writes 0x0 f_pd 0x814e0088 f_signal 0x8c label bbb_heart_beat_led void Kernel::Thread::_mmu_exception(): f_addr 0x81043b40 f_writes 0x1 f_pd 0x8143d088 f_signal 0x86 label timer void Kernel::Thread::_mmu_exception(): f_addr 0x8104ab50 f_writes 0x1 f_pd 0x81474088 f_signal 0x88 label gpio_drv void Kernel::Thread::_mmu_exception(): f_addr 0x81049310 f_writes 0x1 f_pd 0x814a9088 f_signal 0x8a label ctl_module_drv void Kernel::Thread::_mmu_exception(): f_addr 0x8103ada0 f_writes 0x1 f_pd 0x814e0088 f_signal 0x8c label bbb_heart_beat_led What is it that I'm missing?
These are different threads of different programs. Each program is a distinct protection domain with it's own page-table set. When a page-fault is resolved e.g. in the timer at address 0x81000000, that doesn't mean it is resolved in the GPIO driver (gpio_drv) for that address. In the above example each program's binary code is collocated to the virtual address 0x81000000. When booting a scenario the binaries are lying somewhere in physical memory, for example in your case init was located at 0x8113b000 (the ROM FS dump). When the init task gets constructed a mapping between 0x8113b000 (physical) and 0x81000000 (virtual) is assigned. When the first thread of the init program is executed, its instruction pointer is set to 0x81000000, and it'll immediately fault. This is the fault you're observing. It gets resolved by core by inserting an appropriated entry in the page-tables of the init process, this might be a 1MB section entry, or a 4KB page entry, but this depends on the size and alignment of the corresponding physical and virtual area. When another program is started the same procedure recurs. Of course, it'll be another physical address, where the binary address is located, but the same virtual one.
The MMU faults you're observing are ordinary.
The real problem is being caused due to the change to the released code (for 14.05 or 14.08?) for kernel calls which are now split into two tables, for kernel calls that anyone can make, and kernel calls that only core can make. I made what I thought were the appropriate changes to allow my two kernel calls to execute from userland, but, the Control Module register writes appear to be hanging the system. I'm currently investigating the reason for the hang.
You can of course allow every program to access the appropriated registers via the kernel to test its functionality. When things are working, I would nevertheless restrict these calls to core, and provide an appropriated service in core, so you can restrict the access to these registers to drivers that need to touch them via init's configuratiion scripts (access policy).
Good luck & Regards Stefan
Thanks, Bob
On 09/05/2014 01:33 PM, Bob Stewart wrote:
Hi Stefan, The TI Sitara Family of processors Starterware contains code setting up the translation tables... here's the relevant snippet--
/* ** Function to setup MMU. This function Maps three regions ( 1. DDR ** 2. OCMC and 3. Device memory) and enables MMU. */ void MMUConfigAndEnable(void) { /* ** Define DDR memory region of AM335x. DDR can be configured as Normal ** memory with R/W access in user/privileged modes. The cache attributes ** specified here are, ** Inner - Write through, No Write Allocate ** Outer - Write Back, Write Allocate */ REGION regionDdr = { MMU_PGTYPE_SECTION, START_ADDR_DDR, NUM_SECTIONS_DDR, MMU_MEMTYPE_NORMAL_NON_SHAREABLE(MMU_CACHE_WT_NOWA, MMU_CACHE_WB_WA), MMU_REGION_NON_SECURE, MMU_AP_PRV_RW_USR_RW, (unsigned int*)pageTable }; /* ** Define OCMC RAM region of AM335x. Same Attributes of DDR region given. */ REGION regionOcmc = { MMU_PGTYPE_SECTION, START_ADDR_OCMC, NUM_SECTIONS_OCMC, MMU_MEMTYPE_NORMAL_NON_SHAREABLE(MMU_CACHE_WT_NOWA, MMU_CACHE_WB_WA), MMU_REGION_NON_SECURE, MMU_AP_PRV_RW_USR_RW, (unsigned int*)pageTable };
/* ** Define Device Memory Region. The region between OCMC and DDR is ** configured as device memory, with R/W access in user/privileged modes. ** Also, the region is marked 'Execute Never'. */ REGION regionDev = { MMU_PGTYPE_SECTION, START_ADDR_DEV, NUM_SECTIONS_DEV, MMU_MEMTYPE_DEVICE_SHAREABLE, MMU_REGION_NON_SECURE, MMU_AP_PRV_RW_USR_RW | MMU_SECTION_EXEC_NEVER, (unsigned int*)pageTable };
I had to change the memory attributes for device memory to get the kernel to complete initialization. If B is not set the kernel silently page faults after the mmu is turned on.
The .text address of 0x81000000 come from readelf.
Yes I added a printk to get at the mmu exception.
I can see one page fault but it appears to be a page fault on every access to the section entry for 0x81000000.
Bob
On 09/05/2014 08:30 AM, Stefan Kalkowski wrote:
Hi Bob,
On 09/05/2014 01:43 PM, Bob Stewart wrote:
Thanks for the reply Stefan.
For the AM335x processors, the cacheability is required to be set to:
Inner: Write thru, no write allocate Outer: Write back, write allocate.
Are you sure? Can you point me to the relevant information if it's openly available? In the Cortex A8 and AM335x TRM I couldn't find that restriction. I would assume that you don't have to change any memory attributes, as we support Cortex A8 already (although this was error-prone in the past).
So, in 14 .02 I set the Tex, C, and B bits accordingly and that worked fine. I just transferred those setting to 14.08.
With the rework in that tlb area of the kernel for multi-processor support in 14.05 and 14.08, I assumed I was screwing something up in the translation table entry attribute bits.
According to the ROM fs dump "Rom: [8113b000,8117aa24) init". it is the .text section of the program image that starts at 0x81000000.
Don't mix things up: the ROM fs dump gives you the physical memory where the boot modules reside in. The program image layout can be seen by e.g.: objdump, or readelf. In this example they are coincidentally nearby.
After re-reading your log output (I assume you've added the "void Kernel::Thread::_mmu_exception()" printings?) it seems to me like the init process produces a first page-fault, when executing its first instruction, which is normal, and afterwards continues to run - so the pagefault gets actually resolved, right? After that it gets additional page-faults corresponding to its program flow. So what is the actual weird behaviour? I can't see it given your output.
Regards Stefan
I'll dump the contents of the DFSR and see what that tells me later today. I'll also try run/printf as the program image, but the program image I'm using runs fine when built on 14.02.
Thanks for the suggestion.
Bob
On 09/05/2014 03:36 AM, Stefan Kalkowski wrote:
Hi Bob,
On 09/04/2014 03:33 PM, Bob Stewart wrote:
I've never been able to get 14.05 or 14.08 working on my AM335X processor, which is not a big deal as 14.02 has everything I need for the applications I'm using that processor for. But out of curiosity:
The issue on 14.08 may revolve around memory access bit rights in TLB table entries. To get 14.08 to initialize the kernel properly the memory access bits have to be set as Tex = 0, B = 1, and C = 0. These setting seem correct for a shared Device according to the Arm v7 ref manual. With those settings in place, the kernel initializes properly and eventually init runs. During the kernel initialization process Core_pd is called and translations are created for the program image (which starts at 0x81000000) and the core-only io memory regions. The translation table entries for the program image are of section size plus a small page so two entries are inserted in the translation table. The access bits and permission bits for the section entry are correct with the possible exception of the C bit, which in 14.08 appears never to be set and I wondered why that was, when it is used in 14.02.
We reworked a lot regarding ARM caches, shareability etc. within the last months. Nowadays (release 14.08), on all Arm v7 platforms, including Cortex A8, we set the following memory region attributes for normal memory (!not device memory): Tex=0b101, C=0, B=1 That means: normal, inner- and outer-cacheable memory, with write-back,write-allocate caching policy. Which works properly on all our Cortex A8, Cortex A9, and Cortex A15 platforms.
Once the init thread runs, any reference to the translation table entry for the program image, the section entry mentioned above cause a mmu exception as the following partial debug output shows: ... start thread 3 'entrypoint' in program 1 'core' on processor 0/1 start thread 4 'signal' in program 1 'core' on processor 0/1 start thread 5 'pager_activation' in program 1 'core' on processor 0/1 int main(): --- start init --- int main(): transferred 507 MB to init start thread 6 'init' in program 1 'core' on processor 0/1 thread id is 0x7 start thread 7 'init' in program 2 'init' on processor 0/1 void Kernel::Thread::_mmu_exception(): f_addr 0x81000000 f_writes 0x0 f_pd 0x813ed088 f_signal 0x7f label init int main(): --- init created, waiting for exit condition --- void Kernel::Thread::_mmu_exception(): f_addr 0x81045f60 f_writes 0x1 f_pd 0x813ed088 f_signal 0x7f label init void Kernel::Thread::_mmu_exception(): f_addr 0x8102dab8 f_writes 0x0 f_pd 0x813ed088 f_signal 0x7f label init ...
Setting the C bit as it was set in 14.02 makes no difference, which I thought it would and it should be affecting caching behavior.
Any thoughts on this behavoir?
Before thinking about a caching issue, I would investigate whether there is no other issue. Above output shows that the whole kernel/core are fully initialized and the init process is started. When the init process tries to do some "write" operations it fails, right? So there is no problem with the core's translation tables (Core_pd) at all.
First of all, you should identify which kind of MMU exception was triggered by the init process. Therefore, print out the DFSR (data fault status register) directly after the corresponding faults occur. Does the init binary also start at 0x81000000?
Regards Stefan
Thanks, Bob
Slashdot TV. Video for Nerds. Stuff that matters. http://tv.slashdot.org/ _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Slashdot TV. Video for Nerds. Stuff that matters. http://tv.slashdot.org/ _______________________________________________ genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main
Slashdot TV. Video for Nerds. Stuff that matters. http://tv.slashdot.org/
genode-main mailing list genode-main@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/genode-main