Differences between revisions 20 and 21
| Deletions are marked like this. | Additions are marked like this. |
| Line 201: | Line 201: |
|
The complete document for building embedded images is the [attachment:Nto_building_embedded.pdf] |
Setting up and Measuring with QNX and L4
This will be a bunch of the source that i was working on with getting reliable IPC times from QNX and L4.
L4
Working with the released version 2.1.1-patch9 available from the okl4 website.
Building for an ARM gumstix
OKL4 is targeted at ARMv5 platforms and so can work straight off the bat with simply specifying the gumstix machine.
Unfortunately FASS doesn't work correctly out of the box on the ARM platform. This is because elfweaver is responsible for allocating memory regions for the application's stack and heap. Currently there is no special case for ARMv5 which requires a special case and so we are allocated a memsection in the Root server's PD which is outside of the 32M boundaries that FASS requires to work, thus in every switch the TLB is flushed and we fault on the stack upon return. This obviously skews the results of IPC.
Without modifying the elfweaver program itself to realize this we can simply re-link the image when complete. The following script works for the pingpong program used: attachment:patch-weaver.sh
This just patches the weaver.xml file to force the program's stack to a000 and heap to b000 within the 32M bounds.
The pingpong programs used were: attachment:l4pingpong.tar.gz which should be stored to iguana/example/
The makefile to build this is:
pingpong:
tools/build.py project=iguana machine=gumstix toolprefix=arm-linux- scheduling_algorithm=strict example=ping,pong
(./patch-weaver.sh && tools/pyelf/elfweaver merge -obuild/images/image.elf build/images/weaver.xml; true)
install:
arm-linux-objcopy -O binary build/images/image.elf image.bin
scp image.bin paulaner:/tftpboot/okl4_pp.bin
called with:
make pingpong install
Remove Unnecessary Server
To remove the event and timer servers from the image (not completely necessary but good to know) apply the following patch: attachment:removeservers.patch by doing:
patch -p1 < removeservers.patch
from the okl4_2.1.1-patch9 directory.
Performance Counters
OKL4 comes with support for reading the ARM PMU counters however you have to enable CONFIG_PERF. The above removeservers.patch has this in it accidentally but should be fairly obvious to remove.
QNX
Working with svn revision 169376 from the public repository.
General Build instructions
QNX is compiled into a stage directory, by default this is defined by the stage when you install momentics but generally not what we want. We override this with the file qconf-override.mk: attachment:qconf-override.mk. This file currently has my local staging directory which you will want to modify for yourself.
To make use of this we have to export it as an environmental variable
export QCONF_OVERRIDE=/home/jamie/qnx/qconf-override.mk
We now use the make system to install into stage. There are three tunable parameters in the make system:
- CPULIST = the cpu's you want to build for we want arm
- VARIANTLIST = the Varieties you require eg, le/be v4/v6 instrumented etc.
- OSLIST = some sort of compatibility if you dont want the neutrino OS which we always do.
not specifying a parameter means that all the available options are built.
The gumstix is v4 (QNX only specifies v4/v6) le, so having set QCONF_OVERRIDE we can build with:
export VARIANTLIST=le v4 make CPULIST=arm OSLIST=nto hinstall make CPULIST=arm OSLIST=nto install
This can be done from the trunk or any subdirectory with a Makefile in the directory. (Which should be all but the microkernel which must be build along with rest of the procnto system).
Once everything is located in the build directory we have to build the image with
mkifs -vvv -r stage/ imagename.bin
where
- v is verbose level
- r sets a higher precedent stage directory so that if the required files aren't found in stage/ it will look to the system default. Thus either don't export QCONF_OVERRIDE in this terminal or if doing this in a makefile (recommended) use
STAGE=stage
TRUNK=trunk
IMAGE_NAME=image.bin
OVERRIDE=`pwd`/qconf-override.mk
OVERRIDE_CONTENT="USE_INSTALL_ROOT=1\nINSTALL_ROOT_nto=`pwd`/$(STAGE)\nVERSION_REL=6.3.0"
ARM_OPTIONS=OSLIST=nto CPULIST=arm VARIANTLIST=le
override:$(OVERRIDE)
$(OVERRIDE):
echo $(OVERRIDE_CONTENT) > $(OVERRIDE)
headers: $(OVERRIDE)
$(MAKE) QCONF_OVERRIDE=$(OVERRIDE) $(ARM_OPTIONS) -C $(TRUNK)/ hinstall
system: $(OVERRIDE)
$(MAKE) QCONF_OVERRIDE=$(OVERRIDE) $(ARM_OPTIONS) -C $(TRUNK)/ install
kernel: $(OVERRIDE)
$(MAKE) QCONF_OVERRIDE=$(OVERRIDE) $(ARM_OPTIONS) -C $(TRUNK)/services/system install
install:
scp $(IMAGE_NAME) paulaner:/tftpboot/qnx.bin
image:
mkifs -vvv -r $(STAGE)/ $(IMAGE_NAME)
as a base.
Adding a Makefile.dnm to a directory skips the directory which is fine as if necessary it will be sourced from the prebuilt momentics sources. Some things simply dont build and the following script worked for me:
#!/bin/sh touch trunk/utils/s/svn/Makefile.dnm touch trunk/utils/s/sed/Makefile.dnm touch trunk/utils/s/sysctl/Makefile.dnm touch trunk/utils/p/pdksh/Makefile.dnm touch trunk/utils/p/patch/Makefile.dnm touch trunk/utils/r/rtc/qnx4/x86/o.16/Makefile.dnm touch trunk/utils/r/rtc/Makefile.dnm touch trunk/utils/r/rcs/Makefile.dnm touch trunk/utils/d/diff/Makefile.dnm touch trunk/utils/c/cvs/Makefile.dnm touch trunk/utils/g/gawk/Makefile.dnm touch trunk/utils/g/gzip/Makefile.dnm touch trunk/utils/m/mkxfs/Makefile.dnm touch trunk/ports/Makefile.dnm
BSP
The BSP provided for the pxa255 does not work with the gumstix. The version used was created by Peter Weber (<pweber AT qnx DOT com>) for use with a pxa270 and so was backward ported by me with not a whole lot of knowledge re: differences between 270 and 255. I think there may be an issue here with not properly defining all the systems interrupts - leading to the network card not working attachment:qnx-gumstix-bsp.tar.gz. The needed packages are the lib_startus the pxa270 startup and the serial driver.
Again the makefile is set to my local system so will have to be modified.
Debugging
QNX is very proud of their debugging capabilities through using a remote GDB and eclipse. Unfortunately this isn't great for those of us without remote access to the device as i never got the interrupts to fire for incoming network packets. On board debugging is virtually non-existent. I believe this is to do with mistakes in the BSP code, which i played with a bit but didn't feel it worth spending too much time on. This was probably a mistake and would have been a much more reliable method of debugging the kernel.
The skyeye simulator won't work directly giving
SKYEYE ARMul_MRC,CANT UndefInstr CPnum is f, instr ee13cf10
under v1.2 specified to be used by NICTA for use with OKL4, and crashes without reason under 1.2.6
Qemu requires a flash image with uboot as well which i couldn't find with a brief look through disy and the internet.
Issues with timing
The cycle count for XScale architectures is priviledged and to keep consistency across its platforms QNX emulates this with a tick count in the scheduler rather than trapping and reading. But it will work for architectures like x86 as cycle count access is unprivileged. On ARM this has a few issues with initializing the PMU and handling the overflow interrupts. This could probably be a user-space driver issue but i implemented it in kernel so as to be able to retrieve a complete 64bit cycle count rather than have overflows in userspace. The following files worked to provide PMU timings for me:
We get around this by adding a new system call as it is simply easier than trying to re-route or fix up an existing one. The instructions for this are detailed here.
This also provides issues for accurate timing using lmbench. QNX does not provide an accurate timer based off a hardware clock for arm by default and while you can modify lmbench to use the cycle count (more effort than you would think) this would require OKL4 to trap first to OKLinux then to kernel to read the cycle count which i think would skew results when using such small timing measurements.
As recently pointed out the way to do this would then be with the RTC. This should have been done however I had forgotten it, the details should be portable as a driver from the OKL4 package.
Issues with IPC measurements
This was really the point of this work and i'm rather annoyed that i didn't get it to work. I cannot see the reasoning behind the excessive times required to do an IPC message in QNX. QNX uses FCSE, which should mean that (unlike OKL4) the system does not work unless things are correctly allocated within the regions. It uses the Address Space ID to determine the relocation.
More about QNX ARM memory management is found here.
Interesting documents
The QNX Benchmark Methodology provides details of the benchmarks that they believe test the system. Of these the yield test was implemented as it wasn't difficult to do and had more relevance than the others.
This document and others refer to a document called the QNX Neutrino Realtime OS: Kernel Benchmark Results. I cannot find this on google, or anywhere in QNX's documents and heard nothing back from the mailing lists when inquiring about it so while i'm sure it exists i dont think it was ever released.
The complete document for building embedded images is the attachment:Nto_building_embedded.pdf
Navigating the source
Brief look at important locations in the source tree:
- lib/c - C library
- particularily things such as kercalls for auto generated traps.
- services/system - Core procnto files
- memmgr - memory related functions
- There is some screwing around with macros here apparently to allow quick swapping of memory management systems. essentially vmm_fault is fault handler and vmm_* are vm functions and pmm_* is physical.
- memmgr/arm/v4/vmm_aspace.c - handles most FCSE switching
- memmgr - memory related functions
- public/arm - arch related headers
- mmu.h - handles tlb and fault co-processor calls.
- ker - microkernel functionality.
- fast_msg.c - IPC code, everything passes through here - no ASM fastpaths.
- arm/kernel.S - Entry point, ARM data abort and interrupt vectors.
- arm/interrupt.c - kernel.S passes IRQ and most exceptions into handler here.
