FIQ debugger, redux.

I found myself (temporarily, thanks Chris!) in the posession of an OLPC XO 1.75 . Very interesting hardware. The Marvell chip isn't something to particularly rave about (compared to the Xoom's dual Cortex-A9s), but at the end of the day it's a completely open platform (hardware and software), that runs an OS based on Fedora. Plus it uses OpenFirmware instead of some half-baked ROM monitor or a UEFI.. Totally cool. The other great thing is that the kernel is a 3.0-based kernel (not mainline, but w/e), which makes this that much more interesting, especially in light of my interests in MMC flash block core and ARM KGDB/KDB support.

Anyway, while playing with it I had the system wedge up in some totally useless state. Given a lack of a FIQ watchdog or debugger code, I decided to port the FIQ debugger over...

Update: Looks like the XO folks found it useful to hunt for some bugs, cool!

Quick update.

I've been very busy recently with my move to Boston, MA, hence the lack of new content. I have now joined the ranks of VMware employees. Hopefully in the next few weeks I'll have the time to blog about all the interesting things I've stumbled upon and done since my last update :-).

I've been setting up my new personal laptop with Arch Linux, and after migrating my .emacs over
took a look what was in my emacs backups directory. Wow! There is some pretty interesting stuff...from end of High School all the way through college and beyond:
- RM16->PM32 start code for my Bochs32 EDK1 TianoCore port.
- A cassette basic loader and cassette-over-serial ROM emulation for running the MS ROM BASIC in Bochs.
- ELF loader development.
- Beginnings of an RO UEFI NTFS driver.
- Various files from my kernel projects, including ppc7xxx support.
- Various PowerPC and OpenFirmware os-dev related files.
- NES emulator UIC SIGGAME project files.
- LaTeXed homework assignments, scholarship papers, Haskell sources, shell, compilers, etc.
- ZINN Is Not Notes notesfiles replacement from IMSA.

This is exactly why I should convert my entire home directory into a git repo :-).

2.6.40-rc1

2.6.40 development is on, and it's nice to see that it will include a number of MMC subsystem patches that I have written. It feels good to make a little difference.

Andrei Warkentin (17):
     mmc: Reliable write support.
     mmc: quirks: Extends card quirks with MMC/SD quirks matching the CID.
     mmc: core: Rename erase_timeout to cmd_timeout_ms.
     mmc: sdhci: R1B command handling + MMC_CAP_ERASE.
     mmc: core: Allow setting CMD timeout for CMD6 (SWITCH).
     mmc: card: block.c cleanup for host claim/release.
     mmc: MMC boot partitions support.
     mmc: quirks: Support for block quirks.
     mmc: quirks: Fix erase/trim for certain SanDisk cards.
     mmc: core: Fix use of uninitialized data in mmc_cmd_app.
     mmc: Ensure hardware partitions don't mess with mmcblk device naming.
     mmc: quirks: Add/remove quirks conditional support.
     mmc: core: Use CMD23 for multiblock transfers when we can.
     mmc: sdhci: Implement MMC_CAP_CMD23 for SDHCI.
     mmc: core: Block CMD23 support for UHS104/SDXC cards.
     mmc: sdhci: Auto-CMD23 support.
     mmc: sdhci: Auto-CMD23 fixes.

Overview, in no particular order:

• General refactoring of MMC block code.
• MMC block device hardware partition support, in particular implementing support for eMMC 4.3 hardware partitions. eSD partitions still not implemented due to lack of documentation in simplified spec.
• Due to the high potential of bricking a device, "unsafe" partitions like the boot partitions are marked read-only by default, and can be modified only after modifying the 'force_ro' sysfs parameter.
• Changes to the per-card quirks mechanism, which extends matching by CID/name/rev, and allows function-specific quirks tables, so block quirks don't have to pollute core/ code.
• Support for Sandisk eMMC TRIM/ERASE brain damage.
• Proper handling of R1B commands (allowing per-command timeouts).
• Support for CMD23-enabled multiblock transfers, which greatly (30% realistic) improves performance of some cards. Some cards have a slight decrease (unexplained by vendor), and so are blacklisted.
• SDHCI implementation of CMD23 support.
• SDHCI Auto-CMD23 support.
• Support for reliable writes. Exposed as REQ_FUA/REQ_META requests, which some people may find inappropriate, but there isn't a better match and I've yet to see better suggestions (more on this in a later post).

Using GDB to debug UEFI.

GDB, the GNU debugger, is a natural choice for debugging TianoCore EDK2 firmware. In the context of playing around with OVMF (the virtual machine EDK2 target) on QEMU or Xen, you can use the QEMU/Xen debugging support.

On QEMU that means passing the '-s' option. For OVMF that command to invoke would be something like -

fjnh84@fjnh84-desktop:~/mine/edk2/OvmfPkg$ ./build.sh -A IA32 qemu -s
Initializing workspace
/home/fjnh84/mine/edk2/BaseTools
Loading previous configuration from $WORKSPACE/Conf/BuildEnv.sh
WORKSPACE: /home/fjnh84/mine/edk2
EDK_TOOLS_PATH: /home/fjnh84/mine/edk2/BaseTools
using prebuilt tools
Running: qemu -L /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/QEMU -hda fat:/home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32 -s
Could not open option rom 'vapic.bin': No such file or directory
pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin"

For Xen it means running the 'gdbsx' tool, something like -

fjnh84@fjnh84-desktop:~/mine/edk2/OvmfPkg$ gdbsx -a domid 32 1234

In either case, then you would attach a debugger with something like -

(gdb) target remote localhost:1234

As per my earlier post you would then reload UEFI symbols, etc.

Then you start running into problems.

(gdb) print gST
$1 = (EFI_SYSTEM_TABLE **) 0

First of all, the values of .bss globals are bogus. If you look at the address of the symbols, it ends up being the unrelocated value relative to the VMA of the symbol file.

Haven't quite root caused the issue (.bss variables don't seem to be adjusted for the actual objfile location), but a simple workaround is to always link with an ld script that forces all sections to go to .text in the output. Watch out for the the COMMON variables too, the existing script doesn't take care of that.

Since I use the GCC44 toolchain, I modified $(EDK_TOOLS_PATH)/Scripts/gcc4.4-ld-script to be like -

SECTIONS
{
 /* . = 0 + SIZEOF_HEADERS; */
 . = 0x280;
 .text ALIGN(0x20) :
 {
   *(.text .stub .text.* .gnu.linkonce.t.*)
   . = ALIGN(0x20);
   *(.rodata .rodata.* .gnu.linkonce.r.*)
   *(.data .data.* .gnu.linkonce.d.*)
   . = ALIGN(0x20);
   *(.bss .bss.* COMMON)
   . = ALIGN(0x20);
   *(.got .got.*)
   . = ALIGN(0x20);
   *(.rela .rela.*)
   . = ALIGN(0x20);
 } =0x90909090
 /DISCARD/ : {
   *(.note.GNU-stack) *(.gnu_debuglink)
   *(.interp)
   *(.dynsym)
   *(.dynstr)
   *(.dynamic)
   *(.hash)
   *(.comment)
 }

And of course fixed my tools_def.txt like this for IA32 -

DEFINE GCC44_IA32_X64_DLINK_FLAGS    = DEF(GCC44_IA32_X64_DLINK_COMMON) --entry \
$(IMAGE_ENTRY_POINT) -u $(IMAGE_ENTRY_POINT) -Map $(DEST_DIR_DEBUG)/$(BASE_NAME).map \
--script=$(EDK_TOOLS_PATH)/Scripts/gcc4.4-ld-script

Additionally, whatever frame you select prior to dumping the address 'gST', you will always get the same value. Problem is, all UEFI code lives in the same address space, but the different drivers are separate programs. GDB expects globals to be unique, and assumes all loaded symbols are part of the same program. When a global is looked up, gdb scans all partial symtabs starting from the first known objfile, which is the first file that was added using 'add-symbol-file', thus it is a first fit type of deal.

I poked around GDB a bit (7.1, a bit old, but w/e), and decided it was reasonably easy to teach GDB the concept of 'symbol file scope', something often referred to as 'module scope' in other debuggers. Ultimately, that meant -

• Looking in objfile containing current scope block before looking at all objfiles.
• Modifying lookup_symtab code to also handle symbol file names (only handled source names before).
• Fixing location completion so auto-complete works correctly for symbol file names.
• Adding a command to dump loaded symbol files.
• Modify above command to also dump modules matching a particular source file name.

Whereas before GDB had file scope, e.g.-

(gdb) print 'BdsConnect.c'::BdsLibConnectAllDriversToAllControllers
$11 = {void (void)} 0x17eb2619 <BdsLibConnectAllDriversToAllControllers>

Now you also get symbol file scope -

(gdb) print 'BdsDxe.dll'::BdsLibConnectAllDriversToAllControllers
$12 = {void (void)} 0x17eb2619 <BdsLibConnectAllDriversToAllControllers>

The 'list-symbol-files' command is useful too -

(gdb) list-symbol-files DiskIo.c Dispatcher.c EhciDxe.dll
DiskIo.c:
       /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32/MdeModulePkg/Universal/Disk/DiskIoDxe/DiskIoDxe/DEBUG/DiskIoDxe.dll
               .text, [0x17e4c240-0x17e4f440)
Dispatcher.c:
       /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
               .text, [0x17fb1240-0x17fcb8c0)
EhciDxe.dll:
       /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32/MdeModulePkg/Bus/Pci/EhciDxe/EhciDxe/DEBUG/EhciDxe.dll
               .text, [0x17deb240-0x17df3920)

Patch is at https://github.com/andreiw/andreiw-wip/blob/master/gdb/0001-GDB-Initial-prototype-of-symbol-file-scope-module-sc.patch.

Works like a charm :-).

(gdb) print *'DxeCore.dll'::gST
$14 = {Hdr = {Signature = 6076298535811760713, Revision = 131102, HeaderSize = 72, CRC32 = 2051810456, Reserved = 0}, FirmwareVendor = 0x17f7f990,
  FirmwareRevision = 65536, ConsoleInHandle = 0x17c06d90, ConIn = 0x17e67424, ConsoleOutHandle = 0x17c06790, ConOut = 0x17e6752c, StandardErrorHandle = 0x17191690,
  StdErr = 0x17e675cc, RuntimeServices = 0x17f7ff10, BootServices = 0x17fc9ae8, NumberOfTableEntries = 6, ConfigurationTable = 0x17f7ed90}
(gdb) print &'DxeCore.dll'::gST
$15 = (EFI_SYSTEM_TABLE **) 0x17fca6f0
(gdb) print &'IsaBusDxe.dll'::gST
$16 = (EFI_SYSTEM_TABLE **) 0x17e21da4

Something interesting...

http://www.somethinginteresting.org/content/alviro.html

GDB scripting example: reloading symbols for UEFI target

As part of helping the GSoC student with debugging, I got tired of manually groking output and loading module symbols in GDB, because it's a long and painful process. I knew GDB had scripting facilities, so I figured I'd use them. Never in my wildest dreams did I expect Python support (whoo, whoo!).

To be honest, I fully expect every serious place working with UEFI and using GDB already has something like this. But no one has been sharing...and I really don't mind. My Python is not too 1337, but I'm pretty happy with the end result.

GSoC

Here is the latest from Xen UEFI land:

• We can boot to UEFI Shell, finally. The hurdles were -
• Different PCI I/O range compared to Qemu, this resulted in a domain crash when an emulated I/O device tried to register for already registered ports. Boo.
• ACPI timer block hardcoded by Xen at I/O 0xB000. Yes, modifying the "special BAR" has no effect, so UEFI TimerLib code needs to be careful in configuring the timer... if the PCI registers already look configured - use selected values. They are now set once in hvmloader.
• 8259 PIC ExtInt routing needed to be enabled in APIC inside hvmloader, because TianoCore has no APIC support.
• Started making OvmfPkg Xen aware. This was necessarily for at least registering the right I/O ranges. Now, the hypercall pages are populated and a GUIDed HOB containing Xen hypervisor info is published. This will be consumed by the hypervisor DXE.

I see the overall layering as being something like -

• Hypervisor driver - consumes HOB, publishes the EFI_XEN_HYPERCALL_PROTOCOL.
• XenTables driver - exposes Xen ACPI and SMBIOS tables, which were built into hvmloader.
• XenBus driver - consumes Hypervisor driver, creates child nodes for virtual I/O devices, publishes EFI_XEN_BUS_PROTOCOL.
• XenStore driver - exercises XenBus interface, allows SIMPLE_FILE_IO_PROTOCOL access to XenStore nodes.
• Blockfront, Netfront, Fbfront drivers for block, network and video, respectively.

Latest patches as usual at https://github.com/andreiw/andreiw-wip/tree/master/xen/ovmf-support.

Making slides

Two awesome ways of creating view-anywhere slide decks -

• http://meyerweb.com/eric/tools/s5/
• https://github.com/Seldaek/slippy

MSAlumni

Finally registered myself with the Microsoft Alumni Network. It's nice to have the corporate store discounts back :-).

UEFI OVMF/Xen

For anyone interested in the GSoC project status:
- We can boot the 32-bit and the 64-bit OVMF image now inside an HVM domain instead of BIOS
- We end up crashing the HVM domain during PCI bus enumeration (qemu-dm crash?), which is what the student is now investigating...

Xen changes so far included refactoring changes inside hvmloader, adding a new bios_config structure for OVMF images, ensuring that guest physical memory under 4GB is backed with pages, as well as moving out Xen structures from below 4GB where they would collide with the firmware. The hvmloader/bios xenstore key was exposed to xend/xm (hvmbios parameter, values can be rombios, ovmf32 and ovmf64), to faciliate easier testing. Some thought still needs to be put into libxl (xend replacement) changes.
In general OVMF/Xen patches for Xen and TianoCore will be at https://github.com/andreiw/andreiw-wip/tree/master/xen/ovmf-support

Schemix

As part of what I get paid for, I frequently need to tinker with Linux kernel internals or expose them out as part of some prototyping work... Occasionally, the typical cycle of code-compile-reboot-flash-boot-test is a bit too tedious so I was looking for something that would let me interactively poke at the kernel without an external debugger and do little experiments...

http://www.abstractnonsense.com/schemix/

Hopefully it's not too decrepit :). Well, it's for 2.4 :(, but it seems small enough that porting to 2.6 shouldn't too much of a PITA.

LFSR taps

I had to fix up the LFSR code in my block benching tool, and this Xilinx document was very useful for figuring out the primitive polynomials for bits > 16.

http://www.xilinx.com/support/documentation/application_notes/xapp052.pdf

Ended up reading about Galois Fields and Évariste Galois...humbling...humbling...

Mercurial

As part of some Xen stuff I'm doing I have to interact with Mercurial, the SCM used. I'm pretty used to the git work flow, so I was wondering how to do the equivalent of patch squashing.

You need the mq extension.

The equivalent of -

$ git rebase -i HEAD~2

In Hg would look like -

$ hg qimport -g -r -2:-1
$ hg qpop
now at: 2146.diff
$ hg qfold 2147.diff
$ hg qrefresh -m 'new log message'
$ hg qfinish --all

The '-g' flag is a must if you have any binary diffs.

Of course this doesn't help you with reordering commits... :/

GSoC

Today Jordan Justen of the EDK2 TianoCore project announced that four students will be working on UEFI EDK2 projects. I am very excited to be mentoring the OVMF on Xen project, which will enable UEFI firmware for HVM Xen virtual machines.

With luck and lots and lots of work, by the end of this mentorship we will be able to boot into Linux using Xen PV devices (blkback...).

This presents a wonderful opportunity to share my Xen and EDK experiences and help out two projects I love :-).

Haskell and Osdev.

There is a Haskell-written OS! Can't wait to play with this...

House - http://programatica.cs.pdx.edu/House/

Another interesting angle are Linux LKMs in Haskell. This is pretty proof-of-concept... If I ever have time I'll see if I can play with this. Hmmm. I wonder if with new version of GHC this isn't limited to IA-32 anymore.

http://tommd.wordpress.com/2009/09/13/kernel-modules-in-haskell/
http://lwn.net/Articles/352432/

Bringing up UEFI on Linux-ARM devices...

Introduction.

Given that TianoCore is open source and now supporting the ARM architecture, it may be a given that someone may wish to play around with it on their device of choice. Most likely, it's an Android phone, or perhaps some other Linux-using device. That also might imply that you don't have access to boot-loader sources, or the bootloader is "secure", or you simply don't wish to have the possibility of bricking the device totally... It almost likely implies that you might know which SOC chipset is inside, but have no idea for all the power regulators, display controllers, and other logic in the device, so nuking the firmware is definitively not the right option. Better to keep it around so it initializes all the hardware enough that you can interact with it purely by manipulating SoC devices...

So what do you do?

Writing data correctly...

Shamelessly lifting these links from the Android blog. It's just that important.

• http://lwn.net/Articles/322823/
• http://blogs.gnome.org/alexl/2009/03/16/ext4-vs-fsync-my-take/
• http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/
• http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/

Flourish

Looking forward to the Flourish conference this year at my alma mater, the University of Illinois at Chicago. Flourish is an Open Source Conference initially started by one of my friends, Roberto, who has since moved back to Spain. Although, there is definite room to grow to reach the level of Reflections/Projections, there are some interesting speakers.

The Scala talk could be interesting, and I'd attend Icculus's talk... I wish I'd remember about Flourish earlier, maybe I would have presented something interesting as well...

FIQ debugger KGDB support.

ARM-based Android devices in their development life cycle often have a kernel compiled with FIQ debugger support. FIQ debugger allows a developer to gather some data on a hung system via one of the UARTs, even if the system is wedged inside an interrupt handler, by using a FIQ handler.

The FIQ debugger owns the UART port, and allows enabling a console on the port as well (like here https://www.codeaurora.org/gitweb/quic/le/?p=kernel/msm.git;a=commitdiff;h=df5ac72e83df9fd8d735d40d5be464d4b66d6074). A break exits back to the debugger prompt, and a developer can get a backtrace, or run a sysrq action. It would be great, of course, to make this cooperate with KGDB/KDB...

MMC block quirks.

So I'm secretly working on some block MMC quirks for some eMMC devices. It's not that big of a secret, but I'll keep the specifics to myself until I have collected enough data to post to linux-mmc. I already posted there the conclusions I reached based on other people's partial data, but that tends to be insufficient when it comes to convincing others of taking weird-looking code...

Anyway, the point is that occasionally you might wish the generic MMC code to act differently for various devices, whether as a workaround for not complying to the spec, or as a workaround for some smart proprietary intellectual property running inside these devices. The linux-next tree contains a quirks layer by Pierre Tardy, with a slight slant towards SDIO devices, as it was added to resolve some SDIO issue. Here is a patch against that tree that extends the quirk support to MMC/SD devices, allowing matching on CID fields. If any luck has it, it will be merged eventually. It would probably help if I hurried on the MMC block quirks actually using this ;-)... Data collection and analysis is a pain.

Measuring SQLite performance on MMC

I needed (still need to, gah) collect data on how a particular change to the Linux kernel MMC block layer ultimately affects write performance in SQLite. Android tends to use SQLite databases quite a bit, so a simple test on how long it takes to insert some large number of records is probably fair.

I exposed a couple of tunables through debugfs and started collecting data.

Of course, the first set of data was collected using specific synthetic benchmarks that did various kinds of O_SYNC+O_DIRECT I/O to the MMC, but it's nice to be able to see the real-life effect of a particular change, especially in combination with file systems. The problem is that you can get data, but that data is not repeatable within the same configuration. That makes comparing data across different configurations tedious at best.

Some things are pretty obvious - ensuring all mounted file systems are read-only. Ensuring there are no other processes running, etc. Other items don't come immediately to mind (to me at least :-()...

For example, running the same 5000 insertions x 50 times about 10 times on the same file system yields progressively slower data.

Current steps to ensure more-or-less repeatable data gathering:
0) Turn off CPU frequency scaling, RAM frequency scaling, etc, all
    power savings off.
1) Unmount partition containing files on which the SQLite test operates.
2) Perform BLKDISCARD over the partition.
3) Format with desired file system.
4) Mount.
6) Sync && echo 3 > /proc/sys/vm/drop_caches
7) Perform test.
8) Umount.
9) Repeat from (1)

Anything else I could have missed?

Algorithms and Data Structures.

I haven't posted in a while. Unfortunately, not because I hadn't had anything interesting to write lately about, but because of too much interesting stuff happening :-).

In no particular order, some interesting algorithmic links...

1. Bit twiddling hacks at http://graphics.stanford.edu/~seander/bithacks.html
2.  Judy Arrays consumed my imagination, hopefully I'll get to play with them soon.
• http://judy.sourceforge.net/
• http://judy.sourceforge.net/doc/10minutes.htm
• http://wiki.tcl.tk/10096
• http://rusty.ozlabs.org/?p=153
• http://nothings.org/computer/judy/