Thursday, May 26, 2011

2.6.40-rc1

2.6.40 development is on, and it's nice to see that it will include a number of MMC subsystem patches that I have written. It feels good to make a little difference.
Andrei Warkentin (17):
     mmc: Reliable write support.
     mmc: quirks: Extends card quirks with MMC/SD quirks matching the CID.
     mmc: core: Rename erase_timeout to cmd_timeout_ms.
     mmc: sdhci: R1B command handling + MMC_CAP_ERASE.
     mmc: core: Allow setting CMD timeout for CMD6 (SWITCH).
     mmc: card: block.c cleanup for host claim/release.
     mmc: MMC boot partitions support.
     mmc: quirks: Support for block quirks.
     mmc: quirks: Fix erase/trim for certain SanDisk cards.
     mmc: core: Fix use of uninitialized data in mmc_cmd_app.
     mmc: Ensure hardware partitions don't mess with mmcblk device naming.
     mmc: quirks: Add/remove quirks conditional support.
     mmc: core: Use CMD23 for multiblock transfers when we can.
     mmc: sdhci: Implement MMC_CAP_CMD23 for SDHCI.
     mmc: core: Block CMD23 support for UHS104/SDXC cards.
     mmc: sdhci: Auto-CMD23 support.
     mmc: sdhci: Auto-CMD23 fixes.
Overview, in no particular order:
  • General refactoring of MMC block code.
  • MMC block device hardware partition support, in particular implementing support for eMMC 4.3 hardware partitions. eSD partitions still not implemented due to lack of documentation in simplified spec.
  • Due to the high potential of bricking a device, "unsafe" partitions like the boot partitions are marked read-only by default, and can be modified only after modifying the 'force_ro' sysfs parameter.
  • Changes to the per-card quirks mechanism, which extends matching by CID/name/rev, and allows function-specific quirks tables, so block quirks don't have to pollute core/ code.
  • Support for Sandisk eMMC TRIM/ERASE brain damage.
  • Proper handling of R1B commands (allowing per-command timeouts).
  • Support for CMD23-enabled multiblock transfers, which greatly (30% realistic) improves performance of some cards. Some cards have a slight decrease (unexplained by vendor), and so are blacklisted.
  • SDHCI implementation of CMD23 support.
  • SDHCI Auto-CMD23 support.
  • Support for reliable writes. Exposed as REQ_FUA/REQ_META requests, which some people may find inappropriate, but there isn't a better match and I've yet to see better suggestions (more on this in a later post).

Wednesday, May 25, 2011

Using GDB to debug UEFI.

GDB, the GNU debugger, is a natural choice for debugging TianoCore EDK2 firmware. In the context of playing around with OVMF (the virtual machine EDK2 target) on QEMU or Xen, you can use the QEMU/Xen debugging support.

On QEMU that means passing the '-s' option. For OVMF that command to invoke would be something like -
fjnh84@fjnh84-desktop:~/mine/edk2/OvmfPkg$ ./build.sh -A IA32 qemu -s
Initializing workspace
/home/fjnh84/mine/edk2/BaseTools
Loading previous configuration from $WORKSPACE/Conf/BuildEnv.sh
WORKSPACE: /home/fjnh84/mine/edk2
EDK_TOOLS_PATH: /home/fjnh84/mine/edk2/BaseTools
using prebuilt tools
Running: qemu -L /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/QEMU -hda fat:/home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32 -s
Could not open option rom 'vapic.bin': No such file or directory
pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin"
For Xen it means running the 'gdbsx' tool, something like -
fjnh84@fjnh84-desktop:~/mine/edk2/OvmfPkg$ gdbsx -a domid 32 1234
In either case, then you would attach a debugger with something like -
(gdb) target remote localhost:1234
As per my earlier post you would then reload UEFI symbols, etc.

Then you start running into problems.
(gdb) print gST
$1 = (EFI_SYSTEM_TABLE **) 0

First of all, the values of .bss globals are bogus. If you look at the address of the symbols, it ends up being the unrelocated value relative to the VMA of the symbol file.

Haven't quite root caused the issue (.bss variables don't seem to be adjusted for the actual objfile location), but a simple workaround is to always link with an ld script that forces all sections to go to .text in the output. Watch out for the the COMMON variables too, the existing script doesn't take care of that.

Since I use the GCC44 toolchain, I modified $(EDK_TOOLS_PATH)/Scripts/gcc4.4-ld-script to be like -
SECTIONS
{
 /* . = 0 + SIZEOF_HEADERS; */
 . = 0x280;
 .text ALIGN(0x20) :
 {
   *(.text .stub .text.* .gnu.linkonce.t.*)
   . = ALIGN(0x20);
   *(.rodata .rodata.* .gnu.linkonce.r.*)
   *(.data .data.* .gnu.linkonce.d.*)
   . = ALIGN(0x20);
   *(.bss .bss.* COMMON)
   . = ALIGN(0x20);
   *(.got .got.*)
   . = ALIGN(0x20);
   *(.rela .rela.*)
   . = ALIGN(0x20);
 } =0x90909090
 /DISCARD/ : {
   *(.note.GNU-stack) *(.gnu_debuglink)
   *(.interp)
   *(.dynsym)
   *(.dynstr)
   *(.dynamic)
   *(.hash)
   *(.comment)
 }

And of course fixed my tools_def.txt like this for IA32 -
DEFINE GCC44_IA32_X64_DLINK_FLAGS    = DEF(GCC44_IA32_X64_DLINK_COMMON) --entry \
$(IMAGE_ENTRY_POINT) -u $(IMAGE_ENTRY_POINT) -Map $(DEST_DIR_DEBUG)/$(BASE_NAME).map \
--script=$(EDK_TOOLS_PATH)/Scripts/gcc4.4-ld-script

Additionally, whatever frame you select prior to dumping the address 'gST', you will always get the same value. Problem is, all UEFI code lives in the same address space, but the different drivers are separate programs. GDB expects globals to be unique, and assumes all loaded symbols are part of the same program. When a global is looked up, gdb scans all partial symtabs starting from the first known objfile, which is the first file that was added using 'add-symbol-file', thus it is a first fit type of deal.

I poked around GDB a bit (7.1, a bit old, but w/e), and decided it was reasonably easy to teach GDB the concept of 'symbol file scope', something often referred to as 'module scope' in other debuggers. Ultimately, that meant -
  • Looking in objfile containing current scope block before looking at all objfiles.
  • Modifying lookup_symtab code to also handle symbol file names (only handled source names before).
  • Fixing location completion so auto-complete works correctly for symbol file names.
  • Adding a command to dump loaded symbol files.
  • Modify above command to also dump modules matching a particular source file name.
Whereas before GDB had file scope, e.g.-
(gdb) print 'BdsConnect.c'::BdsLibConnectAllDriversToAllControllers
$11 = {void (void)} 0x17eb2619 <BdsLibConnectAllDriversToAllControllers>
Now you also get symbol file scope -
(gdb) print 'BdsDxe.dll'::BdsLibConnectAllDriversToAllControllers
$12 = {void (void)} 0x17eb2619 <BdsLibConnectAllDriversToAllControllers>
The 'list-symbol-files' command is useful too -
(gdb) list-symbol-files DiskIo.c Dispatcher.c EhciDxe.dll
DiskIo.c:
       /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32/MdeModulePkg/Universal/Disk/DiskIoDxe/DiskIoDxe/DEBUG/DiskIoDxe.dll
               .text, [0x17e4c240-0x17e4f440)
Dispatcher.c:
       /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
               .text, [0x17fb1240-0x17fcb8c0)
EhciDxe.dll:
       /home/fjnh84/mine/edk2/Build/OvmfIa32/DEBUG_GCC44/IA32/MdeModulePkg/Bus/Pci/EhciDxe/EhciDxe/DEBUG/EhciDxe.dll
               .text, [0x17deb240-0x17df3920)

Patch is at https://github.com/andreiw/andreiw-wip/blob/master/gdb/0001-GDB-Initial-prototype-of-symbol-file-scope-module-sc.patch.

Works like a charm :-).
(gdb) print *'DxeCore.dll'::gST
$14 = {Hdr = {Signature = 6076298535811760713, Revision = 131102, HeaderSize = 72, CRC32 = 2051810456, Reserved = 0}, FirmwareVendor = 0x17f7f990,
  FirmwareRevision = 65536, ConsoleInHandle = 0x17c06d90, ConIn = 0x17e67424, ConsoleOutHandle = 0x17c06790, ConOut = 0x17e6752c, StandardErrorHandle = 0x17191690,
  StdErr = 0x17e675cc, RuntimeServices = 0x17f7ff10, BootServices = 0x17fc9ae8, NumberOfTableEntries = 6, ConfigurationTable = 0x17f7ed90}
(gdb) print &'DxeCore.dll'::gST
$15 = (EFI_SYSTEM_TABLE **) 0x17fca6f0
(gdb) print &'IsaBusDxe.dll'::gST
$16 = (EFI_SYSTEM_TABLE **) 0x17e21da4

Monday, May 16, 2011

GDB scripting example: reloading symbols for UEFI target

As part of helping the GSoC student with debugging, I got tired of manually groking output and loading module symbols in GDB, because it's a long and painful process. I knew GDB had scripting facilities, so I figured I'd use them. Never in my wildest dreams did I expect Python support (whoo, whoo!).

To be honest, I fully expect every serious place working with UEFI and using GDB already has something like this. But no one has been sharing...and I really don't mind. My Python is not too 1337, but I'm pretty happy with the end result.

GSoC

Here is the latest from Xen UEFI land:
  • We can boot to UEFI Shell, finally. The hurdles were -
    • Different PCI I/O range compared to Qemu, this resulted in a domain crash when an emulated I/O device tried to register for already registered ports. Boo.
    • ACPI timer block hardcoded by Xen at I/O 0xB000. Yes, modifying the "special BAR" has no effect, so UEFI TimerLib code needs to be careful in configuring the timer... if the PCI registers already look configured - use selected values. They are now set once in hvmloader.
    • 8259 PIC ExtInt routing needed to be enabled in APIC inside hvmloader, because TianoCore has no APIC support.
  • Started making OvmfPkg Xen aware. This was necessarily for at least registering the right I/O ranges. Now, the hypercall pages are populated and a GUIDed HOB containing Xen hypervisor info is published. This will be consumed by the hypervisor DXE.
I see the overall layering as being something like -
  • Hypervisor driver - consumes HOB, publishes the EFI_XEN_HYPERCALL_PROTOCOL.
  • XenTables driver - exposes Xen ACPI and SMBIOS tables, which were built into hvmloader.
  • XenBus driver - consumes Hypervisor driver, creates child nodes for virtual I/O devices, publishes EFI_XEN_BUS_PROTOCOL.
  • XenStore driver - exercises XenBus interface, allows SIMPLE_FILE_IO_PROTOCOL access to XenStore nodes.
  • Blockfront, Netfront, Fbfront drivers for block, network and video, respectively.
Latest patches as usual at https://github.com/andreiw/andreiw-wip/tree/master/xen/ovmf-support.

Friday, May 13, 2011

Making slides

Two awesome ways of creating view-anywhere slide decks -

Friday, May 6, 2011

MSAlumni

Finally registered myself with the Microsoft Alumni Network. It's nice to have the corporate store discounts back :-).

UEFI OVMF/Xen

For anyone interested in the GSoC project status:
- We can boot the 32-bit and the 64-bit OVMF image now inside an HVM domain instead of BIOS
- We end up crashing the HVM domain during PCI bus enumeration (qemu-dm crash?), which is what the student is now investigating...

Xen changes so far included refactoring changes inside hvmloader, adding a new bios_config structure for OVMF images, ensuring that guest physical memory under 4GB is backed with pages, as well as moving out Xen structures from below 4GB where they would collide with the firmware. The hvmloader/bios xenstore key was exposed to xend/xm (hvmbios parameter, values can be rombios, ovmf32 and ovmf64), to faciliate easier testing. Some thought still needs to be put into libxl (xend replacement) changes.
In general OVMF/Xen patches for Xen and TianoCore will be at https://github.com/andreiw/andreiw-wip/tree/master/xen/ovmf-support

Schemix

As part of what I get paid for, I frequently need to tinker with Linux kernel internals or expose them out as part of some prototyping work... Occasionally, the typical cycle of code-compile-reboot-flash-boot-test is a bit too tedious so I was looking for something that would let me interactively poke at the kernel without an external debugger and do little experiments...

http://www.abstractnonsense.com/schemix/

Hopefully it's not too decrepit :). Well, it's for 2.4 :(, but it seems small enough that porting to 2.6 shouldn't too much of a PITA.

Thursday, May 5, 2011

LFSR taps

I had to fix up the LFSR code in my block benching tool, and this Xilinx document was very useful for figuring out the primitive polynomials for bits > 16.

http://www.xilinx.com/support/documentation/application_notes/xapp052.pdf

Ended up reading about Galois Fields and Évariste Galois...humbling...humbling...

Mercurial

As part of some Xen stuff I'm doing I have to interact with Mercurial, the SCM used. I'm pretty used to the git work flow, so I was wondering how to do the equivalent of patch squashing.

You need the mq extension.

The equivalent of -

$ git rebase -i HEAD~2

In Hg would look like -

$ hg qimport -g -r -2:-1
$ hg qpop
now at: 2146.diff
$ hg qfold 2147.diff
$ hg qrefresh -m 'new log message'
$ hg qfinish --all

The '-g' flag is a must if you have any binary diffs.

Of course this doesn't help you with reordering commits... :/