Toying around with LE PowerPC64 via the PowerNV QEMU

I've validated that my ppc64le_hello example runs on top of BenH's PowerNV QEMU tree. Runs really snappy!

The only thing that doesn't work is mixed page-size segment support (MPSS, like 16MB in a 4K segment). QEMU does not support MPSS at the moment. Also, QEMU does not implement any of the IBM simulator's crazy Mambo calls.

Exotic QEMU bugs and fixes

I found that the linux-user portion of QEMU has a few bugs around signals. Really, around handling "self-modifying" code and having the code generator step on unmapped memory.

The test is pretty simple.  Have a page of memory containing one instruction which will cause SIGILL to be delivered, followed by a 'ret'. On a SIGILL, unmap the page. On a SIGSEGV, map the page back in. I've two of these tests - one with actual mmap/munmap, and another with mprotect. The tests verify corner conditions in the binary translation logic, with back-to-back signals and an attempt to execute unmapped code.
https://github.com/andreiw/andreiw-wip/blob/master/qemu/tests/sigtest.c
https://github.com/andreiw/andreiw-wip/blob/master/qemu/tests/sigtest_mprotect.c

"self-modifying" code sounds grand, but it's just the signal return path. While newer Linux kernels use VDSO symbols for the restorer (that's the part that does the sigreturn syscall), QEMU still creates an on-the-stack trampoline. When QEMU creates a translation block for the trampoline, it marks the page internally as read-only so that it can detect when the TB should be invalidated. It is this later logic which was short-circuiting and exiting earlier than needed.
That's fixed in https://github.com/andreiw/andreiw-wip/blob/master/qemu/0001-qemu-fix-page_check_range.patch

The second problem is that QEMU doesn't deal very well with being forced to run code that's unmapped. The TCG generator walks over the unmapped memory, gets a SIGSEGV, which attempts delivery of the signal to the translated program (which again, means getting and/or creating more TBs). The problem, though, is that we attempt to reacquire the tcg_ctx.tb_ctx.tb_lock, which we never dropped due to the signal. i.e. after a SIGSEGV here:

#0  disas_a64_insn (s=0x7fffffffdc40, env=<optimized out>) at /target-arm/translate-a64.c:8972
#1  gen_intermediate_code_internal_a64 (cpu=cpu@entry=0x62532200, tb=tb@entry=0x7ffff440b120, search_pc=search_pc@entry=false) at /target-arm/translate-a64.c:9097
#2  0x00000000600d76e5 in gen_intermediate_code_internal (search_pc=false, tb=0x7ffff440b120, cpu=0x62532200) at /target-arm/translate.c:10629
#3  gen_intermediate_code (env=env@entry=0x6253a468, tb=tb@entry=0x7ffff440b120) at /target-arm/translate.c:10904
#4  0x00000000600e4851 in cpu_arm_gen_code (env=env@entry=0x6253a468, tb=tb@entry=0x7ffff440b120, gen_code_size_ptr=gen_code_size_ptr@entry=0x7fffffffdd64) at /translate-all.c:159
#5  0x00000000600e5152 in tb_gen_code (cpu=cpu@entry=0x62532200, pc=pc@entry=4820992, cs_base=cs_base@entry=0, flags=<optimized out>, cflags=cflags@entry=0) at /translate-all.c:973
#6  0x0000000060040e7a in tb_find_slow (flags=<optimized out>, pc=4820992, env=0x6253a468, cs_base=<optimized out>) at /cpu-exec.c:162
#7  tb_find_fast (env=0x6253a468) at /cpu-exec.c:193
#8  cpu_arm_exec (env=env@entry=0x6253a468) at /cpu-exec.c:611
#9  0x000000006005ad2c in cpu_loop (env=env@entry=0x6253a468) at /linux-user/main.c:1015
#10 0x0000000060004dd1 in main (argc=1, argv=<optimized out>, envp=<optimized out>) at /linux-user/main.c:4392

We longjmp back to the CPU loop and deadlock here:

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x000000006012991d in _L_lock_858 ()
#2  0x000000006012978a in __pthread_mutex_lock (mutex=0x604ffa98 <tcg_ctx+350904>) at pthread_mutex_lock.c:61
#3  0x0000000060040bfd in cpu_arm_exec (env=env@entry=0x6253a228) at /cpu-exec.c:610
#4  0x000000006005ad2c in cpu_loop (env=env@entry=0x6253a228) at /linux-user/main.c:1015
#5  0x0000000060004dd1 in main (argc=1, argv=<optimized out>, envp=<optimized out>) at /linux-user/main.c:4392

The solution is to allow tb_gen_code to back out if it knows it can't read the memory. A new exception type is added, EXCP_TB_EFAULT, which then needs to be handled just like an address fault inside cpu_loop.

https://github.com/andreiw/andreiw-wip/blob/master/qemu/0002-qemu-handle-tb_gen_code-getting-called-for-unmapped-.patch
https://github.com/andreiw/andreiw-wip/blob/master/qemu/0003-x86-implement-EXCP_TB_EFAULT.patch

This makes the above tests pass on AArch64 and x86 (32-bit only, since there is no signal handling support for the x86_64-linux-user target at the moment).

Update: Fixes look like they're going in. The TCG deadlock is getting fixed in a simpler way. It is a better and more self-contained fix. http://www.mail-archive.com/qemu-devel@nongnu.org/msg225421.html

Getting PowerPC OpenBIOS to run on QEMU

• You've apt-get installed qemu, but qemu-system-ppc boots to a blank (white or black) screen?
• You've pulled the OpenBIOS SVN, built qemu-openbios.elf, but it boots to a blank screen?

On serial output, you might see "<< set_property: NULL phandle" messages, and the CPU is stuck in a perpetual ISI.

Have no fear. Apparently GCC versions > 4.6 miscompile OpenBIOS, so you need to disable optimization. This is presently set to "-Os" under "Makefile.target". Setting it t "-O0" should do.

I'll probably investigate this deeper after fixing partition-zero booting...

A

iQUIK - the new old PowerMac bootloader for OldWorld machines

About three quarters of a year ago I found a PowerBook 3400c in the trash pile at work. Given my ongoing nostalgia for PowerPC and OpenFirmware, I couldn't pass up the opportunity to reminisce with this PowerBook.

It is by far the oldest mac I've had. The UMAX S900 I dumpster dove for back in 2002 was a dual 604e. This baby is a 603ev. That's three generations behind my old G4 iBook. It's OF is slightly less broken than 1.0.5 on the S900 - at least it doesn't default stdio to serial, and doesn't need nvramc patches to boot from disk. The 3400c booted into MacOS, but clearly it was asking for Linux.

Unfortunately, the best you could do is install via old miBoot floppies, and get a 2.2(!)-based Debian system, just like I had to on the S900 ten whole years ago. Of course you could painfully and slowly upgrade releases, only to realize you can't actually boot a 2.4 kernel with an initrd with the OldWorld bootloader, QUIK. Staring at a "VFS panic" message after investing a week into slowly getting to /that/ point would be bound to enrage most people just toss the aging laptop into trash...

I guess I have an odd fascination with obsolete hardware. I've decided to take ownership of the QUIK code, fixing many of its bugs and limitations. The effort is now called iQUIK and is available for everybody at https://github.com/andreiw/quik.

iQUIK is a Linux bootloader for "OldWorld" PowerMacs. "OldWorld" PowerMacs are all machines that have OpenFirmware, but that don't have built-in USB. Due to a lacking firmware implementation, the typical Yaboot bootloader cannot be used on "OldWorld" machines.

Some highlights:

• You can install and boot Debian Wheezy (the current stable release)
• You can install the boot loader on any medium, including floppy.
• Initrd image work fine, as do 2.2., 2.4 and 2.6 kernels.
• Can boot any kernel/initrd/argument combos without a quik.conf.
• Can list filesystems (only ext2 supported at the moment).
• Works around OF/hw bugs with an innovative shim layer. At least the many bugs on the 3400c.
• Single-stage installation for robustness, using partition-zero boot block and bootstrap partition.No more LILO-style block maps.

You can see the fairly exhaustive documentation at https://github.com/andreiw/quik/blob/master/README. It's only really been tested on a 3400c, although I'm getting a G3 PDQ (that's the last OldWorld PowerBook!) for testing soon.

...but that's not what I wanted to write this post about. I was curious if I could boot iQUIK successfully with the QEMU PowerPC system emulator and OpenBIOS. Turns out yes, but it's a bit painful. The problem is that OpenBIOS doesn't support the mac's "partition-zero" booting, e.g.

0 > boot ata0/ata-disk@0:0

How does "partition-zero" booting work? The mac partition table contains a structure that describes the disk offset of the bootloader, its size and where to load it. The bootloader has to be plain binary (i.e. not XCOFF). iQUIK leverages this by copying the boot code into it's own partition and setting up the boot descriptor in the partition table to point to it. http://www.opensource.apple.com/source/bless/bless-11/README.BOOTING is a pretty good document that describes this more in-depth.

So how to boot on QEMU, then? We could cheat and load the non-munged ELF file the boot code is created from, but I don't have my 3400c nearby to compile with. Instead we will convince OpenBIOS to load our code just like a PowerMac would.

This example assumes:

• You have qemu-system-ppc.
• You have floppy.img from the github quik/distrib.
• You created an install.img: FAT partitioned, ext2 on only FAT partition with the Debian files from http://ftp.nl.debian.org/debian/dists/wheezy/main/installer-powerpc/current/images/powerpc/hd-media/
• Note: take the 'initrd-size' out of yaboot.conf file you downloaded above.

Steps are pretty simple. We launch the simulator like so:

qemu-system-ppc  -hda floppy.img -hdb install.img -serial stdio

At the "0 >" prompt in the framebuffer window we type the following commands to switch the firmware to serial port for input/output. You might wonder why we couldn't use the "prom-env" option to set output-device and input-device? It doesn't work. Heh.

0 > " /pci/mac-io/escc/ch-a" output
0 > " /pci/mac-io/escc/ch-a" input

And now paste the rest of this script into terminal you ran QEMU from. Of course, you could put it on bootable media, but...

0 value ih
\
\ size and load-base need to match
\ SECOND_BASE and SECOND_SIZE in
\ quik/include/layout
\
10000 value size
setenv load-base 3e0000
\
\ boot-file specifies where iQUIK will look
\ for the configuration file. This corresponds to
\ the second ATA disk (-hdb), first FAT partition.
\
setenv boot-file /pci/mac-io/ata-1/disk@1:1/yaboot.conf
\
\ OpenBIOS doesn't know how to parse the MAC "partition-zero"
\ boot descriptor, so we'll manually load it. XXX:2 refers
\ to the Apple_Boostrap partition on -hda, which in the floppy.img
\ image is the first parition (wonky indexing... yes).
\
\ I don't really understand why OpenBIOS needs the seek to 0.
\ Seems like a bug...
\
" /pci/mac-io/ata-1/disk@0:2" open-dev to ih
0 " seek" ih $call-method .
load-base size " read" ih $call-method .
\
\ Based on OpenBIOS libopenbios/bootcode_load.c
\
load-base saved-program-state >sps.entry !
size saved-program-state >sps.file-size !
bootcode saved-program-state >sps.file-type !
-1 state-valid !
\
\ Now can boot. For completeness probably should set
\ /chosen/bootargs but iQUIK can handle the inconsistency.
\
ih close-dev
go

You should now see iQUIK in the framebuffer window (not the serial, because we haven't set the input-device and output-device env variables). You can press the tab key to show the boot labels available. Just press enter to boot the default 'install'.

There you have it.

Of course it would be ideal to just implement bootsector-zero support or even this hack as an nvramrc patch, but unfortunately QEMU PPC has no NVRAM support. Boo...