64-bit ARM OS/Kernel/Systems Development Demo on a Nexus 9

64-bit ARM OS/Kernel/Systems Development on a Nexus 9

The Nexus 9 is based on a 64-bit nVidia K1 chip. At the moment it is the most affordable (price wise) and accessible (unit-wise) platform for exploring OS work on an AArch64 platform. The Nexus 9 allows performing an unlock via "fastboot oem unlock", allowing custom Android images to be booted.
https://github.com/andreiw/nexus9_demo

What this is

This is a small demo, demonstrating how to build and boot arbitrary code on your Nexus 9 and do some basic I/O. The demo demonstrates serial I/O and draws two black diagonal lines on the framebuffer.

What you need - required

• A Nexus 9, unlocked. Search Youtube for walkthroughs
• GNU Make
• An AArch64 GNU toolchain
• ADB/Fastboot tools
• Booting tools from https://github.com/pbatard/bootimg-tools somewhere in your path

What you need - optional

• A headset to RS232 adapter. I've used a cable I've had from the Motorola Xoom project (the first...errr, Nexus, before the devices got called a "Nexus"). You can try http://www.accuvant.com/blog/building-a-nexus-4-uart-debug-cable

How it works

HBOOT, the Nexus bootloader, expects images to be in a certain format. The booted kernel/code must:

• Be 64-bit
• Be binary (not ELF)
• Be linked at 0x80080000
• Be compressed using "gzip"
• Be followed by the binary FDT
• Be contained in an "ANDROID!" boot image.

Some notes:

• The link address appears to be hardcoded in HBOOT. The Android boot image bases and the AArch64 kernel header fields appear to be ignored.
• The boot image can contain an additional ramdisk/initrd/payload.
• The FDT is patched by HBOOT to contain correct linux,initrd-start and linux,initrd-end addresses.

How to build

$ CROSS_COMPILE=aarch64-linux-gnu- make

How to boot

Connect your Android tablet via a USB cable. Optionally connect the UART headphone jack adapter to your computer. The settings are 115200 8-n-1.

$ adb reboot-bootloader
$ fastboot boot nexus9_demo

Actual output of the demo

Hello!
CurrentEL = 0000000000000001
SCTLR_EL1 = 0000000010C5083A
Bye!

Where to go from here

"nexus9_dts" is the decompiled "nexus9_dtb". "nexus9_dtb" was extracted from the Android boot.img.

• https://android.googlesource.com/kernel/tegra/+/android-tegra-3.10/
• https://developer.nvidia.com/tegra-k1-technical-reference-manual

Final thoughts

From studying the Tegra K1 TRM, the K1 should have virtualization support (i.e. EL2). However, the HTC firmware does not allow booting an EL2-enabled OS. All kernels are booted in EL1. This is rather unfortunate and prevents playing around with KVM and Xen on this platform. Perhaps there are some problems with EL2 support. Or perhaps HTC/nVidia/Google were too myopic to allow EL2 access. It's unclear if the "oem unlock" allows reflashing custom unsigned firmware. "nvtboot" seems to enforce signed "Trusted OS" payloads, at least from dumping the strings. The boot flow looks something like this:

• "nvtboot" (32-bit) runs on the AVP/COP.
• "nvtboot" loads "tos" (64-bit) (Trusty aka Secure OS) on the AArch64 chip.
• "tos" loads HBOOT (32-bit).
• HBOOT loads Android and implements the fastboot protocol.

It's unclear how to enter NVFlash/APX mode, or how helpful that would be.

Solaris/PPC

Apparently back around 2006 there was an effort at Sun Labs to get OpenSolaris to work on CHRP(like)  PowerPC machines. And according to the documentation, the kernel could even boot to shell on a G4 Apple.

That effort was called Polaris. It was difficult to find the CDDL-licensed sources, but I've made them available for everyone else to play with at https://github.com/andreiw/polaris

I haven't tried it out or done anything with the sources yet. The Solaris kernel is a pretty amazing piece of software, and a very portable and well-designed one to boot. I am glad Sun open-sourced it before folding, as it's code like this that should be influencing OS R&D for generations to come. It would be interesting to see the Polaris code being used as a base for an AArch64 investigation...

A

What's special about...

addi r0,r1,0x138
ori r0,r0,0x60

...and I/O port 0x92 :-)?

iQUIK update

I now have a 1.5Ghz PowerBook 12" in my possession to test iQUIK with. This is of course a NewWorld, and not a primary target for the iQUIK boot loader...

Couple of observations to be made:

• OpenFirmware 3.0 doesn't support partition zero booting (i.e. hd:0 or CHRP-spec hd:%BOOT). This means that iQUIK cannot be booted the same way as it boots on OldWorlds,  but neither is it required. iQUIK can be booted on NewWorlds the same way as Yaboot, i.e. placing 'iquik.elf' on an HFS+ partition and blessing it. 
• NewWorld OF requires appending ":0" for full-disk access to disk devices

I've also fixed a bug inside partition code that truncated offsets to 32 bits, and improved device path handling and parsing.

In short, though, it works. And it works quite well. So iQUIK now works on OldWorld and NewWorld machines. Yaboot - only on NewWorlds. Of course, Yaboot also supports CHRP machines, network booting and reads all filesystems supported by the underlying OpenFirmware implementation. So there's plenty of work to reach feature parity in that regard.

A

Detecting 'make' environment variables change

While playing with 'iquik' and trying to add a mode to build a reduced-logging version that is smaller, I ran into an interesting question - how do I force a rebuild of everything with a clean?

#
# Example of a Makefile that detects "environment change".
#
# I.e.:
#
# andreiw-lnx:~/src/ make clean
# Cleaning
# andreiw-lnx:~/src/ make 
# Resuming build with env ""
# Building with ""
# andreiw-lnx:~/src/ make CONFIG_EXAMPLE=1
# Cleaning due to env change (was "" now "-DCONFIG_EXAMPLE")
# Cleaning
# Building with "-DCONFIG_EXAMPLE"
# andreiw-lnx:~/src/ make CONFIG_EXAMPLE=1
# Resuming build with env "-DCONFIG_EXAMPLE"
# Building with "-DCONFIG_EXAMPLE"
# andreiw-lnx:~/src
#

ENV_FILE=old_build_env
-include $(ENV_FILE)

#
# Environment definition.
#
ifeq ($(CONFIG_EXAMPLE), 1)
BUILD_FLAGS = -DCONFIG_EXAMPLE
endif
BUILD_ENV = "OLD_BUILD_FLAGS=$(BUILD_FLAGS)"

#
# Detect environment change.
#
ifneq ($(BUILD_FLAGS),$(OLD_BUILD_FLAGS))
 PRETARGET=clean_env
else
 PRETARGET=log_env
endif

all: $(PRETARGET) target

target:
 @echo Building with \"$(BUILD_FLAGS)\"

log_env:
 @echo Resuming build with env \"$(BUILD_FLAGS)\"

log_clean_env:
 @echo Cleaning due to env change \(was \"$(OLD_BUILD_FLAGS)\" now \"$(BUILD_FLAGS)\"\)

clean_env: log_clean_env clean
 @rm -f $(ENV_FILE)
 @echo $(BUILD_ENV) > $(ENV_FILE)

clean:
 @echo Cleaning

Musings on device workarounds and attribution

I was trading war stories with some colleagues today, and remembered the time I was chasing crazy UART bugs.

So I just had to go look at my battlefields of past and reminisce...

Ever look at a random driver and wonder how convoluted weird code gets written? Then you look at the git history and see - nothing useful. No history. It was apparently all written at once, by some crazy smart engineer based on thorough and clean specs, right ;-)?

Like the serial-tegra driver, for example. Ever wonder why UART waits for a certain bit of time after switching baud rate?

I used to work on the Moto Xoom tablet - the first official Android tablet, based around the Tegra 2 SoC. Once upon a time I was investigating a bug around suspend-resume. We were seeing a kernel crash when waking the tablet up occasionally with a paired Bluetooth keyboard. The actual crash was the result of a BlueZ bug that didn't defensively treat BT HCI connect events for devices that weren't disconnected (have a gander at http://www.spinics.net/lists/linux-bluetooth/msg10690.html - yes, a  rogue Bluetooth adapter /can/ crash the system, wonderful, right?)

But why weren't the BT disconnect messages coming through?

The tablet was asleep at the time of the disconnect, and the disconnect woke it up. The Bluetooth host was connected to the CPU via a UART, and the UART needed to be resumed before the BT host could send data. UART resume, among other things, needs to set the baud rate. What was happening, is that the the hardware flow control allowed RX before the baud rate change fully propagated through the UART block. The result is that the received data was corrupted. Oops.

Knowing what was happening didn't mean I had a solution, of course. The docs were useless, and it took another fun half a week to figure out the solution :-). Too bad I can't remember what this fix was for... Probably more BT issues :).

So what point did I want to make? The Tegra HSUART driver "got rewritten" when Tegra 2/3 support was upstreamed. But it's the same code, basically, even down to the code flow and comments. You put in time, sleepless nights and life energy and you can't get basic attribution from some unknown dude at NV.

Behind every line of code is some story. Some feeling of exhilaration, success and victory. I almost made a t-shirt with the fix :-). So always attribute contributions out of solidarity with your fellow hackers. Heh.

BlueZ is a train wreck, though... There. I said it.

MkLinux

The first step to getting MkLinux to run is to get the build tools to run.

Build tools?

The OSF Open Development Environment tools. Which have been very hard to find (https://github.com/slp/mkunity/issues/1). But now you can find them and even build them - https://github.com/andreiw/ode4linux

If I ever find time I'll clean up the code so it doesn't build with a million warnings.

A