64-bit ARM OS/Kernel/Systems Development Demo on an nVidia Shield TV (Tegra X1)

64-bit ARM OS/Kernel/Systems Development on an nVidia Shield TV (Tegra X1)

The Shield TV is based on the 64-bit nVidia X1 chip. Unlike the K1, this is actually a Cortex-A57 based design, instead of being based on the nVidia "Denver" design. That by itself is kind of interesting already. The Shield TV was available much much earlier than the X1-based nVidia development board (Jetson TX1, you can even buy it on Amazon), and costs about a third of the TX1. The Shield TV allows performing an unlock via "fastboot oem unlock", allowing custom OS images to be booted. Unlike the TX1, you don't get a UART (and I haven't found the UART pads yet, either).

What this is

https://github.com/andreiw/shieldTV_demo

This is a small demo, demonstrating how to build and boot arbitrary code on your Tegra Shield TV. Unlike the previous Tegra K1 demo, you get EL2 (hypervisor mode!).

• A Shield TV, unlocked. Search Youtube for walkthroughs.
• Shield OS version >= 1.3.
• GNU Make.
• An AArch64 GNU toolchain.
• ADB/Fastboot tools.
• Bootimg tools (https://github.com/pbatard/bootimg-tools), built and somewhere in your path.
• An HDMI-capable screen. Note, HDMI, not DVI-HDMI adapter. You want the firmware to configure the screen into 1920x1080 mode, otherwise you'll be in 640x480 and we don't want that...

How to build

$ CROSS_COMPILE=aarch64-linux-gnu- make

...should yield 'shieldTV_demo'.

How to boot

1. Connect the Shield TV a USB cable to your dev workstation.
2. Reboot device via:
   

   $ adb reboot-bootloader

   ...you should now see the nVidia splash screen, followed by the boot menu.
3. If OS is 1.3, you can simply:
   

   $ fastboot boot shieldTV_demo

4. If OS is 1.4 or 2.1, you will need to:
   

   $ fastboot flash recovery shieldTV_demo

   ...and then boot the "recovery kernel" by following instructions on screen.

The code will now start. You will see text and some drawn diagonal lines black background. The text should say we're at EL2 and the lines should be green. The drawing will be slow - the MMU is off and the caches are thus disabled.

Let me know if it's interesting to see the MMU setup code.

Final thoughts

The Shield TV is a better deal than the TX1 for the average hobbyist, even with the missing UART. For the price being sold the TX1 should come with a decent amount of RAM, not 1GB more than the Shield TV. nVidia...are you listening? Uncripple your firmware so booting custom images is not a song-and-dance (you broke it in 1.4!) and at least TELL us where the UART pads are on the motherboard. If you're really cool put together an "official" Ubuntu image that runs on the TX1 and the Shield (and fix SCR_EL3.HCE, too).

64-bit ARM OS/Kernel/Systems Development Demo on a Nexus 9

64-bit ARM OS/Kernel/Systems Development on a Nexus 9

The Nexus 9 is based on a 64-bit nVidia K1 chip. At the moment it is the most affordable (price wise) and accessible (unit-wise) platform for exploring OS work on an AArch64 platform. The Nexus 9 allows performing an unlock via "fastboot oem unlock", allowing custom Android images to be booted.
https://github.com/andreiw/nexus9_demo

What this is

This is a small demo, demonstrating how to build and boot arbitrary code on your Nexus 9 and do some basic I/O. The demo demonstrates serial I/O and draws two black diagonal lines on the framebuffer.

What you need - required

• A Nexus 9, unlocked. Search Youtube for walkthroughs
• GNU Make
• An AArch64 GNU toolchain
• ADB/Fastboot tools
• Booting tools from https://github.com/pbatard/bootimg-tools somewhere in your path

What you need - optional

• A headset to RS232 adapter. I've used a cable I've had from the Motorola Xoom project (the first...errr, Nexus, before the devices got called a "Nexus"). You can try http://www.accuvant.com/blog/building-a-nexus-4-uart-debug-cable

How it works

HBOOT, the Nexus bootloader, expects images to be in a certain format. The booted kernel/code must:

• Be 64-bit
• Be binary (not ELF)
• Be linked at 0x80080000
• Be compressed using "gzip"
• Be followed by the binary FDT
• Be contained in an "ANDROID!" boot image.

Some notes:

• The link address appears to be hardcoded in HBOOT. The Android boot image bases and the AArch64 kernel header fields appear to be ignored.
• The boot image can contain an additional ramdisk/initrd/payload.
• The FDT is patched by HBOOT to contain correct linux,initrd-start and linux,initrd-end addresses.

How to build

$ CROSS_COMPILE=aarch64-linux-gnu- make

How to boot

Connect your Android tablet via a USB cable. Optionally connect the UART headphone jack adapter to your computer. The settings are 115200 8-n-1.

$ adb reboot-bootloader
$ fastboot boot nexus9_demo

Actual output of the demo

Hello!
CurrentEL = 0000000000000001
SCTLR_EL1 = 0000000010C5083A
Bye!

Where to go from here

"nexus9_dts" is the decompiled "nexus9_dtb". "nexus9_dtb" was extracted from the Android boot.img.

• https://android.googlesource.com/kernel/tegra/+/android-tegra-3.10/
• https://developer.nvidia.com/tegra-k1-technical-reference-manual

Final thoughts

From studying the Tegra K1 TRM, the K1 should have virtualization support (i.e. EL2). However, the HTC firmware does not allow booting an EL2-enabled OS. All kernels are booted in EL1. This is rather unfortunate and prevents playing around with KVM and Xen on this platform. Perhaps there are some problems with EL2 support. Or perhaps HTC/nVidia/Google were too myopic to allow EL2 access. It's unclear if the "oem unlock" allows reflashing custom unsigned firmware. "nvtboot" seems to enforce signed "Trusted OS" payloads, at least from dumping the strings. The boot flow looks something like this:

• "nvtboot" (32-bit) runs on the AVP/COP.
• "nvtboot" loads "tos" (64-bit) (Trusty aka Secure OS) on the AArch64 chip.
• "tos" loads HBOOT (32-bit).
• HBOOT loads Android and implements the fastboot protocol.

It's unclear how to enter NVFlash/APX mode, or how helpful that would be.

Musings on device workarounds and attribution

I was trading war stories with some colleagues today, and remembered the time I was chasing crazy UART bugs.

So I just had to go look at my battlefields of past and reminisce...

Ever look at a random driver and wonder how convoluted weird code gets written? Then you look at the git history and see - nothing useful. No history. It was apparently all written at once, by some crazy smart engineer based on thorough and clean specs, right ;-)?

Like the serial-tegra driver, for example. Ever wonder why UART waits for a certain bit of time after switching baud rate?

I used to work on the Moto Xoom tablet - the first official Android tablet, based around the Tegra 2 SoC. Once upon a time I was investigating a bug around suspend-resume. We were seeing a kernel crash when waking the tablet up occasionally with a paired Bluetooth keyboard. The actual crash was the result of a BlueZ bug that didn't defensively treat BT HCI connect events for devices that weren't disconnected (have a gander at http://www.spinics.net/lists/linux-bluetooth/msg10690.html - yes, a  rogue Bluetooth adapter /can/ crash the system, wonderful, right?)

But why weren't the BT disconnect messages coming through?

The tablet was asleep at the time of the disconnect, and the disconnect woke it up. The Bluetooth host was connected to the CPU via a UART, and the UART needed to be resumed before the BT host could send data. UART resume, among other things, needs to set the baud rate. What was happening, is that the the hardware flow control allowed RX before the baud rate change fully propagated through the UART block. The result is that the received data was corrupted. Oops.

Knowing what was happening didn't mean I had a solution, of course. The docs were useless, and it took another fun half a week to figure out the solution :-). Too bad I can't remember what this fix was for... Probably more BT issues :).

So what point did I want to make? The Tegra HSUART driver "got rewritten" when Tegra 2/3 support was upstreamed. But it's the same code, basically, even down to the code flow and comments. You put in time, sleepless nights and life energy and you can't get basic attribution from some unknown dude at NV.

Behind every line of code is some story. Some feeling of exhilaration, success and victory. I almost made a t-shirt with the fix :-). So always attribute contributions out of solidarity with your fellow hackers. Heh.

BlueZ is a train wreck, though... There. I said it.

Not such a long way.

Framebuffer support is in, which means that finally, I can almost ditch the serial cable for a lot of work. It also means you don't need a special Xoom (or potentially - some other Tegra2 device) with a serial port. Retail Xoom owners can now proudly do...ummm...something. Not a whole lot, yet ;-(.

Found a page fault abort within __divdi3, but couldn't really debug it since the DABT handler (or should I say - the CPU context save code) is woefully broken. So before I can figure out what's wrong with __divdi3, I need to fix the trap handlers.

But wait, Andrei, how could you have the timer working then??? The IRQ path is completely different (as is the CPU context structure manipulated). Why? No clue... Won't be when I'm done with it ;-).

Btw, ARMv8 will support a division instruction. Until then, I'll have an excuse to read Knuth.

https://plus.google.com/114030412547992895028/posts/Y1MsnBwKCMX

A

Blowing the dust off of my Xoom.

ATAG    (P): 0x00000100
Linked  (V): 0xff008000
Actual  (P): 0x00a00800
Desired (P): 0x00108000

Handing off to C...
[XEN]  __  __            _____  ___   ____    ____  
[XEN]  \ \/ /___ _ __   |___ / / _ \ |___ \  |___ \ 
[XEN]   \  // _ \ '_ \    |_ \| | | |  __) |__ __) |
[XEN]   /  \  __/ | | |  ___) | |_| | / __/|__/ __/ 
[XEN]  /_/\_\___|_| |_| |____(_)___(_)_____| |_____|
[XEN]                                               
[XEN] Xen/ARMv7 virtual machine monitor for ARM architecture
[XEN] Copyright (C) 2012 Andrei Warkentin <andreiw@msalumni.com>
[XEN] Copyright (C) 2007 Samsung Electronics Co, Ltd. All Rights Reserved.
[XEN]  http://www.cl.cam.ac.uk/netos/xen
[XEN]  University of Cambridge Computer Laboratory
[XEN] 
[XEN]  Xen version 3.0.2-2 (andreiw@(none)) (gcc version 4.4.3 (GCC) ) Fri Mar  2 01:59:34 EST 2012
[XEN]  Platform: arm-tegra
[XEN]  GIT SHA: ffd558debcf08dcf59a0c38115906030bf6f261c
[XEN] 
[XEN] TTB PA 0x104000
[XEN] idle_pgd VA 0xff004000
[XEN] xen_pstart 0x0
[XEN] xen_pend 0x40000000
[XEN] _end 0xff03e708
[XEN] _end VA 0x13E708
[XEN] nr_pages needed for all page_infos = 0x500
[XEN] frame table is at 0xff03f000-0xff53f000
[XEN] xenheap_phys_start = 0x648000 (VA 0xff548000)
[XEN] xenheap_phys_end = 0x848000 (VA 0xff748000)
[XEN] looking at bank 0
[XEN]         base - 0x0
[XEN]         end  - 0x40000000
[XEN] calling init_boot_pages on 0x648000-0x40000000
[XEN] Using scheduler: Simple EDF Scheduler (sedf)
[XEN] Initializing ARM FCSE Unit
[XEN] *** LOADING DOMAIN : 0 ***
[XEN] DOM0 image is not a Xen-compatible Elf image.
[XEN] Could not set up DOM0 guest OS
[XEN] 
[XEN] VMM Panic at xensetup.c:357

https://github.com/andreiw/xen3-arm-tegra

Hopefully the next time I work on this won't be in another half a year.