Cross Compiling Open Firmware

This page describes how to cross-compile Open Firmware for a different CPU, using ARM as a case-study. It began as an email from Mitch Bradley, lightly marked-up.

The main files associated with the Forth portion of Open Firmware are described below. This does not include the possibly-numerous files that may be necessary to produce a complete Open Firmware for a given platform, such as platform-specific startup code, ancillary features like memtest, nandblaster, emacs, sound and graphics data, etc.

forth is the "wrapper" that adapts the Forth system to run under an OS like Linux or Windows or whatever. It is a binary executable file that is compiled/linked for a specific CPU instruction set and OS environment. The source code is mostly in forth/wrapper/wrapper.c, with a few support routines in forth/wrapper/wrapper/logger.c and forth/wrapper/zip/*.c . "forth" doesn't actually know anything at all about the Forth programming language - it is just an I/O interface. You run it by saying "forth foo.dic". "foo.dic" is a "dictionary file" containing the OS-independent (but CPU instruction set specific) Forth system. "forth" allocates memory for the dictionary file, reads the file into that memory, and jumps to it like a subroutine, passing as an argument the address of a table of I/O routines.

"forth" must be compiled for the same instruction set as the ".dic" file that it will load, except for special "built-in emulator" versions of "forth". In the built-in emulator case, "forth" is linked with an instruction set emulator library for a different instruction set. Instead of jumping to the memory containing the .dic file image, you point the emulator library at it. This lets you run, for example, a Power PC .dic file on an x86 host system. Built-in emulator versions of "forth" exist for Power PC and SPARC.

For CPUs where I don't have a built-in emulator library (e.g ARM), I typically use QEMU or some other external emulator. In that case, the emulation happens externally to the "forth" wrapper program, so "forth" must be compiled for the target instruction set. For example, to run a Power PC .dic file with QEMU, you need a "forth" that is compiled for the Power PC instruction set, and QEMU takes care of emulating the PPC instruction set for the complete package of the PPC "forth" plus the PPC ".dic" dictionary image.

The advantage of the QEMU approach is that (hopefully) other people will develop and maintain the instruction set emulation code for numerous CPUs. The disadvantage is that you have to figure out how to to compile "forth" for the target instruction set, which can be a challenge because installing gcc cross-compilation setups is an art in itself. Not only do you need the target cross-compiler, you also need a suitable set of libraries for the target.

In the specific case of ARM, my current approach to a cross-compilation (and indeed, cross-execution) environment is "scratchbox" http://www.scratchbox.org/. Once you work out how to get it set up, scratchbox makes it easy to simulate the existence of an ARM Linux system, including the gcc toolchain, libraries, and an ARM execution environment (via an integrated QEMU setup) on an x86 Linux hosts. OFW doesn't specifically depend on scratchbox in any way; it's just the setup I currently use. In the past, I've used other approaches, including the use of an ARM, Ltd emulator in conjunction with a modified wrapper program, and a Linux system running natively on a real ARM CPU.

In principle, since "forth" changes very rarely, it should suffice for someone to compile "forth" once for a given target, and then you could just use the binary under QEMU without recompiling. That sort of works, but sometimes runs afoul of the following factors:

a) Linux is not very good at maintaining long-term binary compatibility for precompiled applications, especially if they are dynamically linked.

b) To run a Linux application under QEMU, you either need to run a full Linux environment under QEMU and then run you app under that (so you can link to the shared libraries and stuff), or else statically-link your app and use QEMU's "user level" emulation - which only works if QEMU is hosted on Linux. Either way, you have an "OS environment" issue to consider and possibly set up.

kernel.dic is the lowest level Forth dictionary file. It implements core ANS Forth, with the file access wordset but few other extensions. It contains the full Forth interpreter/compiler, so it can extend itself with colon definitions and such, and can (via the file wordset) save the extended dictionary into another .dic file. kernel.dic must run on the target system (or an emulation thereof), but it can be compiled (actually "metacompiled") on a host with any instruction set. The source code is in forth/kernel/*.fth (machine-independent parts) and cpu/<instruction_set>/*.fth (machine-dependent parts).

The source code for kernel.dic is written in a combination of Forth and Forth assembler, but it cannot be incrementally compiled with a traditional Forth compiler, because of the chicken-and-egg problem of needing a Forth compiler to compile the Forth compiler. So the solution it to use a metacompiler. A metacompiler is a Forth compiler that, instead of extending itself, creates a Forth dictionary image that is separate from the Forth dictionary that is used to implement the metacompiler itself. The metacompiler itself can run on a different host instruction set than the instruction set for the image it is creating. In particular, Open Firmware's metacompiler can run on any supported instruction set and can compile for any other supported instruction set, even across endianness and word length mismatches. The metacompiler, unlike the incremental compiler, can handle forward references, which is somewhat handy because it's tricky to avoid forward references in kernel source code.

The main restriction of the metacompiler is that CREATE .. DOES> (and CREATE .. ;CODE) words in the kernel source require special handling. In a normal Forth incremental compiler, the CREATE and DOES> clauses share the same environment (instruction set, word length, endianness, alignment, virtual machine data structures, etc.) so the "coupling" of the compile-time and run-time portions is easy. In the metacompiler case, the CREATE portion runs on the host while the DOES> portion runs on the target, so it takes extra effort to set things up so everything works right. In general, the approach I've taken is to restrict the kernel source to "the usual" Forth defining works (variables, constants, values, defer words, code words, colon definitions, and vocabularies), leaving other custom data types to later incrementally-compiled stages. You can, if necessary, add other types of defining works to the metacompiler, but it takes more effort than the usual "CREATE ... DOES> ..." formulation.

tools.dic is an extended Forth dictionary, based on kernel.dic but extended to contain things like the debugger, decompiler, assembler, and other "nice to have" features. It is built by running kernel.dic on the target CPU (or an emulation thereof), incrementally compiling the extension features from source, and saving the extended dictionary into the "tools.dic" file.

builder.dic is an extension of tools.dic that contains tools that are useful for the process of building Open Firmware image. Those tools include the dependency manager (prevents unnecessary recompilation), the tokenizer (makes .fc Fcode binary files from Forth source), and the dropin manager (makes .rom image files from collections of other files). It's also the starting point for running the metacompiler.

builder.dic is the binary artifact that I usually check into the source tree, because it's a convenient point for rebuilding everything else. To rebuild builder.dic, take a fresh checkout, then

(cd cpu/x86/build && make builder.dic)

basefw.dic is an extension of tools.dic that contains basic Open Firmware elaborations such as the device tree and standard support packages. It's a convenient intermediate for the creation of platform-specific Open Firmware images. basefw.dic is CPU instruction set specific but, by and large, independent of particular platform I/O architecture.

prefw.dic is an extension of basefw.dic, an intermediate dictionary, has hardware addresses for the platform, but not all the platform-specific I/O drivers. This is used as the basis for sdkit-arm.

fw.dic is an extension of basefw.dic that contains platform-specific I/O drivers - particularly for soldered-down devices. Drivers for plug-in or probe-able devices are often demand-loaded from dropin modules that are compiled separately and stored in FCode binary form. fw.dic is usually the largest individual component of the set of dropin modules that comprises an OFW .rom image.