[f] [Project Foobazco]

Unsuccessfully predicting the end of the world since 1765.

Bootstrapping Linux on an Indy
Last updated 2000-07-04 2003-04-06

Introduction
Bootstrapping is the process of bringing up an operating system on a new type of system for the first time (alternately, it is the subprocess which consists of building a full cross-compilation environment, or that subprocess which consists or building a full cross-compiler only. We will use it in the largest sense here.). This document covers bootstrapping GNU/Linux on a Silicon Graphics Indy system.
I will address a FAQ right away: "What is cross compilation?" Simply put, cross compilation is the process of running a compiler on a system of one type to generate code for a system of a different type. As you would expect, a cross compiler is the program which does this. Similarly, there exist cross assemblers, linkers, debuggers, and other utilities. In short, a cross toolchain deals with binary files not native to the system on which it runs.
There are three possible systems involved in a bootstrapping operation; the GNU autoconf terminology for them is host, build, and target. The host is the system on which programs are actually run. The build is the system on which a particular package is being built. The target, applicable only to certain packages such as gcc and binutils, is the system for which the software in question generates output (normally, this is the system for which the cross toolchain generates binaries). All three can be different, but this discussion centers on a setup in which Phase 1 binutils and gcc have build=host=sparc-linux and target=mips-linux, Phase 1 kernel and glibc have build=sparc-linux and host=mips-linux, Phase 2 packages have build=sparc-linux and host=target=mips-linux, and Phase 3 packages have build=host=target=mips-linux. Please note that you are free to replace sparc-linux with any supported environment for which you already have an acceptable native toolchain (see below).
There are three main phases to bootstrapping a new system: Building a cross-toolchain, building a native build environment, and finishing up by building a full suite of applications natively. Phase 1 is usually the hardest, because it involves porting the toolchain to your new platform. However, for MIPS targets most of that work has already been done. As an experimenter you need only build the existing software. Phase 2 is difficult because it uses features of the toolchain build environment that are rarely used. With some hacking, however, you can build a native C compiler and binutils. Phase 3 is relatively easy. Most applications written portably can be built out of the box. Some applications will need fixes, possibly major.
I discuss here a step by step approach to bootstrapping Linux on an SGI Indy. By convention, the following machines and software are in use:
  1. pimp is a Sun Ultra 2 with 2 processors running Linux 2.2. It has IP address 192.168.1.1. This sytem will be our cross-host to start the bootstrap. It can be any reasonably well-supported system including sparc-sun-linux, i386-peecee-linux, or even mips-sgi-irix6.5. You will need at least 500MB of disk, maybe more. I recommend finding a nice big 4 or 9 GB disk to use just for this. Your build system should be fairly powerful; glibc especially will be a long build. I recommend a system no less powerful than pimp, with at least 256 MB memory.
  2. ice is our target machine, an SGI Indy with an R4400SC/200 CPU. It has IP address 192.168.1.2 and MAC address 08:00:69:11:22:33.
We will use the following software (note that you can and should try later versions; this is for example only):
  1. GNU binutils 000702 from the CVS repository at anoncvs.cygnus.com
  2. gcc 000703 from the CVS repository at anoncvs.cygnus.com
  3. GNU libc 000622 from the CVS repository at anoncvs.cygnus.com
  4. Linux 2.4 from the CVS repository at oss.sgi.com
  5. Various software from ftp.gnu.org and alpha.gnu.org
What you will need on the host system:
  1. gcc (egcs 1.0.2 or later, 2.95.2 or 2.96 recommended)
  2. binutils 2.9.1 or later (2.9.5.0.22+ or 2.10 recommended)
  3. bison or yacc
  4. Lots of disk space and plenty of memory and cpu power
If you wish to become an active developer (if not, the effort of bootstrapping your own system is probably wasted), I strongly recommend maintaining your own copies of the CVS trees, rather than using snapshots and patching up continuously.
Our ultimate objective is to start with an empty filesystem on ice and end with a functioning Linux installation. Some portions of this are not covered; for example, using an initial NFS root setup to transfer your environment is not covered in detail. It is assumed that you are familiar with the basics of building software from source and with the capabilities and features of the Linux kernel and common Free software.

Phase 1 - the cross toolchain
To begin, we will build a cross toolchain on pimp. We will use the sources binutils-000702.tar.gz, gcc-000703.tar.gz, and glibc-000622.tar.gz, which can be obtained from ftp.foobazco.org/pub/people/wesolows/mips-linux/. Alternately you are free to acquire the current CVS trees and build those instead. We begin by creating two directories on a large, empty disk. I will call the space /s and the two directories crossdev and crossdev-build. 
		$ mkdir /s/crossdev
		$ mkdir /s/crossdev-build
Extract the tarballs into the crossdev-build directory. 
		$ cd /s/crossdev-build
		$ tar xzvf /path/to/gcc-000703.tar.gz
		$ tar xzvf /path/to/glibc-000622.tar.gz
		$ tar xzvf /path/to/binutils-000703.tar.gz
		$ tar xzvf /path/to/linux/linux-2.4.0-test3-pre2.tar.gz
Please note that even the current (as of 4 July 2000) CVS gcc will abort while compiling a kernel. To fix the problem, get and apply gcse-regcopy.diff.gz from the Foobazco FTP site as follows: 
		$ cd egcs-000703
		$ zcat /path/to/gcse-regcopy.diff.gz | patch -p1
Make separate directories for building - that is, build outside the source tree. This requires that you have GNU make (or some other make which properly supports the VPATH feature. If you don't know, get GNU make 3.79). 
		$ mkdir gcc-000703-build
		$ mkdir glibc-000622-build
		$ mkdir binutils-000702-build
Before doing anything else, if you are on an UltraSparc system running Linux, as I am, switch to the 32-bit userland mode: 
		$ sparc32 bash
Some general notes in advance: we will make frequent use of options and environment variables to configure. You are strongly advised to consult each individual program's documentation on the usage and purpose of these arguments. However, for most programs, the following hold:
  • --prefix: The location in the filesystem at which the program assumes it is being installed. Note that for programs which are being cross-compiled, we will not actually install into this location on the build system.
  • --build,--host,--target: These select the type of system(s) which you are working with. See discussion above.
  • CC: This environment variable should be set when cross compiling to a compiler that runs on the build system and generates code for the host system.
  • HOST_CC, BUILD_CC: These should both be set when cross compiling to a native compiler for the build system. Note that not all packages uses there the same way and that the terminology is horribly confusing.
  • RANLIB, AR: When cross compiling, these should be set to the names of the corresponding binutils which run on the build system and operate on binaries for the host system.
  • CFLAGS, LDFLAGS: These variables control additional arguments passed to the compiler and linker, respectively. Note that some packages (incorrectly) pass LDFLAGS to the linker when linking binaries to run on both the build and host systems. For this reason LDFLAGS should be used with extreme care when cross compiling.
We must build binutils first. This step is easy: 
		$ cd binutils-000622-build
		$ ../crossdev/binutils-000622/configure 
		  --prefix=/s/crossdev --target=mips-linux
		$ make MAKE="make -j3"
This should go smoothly. If it does not, your host's toolchain may be broken or you may have found a bug in binutils. When it has completed, just install it: 
		$ make install
Please note: you will not ever need root access on the host system during this process (provided that the system has sufficient user-writable disk space). If you install into a public location as root, you are on your own. If you do so and make a mistake you can destroy your host system. This is especially true of the glibc install.

The next step is building gcc. This should be fairly straightforward as well. Please note that you should not use gcc 2.95.x, and you cannot use egcs < 1.0.2. I recommend using the 3.2 or 3.3 CVS tree, despite some lingering issues. Before going on, it's a good idea to add a PATH entry for your new development tools. 

		$ PATH=$PATH:/s/crossdev/bin
Now build gcc. 
		$ cd ../gcc-000622-build
		$ AR=mips-linux-ar RANLIB=mips-linux-ranlib 
		  ../gcc-000622/configure --prefix=/s/crossdev 
		  --target=mips-linux --with-newlib --disable-threads
		  --disable-shared
		$ make -C libiberty LANGUAGES=c MAKE="make -j3"
		$ make -C gcc LANGUAGES=c MAKE="make -j3"
The use of --disable-threads and --disable-shared is only needed the first time you build a gcc (eg before you have glibc built). Now install gcc (the build should have worked; if not you may have found a gcc bug): 
		$ make -C libiberty install LANGUAGES=c
		$ make -C gcc install LANGUAGES=c
You now have a basic cross-toolchain. The next step is to build glibc. However, in order to do that you will need some headers from the kernel. Specifically, the include/linux and include/asm-mips directories. We also need include/linux/version.h so we will need to unpack the kernel and configure it. Since we're going to do this anway, we might as well put off building glibc and just build a kernel. We already have sufficient tools to do so (glibc is not needed to build kernels). When you configure your kernel, I recommend not adding much of anything. If you do, your kernel may not boot. You can probably remove features you know you won't need. Make sure you don't disable kernel-level autoconfig or root fs on NFS as you will need them later. 
		$ cd ../linux
		$ make menuconfig
		$ make dep boot MAKE="make -j3" CROSS_COMPILE=mips-linux-
And copy the headers: 
		$ cd include
		$ mkdir /s/crossdev/mips-linux/include
		$ cp -rd linux /s/crossdev/mips-linux/include
		$ cp -rd asm-mips /s/crossdev/mips-linux/include
		$ cp -rd asm-generic /s/crossdev/mips-linux/include
		$ mv /s/crossdev/mips-linux/include/asm-mips
		  /s/crossdev/mips-linux/include/asm
Save your ELF and ECOFF kernels (/s/crossdev-build/linux/vmlinux and /s/crossdev-build/linux/arch/mips/boot/vmlinux.ecoff) in a safe place for later; we'll use them to boot your system. Now we can build glibc. 
		$ cd ../glibc-000622-build
		$ AR=mips-linux-ar RANLIB=mips-linux-ranlib CC=mips-linux-gcc 
		  HOST_CC=cc BUILD_CC=cc ../glibc-000622/configure 
		  --prefix=/usr --host=mips-linux --enable-add-ons --with-elf 
		  --disable-profile 
		  --with-headers=/s/crossdev/mips-linux/include
		$ make
Now install glibc. It's important to get this right - if you don't give the right arguments, it will try (and hopefully fail) to overwrite your system's native libraries. 
		$ mkdir ../glibc-000622-inst
		$ make install install_root=/s/crossdev-build/glibc-000622-inst
Now we need to use this glibc for two things: To cross-compile against, and to use natively. To get the native set, we just want a full image of the installation: 
		$ cd /s/crossdev-build/glibc-000622-inst
		$ tar czf ../glibc-000622-inst.tar.gz .
Later on we'll extract this into our new root filesystem. To get the cross-build library set up, just copy the libraries and headers into the cross-compiler's search path: 
		$ cd lib && find . -print | cpio -pumd 
		  /s/crossdev/mips-linux/lib
		$ cd ../usr/lib && find . -print | cpio -pumd 
		  /s/crossdev/mips-linux/lib
		$ cd ../include && find . -print | cpio -pumd 
		  /s/crossdev/mips-linux/include
That should conclude Phase 1. You are now ready to build native programs.

Phase 2: the native core
We now use our cross toolchain to build a set of utilities sufficient for building programs natively on ice. This set generally includes a C compiler, binutils, fileutils, textutils, diffutils, sh-utils, grep, sed, make, tar, gzip, and util-linux. You may want additional packages, but remember that our goal is to build just enough that we can then build natively whatever we wish.
In the list above, there are three major classes of packages: gcc, util-linux, and everything else. The "everything else" list will generally build all in the same fashion, and will do so fairly cleanly. For these packages, use the following build commands: 
		$ CC=mips-linux-gcc AR=mips-linux-ar RANLIB=mips-linux-ranlib 
		  LD=mips-linux-ld BUILD_CC=cc HOST_CC=cc CC_FOR_BUILD=cc
		  CFLAGS="-I/s/crossdev/mips-linux/include" ./configure 
		  --prefix=/usr --disable-largefile
		$ make
		$ make install prefix=/s/crossdev-build/native/package-name/usr
		$ cd /s/crossdev-build/native/package-name
		$ tar czf ../package-name.tar.gz .
Note that you can adjust the --prefix argument to reflect the location you want the programs to live in. You need --disable-largefile for the moment; once the remaining issues with stat64 and friends are worked out that argument will no longer be necessary. Some of these packages' configure script will fail with an error about setvbuf while cross-compiling. You can resolve this problem by placing the following line into config.cache: 
		ac_cv_func_setvbuf_reversed=${ac_cv_func_setvbuf_reversed=no}
Again, note that you need not and should not install as root, except for the sh-utils package, which wants to setuid the passwd program. Make very sure you properly specify the prefix= variable when installing; otherwise native programs may be overwritten.
Unfortunately, gcc and util-linux are trickier. Util-linux was never really meant to be cross-compiled, and it's a portability nightmare anyway. Nevertheless, I've put together a patch that allows this. Note that there are other patches for util-linux available there; I recommend you get and apply them all. Once you've done this, actually building util-linux should be pretty straightforward. 
		$ CC=mips-linux-gcc AR=mips-linux-ar RANLIB=mips-linux-ranlib 
		  CFLAGS="-I/s/crossdev/mips-linux/include -I../lib -I. -O2" 
		  ./configure
		$ make CC=mips-linux-gcc AR=mips-linux-ar 
		  RANLIB=mips-linux-ranlib 
		  CFLAGS="-I/s/crossdev/mips-linux/include -I../lib -I. -O2"
		$ make CC=mips-linux-gcc AR=mips-linux-ar 
		  RANLIB=mips-linux-ranlib 
		  CFLAGS="-I/s/crossdev/mips-linux/include -I../lib -I. -O2" 
		  install DESTDIR=/s/crossdev-build/native/ul-inst
		$ cd /s/crossdev-build/native/ul-inst/bin
		$ mv * ../usr/bin
		$ cd ..
		$ rmdir bin
		$ tar czf ../util-linux.tar.gz .
Building gcc used to be the worst part of all. This method seems to work well, however, and if you use sufficiently new versions it should steer you away from trouble. Don't forget to patch the sources as above. 
		$ CC="mips-linux-gcc 
			-I/scratch/crossdev/mips-linux/mips-linux/include" 
		  CC_FOR_BUILD=cc 
		  GCC_FOR_TARGET="mips-linux-gcc -I/s/crossdev/mips-linux/include" 
		  AR=mips-linux-ar RANLIB=mips-linux-ranlib LD=mips-linux-ld 
		  ../egcs-000703/configure --prefix=/usr --enable-threads 
		  --enable-languages=c --host=mips-linux --target=mips-linux
		  --build=`../egcs-000703/config.guess`
		$ make
		$ make install prefix=/s/crossdev-build/native/gcc/usr
		$ cd /s/crossdev-build.native/gcc
		$ tar xzvf ../gcc.tar.gz .
At least with the versions described within, several files will not be compiled correctly; they will fail to assemble. You can work around this by waiting for these files to come up, adding -Os to the CFLAGS variable, compiling that file, then killing the build process, removing -Os, and continuing. Note that building with -Os takes much longer than normal building. Hopefully this problem will be fixed soon. The last thing you will need is a statically linked copy of bash. I recommend obtaining one from the several available distributions. At this point you should extract the glibc package from phase 1 and all the native packages from phase 2 into a directory on your NFS server, for use as a root filesystem. You can do this as follows, assuming your NFS-exported directory will be called /nfs: 
		$ cd /nfs
		$ tar xzvf /s/crossdev-build/glibc-000622-inst/glibc-000622-inst.tar.gz
		$ tar xzvf /s/crossdev-build/native/pacakge-name.tar.gz
		...
		$ cp /tmp/static-bash usr/bin/bash
		$ cd bin
		$ mv * ../usr/bin
		$ cd ..
		$ rmdir bin
		$ ln -s usr/bin bin
		$ mkdir proc
You will also need to acquire a /dev/tree from somewhere, or use mknod to create a few required entries. This should set up an appropriate root filesystem to start with. Boot the kernel built in phase 1 using this root filesystem (the details of doing this are outside the scope of this document but may be found in the Simple Linux/MIPS installation HOWTO). Don't forget to use the init=/bin/bash option, since init isn't set up or available yet. If all goes well, you should get a shell prompt. At this point I recommend you follow the basic directions on setting up your system to boot from its own disk as found in the aforementioned HOWTO. Note that you will need to obtain or write your own init scripts, and move /sbin/simpleinit to /sbin/init, and possibly do other things as well. Once your system is up and running from local disk, you should add a non-root account for yourself and give yourself a home directory with at least a few hundred MB free. When your non-root account is working, you're ready to proceed to phase 3.

Phase 3: The spit and polish
Nowhere else will you see critical system components like the init scripts termed "mere spit and polish." The parts of the system you have built so far are, if you're lucky, sufficient to have a working development environment - that is, to complete the bootstrap procedure. Thus your taska at this point involve building native versions of all the software you want to use. This is usually easier than cross-building, so I will not go into details here. Consult the documentation that comes with each piece of software you want to use.

( top ) ( home )