Building tiny systems with embedded NetBSD By Brian Rose Overview NetBSD is an extremely flexible operating system that is designed to be portable across various architectures. This feature makes it attractive for embedded developers. In this article, I will demonstrate a process for creating a very small kernel that can boot, either to a shell prompt or to a login screen. Booting to the shell prompt allows the developer to quickly give life to a system and perform some basic interactions. The shell itself can be a powerful management tool, combined with the right collection of programs. Doing this requires only the kernel and two binaries. Booting to a login prompt provides a little added security by controlling access to the machine. This would be useful in devices where the console can be accessed by the user, and you need to control what the user has access to. There are a few added files in this setup, and to keep things small, you will need to manually trim some of them. But this is still a very easy process. The minimum single user system The first stage of embedded development usually involves getting a skeleton operating system up and running. Assuming you are using one of NetBSD many supported platforms, this is quite easy. I used a PC booting off of a floppy drive. But this method can also be easily adapted to handle diskless clients that boot off a network. At an absolute minimum, you will need the kernel, the /dev nodes for your system, and a /sbin/init file. You can craft the init program yourself, putting all of your application code into it. This may be a good solution when you only have one application that you would like to run. Another option is to use the stock init program and a shell to call your application. This is a bit easier than writing your own init program, and it also allows you to add functionality to your system by simply adding more tools from the stock system. The shell itself is a powerful tool that can make future development much easier. This is the method that will be illustrated here. Building the kernel The first thing that needs to be done is to build a kernel with a built in ramdisk. The ramdisk will hold the kernel's root filesystem. A good kernel configuration to start with is the INSTALL_TINY configuration. This kernel has all the important stuff in it, without a lot of bloat. Make sure that it has the following lines in it. in sys/arch/[arch]/conf $ cp INSTALL_TINY MYTINY $ vi MYTINY # Enable the hooks used for initializing the root memory-disk. options MEMORY_DISK_HOOKS options MEMORY_DISK_IS_ROOT # force root on memory disk options MEMORY_DISK_SERVER=1 # make the ramdisk writeable #options MEMORY_DISK_ROOT_SIZE=2880 # 1.44M, same as a floppy options MEMORY_DISK_ROOT_SIZE=8192 # 4Meg options MEMORY_RBFLAGS=0 # don't force single user This configuration includes ramdisk support, has the kernel boot from the ramdisk, allows the ramdisk to be written to, sets the size a 8192 sectors (512 bytes each), and allows the kernel to boot into multiuser mode (if the files are available). You will also need to check the ramdisk driver to make sure that it supports the MEMORY_RBFLAGS option. In my 1.6.0 system, it did not, so I needed to add the following code. in sys/dev/md_root.c /* * This is called during open (i.e. mountroot) */ ## #ifndef MEMORY_RBFLAGS ## #define MEMORY_RBFLAGS RB_SINGLE ## #endif void md_open_hook(int unit, struct md_conf *md) { // ............................... added code if (unit == 0 && (MEMORY_RBFLAGS & RB_SINGLE) ) { /* The root ramdisk only works single-user. */ boothowto |= RB_SINGLE; } } Once this is done, build your kernel. $ config MYTINY $ cp ../compile/MYTINY $ make depend; make Creating the crunched binaries Although you could copy the binaries from your host into your mini filesystem, a more efficient way (sizewise) to do this is to use a utility called crunchgen. Many programs in the NetBSD system are linked statically. For example, each program that uses the utility library (libutil) has one copy of the library linked to it. Several programs on the system produce redundant copies of the libraries used by the programs. Crunchgen takes the programs' object files and merges them into one uber-program. This crunched binary is then linked to the libraries, so only one copy of the library is needed for the whole system. "But how do I use the different programs?", you ask. The answer lies in hard links. A hard link is like a filesystem's alias for a program. For example, If I have a program called print_my_error, I can link that to the alias myerr. Then when I invoke myerr, the filesystem simply follows the link and runs the program print_my_error. I can even make the two program names have slightly different behavior. This is done by having print_my_error look at argv[0] that is passed to the main() function. If it is myerr, then I can have it do a special task. This is how crunchgen works. It takes all of its constituent programs and globs them together. Then in the crunched binary's main routine, there is logic that examines argv[0] and calls the main routine for the appropriate constituent. To make a crunched binary, you need to know what binaries you want, where their sources are located, and what libraries they use. Once you settle on what binaries to use, simply search the /usr/src folder for the sources. The layout of /usr/src is straightforward. To look for the source for /sbin/init, you would look in /usr/src/sbin/init. To find the /usr/bin/login, go to /usr/src/usr.bin/login. To find out which libraries are used, open up the program's Makefile, and look for the LDADD lines. You should see entries like "-lutil -lcrypt". Sometimes, programs in the NetBSD distribution are simply hardlinks to another program. For example, mount_mfs is an alias for newfs. You will find no /usr/src/sbin/mount_mfs directory. So how do you find out where the sources are for mount_mfs? Search through the makefiles for "LINKS= ${BINDIR}/newfs ${BINDIR}/mount_mfs". This shows you that the Makefile will link mount_mfs to newfs. $ find /usr/src -name Makefile -exec grep mount_mfs {} \; MAN= newfs.8 mount_mfs.8 LINKS= ${BINDIR}/newfs ${BINDIR}/mount_mfs MLINKS= mount_mfs.8 mfs.8 Armed with this information, you can now create your crunchgen configuration file. This is just a list of the above information. There are some additional features, which I will outline below. srcdirs /usr/src/bin /usr/src/sbin progs init sh reboot ls ln sh -sh special init objpaths /usr/src/sbin/init/obj/init.smallprog.o # libraries used by the programs # ---------------- Minimum single user files # init : -lutil -lcrypt # sh : -ll -ledit -ltermcap # ---------------- Useful utilities # ls : - # reboot : -lutil # libs -lutil -lcrypt -ll -ledit -ltermcap The first line tells crunchgen where to look for sources. It looks for the sources by appending the program name onto the listed directories. The progs line is the list of programs that you want included in your binary. The ln lines tell crunchgen about aliases that are used for some of the programs. The shell is sometimes invoked as -sh, so the crunched binary will recognize "-sh" as "sh". Also, as noted above, some binaries are simply aliases for other programs. The crunched binary needs to be on the lookout for these aliases as well. The special line tells crunchgen that init should not be built from source, but rather just use the specified object file. In this case, I built the init program with the SMALLPROG #define, so I don't get the annoying "Enter pathname of shell or RETURN for sh:" prompt when the system starts in single user mode. Instead, it will silently drop to the shell prompt. To do this, I build my special init program this way. $ cd /usr/src/sbin/init $ make -D SMALLPROG $ cp obj/init.o obj/init.smallprog.o This functionality makes it easy to customize your system without too much trouble. You could create a customized version of a standard program and use that in your embedded builds, while keeping the original for other purposes. Finally, the last line tells crunchgen which libraries to link to. Again, you can get this information by looking through the Makefiles for the constituent programs and noting the LDADD lines. I've had trouble with library ordering, so if you have errors indicating unresolved externals, and you know you included the library, try moving the "missing" library closer to the front of the list. This worked for me. Once your crunchgen file is crafted, build it with the following commands. The finished executable will be named after your conf file, but without the conf extension. $ crunchgen -m Makefile mytiny.conf $ make -f Makefile objs exe Continue to page 2: Creating the filesystem image and the minimum multi user system. Building tiny systems with embedded NetBSD (page 2) Back to page one Creating the filesystem image Now that our binaries are all wrapped up, the next thing to do is to populate the filesystem. This involves linking all the programs to the crunched binary and copying or creating the necessary data files. This is easily done with the following commands. Don't forget to set the permissions to root:wheel! $ mkdir files/bin files/sbin files/dev $ cp /dev/MAKEDEV files/dev $ cd files/dev $ ./MAKEDEV floppy ramdisk wscons $ cd ../.. $ cp work/mytiny files/sbin/init $ ln files/sbin/init files/bin/sh $ ln files/sbin/init files/bin/ls $ ln files/sbin/init files/sbin/reboot $ su root # chown -R root:wheel files # exit With the filesystem populated, we can wrap it up into an image file that can be embedded into a kernel. This is done with the makefs command. This tool lets you take a directory and bundle it up into a single file. Make sure that your size is not larger than the size you specified in your kernel configuration. Once the image is created, link it to the kernel with the mdsetimage command. Your kernel is now ready to go! You can compress your kernel if you like. The standard NetBSD bootloader knows how to decompress gzipped kernels. Remember, your spartan OS includes 4 megabytes of mostly empty space in the filesystem. I compressed my kernel down to 825,544 bytes. # makefs -s 4m -t ffs crunch.image files # mdsetimage netbsd.ramdisk crunch.image # gzip -c netbsd.ramdisk > netbsd # ls -l netbsd -rw-r--rw- 1 brose users 825544 Aug 24 22:28 netbsd When you boot this kernel, you will see the normal output and then it will present you with a shell prompt. At that time, you can do whatever it is you need to do. My example will let you move around the file structure, list the contents with ls, and reboot. Not very useful, but it is a good starting point. The minimum multi user system A multi user system is built in the same manner, but you need a few extra programs and data files. I also use the standard init program. Notice that I commented out the special init line, so that I use the stock init for the multiuser configuration. You may want to go back to your init source and make a fresh init, just in case the .o files are from your single user build. $ cd /usr/src/sbin/init $ make clean; make Here's my crunchgen configuration. srcdirs /usr/src/bin /usr/src/sbin /usr/src/usr.bin /usr/src/usr.sbin /usr/src/libexec progs init mount newfs mount_ffs sh ttyflags getty pwd_mkdb passwd login reboot ls ln sh -sh ln newfs mount_mfs # special init objpaths /usr/src/sbin/init/init.smallprog.o # libraries used by the programs # ---------------- Minimum single user files # init : -lutil -lcrypt # mount : # newfs : -lutil # mount_ffs : # sh : -ll -ledit -ltermcap # ---------------- Minimum multiuser files # ttyflags : # getty : -lutil -ltermcap # pwd_mkdb : -lutil # passwd : -lrpcsvc -lcrypt -lutil -lkrb5 -lcrypto -lasn1 -lcom_err -lroken # login : -lutil -lcrypt -lskey -lkrb5 -lasn1 -lkrb -lcrypto -lroken -lcom_err # ---------------- Useful utilities # ls : # reboot : -lutil # umount : # libs -lutil -ll -ledit -ltermcap -lcrypt -lrpcsvc -lkrb5 -lkrb -lcrypto -lasn1 -lcom_err -lroken -lskey And my shell commands for populating the filesystem. mkdir files/bin files/sbin files/usr files/etc files/var files/dev files/tmp files/root files/home mkdir files/usr/bin files/usr/sbin files/usr/libexec mkdir files/var/run files/var/db files/var/crash mkdir files/usr/share mkdir files/usr/share/misc cp /dev/MAKEDEV files/dev cd files/dev ./MAKEDEV floppy ramdisk wscons cd ../.. echo "/dev/md0a / ffs rw 1 1" > files/etc/fstab echo "echo Initializing system..." > files/etc/rc echo "export PATH=/sbin:/bin:/usr/sbin:/usr/bin" >> files/etc/rc echo "mount -ua" >> files/etc/rc echo "ttyflags -a" >> files/etc/rc cp work/mytiny files/sbin/init ln files/sbin/init files/bin/ls ln files/sbin/init files/sbin/mount ln files/sbin/init files/sbin/mount_ffs ln files/sbin/init files/sbin/mount_mfs ln files/sbin/init files/sbin/umount ln files/sbin/init files/bin/sh ln files/sbin/init files/usr/libexec/getty ln files/sbin/init files/sbin/ttyflags ln files/sbin/init files/sbin/pwd_mkdb ln files/sbin/init files/usr/bin/passwd ln files/sbin/init files/usr/bin/login ln files/sbin/init files/sbin/reboot ln files/sbin/init files/sbin/newfs cp /etc/ttys files/etc cp /etc/master.passwd files/etc cp /etc/pwd.db files/etc cp /etc/spwd.db files/etc cp /etc/passwd files/etc cp termcap.mini files/usr/share/misc/termcap cp /etc/gettytab files/etc The termcap.mini file is simply a hand trimmed version of the termcap file. This is used by the getty program when initializing the console. Since it is very large (over 500k) and most of it is useless, you can trim out the terminal types that you don't use. I trimmed mine down to about 8k. Once you have done this, simply place this filesystem into a kernel and boot. As you can see, a multi user setup is a bit larger. But most of the extra space is from incorporating the extra libraries for the login and getty programs. As you add more code to the system, your code growth should not be as dramatic. # makefs -s 4m -t ffs crunch.image files # mdsetimage netbsd.ramdisk crunch.image # gzip -c netbsd.ramdisk > netbsd # ls -l netbsd -rw-r--rw- 1 brose users 1066057 Aug 24 22:28 netbsd Brian Rose is an electrical engineer who has developed embedded software for the telecom and video distribution industries. He is currently on involuntary hiatus (layoff) and pondering the benefits of being a scuba instructor at Mexican resorts. Copyright (C) 2003 by Brian Rose, all rights reserved. Reproduced by BSD Newsletter.com with permission of the author.