How ubuntu server boots up: V init vs upstream

How many time did I boot my labtop today? Well, I booted up twice, usually I never move my ass out, but shockingly today I just did. Don’t get too curious about where did I go, because thats not the point, the point of curiosity is what happens when a ubuntu server boots up. Thats what we will try to figure out in this blog. Unlike any other Linux distributions, Ubuntu uses a different startup process for services, known as Upstart. As it has backward compatibility, the difference remain unnoticed most of the time.

When Ubuntu server starts up, first thing it does is it starts up GRUB boot loader. GRUB boot loader stays at least partially at the boot code on the Master Boot Record (the first 512 bytes of hard drive). It selects which Linux kernel the system will boot from and which options to uses when it boots.

When we look at /boot/grub/grub.cfg or /etc/default/grub, we see references of a program called update-grub. This is a helper program that automates the update of GRUB configuration file when new kernels are added. It executes number of configuration scripts stored at /etc/grub.d. When we select a kernel to boot from GRUB menu, it loads the kernel into memory along with its initrd file (initial RAM disk). The initrd file is actually a gzipped cpio archive known as an initramfs file under Ubuntu.  example gz file is initrd.img-2.6.32-14-generic-pae.

When a kernel boots, it needs to be able to at least mount the root file system so that it can access basic configuration files, kernel modules, and system binaries.

Now with the increase of hardware and supporting file systems, it makes sense to support them only if that is necessary. It keeps kernel smaller and flexible.

It needs to accessthe files to mount root file system. The initramfs file provides the kernel the essential kernel modules and system binaries it needs to have to mount theroot file system and complete the boot process.  Grub provide the information about root file system.

When kernel boots, it extracts the initramfs into RAM and runs a script called init. It basically creates some system mount points and mounts the actual root partition. Finally, after this init script has mountedthe real root file system, its last task is to run the /sbin/init program on theroot file system, which starts the next phase of the boot process.

The /sbin/init program is the parent process of every program running onthe system. This process always has a PID of 1 and is responsible for starting the rest of the processes that make up a running Linux system.

UNIX like OS has few standards to initialize, most of the known distributions are using System V init model but Ubuntu Server has switched to a systemknown as Upstart. Ubuntu still some features of System V init such as runlevels and /etc/rc?.d directories for backwardcompatibility. The good thing about upstart is that it manages everything under the hood.

In this V init system, different system states are  known as runlevels. When V init system starts it goes through the configuration file located at /etc/inittab and discovers its default runlevel. Then it enters to that runlevel and starts processes that has been configured to run at that runlevel. Runlevels are labeled by numbers ranging from 0 to 6. For an instance, runlevel 0 is reserved for a halted system state. When we enter runlevel 0, the system shuts down all running processes, unmounts all file systems, and powers off. Likewise, runlevel 6 is reserved for rebooting the machine. Runlevel 1 is reserved for single-user mode a state where only a single user can log in to the system with only few process running which comes very handy for digonosis. Even in the default GRUB menu you will notice a recovery mode option that boots you into runlevel 1.

Runlevels 2 through 5 are left for distributions and us to define. So we can create our own runlevels. Traditionally in Linux distributions one runlevel is allocated for graphical desktop (eg. runlevel 5 of Red Hat) and another runlevel for a system with no graphics (Eg. runlevel 3 of RedHat). User also has scope to create his own run level, for instance, maybe starting up a system without network access some time could come handy, so we can define it as a runlevel . In that case we need to pass an argument at boot prompt to override the default runlevel with desired runlevel. Once the system is booted, we can change the current runlevel with the init commands using sudo init 1.

/etc/init.d directory contains all of the start-up scripts of all services of all runlevels. This scripts usually contains start and stop commands.

After the choice of runlevel, init goes to /etc/rcS.d and runs each script that begins with an S in numerical order with start as an argument. Finally init is finished but stays running in the background, waiting for the runlevel to change.

init scripts has few draw backs for an instance, if a service dies in before completing the task it does not automatically starts the process. So we need another tool to monitor this process succeedeed or not. Init scripts is are generally affected by either change in runlevel or when the system starts up but some reason not executed. A perfect example would be Init scripts that depend on a network connection. On Ubuntu the init script that establishes the network connection is called networking. As we know it follow a neurmaric sequence, any init scripts that depend on a network connection are named with a higher number than this init script to ensure they run after the networking script has run. Lets imagine a situation where you boot up your system at the time when your network cable is unplugged. So in V init system, network init will run and failed and other connection will time out one by one.

It was designed not only to address some of the shortcomings of the System V init process, but also to provide a more robust system for managing services. Upstart solves this problem because upstart is event driven. Upstart can be configured to take action based on those events. Some sample events might be system start-up, system shutdown, the Ctrl-Alt-Del sequence being pressed, the runlevel changing, or an Upstart script starting or stopping. Upstart also constantly monitors the system for certain events to occur, and when they do,

Upstart does not completely replace System V init, or the functionality of init and the /etc/inittab files or changes of runlevels, but instead more core functionality is being ported to Upstart scripts. The difference is that Upstart now starts and stops services when runlevels changes. Upstart script are defined with either the script or exec options. Exec option keeps track of its PID. The convention is to keep track of these PIDs in the /var/run/ directory. With the script option, Upstart treats the lines that follow as a shell script until it reaches the end script line. Upstart provides methods to check the status of Upstart jobs and start and stop them as appropriate. We can check the status, start, and stop Upstart scripts with the appropriately named status, start, and stop commands. For example we can use sudo /etc/init.d/networking status, ubuntu short hand of this command is sudo service networking status. To disable an init script from starting up, we need to use sudo update-rc.d -f servicename remove, and to enable a script we need to use sudo update-rc.d servicename defaults. When we need to write out own script, we should start from the sceleton provided by ubuntu at /etc/init.d/skeleton. init scripts reside in /etc/init.d and have symlinks in /etc/rc?.d directories, where ? is a number representing a runlevel. So when we create our own script we need to choose a value for rc?.d wisely. The lower the value early it runs. We need to be careful about the dependency.

In Ubunutu services are managed two ways, either through init scripts or using xinetd. Xinetd is an updated and resource efficient version of the classic inetd service. When a system boots a init scripted service starts, the service could sit idly for decades before it gets accessed, wasting valuable server resources. On the other hand xinetd listen to ports its child services uses. If a connection is made on one of the ports, xinetd will then spawn the service that corresponds to that port, and once the connection is finished, the service exits until it is needed again.

Thanks:
1. BRACU Ayesha Abed Library
2. Kyle Rankin and Benjamin Mako Hill
3. My boredom 😛