The Yocto Project

The Yocto Project is a more complex beast than Buildroot. Not only can it build toolchains, bootloaders, kernels, and root filesystems, as Buildroot can, but it can generate an entire Linux distribution for you, with binary packages that can be installed at runtime.

The Yocto Project is primarily a group of recipes, similar to Buildroot packages but written using a combination of Python and shell script, and a task scheduler called BitBake that produces whatever you have configured, from the recipes.

There is plenty of online documentation at https://www.yoctoproject.org/.

Background

The structure of the Yocto Project makes more sense if you look at the background first. Its roots are in OpenEmbedded, http://openembedded.org/ which, in turn, grew out of a number of projects to port Linux to various hand-held computers, including the Sharp Zaurus and Compaq iPaq. OpenEmbedded came to life in 2003 as the build system for those hand-held computers but quickly expanded to encompass other embedded boards. It was developed and continues to be developed by an enthusiastic community of programmers.

The OpenEmbedded project set out to create a set of binary packages using the compact .ipk format, which could then be combined in various ways to create a target system and be installed on the target at runtime. It did this by creating recipes for each piece of software and using BitBake as the task scheduler. It was, and is, very flexible. By supplying the right metadata, you can create an entire Linux distribution to your own specification. One that is fairly well known is The Ångström Distribution, http://www.angstrom-distribution.org, but there are many others.

At some time in 2005 Richard Purdie, then a developer at OpenedHand, created a fork of OpenEmbedded which had a more conservative choice of packages and created releases that were stable over a period of time. He named it Poky, after the Japanese snack (if you are worried about these things, Poky is pronounced to rhyme with hockey). Although Poky was a fork, OpenEmbedded and Poky continued to run alongside each other, sharing updates and keeping the architectures more or less in step. Intel brought out OpenedHand in 2008 and they transferred Poky Linux to the Linux Foundation in 2010 when they formed the Yocto Project.

Since 2010, the common components of OpenEmbedded and Poky have been combined into a separate project known as OpenEmbedded core, or just oe-core.

Therefore, the Yocto Project collects together several components, the most important of which are the following:

Strictly speaking, the Yocto Project is an umbrella for these sub-projects. It uses OpenEmbedded as its build system, and Poky as its default configuration and reference environment. However, people often use the term "the Yocto Project" to refer to the build system alone. I feel that it is too late for me to turn this tide, so for brevity I will do the same. I apologise in advance to the developers of OpenEmbedded.

The Yocto Project provides a stable base which can be used as it is or which can be extended using meta layers, which I will discuss later in this chapter. Many SoC vendors provide board support packages for their devices in this way. Meta layers can also be used to create extended, or just different, build systems. Some are open source, such as the Angstrom Project, others are commercial, such as MontaVista Carrier Grade Edition, Mentor Embedded Linux, and Wind River Linux. The Yocto Project has a branding and compatibility testing scheme to ensure that there is interoperability between components. You will see statements like Yocto Project Compatible 1.7 on various web pages.

Consequently, you should think of the Yocto Project as the foundation of a whole sector of embedded Linux, as well as being a complete build system in its own right. You may be wondering about the name, yocto. A yocto is the SI prefix for 10-24, in the same way that micro is 10-6. Why name the project yocto? It was partly to indicate that it could build very small Linux systems (although, to be fair, so can other build systems), but also, perhaps, to steal a march on the Ångström distribution which is based on OpenEmbedded. An Ångström is 10-10. That's huge, compared to a yocto!

Stable releases and support

Usually, there is a release of the Yocto Project every six months, in April and October. They are principally known by the code name, but it is useful to know the version numbers of the Yocto Project and Poky as well. Here is a table of the four most recent releases at the time of writing:

Code name

Release date

Yocto version

Poky version

Fido

April 2015

1.8

13

Dizzy

October 2014

1.7

12

Daisy

April 2014

1.6

11

Dora

October 2013

1.5

10

The stable releases are supported with security and critical bug fixes for the current release cycle and the next cycle, that is approximately twelve months after release. No toolchain or kernel version changes are allowed for these updates. As with Buildroot, if you want continued support, you can update to the next stable release or you can backport changes to your version. You also also have the option of commercial support for periods of several years with the Yocto Project from operating system vendors such as Mentor Graphics, Wind River, and many others.

Installing the Yocto Project

To get a copy of the Yocto Project, you can either clone the repository, choosing the code name as the branch which is fido in this case:

$ git clone -b fido git://git.yoctoproject.org/poky.git

You can also download the archive from http://downloads.yoctoproject.org/releases/yocto/yocto-1.8/poky-fido-13.0.0.tar.bz2.

In the first case, you will find everything in the poky directory, in the second case, poky-fido-13.0.0/.

In addition, you should read the section titled System Requirements from the Yocto Project Reference Manual (http://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#detailed-supported-distros) and, in particular, you should make sure that the packages listed there are installed on your host computer.

Configuring

As with Buildroot, let's begin with a build for the ARM QEMU emulator. Begin by sourcing a script to set up the environment:

$ cd poky
$ source oe-init-build-env

That creates a working directory for you named build and makes it the current directory. All of the configuration, intermediate, and deployable files will be put in this directory. You must source this script each time you want to work on this project.

You can choose a different working directory by adding it as a parameter to oe-init-build-env, for example:

$ source oe-init-build-env build-qemuarm

That will put you into the build-qemuarm directory. You can then have several projects on the go at the same time: you choose which one you want to work with through the parameter to oe-init-build-env.

Initially, the build directory contains only one subdirectory named conf, which contains the configuration files for this project:

For now, we just need to set the MACHINE variable in local.conf to qemuarm by removing the comment character at the start of this line:

MACHINE ?= "qemuarm"

Building

To actually perform the build, you need to run bitbake, telling it which root filesystem image you want to create. Some common images are as follows:

By giving BitBake the final target, it will work backwards and build all the dependencies first, beginning with the toolchain. For now, we just want to create a minimal image to see whether or not it works:

$ bitbake core-image-minimal

The build is likely to take some time, maybe more than an hour. When it is complete, you will find several new directories in the build directory including build/downloads, which contains all the source downloaded for the build, and build/tmp which contains most of the build artifacts. You should see the following in tmp:

Running

When you build a QEMU target, an internal version of QEMU is generated, which removes the need to install the QEMU package for your distribution and thus avoids version dependencies. There is a wrapper script named runqemu for this internal QEMU.

To run the QEMU emulation, make sure that you have sourced oe-init-build-env and then just type:

$ runqemu qemuarm

In this case, QEMU has been configured with a graphic console so that the boot messages and login prompt appear in the black framebuffer screen:

Running

You can log on as root, without a password. You can close down QEMU by closing the framebuffer window. You can launch QEMU without the graphic window by adding nographic to the command line:

$ runqemu qemuarm nographic

In this case, you close QEMU using the key sequence Ctrl + A + X.

The runqemu script has many other options, type runqemu help for more information.

Layers

The metadata for the Yocto Project is structured into layers, by convention, each with a name beginning with meta. The core layers of the Yocto Project are as follows:

The list of layers in which BitBake searches for recipes is stored in <your build directory>/conf/bblayers.conf and, by default, includes all three layers mentioned in the preceding list.

By structuring the recipes and other configuration data in this way, it is very easy to extend the Yocto Project by adding new layers. Additional layers are available from SoC manufacturers, the Yocto Project itself, and a wide range of people wishing to add value to the Yocto Project and OpenEmbedded. There is a useful list of layers at http://layers.openembedded.org. Here are some examples:

  • meta-angstrom: The Ångström distribution
  • meta-qt5: Qt5 libraries and utilities
  • meta-fsl-arm: BSPs for Freescale ARM-based SoCs
  • meta-fsl-ppc: BSPs for Freescale PowerPC-based SoCs
  • meta-intel: BSPs for Intel CPUs and SoCs
  • meta-ti: BSPs for TI ARM-based SoCs

Adding a layer is as simple as copying the meta directory into a suitable location, usually alongside the default meta layers, and adding it to bblayers.conf. Just make sure it is compatible with the version of the Yocto Project you are using.

To illustrate the way layers work, let's create a layer for our Nova board which we can use for the remainder of the chapter as we add features. Each meta layer has to have at least one configuration file, conf/layer.conf, and should also have a README file and a license. There is a handy helper script that does the basics for us:

$ cd poky
$ scripts/yocto-layer create nova

The script asks for a priority, and if you want to create sample recipes. In the example here, I just accepted the defaults:

Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n]
Would you like to have an example bbappend file created? (y/n) [default: n]
New layer created in meta-nova.
Don't forget to add it to your BBLAYERS (for details see meta-nova\README).

That will create a layer named meta-nova with a conf/layer.conf, an outline README and a MIT license in COPYING.MIT. The layer.conf file looks like this:

# We have a conf and classes directory, add to BBPATH
BBPATH .= ":${LAYERDIR}"

# We have recipes-* directories, add to BBFILES
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
${LAYERDIR}/recipes-*/*/*.bbappend"

BBFILE_COLLECTIONS += "nova"
BBFILE_PATTERN_nova = "^${LAYERDIR}/"
BBFILE_PRIORITY_nova = "6"

It adds itself to BBPATH and the recipes it contains to BBFILES. From looking at the code, you can see that the recipes are found in the directories with names beginning recipes- and have file names ending in .bb (for normal BitBake recipes), or .bbappend (for recipes that extend existing normal recipes by adding and overriding instructions). This layer has the name nova which is added to the list of layers in BBFILE_COLLECTIONS and it has a priority of 6. The layer priority is used if the same recipe appears in several layers: the one in the layer with the highest priority wins.

Since you are about to build a new configuration, it is best to begin by creating a new build directory named build-nova:

$ cd ~/poky
$ . oe-init-build-env build-nova

Now you need to add this layer to your build configuration, conf/bblayers.conf:

LCONF_VERSION = "6"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /home/chris/poky/meta \
  /home/chris/poky/meta-yocto \
  /home/chris/poky/meta-yocto-bsp \
  /home/chris/poky/meta-nova \
  "
BBLAYERS_NON_REMOVABLE ?= " \
  /home/chris/poky/meta \
  /home/chris/poky/meta-yocto \"

You can confirm that it is set up correctly by using another helper script:

$ bitbake-layers show-layers
layer                 path                     priority
==========================================================
meta              /home/chris/poky/meta            5
meta-yocto        /home/chris/poky/meta-yocto      5
meta-yocto-bsp    /home/chris/poky/meta-yocto-bsp  5
meta-nova         /home/chris/poky/meta-nova       6

There you can see the new layer. It has priority 6 which means that we could override recipes in the other layers, which have a lower priority.

At this point it would be a good idea to run a build, using this empty layer. The final target will be the Nova board but, for now, build for a BeagelBone Black by removing the comment before MACHINE ?= "beaglebone" in conf/local.conf. Then, build a small image using bitbake core-image-minimal as before.

As well as recipes, layers may contain BitBake classes, configuration files for machines, distributions, and more. I will look at recipes next and show you how to create a customized image and how to create a package.

BitBake processes metadata of several different types, which include the following:

A recipe is a collection of tasks written in a combination of Python and shell code. The tasks have names like do_fetch, do_unpack, do_patch, do_configure, do_compile, do_install, and so on. You use BitBake to execute these tasks.

The default task is do_build, so that you are running the build task for that recipe. You can list the tasks available in a recipe by running bitbake core-image-minimal like this:

$ bitbake -c listtasks core-image-minimal

The -c option allows you to specify the task, missing off the do_ part. A common use is -c fetch to get the code needed by a recipe:

$ bitbake -c fetch busybox

You can also use fetchall to get the code for the target and all the dependencies:

$ bitbake -c fetchall core-image-minimal

The recipe files are are usually named <package-name>_version.bb. They may have dependencies on other recipes, which would allow BitBake to work out all the subtasks that need to be executed to complete the top level job. Unfortunately, I don't have the space in this book to describe the dependency mechanism, but you will find a full description in the Yocto Project documentation.

As an example, to create a recipe for our helloworld program in meta-nova, you would create a directory structure like this:

meta-nova/recipes-local/helloworld
├── files
│   └── helloworld.c
└── helloworld_1.0.bb

The recipe is helloworld_1.0.bb and the source is local to the recipe directory in the subdirectory files. The recipe contains these instructions:

DESCRIPTION = "A friendly program that prints Hello World!"
PRIORITY = "optional"
SECTION = "examples"

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/GPL-2.0;md5=801f80980d171dd6425610833a22dbe6"

SRC_URI = "file://helloworld.c"
S = "${WORKDIR}"

do_compile() {
  ${CC} ${CFLAGS} -o helloworld helloworld.c
}

do_install() {
  install -d ${D}${bindir}
  install -m 0755 helloworld ${D}${bindir}
}

The location of the source code is set by SRC_URI: in this case it will search directories, files, helloworld, and helloworld-1.0 in the recipe directory. The only tasks that need to be defined are do_compile and do_install, which compile the one source file simply and install it into the target root filesystem: ${D} expands to the staging area of the target device and ${bindir} to the default binary directory, /usr/bin.

Every recipe has a license, defined by LICENSE, which is set to GPLv2 here. The file containing the text of the license and a checksum is defined by LIC_FILES_CHKSUM. BitBake will terminate the build if the checksum does not match, indicating that the license has changed in some way. The license file may be part of the package or it may point to one of the standard license texts in meta/files/common-licenses, as is the case here.

By default, commercial licenses are disallowed, but it is easy to enable them. You need to specify the license in the recipe, as shown here:

LICENSE_FLAGS = "commercial"

Then, in your conf/local.conf, you would explicitly allow this license, like so:

LICENSE_FLAGS_WHITELIST = "commercial"

To make sure that it compiles correctly, you can ask BitBake to build it, like so:

$ bitbake  helloworld

If all goes well, you should see that it has created a working directory for it in tmp/work/cortexa8hf-vfp-neon-poky-linux-gnueabi/helloworld/.

You should also see there is an RPM package for it in tmp/deploy/rpm/cortexa8hf_vfp_neon/helloworld-1.0-r0.cortexa8hf_vfp_neon.rpm.

It is not part of the target image yet, though. The list of packages to be installed is held in a variable named IMAGE_INSTALL. You can append to the end of that list by adding this line to your conf/local.conf:

IMAGE_INSTALL_append = " helloworld"

Note that there has to be a space between the first double quote and the first package name. Now, the package will be added to any image that you bitbake:

$ bitbake core-image-minimal

If you look in tmp/deploy/images/beaglebone/core-image-minimal-beaglebone.tar.bz2 you will see that /usr/bin/helloworld has indeed been installed.

Customizing images via local.conf

You may often want to add a package to an image during development or tweak it in other ways. As shown previously, you can simply append to the list of packages to be installed by adding a statement like this:

IMAGE_INSTALL_append = " strace helloworld"

It should be no surprise that you can also do the opposite: you can remove a package using this syntax:

IMAGE_INSTALL_remove = "someapp"

You can make more sweeping changes via EXTRA_IMAGE_FEATURES. There are too many to list here, I recommend you look at the Image Features section of the Yocto Project Reference Manual and the code in meta/classes/core-image.bbclass. Here is a short list which should give you an idea of the features you can enable:

Writing an image recipe

The problem with making changes to local.conf is that they are, well, local. If you want to create an image that is to be shared with other developers, or to be loaded onto a production system, then you should put the changes into an image recipe.

An image recipe contains instructions about how to create the image files for a target, including the bootloader, the kernel, and the root filesystem images. You can get a list of the images that are available by using this command:

$ ls meta*/recipes*/images/*.bb

The recipe for core-image-minimal is in meta/recipes-core/images/core-image-minimal.bb.

A simple approach is to take an existing image recipe and modify it using statements similar to those you used in local.conf.

For example, imagine that you want an image that is the same as core-image-minimal but includes your helloworld program and the strace utility. You can do that with a two-line recipe file which includes (using the require keyword) the base image and adds the packages you want. It is conventional to put the image in a directory named images, so add the recipe nova-image.bb with this content in meta-nova/recipes-local/images:

require recipes-core/images/core-image-minimal.bb
IMAGE_INSTALL += "helloworld strace"

Now you can remove the IMAGE_INSTALL_append line from your local.conf and build it using:

$ bitbake nova-image

If you want to go further and take total control of the contents of the root filesystem, you can start from scratch with an empty IMAGE_INSTALL variable and populate it like this:

SUMMARY = "A small image with helloworld and strace packages" IMAGE_INSTALL = "packagegroup-core-boot helloworld strace"
IMAGE_LINGUAS = " "
LICENSE = "MIT"
IMAGE_ROOTFS_SIZE ?= "8192"
inherit core-image

IMAGE_LINGUAS contains a list of glibc locales to be installed in the target image. They can take up a lot of space so, in this case, we set the list to be empty, which is fine so long as we do not need locale-dependent library functions. IMAGE_ROOTFS_SIZE is the size of the resulting disk image, in KiB. Most of the work is done by the core-image class which we inherit at the end.

Creating an SDK

It is very useful to be able to create a standalone toolchain that other developers can install, avoiding the need for everyone in the team to have a full installation of the Yocto Project. Ideally, you want the toolchain to include development libraries and header files for all the libraries installed on the target. You can do that for any image using the populate_sdk task, as shown here:

$ bitbake nova-image -c populate_sdk

The result is a self-installing shell script in tmp/deploy/sdk named:

poky-<c_library>-<host_machine>-<target_image><target_machine>-toolchain-<version>.sh

Here is an example:

poky-glibc-x86_64-nova-image-cortexa8hf-vfp-neon-toolchain-1.8.1.sh

Note that, by default, the toolchain does not include static libraries. You can enable them individually by adding lines like this to your local.conf or the image recipe:

TOOLCHAIN_TARGET_TASK_append = " glibc-staticdev"

You can also enable them globally as shown:

SDKIMAGE_FEATURES_append = " staticdev-pkgs"

If you only want a basic toolchain with just C and C++ cross compilers, the C library and header files, you can instead run:

$ bitbake meta-toolchain

To install the SDK, just run the shell script. The default install directory is /opt/poky, but the install script allows you to change that:

$ tmp/deploy/sdk/poky-glibc-x86_64-nova-image-cortexa8hf-vfp-neon-toolchain-1.8.1.sh

Enter target directory for SDK (default: /opt/poky/1.8.1):

You are about to install the SDK to "/opt/poky/1.8.1". Proceed[Y/n]?

[sudo] password for chris:

Extracting SDK...done

Setting it up...done

SDK has been successfully set up and is ready to be used.

To make use of the toolchain, first source the environment set up script:

. /opt/poky/1.8.1/environment-setup-cortexa8hf-vfp-neon-poky-linux-gnueabi

Toolchains generated in this way are not configured with a valid sysroot:

$ arm-poky-linux-gnueabi-gcc -print-sysroot

/not/exist

Consequently, if you try to cross compile as I have shown in previous chapters, it will fail like this:

$ arm-poky-linux-gnueabi-gcc helloworld.c -o helloworld

helloworld.c:1:19: fatal error: stdio.h: No such file or directory

#include <stdio.h>

                   ^

compilation terminated.

This is because the compiler has been configured to be generic to a wide range of ARM processors, and the fine tuning is done when you launch it using the right set of gcc flags. So long as you use $CC to compile, everything should work fine:

$ $CC helloworld.c -o helloworld

License audit

The Yocto Project insists that each package has a license. A copy of the license is in tmp/deploy/licenses/[packagenam.e] for each package, as it is built. In addition, a summary of the packages and licenses used in an image are in the <image name>-<machine name>-<date stamp> directory. This is shown here:

$ ls tmp/deploy/licenses/nova-image-beaglebone-20151104150124
license.manifest  package.manifest

The first file lists the licenses used by each package, the second lists the package names only.