KBEA-00166 - Using eMake to Accelerate BitBake Builds

ElectricAccelerator uses the bemake wrapper utility to run Electric Make (eMake) in BitBake builds. This utility replaces GNU Make (gmake) with eMake do_compile build steps and runs gmake for other types of build steps.

Setting Up Your BitBake Build for Use with eMake

To set up your BitBake build for use with eMake:

  1. Set up your build environment by sourcing your BitBake environment setup script.

    For example, the following command sources a script named oe-init-build-env and specifies a build directory named rpi-test-build:

    source poky/oe-init-build-env rpi-test-build

    If you do not specify a build directory, it defaults to a directory named build in your current working directory.

  2. Choose or create a directory (such as your build directory) to contain eMake build assets.

    Examples of build assets are annotation files and history files.

  3. Update your BitBake conf/local.conf file with variables for eMake.

    This is explained in the “Setting the Required Variables in Your BitBake Configuration File” section below.

  4. (Optional) Create a custom blacklist or whitelist file in the eMake build asset directory.

    This is explained in “Creating a Custom Blacklist” and “Creating a Custom Whitelist” below.

Setting the Required Variables in Your BitBake Build Configuration File

To set the variables that eMake requires, you add them to your BitBake conf/local.conf file. This configuration file contains all the local user configurations for your BitBake build environment.

Setting the MAKE and EMAKE Variables

Set the MAKE variable in conf/local.conf to point to the ElectricAccelerator bemake executable, and also set the EMAKE variable to the same value. For example:

MAKE = "/opt/ecloud/i686_Linux/64/bin/bemake"
export EMAKE="/opt/ecloud/i686_Linux/64/bin/emake"

Setting the EMAKE_ASSET_BASE Variable

Set the EMAKE_ASSET_BASE variable to a directory of your choice. For example:

export EMAKE_ASSET_BASE = "/home/ubuntu/src/morty/rpi-test-build"

The conf/local.conf file is also the place to insert important eMake settings such as EMAKEFLAGS and EMAKE_ROOT. Note that the eMake asset directory might grow large with annotation files and cached build files.

Examples of Variables to Set

MAKE = "/opt/ecloud/i686_Linux/64/bin/bemake"
export EMAKE="/opt/ecloud/i686_Linux/64/bin/emake"
export EMAKE_ASSET_BASE = "/home/ubuntu/src/morty/rpi-test-build"
export EMAKEFLAGS = "--emake-jobcache=gcc --emake-resource=ubu1604"
export EMAKE_ROOT="/home/ubuntu/"
export BB_ENV_EXTRAWHITE += "EMAKEFLAGS EMAKE_ASSET_BASE EMAKE_ROOT"

(Optional) Selecting Packages to Build With eMake

Some packages do not build quickly or successfully with eMake for a variety of reasons, so you can still build them with gmake instead. You can specify these “gmake only” packages via blacklisting or whitelisting. These two methods are mutually exclusive, and you cannot combine them.

Creating a Custom Blacklist

Blacklisting means providing a list of packages that should not be built with eMake (and therefore must be built with gmake). Blacklisting is enabled for bemake by default. Examples of packages that are blacklisted by default are perl and glibc

The default list of blacklisted packages is at the head of the bemake script at /opt/ecloud/i686_Linux/64/bin/bemake. Do not edit this script.

To use your own blacklist (which ignores the default blacklist), create the following file:

$EMAKE_ASSET_BASE/bemake_package_blacklist

Each line in the file must be a regular expression as understood by the BASH =~ operator. The package names are automatically anchored at the start and end so that a regular expression such as bla does not match a package named blablah.

Creating a Custom Whitelist

Whitelisting means listing those packages that must be built by eMake so that any other packages will be built with gmake. Enabling whitelisting overrides any form of blacklisting completely.

To use your own whitelist:

  1. Create the following file:

    $EMAKE_ASSET_BASE/bemake_package_whitelist

  2. Add your entries to the file.

    For example, the following entries:

    linux.*
    openssh
    gcc

    will build only Linux kernel packages, openssh, and gcc with eMake. Note that the existence of this whitelist file causes the default blacklist entries and all other blacklist entries to be ignored.

Viewing bemake Annotation and Log Files

Annotation files for bemake are in $EMAKE_ASSET_BASE/<package_name>. The eMake JobCache and history files are also here.

The log file is $EMAKE_ASSET_BASE/bemake.log. This file shows the choice made for each package (bemake or gmake). bemake cannot know when a build starts or stops, so this log will grow with each build. Therefore, you must monitor its size.

Have more questions? Submit a request

Comments

Powered by Zendesk