This KB article applies to ElectricAccelerator 11.0 and newer versions. For information about this functionality in 10.1, see KBEA-00166 - Using eMake to Accelerate BitBake Builds Using ElectricAccelerator 10.1.
ElectricAccelerator 11.0 provides a new command, ebitbake, which applies acceleration to Yocto builds. It provides caching for BitBake do_configure tasks and caching and build acceleration for do_compile steps.
Make sure that you have installed:
- ElectricAccelerator 11.0 or later
- ElectricAccelerator JobCache licenses for the number of agents to use in the build
- Yocto version between 2.2.4 (latest current Morty) and 2.6.0 (Thud)
(Optional) Optimizing the JobCache Hit Rate
You can increase the number of JobCache hits by creating certain directories before running ebitbake. The following example shows how to create these directories from inside the build directory:
mkdir -p tmp/sysroots/x86_64-linux/lib
mkdir -p tmp/sysroots/x86_64-linux/usr/include/sys
mkdir -p tmp/sysroots-uninative/x86_64-linux/usr/bin
mkdir -p tmp/sysroots/x86_64-linux/usr/bin/x86_64-linux
mkdir -p tmp/sysroots/x86_64-linux/usr/sbin
mkdir -p tmp/sysroots/x86_64-linux/sbin
mkdir -p tmp/sysroots/x86_64-linux/bin
Ensuring Consistent Environment Variables
Your environment variables should be exactly the same in successive builds with one exception: Paths in the eMake root are replaced with numbers that correspond to their position in the eMake root. This means that even logging in with a different terminal emulator can cause the cache to miss. For example, TERM=xterm-256color versus TERM=xterm.
Another important issue is the contents of the PATH environment variable. After you source poky/oe-init-build-env, you must prepend /opt/ecloud/i686_Linux/64/bin. This must be done the same way for every build to avoid an excessive JobCache miss rate.
Starting a Build
- Source the oe-init-build-env script as normal.
. poky/oe-init-build-env mybuilddir/
- Put ebitbake at the front of the path.
- Edit your BitBake conf/local.conf file to set the eMake environment variables.
This step must be be done only for your first build.
• EMAKEFLAGS must specify the Cluster Manager to use:
The host can be either an IP address or a machine name. For example:
• (Optional) EMAKE_ROOT is set by default to include the build directory and the poky directory, but you can change this variable in the file.
• (Optional) EMAKE_ASSET_BASE is set by default to be emakeassets within the build directory. This contains the cache data, annotation files, and debug logs. You can change this variable to point to another location in the file. Note that the eMake asset directory might grow large with annotation files and cached build files.
- Run your build as normal but replace the command bitbake with ebitbake.
The ebitbake command creates an eBitBake build ID (which is important later for reporting). For example, EBITBAKE_BUILD_ID = 99937022.
(Optional) Selecting Packages to Build With eMake
Some packages do not build quickly or successfully with eMake for a variety of reasons, so you can still build them with gmake instead. You can specify these “gmake only” packages via blacklisting or whitelisting. These two methods are mutually exclusive, and you cannot combine them.
Blacklisting means providing a list of packages that should not be built with eMake (and therefore must be built with gmake). Blacklisting is enabled for bemake by default. Examples of packages that are blacklisted by default are perl and glibc.
The default list of blacklisted packages is at the head of the bemake script at /opt/ecloud/i686_Linux/64/bin/bemake. Do not edit this script.
To use your own blacklist (which ignores the default blacklist), create the following file:
Each line in the file must be a regular expression as understood by the BASH =~ operator. The package names are automatically anchored at the start and end so that a regular expression such as bla does not match a package named blablah.
Whitelisting means listing those packages that must be built by eMake so that any other packages will be built with gmake. Enabling whitelisting overrides any form of blacklisting completely.
To use your own whitelist:
- Create the following file:
- Add your entries to the file.
For example, the following entries:
will build only Linux kernel packages, openssh, and gcc with eMake. Note that the existence of this whitelist file causes the default blacklist entries and all other blacklist entries to be ignored.
Using Job Caching and “Learning” Builds when the Build Directory is New Each Time
Some continuous integration systems create a new build directory for each build. You can use the cached files and eMake history from a previous build by setting the EMAKE_ASSET_BASE environment variable in your BitBake conf/local.conf file. You must do this before starting a learning build.
The steps are the same for starting a build, except that when editing conf/local.conf, you must also add a line similar to:
This location must be writable and must have enough space to store a large cache of the build outputs and the annotation files.
Note: This functionality is not suitable for simultaneous access by more than one build.
Viewing bemake Annotation and Log Files
Annotation files for bemake are in $EMAKE_ASSET_BASE/<package_name>. The JobCache and history files are also here.
The log file is $EMAKE_ASSET_BASE/bemake.log. This file shows the choice made for each package (bemake or gmake). bemake cannot know when a build starts or stops, so this log will grow with each build. Therefore, you must monitor its size.
Analyzing a Build Using ElectricInsight
Each BitBake package and task that is accelerated by ElectricAccelerator creates an eMake annotation file. You can combine these files and BitBake buildstats data into a single ”pseudo-annotation” file to create an annotation view of the entire build by using the bb2anno utility. You can then load this file into ElectricInsight (”Insight”) to run certain reports to see BitBake dependency information and to perform aggregate analysis.
The bb2anno utility is in the /opt/ecloud/i686_Linux/unsupported/ directory by default. Following is the bb2anno command usage:
bb2anno -id <ebitbake_build_ID> -annoroot <path_to_anno_files> -buildstats <path_to_buildstats> -buildlogs <path_build_logs>
The following example shows how to enter the bb2anno command to create a compressed pseudo-annotation file:
~/bb2anno -id 99937022 -annoroot emakeassets -buildstats tmp/buildstats/20181212102014/ -buildlogs tmp/work | gzip > combined_99937022.anno.gz
Using Insight to view a BitBake pseudo-annotation file requires Insight version 5.5 (which is bundled with Accelerator) and an Insight 5.5 license to generate reports from the file. For details about analyzing a build using Insight, see the ElectricInsight 5.5 User Guide at http://docs.electric-cloud.com/accelerator_doc/AcceleratorIndex.html.
Unsupported Yocto Project Modules
In ElectricAccelerator versions 11.0 and later, bemake cannot build the following modules:
ElectricAccelerator 11.0 and newer versions.