KBEA-00098 - When to use each of the eMake history file options

Summary

eMake supports three different options for --emake-history. This article provides guidelines for when to use each history file option.

Solution

eMake uses a history file in order to track dependencies between tasks for a build, using information gathered from previous builds to build successfully in parallel in later builds. Several different eMake options are available to manipulate the history file; the Electric Make Users Guide provides the following descriptions:

  1. Merge - By default, Electric Make merges any new dependencies it discovers into the existing history file. In this way, the history file automatically evolves as your makefiles change, learning dependencies that accelerate the builds you run.
  2. Create - If --emake-history is set to create, the old history file contents are completely overwritten by new dependencies discovered in the run that just completed. Use this setting to start a fresh history file, effectively eliminating any stale information from the file.
  3. Read - If --emake-history is set to read, no data is written to the history file at build completion, and any new dependencies Electric Make discovered are discarded. Use this setting when you are sharing a single, static copy of the history file between developers.

In all three cases, eMake reads the history file (if it exists), but the differences among the three options revolve around what is written back to the history file (if anything). The behavior of each can be leveraged differently, depending on your use case.

Per-User History Files [ec:merge]

This is the default scenario, where all developers maintain their own builds and their own history files. It is usually recommended for most environments, although this article describes several other instances where it may not be suitable. With per-history history files, merge is used to iteratively build them, allowing for accelerated builds within a relatively stable development environment. However, because eMake's history file maintains implicit dependencies that are rarely purged automatically, dependency changes within the development environment can lead to serialization over time, which is when a history file purge can be useful (described below).

Centrally Maintained History Files [ec:read]

With centrally maintained history files, a designated group (generally configuration management and/or release engineering) generates history files and stores them on a central server. From there, automated mechanisms (such as scripts) select appropriate history files for users in the read mode (so that individual users do not update the shared files). Depending on how the history files are generated, they might be subject to the same serialization issues as per-user history files, so history file purges (described below) may also be occasionally useful.

Hybrid History Files [ec:merge]

A hybrid strategy of the per-user history files and centrally maintained history files can be used for improved performance and flexibility, at the cost of increased complexity. As in the centrally maintained history file approach, a designated group generates history files and distributes them. But rather than leaving the history file remote as in the centrally maintained case, those history files are copied locally and modified by the user with the default merge option.

Developer Preflight Builds [ec:read]

If a centralized build infrastructure is available for developer preflight builds, history files can minimize the amount of time each build occupies on that infrastructure. But, the different preflight builds should not affect each other's dependency trees, so read is the preferred mode here (with routine history file updates).

Excessive Serialization/History File Purge [ec:create]

As a development environment evolves over time, dependencies are added and removed. merge handles the dependency additions, but dependency removals can cause unnecessary serialization. For example, if target A depends upon target B, and a history file is generated, even if that dependency is removed, as long as the history file is only updated via merge, A will continue to wait for B to complete before running. If too many of these false dependency changes are accumulate, your build times may be impacted negatively.

The recommended way to regenerate your history file is to use the create option, rather than by deleting your history file. This way, your eMake run that regenerates the history file can take advantage of any history you have accumulated thus far, allowing for some parallelization that would not be available otherwise.

Dependency churn is generally signaled by Makefile changes, but there is no simple "one size fits all" approach. However, developers are usually aware of major dependency removals, which are not automatically detected by eMake, so developer input in purging the history file can be useful. In addition, regenerating the history file on a semi-regular basis is a viable option, particularly if you experience severe Makefile churn, because create will leverage your existing history file.

Shifting emake Roots [ec:create]

This is an uncommon occurrence, but may impact some users.

History files contain paths to targets relative to the emake root. As such, if the emake root and source trees change locations relative to each other, the history file should be regenerated with --emake-history=create. For example, if project A were refactored into projects A and B and moved beneath directory X, and the emake root moved up a directory from A to X (so that B would now fall under the emake root as well), the old history file would no longer be valid and would need to be recreated.

Outdated/Corrupt History File [ec:create]

This is an uncommon occurrence, but may impact some users.

eMake may complain about the history file being unusable (such as eMake errors EC1038 through EC1045), in which case the history file should be regenerated with --emake-history=create.

Applies to

  • Product versions: All
  • OS versions: All
Have more questions? Submit a request

Comments

Powered by Zendesk