KBEA-00140 - Managing History Files

Summary

What is a History File?

• A record of actual dependencies discovered during a build
• Contains missing dependencies that could break a non-emake parallel build

          Makefile + History = Complete set of dependencies for a build

• Its a performance optimization for subsequent builds
• Allows Accelerator to avoid conflicts

 Accelerator deduces dependencies on-the-fly: watching all file accesses and detecting out-of-order (e.g. write after read) steps. These are called conflicts. Accelerator will detect and correct conflicts and record the discovered dependencies in the history file.

 The History File is a readable (text) file, but in a special format. It's capable of recording simple dependencies AND dependencies across make instances. It's senstive to changes in the makefile structure, not source code changes.

Solution

Managing the History File

Emake command line options:

• --emake-historyfile=<pathname>
• --emake-history=[read|create|merge]

              read - existing history information is used, but new dependency information is discarded

              create - existing history information is used, new dependency information overwrites an existing history file.

              merge - existing history information is used, new dependency information is merged into the existing history file.

Models for History File Maintenance

1. Per-User

•   Everyone makes their own history file, no sharing
•   Same for production and developer builds
•   Each user manages the storage themselves
•   Builds run with merge mode
•   This is the default configuration
•   emake.data is written where emake is invoked and will merge history
•   This was the model we originally had in mind when designing the history mechanism
  

Pros

• Easy, no overhead for IT/Release
• Just works out of the box

Cons

• Users may need to manage multiple history files
• Duplicated work: each user has to create history
  

2. Centrally Maintained

•  History is created by production and made centrally available
•  Shared file system
•  Stored on the Cluster Manager server
•  Using Commander, can be made a property
•  Users build using scripts or automated process that chooses an appropriate history file
•  Builds run with read mode
•  Completely hidden from users

Pros

• Users dont need to understand history
• Reduces duplicated work

Cons

• Have to design a mechanism to match history files to builds
• Slow to respond to source changes that require history updates (only updates when a new file is distributed)

3. Hybrid

Various mixes of the above models

• A reference history file is created by production (e.g. nightly) build
• Users get reference history file as needed and update locally on each build
• Users run in merge mode
• Production builds also run in merge mode and fresh history files are made available

Pros

• Best performance
• Flexibility

Cons

• Configuration effort is shared between users and Release
  

General Guidelines

• Small deltas
• When possible, let history live where the development happens
• Continuous feedback
• Accelerator is designed to learn from previous builds
• Possibly use history from CI build for production

Applies to

• Product versions: All
• OS versions: All