KBEC-00038 - Improving CloudBees Flow server performance by tuning memory allocation

Description

CloudBees Flow is distributed to work with a minimum memory configuration of 2 GB. Most modern server configurations have the ability to add more memory, which CloudBees Flow can be tuned to use. Also, you can tune CloudBees Flow to use less memory to prevent disk thrashing that is associated with excess paging on a 1 GB memory configuration. You can tune your configuration's memory to improve CloudBees Flow server performance.

 

Java (JRE) solution

The CloudBees Flow server is a Java application. As the CloudBees Flow load increases, the small, default Java heap size is exhausted, which results in the following commander.log file message:

When the CloudBees Flow server starts, a configuration file specifies the default Java heap size. On 32-bit systems, heap size can be set to a maximum of 1500 MB. If you have a large memory machine, increase maxmemory to the 1500 MB maximum. On small memory machines, specify a memory allocation as low as 256 MB to prevent thrashing.

Here is a list of JVM memory considerations:

  1. Out of the box, the CloudBees FlowJVM heap size is configured to use 25% of the physical memory on the machine.
  2. In addition to the heap size, there's memory used by code, stacks, threads, etc., so the grand total usage may exceed 25%.
  3. Once the JVM has allocated memory from the OS, it never gives it back.
  4. CloudBees Flow will configure it to allocate the 25% right away. Growing the heap can be expensive, and this is a server product, so that makes sense as a default.
  5. The JVM will tend to accumulate garbage until it nears the heap size limit, at which time it will garbage collect.
  6. Because of this behavior, it often looks like the JVM is using more memory than it actually is, especially if you're looking at the virtual size of the process, as opposed to the RSS/working set.
  7. On an 8GB machine, a 2.x GB is expected, based on the 25% number above.

After changing the configuration file, restart the CloudBees Flow server. On Linux, use the script "commander restart" in /etc/init.d. On Windows, use the Control Panel, Services tab to restart the CloudBees Flow server.

For version 8.5.0 and above, the installer will recognize what type of install is taking place and will apply the following settings:

For the standalone agent install the memory values will be:

  • init: 16MB
  • max: 64MB

If any other components (server, web server, repository) is also installed, then memory values will be:

  • init: 256MB
  • max: 512MB

Java configuration example

The Java heap size attribute is located in the wrapper.conf file within the CloudBees Flow server/conf directory.

As distributed, the Java heap size is a percentage of memory.

Notes:

  1. You can configure the % or MB via ecconfigure and we should persist these values across an upgrade.

    ./ecconfigure --serverMaxMemory 30
    ./ecconfigure --serverMaxMemoryMB 1000
  2. The wrapper memory values are a percentage of memory instead of a fixed memory value. This facilitates an easy general purpose install. On enterprise production CloudBees Flow installs, it is best to place a physical number for the Java memory allocation.
  3. On Linux use the -Xmx command to determine the maximum memory setting of Java.
    java -Xmx2500m # gives usage page (which is OK)
    java -Xmx3000m # produces error message "The specified size exceeds the maximum representable size"


  4. The number of threads allocated in the server varies based on the number of cores in the system and how thread stacks consume address space.

Here is the extract of text from the file wrapper.conf where the changes are to be made:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=512

# Maximum Java Heap Size (in MB)
# If this is set too high, java will not run at all and the
# Commander server will not start. At least for 32-bit Windows,
# the limit is about 1500 MB.
wrapper.java.maxmemory=512

32-bit systems notes

There are memory limitations considerations and it can be expanded to a maximum of 1500 MB on Windows and perhaps more on large memory 32-bit Linux machines or decreased to 256 MB for small memory machines

Systems with many cores (say 8 or more), lower the maximum heap size on the 32-bit JVM by around 100 MB to prevent the JVM from running out of address space when there really is enough memory available. On Windows, this means lower it to 1400 MB and on Linux lower it to 2300 MB.

 

 

Have more questions? Submit a request

Comments

Powered by Zendesk