ElectricFlow Automation Release Process

Problem

This article discusses the following:

• Suggestions for implementing a release process for ElectricFlow automation projects.
• Issues that can affect Flow server performance. Examples are from actual customer usage, and they are presented here to help you understand some of the issues that can cause problems in Flow.
  You should consider these items when implementing new automation initiatives in Flow. As additional issues are found, this article will be updated.

Solution

Flow Release Process Recommendations

Consider having two Flow servers, one for a development environment and one for a production environment. This allows you to test any changes in a non-critical environment, ensuring that your productions systems are not affected. Just as you review and test your product code before it is released into your own production systems, you should also review and test Flow changes before releasing it into your Flow production system. This will ensure that your production environment is not at risk and will not delay your own product releases.

Why do these types of issues occur? For the same reason that they occur in our customers released products, developers are often focused on solving a particular problem, and when they find a solution to that single problem they do not always evaluate all of the consequences of that solution. For this reason, having a code review of the changes being done to any Flow project is also recommended. By having another knowledgeable programmer (in this instance a Flow programmer) review the changes, it will be more likely that changes that can have a negative impact can be caught before they are released into production.

By establishing a release process for Flow changes, just as you have a release process for your own code changes, you will have a more stable environment from which to perform your builds and releases. As stated previously, the Flow product is a programming environment, and the changes to it should have their own defined release process, to ensure that stability and integrity of your Build and Release Process.

Considerations for your Flow release process:

1)      Have two servers, one for development and one for production, and promote changes from one to the next only after being reviewed and validated. Define and document the promotion process so that all participants are aware of the requirements.

2)      Have a team of reviewers, who meet regularly, review and approve new code adjustments.

3)      Determine what changes require review and which do not to ensure that the process is not overly restrictive.

4)      Implement a crises change policy to allow for changes to be implemented quickly to address problems that may slip through inadvertently.

5)      Ensure that the process includes backing up the system before making any changes, which allows for an easier restore to the earlier instance of the Flow environment.

The following are cases that have been encountered that should be considered during review:

1)      Using FindObjects in a query: maxIDs =0 means you are getting everything in the system. This might bog down your database, or worse, bring down your system, especially if you are querying on all jobs or job-steps in your system. When implementing a query, keep in mind the potential scope of the returned items – or this could cause problems with your system.

2)      When creating jobs that spawn other jobs, you can quickly overload your agents, database, or memory. Be sure to keep a close eye on any procedures that are written that dynamically spawn other jobs; ensure that there is some limit imposed in your steps.

3)      Be advised that if your hostname doesnt match in the conf files, it could cause issues (localhost vs. the machine name for example). If the machines response to hostname differs from conf/commander.properties value COMMANDER_SERVER_NAME, machines wont be able to communicate with each other.  A prototypical example is when installing on the system and the COMMANDER_SERVER_NAME is prod1.electric-cloud.com but that name doesnt resolve locally. 

4)      ACLs – Access Control List changes can often have unexpected consequences, causing users to lose the ability to run commands that they were previously able to run. Be careful when making changes and be sure to fully test any ACL changes.

5)      Firewalls – Firewalls are configured in slightly different ways on all systems. Some people think that ping is sufficient, but it is not. Various ports need to be available, including 80, 443 and others per our documentation. When misconfigured, a user may be able to ping and even add an agent, but cant use it.

6)      SE Linux – This causes issues and Electric Cloud recommends disabling it whenever possible. 

7)      Share permissions on Windows (in addition to file permissions) – The EC agents must have adequate permissions on network shares or workspaces if they are to read/write files.

8)      Change in PATH on Windows requires an agent restart to take effect – When an application is installed, anytime an environment variable must be modified (Ant, Maven, Java, etc.) a restart is required.

9)      MS SQL named instances (vs. default one on static port) --  Administrators can define access to MySQL DB through different ways:

• Default: works
• Different port: works if static
• Instance name: it generates a dynamic port but our config file doesnt accept the syntax with putting an instance name. The workaround is to set up a static port for this instance and access it that way (or revert to default)

10)    Network Permissions – Agents must be able to connect to and write to the location where the workspace is defined. When using multi-OS environments, be sure that all connections are possible and permissions are set properly. The use of credentials can be helpful in this situation.

11)    The service account has a home directory that is typically not in C:\Users\<> or C:\Documents and Settings\<> or the equivalent, so configuring them by logging in and running a process doesnt work.

Using Plugins to Version

If the user creates their project as a plugin and iterates on the version, they can have multiple versions of the same plugin/project and can easily roll back by promoting the desired version. They can also have live reference sub-procedure calls (by leaving out the version number) or explicit version calls (by including the version number). Basically, use the built-in plugin versioning capabilities to version a project.