Installing and Configuring SL1 PowerFlow

 

This section describes how to install, upgrade, and configure PowerFlow, and also how to set up security for PowerFlow.

PowerFlow Architecture

This topic describes the different aspects of PowerFlow architecture.

PowerFlow Container Architecture

PowerFlow is a collection of purpose-built containers that are charged to pass information to and from SL1. Building PowerFlow architecture in containers allows you to add more processes to handle the workload as needed.

The following diagram describes the container architecture for PowerFlow:

PowerFlow includes the following containers:

• GUI. The GUI container provides the user interface for PowerFlow.
• REST API. The REST API container provides access to the Content Store on the PowerFlow instance.
• Content Store. The Content Store container is basically a database service that contains all the reusable steps, applications, and containers in the PowerFlow instance.
• Step Runners. Step Runner containers execute steps independently of other Step Runners. All Step Runners belong to a Worker Pool and can run steps in order, based on the instructions in the applications. By default there are five Step Runners (worker nodes) include in the PowerFlow platform. PowerFlow users can scale up or scale down the number of worker nodes, based on the workload requirements.

Integration Workflow

The following high-level diagram for a ServiceNow Integration provides an example of how PowerFlow communicates with both the SL1 Central Database and the third-party (ServiceNow) APIs:

The workflow includes the following components and their communication methods:

• SL1 Central Database. PowerFlow communicates with the SL1 database over port 7706.
• SL1 REST API. PowerFlow communicates with the SL1 REST API over port 443.
• GraphQL. PowerFlow communicates with GraphQL over port 443.
• ServiceNow Base PowerPack. In this example, the Run Book Automations from the ServiceNow Base PowerPack (and other SL1 PowerPacks) communicate with PowerFlow over port 443.
• PowerFlow. PowerFlow communicates with both the SL1 Central Database and an external endpoint.
• ServiceNow API. In this example, the ServiceNow applications in PowerFlow communicate with the ServiceNow API over port 443.

PowerFlow both pulls data from SL1 and has data pushed to it from SL1. PowerFlow both sends and retrieves information to and from ServiceNow, but PowerFlow is originating the requests.

High-Availability, Off-site Backup, and Proxy Architecture

You can deploy PowerFlow as a High Availability cluster, which requires at least three nodes to achieve automatic failover. While PowerFlow can be deployed as a single node, the single-node option does not provide redundancy through High Availability. PowerFlow also supports off-site backup and connection through a proxy server.

The following diagram describes these different configurations:

• High Availability for PowerFlow is a cluster of PowerFlow nodes with a Load Balancer managing the workload. In the above scenario, if one PowerFlow node fails, the workload will be redistributed to the remaining PowerFlow nodes. High Availability provides local redundancy. For more information, see Appendix A: Configuring PowerFlow for High Availability.
• Off-site Backup can be configured by using PowerFlow to back up and recover data in the Couchbase database. The backup process creates a backup file and sends that file using Secure Copy Protocol (SCP) to a user-defined, off-site destination system. You can then use the backup file from the remote system and restore its content. For more information, see Creating a Backup.
• A Proxy Server is a dedicated computer or software system running as an intermediary. The proxy server in the above scenario handles the requests between PowerFlow and the third-party application. For more information, see Configuring a Proxy Server.

In addition, you can deploy PowerFlow in a multi-tenant environment that supports multiple customers in a highly available fashion. After the initial High Availability (HA) core services are deployed, the multi-tenant environment differs in the deployment and placement of workers and use of custom queues. For more information, see Appendix B: Configuring PowerFlow for Multi-tenant Environments.

There is no support for active or passive Disaster Recovery. ScienceLogic recommends that your PowerFlow Disaster Recovery plans include regular backups and restoring from backup. For more information, see Creating a Backup.

Reviewing Your Deployment Architecture

Review the following aspects of your architecture before deploying PowerFlow:

1. How many SL1 stacks will you use to integrate with the third-party platform (such as ServiceNow, Cherwell, or Restorepoint)?
2. What is a good estimate of the number of devices across all of your SL1 stacks?
3. How many data centers will you use?
4. Specify the location of each data center.
5. What is the latency between each data center? (Latency must be less than 80 ms.)
6. How many SL1 stacks are in each data center?
7. Are there any restrictions on data replication across regions?
8. What is the location of the third-party platform (if applicable)?
9. What is the VIP for Cluster Node Management?

Based on the above list, ScienceLogic recommends the following deployment paths:

• For question A, if you answered three or fewer SL1 stacks, consider a standard High-Availability deployment. For more information, see Appendix A: Configuring PowerFlow for High Availability.
• For question A, if you answered more than three SL1 stacks to question A, consider configuring PowerFlow in a multi-tenant configuration. For more information, see Appendix B: Configuring PowerFlow for Multi-tenant Environments.
• For question G, if you answered "Yes" to data replication restrictions, consider the following deployment options:

• Deploy separate PowerFlow clusters per region. This deployment requires more management of PowerFlow clusters, but it ensures that the data is completely separated between regions. This deployment also ensures that if a single region goes down, you only lose operations for that region.
• Deploy a single PowerFlow cluster in the restrictive region. This deployment is easier to manage, as you are only dealing with a single PowerFlow cluster. As an example, if Europe has a law that requires that data in Europe cannot be replicated to the United States, but that law does not prevent data from the United States from coming into Europe, you can deploy a single PowerFlow cluster in Europe to satisfy the law requirements.

• If you are deploying a multi-tenant configuration, check to see if your environment meets one the following:

• You have three or more data centers and the latency between each data center is less than 80 ms (question E), consider deploying a multi-tenant PowerFlow where each node is in a separate data center to ensure data center resiliency. This deployment ensures that if a single data center goes down, PowerFlow will remain operational.
• You have only two data centers and the latency between data centers is less than 80 ms, consider deploying a multi-tenant PowerFlow where two nodes are in one data center and the other node is in the other data center. This deployment does not ensure data center resiliency, but it does provide standard High Availability if a single node goes down. If the data center with one node goes down, PowerFlow will remain operational. However, if the data center with two nodes goes down, PowerFlow will no longer remain operational.
• You have only two data centers but the latency between data centers is more than 80 ms. In this situation, you can still deploy a multi-tenant PowerFlow, but all nodes must be located in a single data center. This deployment still provides standard High Availability so that, if a single node goes down, the other two nodes ensure PowerFlow operations. If you require more resiliency than a single-node failure, you can deploy five nodes, which will ensure resiliency with two down nodes. However, if the data center goes down, PowerFlow will not be operational.
• You only have one data center, you can still deploy a multi-tenant PowerFlow, but all nodes are located in a single data center. This deployment still provides standard High Availability so that, if a single node goes down, the other two nodes ensure PowerFlow operations. If you require more resiliency than a single-node failure, you can deploy five nodes, which will ensure resiliency with two down nodes. However, if the data center goes down, PowerFlow will not be operational.

System Requirements

PowerFlow itself does not have specific minimum required versions for SL1 or AP2. However, certain PowerFlow SyncPacks have minimum version dependencies, which are listed on the Dependencies for SL1 PowerFlow SyncPacks page.

Ports

The following table lists the PowerFlow ingress requirements:

Source	Port	Purpose
SL1 host	443	SL1 run book actions and connections to PowerFlow
User client	3141	Devpi access
User client	443	PowerFlow API
User client	5556	Dex Server: enable authentication for PowerFlow
User client	8091	Couchbase Dashboard
User client	15672	RabbitMQ Dashboard
User client	22	SSH access

The following table lists the PowerFlow egress requirements:

Destination	Port	Purpose
SL1 host	7706	Connecting PowerFlow to SL1 Database Server
SL1 host	443	Connecting PowerFlow to SL1 API

Additional Considerations

Review the following list of considerations and settings before installing PowerFlow:

• ScienceLogic highly recommends that you disable all firewall session-limiting policies. Firewalls will drop HTTPS requests, which results in data loss.
• Starting with PowerFlow version 3.0.0, the minimum storage size for the initial partitions is 60 GB. Anything less will cause the automated installation to stop and wait for user input. You can use the tmux application to navigate to the other panes and view the logs. In addition, at 100 GB and above, PowerFlow will no longer allocate all of the storage space, so you will need to allocate the rest of the space based on your specific needs.
• PowerFlow clusters do not support vMotion or snapshots while the cluster is running. Performing a vMotion or snapshot on a running PowerFlow cluster will cause network interrupts between nodes, and will render clusters inoperable.
• The site administrator is responsible for configuring the host, hardware, and virtualization configuration for the PowerFlow server or cluster. If you are running a cluster in a VMware environment, be sure to install open-vm-tools and disable vMotion.
• You can configure one or more SL1 systems to use PowerFlow to sync with a single instance of a third-party application like ServiceNow or Cherwell. You cannot configure one SL1 system to use PowerFlow to sync with multiple instances of a third-party application like ServiceNow or Cherwell. The relationship between SL1 and the third-party application can be either one-to-one or many-to-one, but not one-to-many.
• The default internal network used by PowerFlow services is 172.21.0.1/16. Please ensure that this range does not conflict with any other IP addresses on your network. If needed, you can change this subnet in the docker-compose.yml file.

For more information about system requirements for your PowerFlow environment, see the System Requirements page at the ScienceLogic Support site at https://support.sciencelogic.com/s/system-requirements.

Hardened Operating System

The operating system for PowerFlow is pre-hardened by default, with firewalls configured only for essential port access and all services and processes running inside Docker containers, communicating on a secure, encrypted overlay network between nodes. Please refer to the table, above, for more information on essential ports.

You can apply additional Linux hardening policies or package updates as long as Docker and its network communications are operational.

The PowerFlow operating system is an Oracle Linux distribution, and all patches are provided within the standard Oracle Linux repositories. The patches are not provided by ScienceLogic.

Additional Prerequisites for PowerFlow

To work with PowerFlow, ScienceLogic recommends that you have knowledge of the following:

• Linux and vi (or another text editor).
• Python.
• Postman or another API tool for interacting with the PowerFlow API.
• Couchbase (Community Edition). For more information, see Helpful Couchbase Commands.
• Docker. For more information, see Helpful Docker Commands and https://docs.docker.com/engine/reference/commandline/cli/.

The most direct way of accessing the most recent containers of PowerFlow is by downloading the latest RPM file from the ScienceLogic Support Portal. As a separate option, you can also access the PowerFlow containers directly through Docker Hub. To access the containers through Docker Hub, you must have a Docker Hub ID and enable permissions to pull the containers from Docker Hub. To get permissions, contact your ScienceLogic Customer Success Manager.

Installing PowerFlow

Due to the upcoming end of support for Oracle Linux 7, ScienceLogic strongly urges users to upgrade to Oracle Linux 8 (OL8). As such, only the OL8-based package and upgrade path is defined and provided. If you have extenuating circumstances and want to obtain an OL7-based install for PowerFlow 3.0.0, please contact your CSM or ScienceLogic support.

Installing PowerFlow for the First Time

You can install PowerFlow for the first time in the following ways:

• Via ISO to a server on your network
• Via RPM to a cloud-based server

If you are installing PowerFlow in a clustered environment, see Configuring the PowerFlow System for High Availability.

Upgrading an Existing PowerFlow System

• If you are upgrading an existing version of PowerFlow to version 3.0.0 or later, the steps are slightly different, because you will need to convert the operating system to Oracle Linux 8. For more information, see Converting PowerFlow to Oracle Linux 8 (OL8).
• If you are upgrading an existing version of PowerFlow to a version before version 3.0.0, see Upgrading PowerFlow.

The site administrator is responsible for configuring the host, hardware, and virtualization configuration for the PowerFlow server or cluster. If you are running a cluster in a VMware environment, be sure to install open-vm-tools and disable vMotion.

Installing PowerFlow via ISO

Due to the upcoming end of support for Oracle Linux 7, ScienceLogic strongly urges users to upgrade to Oracle Linux 8 (OL8). As such, only the OL8-based package and upgrade path is defined and provided. If you have extenuating circumstances and want to obtain an OL7-based install for PowerFlow 3.0.0, please contact your CSM or ScienceLogic support.

Locating the ISO Image

To locate the PowerFlow ISO image:

1. Go to the ScienceLogic Support site at https://support.sciencelogic.com/s/.
2. Click the Product Downloads tab and select PowerFlow Platform. The PowerFlow page appears.
3. Click the link to the current release. The Release Version page appears.
4. In the Release Files section, click the ISO link for the PowerFlow image. A Release File page appears.
5. Click Download File at the bottom of the Release File page.

Installing from the ISO Image

When installing PowerFlow from an ISO, you can now install open-vm-tools by selecting Yes to "Installing Into a VMware Environment" option during the installation wizard.

To install PowerFlow via ISO image:

1. Download the latest PowerFlow ISO file to your computer or a virtual machine center.

2. Using your hypervisor or bare-metal (single-tenant) server of choice, mount and boot from the PowerFlow ISO. The PowerFlow Installation window appears:

   

3. Select Install PowerFlow. The Military Unique Deployment window appears.

4. Select Yes only if you require a Military Unique Deployment (MUD) of the PowerFlow system. In most situations, you would select the default option of No. After the installer loads, the Network Configuration window appears.

5. Complete the following fields:

   • IP Address. Type the primary IP address of the PowerFlow server.
   • Netmask. Type the netmask for the primary IP address of the PowerFlow server.
   • Gateway. Type the IP address for the network gateway.
   • DNS Server. Type the IP address for the primary nameserver.
   • Hostname. Type the hostname for PowerFlow.

5. Press Continue. The Root Password window appears.

6. Type the password you want to set for the root user on the PowerFlow host (and the service account password) and press Enter. The password must be at least six characters and no more than 24 characters, and all special characters are supported.

   You use this password to log into the PowerFlow user interface, to SSH to the PowerFlow server, and to verify API requests and database actions. This password is set as both the "Linux host isadmin" user and in the /etc/iservices/is_pass file that is mounted into the PowerFlow stack as a "Docker secret". Because it is mounted as a secret, all necessary containers are aware of this password in a secure manner. For more information, see Changing the PowerFlow Password.

   To avoid authentication issues, do not use the dollar sign ($) character as the first character in any of the passwords related to PowerFlow. You can use the $ character elsewhere in the password if needed.

7. Type the password for the root user again and press Enter. The PowerFlow installer runs, and the system reboots automatically. This process will take a few minutes.

8. After the installation scripts run and the system reboots, SSH into your system using PuTTY or a similar application. The default username for the system is isadmin.

9. To start the Docker services, change directory to run the following commands:

   cd /opt/iservices/scripts

   ./pull_start_iservices.sh

   

   This process will take a few minutes to complete.

10. To validate that iservices is running, run the following command to view each service and the service versions for services throughout the whole stack:

    docker service ls

11. Navigate to the PowerFlow user interface using your browser. The address of the PowerFlow user interface is:

    https://<IP address entered during installation>
    

12. Log in with the default username of isadmin and the password you specified in step 6.

13. After installation, you must license your PowerFlow system if you want to enable all of the features. For more information, see Licensing PowerFlow.

    If you are licensing a PowerFlow High Availability cluster, you can run the licensing process on any node in the cluster once the cluster is ready. The node does not have to be the leader, and the licensing process does not have to be run on all nodes in the Swarm. If you are setting up High Availability for the PowerFlow on a multiple-node cluster, see Preparing the PowerFlow System for High Availability.

    The HOST_ADDRESS value in the /etc/iservices/isconfig.yml file should be the fully qualified domain name (FQDN) of either the host if there is no load balancer, or the FQDN of the load balancer if one exists. If you change the HOST_ADDRESS value, you will need to restart the PowerFlow stack.

Troubleshooting the ISO Installation

To verify that your stack is deployed, view your Couchbase logs by executing the following command:

docker service logs --follow iservices_couchbase

If no services are found to be running, run the following command to start them:

docker stack deploy -c docker-compose.yml iservices

To add or remove additional workers, run the following command: 

docker service scale iservices_steprunner=10

Installing PowerFlow via RPM to a Cloud-based Environment

Due to the upcoming end of support for Oracle Linux 7, ScienceLogic strongly urges users to upgrade to Oracle Linux 8 (OL8). As such, only the OL8-based package and upgrade path is defined and provided. If you have extenuating circumstances and want to obtain an OL7-based install for PowerFlow 3.0.0, please contact your CSM or ScienceLogic support.

Considerations for the RPM Installation

• The PowerFlow version 3.0.0 and later RPM is OL8-based. As a result, you cannot install the PowerFlow 3.0.0 RPM in an OL7 environment.
• If you install the PowerFlow 3.0.0 RPM on any operating system other than Oracle Linux 8, ScienceLogic will only support the running application and associated containers. ScienceLogic will not assist with issues related to host configuration for operating systems other than Oracle Linux 8 (or Oracle Linux 7 for systems before PowerFlow version 3.0.0).
• If you are deploying PowerFlow without a load balancer, you can only use the deployed IP address as the management user interface. If you use another node to log in to the PowerFlow system, you will get an internal server error. Also, if the deployed node is down, you must redeploy the system using the IP address for another active node to access the management user interface.
• The HOST_ADDRESS value in the /etc/iservices/isconfig.yml file should be the fully qualified domain name (FQDN) of either the host if there is no load balancer, or the FQDN of the load balancer if one exists. If you change the HOST_ADDRESS value, you will need to restart the PowerFlow stack.
• If you are installing the RPM in a cluster configuration, and you want to distribute traffic between the nodes, a load balancer is required.
• If you install the PowerFlow system in a cloud-based environment using a method other than an ISO install, you are responsible for setting up and configuring the requirements of the cloud-based environment.

Locating the RPM file

To locate the PowerFlow RPM file:

1. Go to the ScienceLogic Support site at https://support.sciencelogic.com/s/.
2. Click the Product Downloads tab and select PowerFlow. The PowerFlow page appears.
3. Click the link to the current release. The Release Version page appears.
4. In the Release Files section, click the RPM link for the PowerFlow image. A Release File page appears.
5. Click Download File at the bottom of the Release File page.

Installing from the RPM File

You can also install PowerFlow on other cloud-based environments, such as Microsoft Azure. For other cloud-based deployments, the process is essentially the same as the following steps: PowerFlow provides the containers, and the cloud-based environment provides the operating system and server.

You can install PowerFlow version 3.0.0 or later on any Oracle Linux 8 (OL8) operating system, even in the cloud, as long as you meet all of the operating system requirements. These requirements include CPU, memory, Docker and a docker-compose file installed, and open firewall settings. When these requirements are met, you can install the RPM and begin to deploy the stack as usual.

Previous versions of PowerFlow before version 3.0.0 can use Oracle Linux 7.x, but ScienceLogic strongly recommends that you convert the operating system to OL8 as soon as possible. The steps below are specific for PowerFlow version 3.0.0 or later.

The PowerFlow version 3.0.0 and later RPM is OL8-based. As a result, you cannot install the PowerFlow 3.0.0 RPM in an OL7 environment.

The following procedure describes how to install PowerFlow via RPM to Amazon Web Service (AWS) EC2. The ec2-user must belong to the iservices group.

To install a single-node PowerFlow version 3.0.0 via RPM to a cloud-based environment (using AWS as an example):

1. In Amazon Web Service (AWS) EC2, click Launch instance. The Launch an instance page appears.

   If you are installing PowerFlow to another cloud-based environment, such as Microsoft Azure, set up the operating system and server, and then go to step 7.

2. Deploy a new Oracle Linux 8.0 virtual machine by searching for ami-02a7419f257858fad in the Search our full catalog field in the Application and OS Images section.

3. Press Enter. The Choose an Amazon Machine Image (AMI) page appears.

4. Click the AWS Marketplace AMIs tab and click Select for an AMI file with a name that fits the following pattern: OL8.8-x86_64_HVM-*

   For example:

   

5. From the Choose an Instance Type page, select at least a t2.xlarge AMI instance, depending on your configuration:

• Single-node deployments. The minimum is t2.xlarge (four CPUs with 16 GB memory), and ScienceLogic recommends t2.2xlarge (8 CPUs with 32 GB memory).
• Cluster deployments. Cluster deployments depend on the type of node you are deploying. Refer to the separate multi-tenant environment guide for more sizing information. ScienceLogic recommends that you allocate at least 50 GB or more for storage.

6. Go to the Step 6: Configure Security Group page and define the security group:

• Inbound port 443 needs to be exposed to any of the systems that you intend to integrate.

• For access to the PowerFlow user interface, add the following ports to the security group:

  • 15672 TCP for RabbitMQ
  • 5556 for Dex Server authentication
  • 3141 for Devpi access

  For more information about ports, see the System Requirements.

• Port 8091 is exposed through https. ScienceLogic recommends that you make port 8091 available externally to help with troubleshooting:

  

7. Upload the sl1-powerflow-3.x.x-1.el8.x86_64.rpm file to the PowerFlow server using SFTP or SCP.

8. Enable the necessary repositories by running the following commands on the PowerFlow system:

   sudo dnf install yum-utils

   sudo dnf config-manager --enable ol8_baseos_latest

   sudo dnf config-manager --enable ol8_appstream

   sudo dnf config-manager --enable ol8_addons

9. Run the following commands on the server instance to upgrade to Python 3.11

   sudo dnf update

   sudo dnf install python3.11-pip

10. Ensure that the latest required packages are installed by running the following commands on the server instance:

    sudo dnf install wget

     

    wget --no-check-certificate https://download.docker.com/linux/centos/8/x86_64/stable/Packages/containerd.io-1.6.28-3.1.el8.x86_64.rpm

    wget --no-check-certificate https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-ce-25.0.3-1.el8.x86_64.rpm

    wget --no-check-certificate https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-ce-cli-25.0.3-1.el8.x86_64.rpm

    wget --no-check-certificate https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-ce-rootless-extras-25.0.3-1.el8.x86_64.rpm

    wget --no-check-certificate https://download.docker.com/linux/centos/8/x86_64/stable/Packages/docker-compose-plugin-2.24.6-1.el8.x86_64.rpm

     

    sudo dnf install -y containerd.io-1.6.28-3.1.el8.x86_64.rpm docker-ce-25.0.3-1.el8.x86_64.rpm docker-ce-cli-25.0.3-1.el8.x86_64.rpm docker-ce-rootless-extras-25.0.3-1.el8.x86_64.rpm docker-compose-plugin-2.24.6-1.el8.x86_64.rpm

You might need to remove spaces from the code that you copy and paste from this manual. For example, in instances such as the wget command, above, line breaks were added to long lines of code to ensure proper pagination in the document.

You will need to update both instances of the Docker version in this command if there is a more recent version of Docker CE on the Docker Download page: https://download.docker.com/linux/centos/7/x86_64/stable/Packages/.

Do not change the version of pip from 20.2.4. This version is required for PowerFlow.

11. Create the Docker group:

    sudo groupadd docker

12. Add your admin user to the Docker group and the wheel group:

    sudo usermod -aG docker $USER

    sudo usermod -aG wheel $USER

    where $USER is the isadmin user name or the ec2-user in AWS. The ec2-user should belong to the iservices group, which is created as part of this RPM installation process.

13. Log out and log back in to ensure that your group membership is re-evaluated.

14. Run the following commands for the configuration updates:

    sudo setenforce 0

    sudo vi /etc/selinux/config

    SELINUX=permissive

    If changing the SELINUX=permissive configuration does not work, replace it with SELINUX=disabled.

15. Run the following firewall commands as "sudo":

    sudo firewall-cmd --add-port=2376/tcp --permanent

    sudo firewall-cmd --add-port=2377/tcp --permanent

    sudo firewall-cmd --add-port=7946/tcp --permanent

    sudo firewall-cmd --add-port=7946/udp --permanent

    sudo firewall-cmd --add-port=4789/udp --permanent

    sudo firewall-cmd --add-protocol=esp --permanent

    sudo firewall-cmd --reload

    To view a list of all ports, run the following command: firewall-cmd --list-all.

    If you copy and paste any of the commands with a --, make sure the two hyphens are entered as hyphens and not special characters.

16. Install the remaining Python packages needed for the PowerFlow RPM file:

    sudo dnf update

    sudo rpm -qa|grep python3-pyyaml

    sudo dnf remove python3-pyyaml

    sudo pip3 install --no-build-isolation wheel

    sudo pip3 install requests==2.27.1

    sudo pip3 install --no-build-isolation pyyaml==5.4.1

    sudo pip3 install --no-build-isolation docker-compose

17. Copy the PowerFlow RPM to the instance of installation and install the RPM:

    sudo dnf install sl1-powerflow-3.X.X-1.el8.x86_64.rpm

    systemctl restart docker

    If an OL8 (hardened) image was used, the /tmp mount point might have been mounted using the noexec flag. If that is the case, run the following steps to install the RPM:
    
    mkdir -p $HOME/tmp
    TMPDIR=$HOME/tmp dnf install sl1-powerflow-3.X.X-1.el8.x86_64.rpm
    systemctl restart docker

18. Create a password for PowerFlow:

    printf '<password>' > /etc/iservices/is_pass

    where <password> is a new, secure password.

19. Pull and start iservices to start PowerFlow:

    sudo /opt/iservices/scripts/pull_start_iservices.sh

For an AWS deployment, ScienceLogic recommends that you switch to an Amazon EC2 user as soon as possible instead of running all the commands on root.

For a clustered PowerFlow environment, you must install the PowerFlow RPM on every server that you plan to cluster into PowerFlow. You can load the Docker images for the services onto each server locally by running /opt/iservices/scripts/pull_start_iservices.sh. Installing the RPM onto each server ensures that the PowerFlow containers and necessary data are available on all servers in the cluster.

After installation, you must license your PowerFlow system to enable all of the features. Licensing is required for production systems only, not for test systems. For more information, see Licensing PowerFlow.

Troubleshooting a Cloud Deployment of PowerFlow

After completing the AWS setup instructions, if none of the services start and you see the following error during troubleshooting, you will need to restart Docker after installing the RPM installation.

sudo docker service ps iservices_couchbase --no-trunc

"error creating external connectivity network: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i docker_gwbridge -j RETURN: iptables: No chain/target/match by that name."

Converting PowerFlow to Oracle Linux 8 (OL8)

Starting with version 3.0.0 of PowerFlow, you can convert your PowerFlow system to Oracle Linux 8. ScienceLogic strongly recommends you make the conversion to OL8 as soon as possible, as OL7 is going End of Life (EOL) near the end of 2024.

Complete the upgrade steps in the following order:

1. If needed (see the tables, below), back up the PowerFlow system.
2. Use the automated upgrade scripts to convert the operating system to Oracle Linux 8 (OL8).
3. Install PowerFlow version 3.0.0 or later using the .iso file.
4. If needed, restore the PowerFlow system.

Upgrade Options for Converting from PowerFlow 2.x (OL7) to PowerFlow 3.x or Later (OL8)

Select one of the following options to upgrade from an older version of PowerFlow running OL7 to PowerFlow version 3.0.0 or later running OL8:

Upgrade Option	Requirements	Implications, Downtime, Other Considerations
Automated upgrade scripts	About 82 GB free space is required in root(/) or the /var/data or any other partition. Also, an extra 10 GB is required in root(/).	Minimum PowerFlow version to upgrade from: 2.3.0 This approach will likely cause about 50-60 minutes of downtime.
Backup, install, and restore, using a separate system	Identical, secondary environment for installing the PowerFlow 3.0.0 .iso file	This approach allows for the existing PowerFlow system to continue running while you deploy and configure a new OL8 based system. Once data is fully restored on the new system, you may switch the load balancer configuration to point to the new system, virtually eliminating downtime. This approach also allows you to use the old PowerFlow system as a fallback or rollback option.
Backup, re-install, and restore, using the same system	A separate file store where backup and configuration files can be stored temporarily	This approach will incur downtime as your existing system will be re-installed to PowerFlow 3.0.0 and restored with the data from the previous version. This approach does not allow you to roll back or switch back over to the older version of PowerFlow.

Upgrade Paths Based on PowerFlow Environments

Your upgrade options depend on your PowerFlow environment, so review the following table before beginning the upgrade and conversion process.

Environment Type	Upgrade Options	Recommended Option	Additional Notes
Internet-connected, on-premises	Use the automated upgrade scripts. OR Run a back up, re-install, and restore.	Use the automated upgrade scripts.	None
Offline on-premises	Use the offline version of the automated upgrade scripts. OR Run a back up, re-install, and restore.	Use the automated upgrade scripts.	None
MUD installation (FIPS-enforced)	Run a back up, re-install, and restore.	Run a back up, re-install, and restore.	The Oracle Linux Leapp utility does not support the automated upgrade of FIPS systems, and recommends a new installation. For more information, see the Red Hat documentation.
AWS-based cloud installation	Use the automated upgrade scripts.	Use the automated upgrade scripts.	The automated upgrade scripts will likely work for AWS environments, but due to potential environmental differences between chosen AMIs, there might be other package updates or requirements. If you encounter environmental problems, you should consider using the back up, re-install, and restore process instead.

Automated Upgrade Scripts

The PowerFlow automated upgrade scripts are delivered within the PowerFlow 3.0.0 ISO file, which is available at the ScienceLogic Support Site at https://support.sciencelogic.com/s/powerflow.

Follow the instructions below to run the scripts to convert the operating system to OL8.

About 82 GB free space is required in root(/) or the /var/data or any other partition. Also, an extra 10 GB is required in root(/). You can upgrade from PowerFlow version 2.3.0 or later. This approach will likely cause about 30-50 minutes of downtime.

These upgrade scripts use the Oracle Linux Leapp utility. For more information, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/upgrading_from_rhel_7_to_rhel_8/index.

The upgrade_OL7_to_OL8.sh Script

The upgrade_OL7_to_OL8.sh script executes the OL8 Upgrade steps that are listed in the Oracle documentation: https://docs.oracle.com/en/operating-systems/oracle-linux/8/leapp/leapp-UpgradingtheSystem.html#chap-leapp-upgrade.

The logs created by this script are located in the /tmp directory. You can review the logs for details about the upgrade process at /tmp/pf_local_ol8_upgrade.log. You will need to manually remove the logs after the upgrade finishes.

The ol8_post_upgrade_actions.sh Script

The ol8_post_upgrade_actions.sh script executes the post-upgrade takes that are listed in the Oracle documentation: https://docs.oracle.com/en/operating-systems/oracle-linux/8/leapp/leapp-CompletingPostupgradeTasks.html#post-upgrade.

The script also verifies if the PowerFlow RPM and their dependencies were updated.

After running both scripts, follow the instructions below for the upgrade, based on your system configuration:

• Single-node Upgrade
• Cluster Upgrade
• AWS EC2 Instance Upgrade

Single-node Upgrade

For a single-node upgrade:

1. Make backups of the PowerFlow configuration files:

• All the files on the directory /opt/iservices/scripts
• All the files on the directory /etc/iservices

2. Connect the PowerFlow 3.0.0 ISO file to the System.

3. Create the directory /mnt/tmpISMount to mount the ISO file.

4. Run the following command to mount the PowerFlow ISO file onto the system:

   sudo mount -o loop /dev/cdrom /mnt/tmpISMount

5. Run the upgrade script using sudo (the script runs in offline mode as default):

   sudo /mnt/tmpISMount/scripts/ol8_upgrade/upgrade_OL7_to_OL8.sh

Add the following arguments as needed.

--reboot:       To reboot the system automatically to finish the Upgrade process. Disabled by default.
--online:       To execute the Upgrade process in online mode. Disabled by default.
                If the System is offline and this option is used the Upgrade will fail.
--remove_stack: To remove the PowerFlow Stack if it is running. Disabled by default.
--upgrade_path: To provide a path to execute the Upgrade process. 35GB of disk space is needed. Default path: /
                Example: --upgrade_path=/var/data
--iso_path:     To provide the path where the PF ISO was mounted. Default path: /mnt/tmpISMount
                Example: --iso_path=/mnt/tmpISMount
--help:         To display this menu"

6. Reboot the system manually if you did not run the script with the reboot option. Do not interrupt the upgrade process that continues to run when the system is restarting.

7. After the system finishes the upgrade, connect to the system again and run the post-upgrade script:

   sudo /tmp/ol8_post_upgrade_actions.sh

   If the post-upgrade script was removed, connect and mount the PowerFlow version 3.0.0 ISO again and execute the script from the ISO:

   sudo /mnt/tmpISMount/scripts/ol8_upgrade/ol8_post_upgrade_actions.sh

   This script verifies if your system is OL8 and validates PowerFlow rpm was installed successfully. If the upgrade was successful, the script removes the files that were created for the upgrade, but it will ask for a confirmation to do so.

8. After the post-upgrade is executed, be sure to validate that the docker-compose file has the correct information.

Cluster Upgrade

For a cluster upgrade:

1. Remove the PowerFlow stack.

2. For each node, follow all of the steps in the Single-node Upgrade above.

3. After the upgrade and post-upgrade processes have finished, be sure to validate that the docker-compose file has the correct information.

4. Start the PowerFlow stack in the main node by running the following command:

   docker stack deploy -c /opt/iservices/scripts/docker-compose.yml iservices

AWS EC2 Instance Upgrade

Remove the PowerFlow stack in a clustered environment before starting the upgrade. After upgrading all the nodes, start the PowerFlow stack in the main node.

To perform an in-place upgrade with AWS EC2 instances:

1. Make backups of the PowerFlow configuration files:

• All the files on the directory /opt/iservices/scripts
• All the files on the directory /etc/iservices

2. Make sure the Oracle Linux RPM has all of the latest packages; this command searches all available Oracle Linux repos and updates any Oracle Linux packages it might need:

   sudo yum update -y

3. Run the following commands to download and install the following Oracle Linux RPM: https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/libreport-filesystem-2.1.11-53.0.3.el7.x86_64.rpm.

   wget https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/libreport-filesystem-2.1.11-53.0.3.el7.x86_64.rpm

   sudo yum install libreport-filesystem-2.1.11-53.0.3.el7.x86_64.rpm

4. Remove the uname26 package:

   sudo yum remove uname26

5. Set PermitRootLogin to no in /etc/ssh/sshd_config and run systemctl restart sshd.

6. Connect the PowerFlow 3.0.0 ISO file to the System.

7. Create the directory /mnt/tmpISMount to mount the ISO file.

8. Run the following command to mount the PowerFlow ISO file onto the system:

   sudo mount -o loop /dev/cdrom /mnt/tmpISMount

9. Run the first upgrade script with the following command:

   sudo /mnt/tmpISMount/scripts/ol8_upgrade/upgrade_OL7_to_OL8.sh --online --remove_stack

   Add the following arguments as needed.

   --reboot:       To reboot the system automatically to finish the Upgrade process. Disabled by default.
   --online:       To execute the Upgrade process in online mode. Disabled by default.
                   If the System is offline and this option is used the Upgrade will fail.
   --remove_stack: To remove the PowerFlow Stack if it is running. Disabled by default.
   --upgrade_path: To provide a path to execute the Upgrade process. 35GB of disk space is needed. Default path: /
                   Example: --upgrade_path=/var/data
   --iso_path:     To provide the path where the PF ISO was mounted. Default path: /mnt/tmpISMount
                   Example: --iso_path=/mnt/tmpISMount
   --help:         To display this menu"

10. When the upgrade is complete, edit /etc/default/grub by adding GRUB_SERIAL_COMMAND="serial --speed=115200" and GRUB_TIMEOUT=20, and then run grub2-mkconfig:

    sudo vim /etc/default/grub

    // edit and save the config

    sudo grub2-mkconfig -o /boot/grub2/grub.cfg

    For more information, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console-prerequisites.html.

11. In AWS, use the Serial Connect service to connect to the EC2 instance and reboot.

12. When the instance starts, you are given the option to select a boot OS. Select the option marked upgrade.

13. When the upgrade has completed, run the post-upgrade script:

    sudo /tmp/ol8_post_upgrade_actions.sh

14. Revert the changes made to grub and recreate the grub.cfg file.

Back Up, Re-install, and Restore Your PowerFlow System

If you are backing up, installing, and restoring using the same system, you will need a separate file store where backup and configuration files can be stored temporarily. This approach will incur downtime as your existing system will be re-installed to 3.0.0 and restored with the data from previous. This approach does not allow you to roll back or switch back over to the older version.

If you are backing up, installing, and restoring using a separate system, you will need an identical, secondary environment for installing the PowerFlow 3.0.0 .iso file. This approach allows for the existing PF system to continue running/operating while you deploy and configure a new OL8 based system. Once data is fully restored on the new system, you may switch load balancer config to point to the new system, virtually eliminating downtime. This approach also allows you to use the old system as a fallback/rollback option

Use the following steps to back up your current PowerFlow configuration after you have converted the operating system to OL8, but before you upgrade to PowerFlow version 3.0.0 or later:

1. Use the "PowerFlow Backup" application in the PowerFlow user interface to create a backup file and send that file using secure copy protocol (SCP) to a destination system. For more information, see Backing up and Restoring PowerFlow Data.

   The backup and restore applications are application-level backup and restore tools. For full-system backups, you will need to do a filesystem-level backup to ensure that you get the encryption key that was used to encrypt configuration objects as well as other files used to describe the environment, including the /etc/iservices directory, the docker-compose.yml file and the docker-compose-override.yml file.

2. Install a "fresh" version of PowerFlow using the .iso file. For more information, see Installing PowerFlow via ISO.

3. Use the "PowerFlow Restore" application in the PowerFlow user interface to retrieve your backup file (from step 1, above) from the remote system and restore its content. For more information, see Restoring PowerFlow Data.

Upgrading PowerFlow (not for PowerFlow 3.0.0)

This topic is only relevant for users that are upgrading an existing version of PowerFlow to a version before version 3.0.0. If you are upgrading to PowerFlow version 3.0.0 or later, see Converting PowerFlow to Oracle Linux 8 (OL8).

ScienceLogic releases a major update to PowerFlow every six months. ScienceLogic also releases a monthly maintenance release (MMR) as needed to address major customer-facing bugs. If there are no major bugs to be addressed via MMR, the MMR will not be produced for the month. Security updates are included in an MMR only if an MMR is planned to be released.

You should always upgrade to the most recent release of PowerFlow.

For upgrades from PowerFlow version 2.2.x systems that have the localpkg_gpgcheck=1option enabled in /etc/yum.conf, the SL RPM Public Key is required. Please contact your ScienceLogic Customer Success Manager (CSM) or create a new Service Request case at https://support.sciencelogic.com/s in the "PowerFlow" category to request access to that key.

Please note that you can use the latest RPM if you only need to upgrade the PowerFlow application.

If you need the most recent, stable version of the Oracle Linux 7 operating system (OS), you can upgrade by mounting the latest ISO to an existing PowerFlow system. If there are OS vulnerabilities discovered in PowerFlow, you will need to either patch the vulnerability yourself using yum or wait for the next PowerFlow ISO. For more information, see Upgrading OS Packages (for Offline Deployments Only).

When a yum update is performed, there is no risk of PowerFlow operations being affected as long as Docker or networking services are not included in the updates.

If you are upgrading to Couchbase version 6.6.0, see Upgrading to Couchbase Version 6.6.0.

Considerations for Upgrading PowerFlow

• If you made any modifications to the nginx configuration or to other service configuration files outside of the docker-compose.yml file, you will need to modify those custom configurations before upgrading, or contact ScienceLogic Support to prevent the loss of those modifications.
• If you are deploying PowerFlow without a load balancer, you can only use the deployed IP address as the management user interface. If you use another node to log in to the PowerFlow system, you will get an internal server error. Also, if the deployed node is down, you must redeploy the system using the IP address for another active node to access the management user interface.
• In PowerFlow 2.2.2 and older, the gui services could run on any node and provide traffic to the whole cluster. Starting with PowerFlow 2.3.0, Military Unique Deployment (MUD) system requirements required security updates. As a result, the gui services must run only on the nodes that are receiving the initial requests. The gui services have to be running on the nodes to which the load balance is routing, or the host_address.
• If you are upgrading from PowerFlow 2.2.2 or older, when you upgrade to this version of PowerFlow, ensure that the PowerFlow platform is stable and that you do not wish to roll back to a previous version before you start installing or updating SyncPacks. After you update the content on the PowerFlow platform, rolling back to a previous platform version requires restoring from a backup. You can roll back PowerFlow versions, but you should not roll back the database version.

Deploying PowerFlow as a MUD System (Optional)

Starting with PowerFlow version 2.3.0, you can deploy PowerFlow as a Military Unique Deployment (MUD) system. Please note the following criteria:

• You cannot convert a 2.3.0 or later non-MUD PowerFlow system to a MUD system.
• If you want to upgrade from an older (non-MUD) PowerFlow system to a MUD system, you will need to run a backup and restore to the new deployment.
• Upgrading from a 2.2.x system to a 2.3.0 or later non-MUD system is fully supported.

Steps for Upgrading PowerFlow

1. Validate the cluster configuration
2. Locate the RPM or ISO file for upgrading
3. Upgrade OS packages (for offline deployments only)
4. Download updates to address common vulnerabilities and exposures (CVEs)
5. Install the audit package for audit logging
6. Upgrade from version 2.2.0 or later (choose one of the following):

• Single-node upgrade
• Cluster upgrade with short downtime
• Rolling cluster upgrade with no downtime

Validating the Cluster Configuration

Perform the following tasks to ensure the cluster configuration is valid for this version of PowerFlow:

1. Ensure that the GUI service is constrained to, and is running, the nodes that are expected to receive traffic.

2. After the system is running, run the powerflowcontrol(pfctl) healthcheck and autoheal actions to address any missed role-based access control (RBAC) or index updates included with this version.

3. If your system is using a Load Balancer, update the settings to forward traffic for Couchbase (:8091) and RabbitMQ (:15672). Review the example configuration generated by the following command:

   pfctl --host <host_IP_1> user:pass --host <host_IP_2> user:pass --host <host_IP_3> user:pass cluster-action --action generate_haproxy_config

   where the <host_IP> values are the IP address of the hosts.

   or

   pfctl --config pfctl.yml cluster-action --action generate_haproxy_config

4. If your system is using a Load Balancer, open up firewall ports to accept traffic via 8091 and 15672 on the Load Balancer.

Locating the RPM or ISO File for Upgrading

As a best practice, you should always upgrade to the most recent version of PowerFlow that is currently available at https://support.sciencelogic.com/s/. Versions of PowerFlow before version 2.2.0 are no longer supported by ScienceLogic.

To locate and download the PowerFlow RPM file:

1. Go to the ScienceLogic Support site at https://support.sciencelogic.com/s/.
2. Click the Product Downloads tab and select PowerFlow. The PowerFlow page appears.
3. Click the link to the current release. The Release Version page appears.
4. In the Release Files section, click the RPM or ISO link. A Release File page appears.
5. Click Download File at the bottom of the Release File page.

Upgrading OS Packages (for Offline Deployments Only)

Upgrading OS packages for an offline deployment require you to mount the ISO and update the packages that are shipped with the ISO.

1. Mount the PowerFlow ISO onto the system:

   mount -o loop /dev/cdrom /mnt/tmpISMount

2. After you mount the ISO, add a new repository file to access the ISO as if it were a yum repository. Create a /etc/yum.repos.d/localiso.repo file with the following contents:

   [localISISOMount]

   name=Locally mounted IS ISO for packages

   enabled=1

   baseurl=file:///mnt/tmpISMount

   gpgcheck=0

   After you create and save this file, the Linux system can install packages from the PowerFlow ISO.

3. Optionally, you can import the latest GNU Privacy Guard (GPG) key to verify the packages by running the following commands:

   rpm --import /mnt/repo_keys/RPM-GPG-KEY-Oracle

   rpm --import /mnt/tmpISMount/repo_keys/RPM-GPG-KEY-Docker-ce

   rpm --import /mnt/tmpISMount/repo_keys/RPM-GPG-KEY-EPEL-7

4. To disable other repos except the local ISO mount repo, run the following command:

   yum --disablerepo="*" --enablerepo="localISISOMount" update

5. Run the following command to update and install the host-level packages:

   yum update

Downloading Updates to Address Common Vulnerabilities and Exposures (CVEs)

This release includes updates that address the common vulnerabilities and exposures (CVEs) identified since the last release of PowerFlow. If you are using PowerFlow version 2.3.0 or older, you can run a sudo yum update to address these CVEs.

If you do not have Internet access to the Oracle Linux 7 repos used by the yum update, you can manually download the packages directly from the repository links below and copy them to your PowerFlow system:

• https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/glibc-2.17-325.0.3.el7_9.x86_64.rpm

• https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/glibc-common-2.17-325.0.3.el7_9.x86_64.rpm

• https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/glibc-devel-2.17-325.0.3.el7_9.x86_64.rpm

• https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/glibc-headers-2.17-325.0.3.el7_9.x86_64.rpm

Installing the Audit Package for Audit Logging

To upgrade to the latest version of PowerFlow from version 2.2.0 or later, you will need to install the audit package used for audit logging in the latest release.

1. If your PowerFlow system is connected to the Internet, run the following command to download and install the audit package:

   sudo yum install <sl1-powerflow-rpm-location>

2. If your PowerFlow system is not connected to the Internet, you can upgrade using the packages on the ISO file:

   1. Mount the 2.x.x ISO file onto your PowerFlow system by running the relevant mount command for your system. For example:

      mount -oloop /home/isadmin/2.x.x.iso /home/isadmin/2.x.xiso-iso-mount-point/

      where /home/isadmin/2.x.x.iso is the path to your ISO file that you have uploaded, and /home/isadmin/2.x.xiso-iso-mount-point/ is the directory created for the purposes of mounting the ISO locally.

   2. Run the following commands:

      sudo yum install 2.x.xiso-iso-mount-point/audit-2.8.5-4.el7.x86_64.rpm

      sudo yum install 2.x.xiso-iso-mount-point/audit-libs-2.8.5-4.el7.x86_64.rpm

      sudo yum install 2.x.xiso-iso-mount-point/audit-libs-python-2.8.5-4.el7.x86_64.rpm

   3. Create the following file: /etc/yum.repos.d/local-yum-repo.repo.

   4. Add the following lines to the new local-yum-repo.repo file:

      [local-yum-repo]

      name=yum repo from mounted ISO

      baseurl=file:///mnt/tmp_install_mount/

      enabled=1

      gpgcheck=0

   5. Run the following command:

      sudo yum update --disablerepo=* --enablerepo=local-yum-repo

3. You can also manually download and install the audit package from the following links:

   • https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/audit-2.8.4-4.el7.x86_64.rpm

   • https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/audit-libs-2.8.4-4.el7.x86_64.rpm

   • https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/audit-libs-python-2.8.4-4.el7.x86_64.rpm

4. Continue to the following procedure, which is the standard upgrade process used in previous releases.

Upgrading from Version 2.2.0 or Later

Depending on your PowerFlow environment and your company's requirements, select one of the following upgrade options:

• Single-node Upgrade
• Cluster Upgrade with Short Downtime
• Rolling Cluster Upgrade with No Downtime

Perform the steps in the following procedure in the order listed below to ensure a proper upgrade.

Single-node Upgrade

To upgrade a single-node PowerFlow system from version 2.2.0 or later:

1. Make a copy of the docker-compose file that you used to deploy PowerFlow (in case you need to roll back to the previous version).

2. Either go to the console of the PowerFlow system or use SSH to access the server.

3. Log in as isadmin with the appropriate (root) password. You must be root to upgrade using the RPM file.

4. Download the PowerFlow RPM and copy the RPM file to the PowerFlow system.

5. Type the following at the command line:

   sudo yum upgrade <full_path_of_rpm>

   where full_path_of_rpm is the name and path of the RPM file, such as /home/isadmin/sl1-powerflow-2.x.x-1.x86_64.

6. After the RPM is installed, run the following Docker command:

   docker stack rm iservices

   After you run this command, the stack is no longer running.

   If you want to upgrade your services in place, without bringing them down, you may skip this step. Please note that skipping this step might cause the services to take longer to update.

7. If the upgrade process recommends restarting Docker, run the following command:

   systemctl restart docker

   If you restart Docker for this step, you should skip step 10, below.

8. Re-deploy the Docker stack to update the containers:

   docker stack deploy -c /opt/iservices/scripts/docker-compose.yml iservices

9. After you re-deploy the Docker stack, the services automatically update themselves. Wait a few minutes to ensure that all services are updated and running before using the system.

10. If the upgrade process recommends restarting Docker, run the following command:

    systemctl restart docker

    If you restarted Docker after the RPM was installed and before the stack was redeployed with the new docker-compose file, you can run the following powerflowcontrol (pfctl) action to correct potential upgrade issues:

    pfctl --host pf-node-ip '<username>:<password>' node-action --action modify_iservices_volumes_owner

    After you run the above pfctl action, you will need to restart the syncpacks_steprunner service.

11. To view updates to each service and the service versions for services throughout the whole stack, type the following at the command line:

    docker service ls

    Each service now uses the new version of PowerFlow.

Cluster Upgrade with Short Downtime

If you are running PowerFlow in a clustered environment, you should install the RPM on all nodes in the cluster for the upgrade. The order in which you install the RPM on the nodes does not matter. Installing the RPM on each node simply makes the latest PowerFlow container images and scripts available to the system.

Please note that installing the RPM on the nodes does not change the version of the currently running PowerFlow application stack. The new version of PowerFlow is only deployed when you run the docker stack deploy command on the new docker-compose file that is generated after you install the RPM.

The following upgrade procedure for a clustered PowerFlow environment results in only five to ten minutes of downtime.

For more information, including extensive examples, see PowerFlow Multi-tenant Upgrade Process in Appendix B: Configuring the PowerFlow System for Multi-tenant Environments.

To perform a cluster upgrade with short downtime:

1. Make a copy of the docker-compose file that you used to deploy PowerFlow (in case you need to roll back to the previous version).

2. Use SSH to access the node.

3. Log in as isadmin with the appropriate (root) password. You must be root to upgrade using the RPM file.

4. Copy the RPM file to each node.

5. Install the RPM on all nodes in the cluster by typing the following command at the command line for each node:

   sudo yum upgrade <full_path_of_rpm>

   where full_path_of_rpm is the name and path of the RPM file, such as /home/isadmin/sl1-powerflow-2.x.x-1.x86_64. For more information, see Installing the PowerFlow RPM.

   Installing the RPM on all the nodes makes the containers available and updates the docker-compose file on each node, but the existing PowerFlow version will continue running.

6. After you have installed the RPM on all of the nodes, open the new docker-compose.yml file at /opt/iservices/scripts/ and confirm that the versions of Couchbase, RabbitMQ, and any custom workers are using the latest, updated version numbers.

7. After the RPM is installed on the nodes, run the following Docker command:

   docker stack rm iservices

   After you run this command, the stack is no longer running.

   If you want to upgrade your services in place, without bringing them down, you may skip this step. Please note that skipping this step might cause the services to take longer to update.

8. If the upgrade process recommends restarting Docker, run the following command:

   systemctl restart docker

   If you restart Docker for this step, you should skip step 11, below.

9. Re-deploy the Docker stack with the docker-compose file you reviewed in step 6:

   docker stack deploy -c /opt/iservices/scripts/docker-compose.yml iservices

   The time between removing the old version of the stack and deploying the new version of the stack is the only period of down time.

10. After you re-deploy the Docker stack, the services automatically update themselves. Wait a few minutes to ensure that all services are updated and running before using the system.

11. If the upgrade process recommends restarting Docker, run the following command:

    systemctl restart docker

    If you restarted Docker after the RPM was installed and before the stack was redeployed with the new docker-compose file, you can run the following powerflowcontrol (pfctl) action to correct potential upgrade issues. This node-action should be run over all of the cluster nodes, including the manager and worker nodes. For example:

    pfctl --host <pf-node1-ip> '<username>:<password>' -–host <pf-node2-ip> '<username>:<password>' -–host <pf-node3-ip> '<username>:<password>' node-action --action modify_iservices_volumes_owner

    Depending on your system settings, you might need to add the following line before the pfctl in the above command: CRYPTOGRAPHY_ALLOW_OPENSSL_102=1

    After you run the above pfctl action, you will need to restart the syncpacks_steprunner service.

12. To view updates to each service and the service versions for services throughout the whole stack, type the following at the command line:

    docker service ls

    Each service now uses the new version of PowerFlow.

Rolling Cluster Upgrade with No Downtime

For a clustered PowerFlow environment, the following rolling cluster update results in no downtime, but the process requires intermediate compose updates.

For more information, including extensive examples, see PowerFlow Multi-tenant Upgrade Process in Appendix B: Configuring the PowerFlow System for Multi-tenant Environments.

To perform a rolling cluster upgrade with no downtime:

1. Make a copy of the docker-compose file that you used to deploy PowerFlow (in case you need to roll back to the previous version).

2. Use SSH to access the node.

3. Log in as isadmin with the appropriate (root) password. You must be root to upgrade using the RPM file.

4. Copy the RPM file to each node.

5. Install the RPM on all nodes in the cluster by typing the following command at the command line for each node:

   sudo yum upgrade <full_path_of_rpm>

   where full_path_of_rpm is the name and path of the RPM file, such as /home/isadmin/sl1-powerflow-2.x.x-1.x86_64. For more information, see Installing the PowerFlow RPM.

   Installing the RPM on all the nodes makes the containers available and updates the docker-compose file on each node, but the existing PowerFlow version will continue running.

6. After the RPM is installed on the nodes, remove the "core nodes" one by one to cause a failover, and then re-add a new version of the same node without taking down the stack:

   1. Access the Couchbase user interface at https://<IP of PowerFlow>:8091.

   2. On the Servers tab, select a single database node and click Failover. Select Graceful Failover. Manually failing over before updating ensures that the system is still operational when the container comes down.

   3. Modify the /opt/iservices/scripts/docker-compose.yml file at that you used to deploy PowerFlow, and change just one of the Couchbase services and RabbitMQ services to use the new version (the same Couchbase server you previously failed over).

   4. Deploy the docker-compose file with the new updated Couchbase server.

   5. Wait for the new instance of Couchbase to join the existing cluster as the latest version. When it has joined, that core node is updated.

7. Continue failing over nodes and re-deploying them with the new PowerFlow version until all core nodes are updated.
8. After the core nodes are updated, you can proceed similarly with each individual worker nodes. You can update these nodes in groups if that is faster. You do not need to fail over the worker nodes; you can just change the services and images.
9. At the end of the node-by-node upgrade, your existing docker-compose should contain all of the new versions specified by the latest docker-compose file shipped with the RPM.

Troubleshooting the Upgrade from Version 2.2.0 or Later

After upgrading to version 2.4.0 or later, you might encounter an issue with a missing role-based access control (RBAC) permission that prevents the PowerFlow user interface from accessing Flower or RabbitMQ. 

You can resolve this scenario by running one of the following steps:

• If you just upgraded to PowerFlow version 2.4.0, run this powerflowcontrol (pfctl) command:

  pfctl --host <ip_address> <username>:<password> node-action --action upload_default_content

• For PowerFlow version 2.4.1 and later, run the pfctl healthcheck and autoheal actions.

For more information, see Using the powerflowcontrol (pfctl) Command-line Utility.

Uploading Custom Dependencies to the PyPI Server with the iscli Tool

You can use the PowerFlow command-line tool (iscli) to upload custom dependencies to the PowerFlow local Python Package Index (PyPI) Server:

1. Copy the Python package to the pypiserver container.

2. Exec into the container and run the following commands:

   devpi login isadmin

   devpi use http://127.0.0.1:3141/isadmin/dependencies

   devpi upload <location of your package dependencies>

Troubleshooting Upgrade Issues

The following topics describe issues that might occur after the upgrade to version 2.2.0 or later, and how to address those issues.

Cannot mount the virtual environment, or the virtual environment is not accessible

If the docker container does not properly mount the virtual environment, or the virtual environment is not accessible to the environment, you might need to remove and re-deploy the service to resolve the issue.

After upgrading, the syncpacks_steprunner service fails to run

This error flow tends to happen when the syncpacks_steprunner service is deployed, but the database is not yet updated with the indexes necessary for the SyncPack processes to query the database. In most deployments, the index should be automatically created. If the index is not automatically created, which it might do in a clusterd configuration, you can resolve this issue by manually creating the indexes.

In this situation, if you check the logs, you will most likely see the following message:

couchbase.exceptions.HTTPError: <RC=0x3B[HTTP Operation failed. Inspect status code for details], HTTP Request failed. Examine 'objextra' for full result, Results=1, C Source=(src/http.c,144), OBJ=ViewResult<rc=0x3B[HTTP Operation failed. Inspect status code for details], value={'requestID': '57ad959d-bafb-46a1-9ede-f80f692b0dd7', 'errors': [{'code': 4000, 'msg': 'No index available on keyspace content that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online.'}], 'status': 'fatal', 'metrics': {'elapsedTime': '5.423085ms', 'executionTime': '5.344487ms', 'resultCount': 0, 'resultSize': 0, 'errorCount': 1}}, http_status=404, tracing_context=0, tracing_output=None>, Tracing Output={":nokey:0": null}>

To address this issue, wait a few minutes for the index to be populated. If you are still getting an error after the database has been running for a few minutes, you can manually update the indexes by running the following command:

initialize_couchbase -s

Creating a primary index is only for troubleshooting, and primary indexes should not be left on the system.

Upgrading to Couchbase Version 6.6.0

This section contains a set of upgrade considerations for moving to Couchbase version 6.6.0 (Community Edition). Version 2.6.0 of the PowerFlow Platform includes Couchbase version 6.6.0 (Community Edition).

PowerFlow Supported Upgrade Paths

The following constraints are based on the Couchbase version used by the different versions of PowerFlow. For more information, see Couchbase Supported community upgrade paths.

Starting Versions	Path to Current Versions
PowerFlow 2.0.x (Couchbase 5.1.1)	2.0.x (Couchbase 5.1.1) -> 2.1.x to 2.5.x (Couchbase 6.0.2) -> 2.6.0(Couchbase 6.6.0)
PowerFlow 2.1.x (Couchbase 6.0.2)	2.1.x (Couchbase 6.0.2) -> 2.6.0 (Couchbase 6.6.0)

Logs Buckets

When upgrading to Couchbase version 6.6.0, the number of documents in the logs bucket could make the upgrade take longer, as a namespace upgrade is needed.

ScienceLogic recommends that you flush the logs bucket if there are more than 300,000 documents that are taking up close to 2 GB of space in every node. Flushing the logs bucket will speed up the upgrade process. Otherwise, migrating a logs bucket of that size would take two to three minutes per node.

Do not interrupt the upgrade process, as that can corrupt documents. Please wait until the upgrade finishes running.

Run the following command to flush the logs bucket after the PowerFlow version 2.6.0 RPM was installed, but before redeploying the PowerFlow Stack:

pfctl --host <hostname><username>:<password> node-action --action flush_logs_bucket

Alternately, you can flush the logs bucket manually using the Couchbase user interface.

Downgrading

Downgrades from Couchbase 6.6.x are not supported because the namespace is upgraded.

Licensing PowerFlow

Before users can access all of the features of PowerFlow, the Administrator user must license the PowerFlow instance through the ScienceLogic Support site. For more information about accessing PowerFlow files at the ScienceLogic Support site, see the following Knowledge Base article: SL1 PowerFlow Download and Licensing.

When you log in to the PowerFlow system, a notification appears at the bottom right of the screen that states how much time is left in your PowerFlow license. The notification displays with a green background if your license is current, yellow if you have ten days or less in your license, and red if your license has expired. You need to click the Close icon () to close this notification.

You can also track your licensing information on the About page (username menu > About). You can still log into a system with an expired license, but you cannot create or schedule PowerFlow applications.

The administrator and all users cannot access certain production-level capabilities until the administrator licenses the instance. For example, users cannot create schedules or upload PowerFlow applications and steps that are not part of a SyncPack until PowerFlow has been licensed.

If you are not deploying PowerFlow on a production or pre-production environment, you can skip the licensing process.

If you are licensing a PowerFlow High Availability cluster, you can run the following licensing process on any node in the cluster. The node does not have to be the leader, and the licensing process does not have to be run on all nodes in the Swarm.

Licensing a PowerFlow System

To license a PowerFlow system:

1. Run the following command on your PowerFlow system to generate the .iskey license file:

   iscli --license --customer "<user_name>" --email <user_email>

   where <user_name> is the first and last name of the user, and <user_email> is the user's email address. For example:

   iscli --license --customer "John Doe" --email jdoe@sciencelogic.com

2. Run an ls command to locate the new license file: customer_key.iskey.

3. Using WinSCP or another utility, copy the .iskey license file to your local machine.

4. Go to the PowerFlow License Request page at the ScienceLogic Support site: https://support.sciencelogic.com/s/integration-service-license-request:

   

5. For Step 2 of the "Generate License File" process, select the PowerFlow record you want to license.

   You already covered Step 1 of the "Generate License File" process in steps 1-3 of this procedure.

6. Scroll down to Step 3 of the "Generate License File" process and upload the .iskey license file you created in steps 1-3 of this procedure.

7. Click Upload Files.

8. After uploading the license file, click Generate PowerFlow License. A new Licensing page appears:

   

9. Click the .crt file in the Files pane to download the new .crt license file.

10. Using WinSCP or another file-transfer utility, copy the .crt license file to your PowerFlow system.

11. Upload the .crt license file to the PowerFlow server by running the following command on that server:

    iscli -l -u -f ./<license_name>.crt -H <IP_address> -U <user_name> -p <user_password>

    where <license_name> is the system-generated name for the .crt file, <IP_address> is the IP address of the PowerFlow system, <user_name> is the user name, and <user_password> is the user password. For example:

    iscli -l -u -f ./aCx0x000000CabNCAS.crt -H 10.2.33.1 -U isadmin -p passw0rd

ScienceLogic determines the duration of the license key, not the customer.

If you have any issues licensing your PowerFlow system, please contact your ScienceLogic Customer Success Manager (CSM) or open a new Service Request case under the "Integration Service" category.

Licensing Solution Types

The licensing for the PowerFlow platform was separated into three solution types:

• Standard: This solution lets you import and install SyncPacks published by ScienceLogic and ScienceLogic Professional Services, and to run and schedule PowerFlow applications from those SyncPacks. You cannot customize or create PowerFlow applications or steps with this solution type. Features that are not available display in gray text in the user interface.

• Advanced: This solution contains all of the Standard features, and you can also build your own SyncPacks and upload custom applications and steps using the command-line interface. You can create PowerFlow applications using the PowerFlow command-line interface, but you cannot create and edit applications or steps using the PowerFlow builder in the user interface.

• Premium: This solution contains all of the Advanced features, and you can also use the PowerFlow builder, the low-code/no-code, drag-and-drop interface, to create and edit PowerFlow applications and steps.

A yellow text box appears in the PowerFlow user interface when the license is close to expiring, displaying how many days are left before the license expires. The license status and expiration date also displays on the About page in the PowerFlow user interface.

An unlicensed system will not be able to create PowerFlow applications, steps, or schedules. Unlicensed systems will only be able to run applications that are installed manually through SyncPacks.

Features that are locked by licensing solution type are grayed out. If you click on a grayed-out feature, the user interface will display a notification prompting you to upgrade your license.

Configuring a Proxy Server

To configure PowerFlow to use a proxy server:

1. Either go to the console of the PowerFlow system or use SSH to access the PowerFlow server.

2. Log in as isadmin with the appropriate password.

3. Using a text editor like vi, edit the file /opt/iservices/scripts/docker-compose-override.yml.

   PowerFlow uses a docker-compose-override.yml file to persistently store user-specific configurations for containers, such as proxy settings, replica settings, additional node settings, and deploy constraints. The user-specific changes are kept in this file so that they can be re-applied when the /opt/iservices/scripts/docker-compose.yml file is completely replaced on an RPM upgrade, ensuring that no user-specific configurations are lost.

4. In the environment section of the steprunner service, add the following lines:

   services:
     steprunner:
       environment:
   	https_proxy: "<proxy_host>"
   	http_proxy: "<proxy_host>"
   	no_proxy: ".isnet"

   If your proxy appears to only use HTTP and not HTTPS, you will need to use http in both https_proxy environment variables.

   If you do not want to use more than one proxy location, you can use the no_proxy setting to specify all of the locations, separated by commas and surrounds by quotation marks. For example: no_proxy: ".isnet,10.1.1.100,10.1.1.101"
   

   If you want to access external pypi packages while using a proxy, be sure to include pypi.org and files.pythonhosted.org to this section to ensure the proxy enables those locations.

5. In the environment section of the syncpacks_steprunner service, add the following lines:

   services:
     syncpacks_steprunner:
       environment:
   	https_proxy: "<proxy_host>"
   	http_proxy: "<proxy_host>"
   	no_proxy: ".isnet"

   If your proxy appears to only use HTTP and not HTTPS, you will need to use http in both https_proxy environment variables.

   If you want to access external pypi packages while using a proxy, be sure to include pypi.org and files.pythonhosted.org to this section to ensure the proxy enables those locations.

6. Save the settings in the file and then run the /opt/iservices/scripts/compose_override.sh script.

   The compose_override.sh script validates that the configured docker-compose.yml and docker-compose-override.yml files are syntactically correct. If the settings are correct, the script applies the settings to your existing docker-compose.yml file that is used to actually deploy.

7. Re-deploy the steprunners to use this change by typing the following commands:

   docker service rm iservices_steprunner

   docker stack deploy -c /opt/iservices/scripts/docker-compose.yml iservices

Changing the PowerFlow System Password

The PowerFlow system uses two primary passwords. For consistency, both passwords are the same after you install PowerFlow, but you can change them to separate passwords as needed.

To avoid authentication issues, do not use the dollar sign ($) character as the first character in any of the passwords related to PowerFlow. You can use the $ character elsewhere in the password if needed.

PowerFlow uses the following passwords:

• The PowerFlow Administrator (isadmin) user password. This is the password that you set during the PowerFlow ISO installation process, and it is only used by the default local Administrator user (isadmin). You use this password to log into the PowerFlow user interface and to verify API requests and database actions. This password is set as both the "Linux host isadmin" user and in the /etc/iservices/is_pass file that is mounted into the PowerFlow stack as a "Docker secret". Because it is mounted as a secret, all necessary containers are aware of this password in a secure manner. Alternatively, you can enable third-party authentication, such as LDAP or AD, and authenticate with credentials other than isadmin. However, you will need to set the user policies for those LDAP users first with the default isadmin user. For more information, see Managing Users in PowerFlow.
• The Linux Host OS SSH password. This is the password you use to SSH and to log in to isadmin. You can change this password using the standard Linux passwd command or another credential management application to manage this user. You can also disable this Linux user and add your own user if you want. The PowerFlow containers and applications do not use or know this Linux login password, and this password does not need to be the same between nodes in a cluster. This is a standard Linux Host OS password.

Starting in PowerFlow version 3.0.0, you can use the following command to update the PowerFlow Administrator (isadmin) user password:

pfctl password set

This command replaces the ispasswd script from earlier releases of PowerFlow, which was found in /opt/iservices/scripts/ispasswd. The ispasswd script will be deprecated in a future release. 

To change the PowerFlow Administrator (isadmin) user password using the ispasswd script:

1. You can change the mounted isadmin password secret (which is used to authenticate via API by default) and the Couchbase credentials on the PowerFlow stack by running the ispasswd script on any node running PowerFlow in the stack:

   /opt/iservices/scripts/ispasswd

2. Follow the prompts to reset the password. The password must be at least six characters and no more than 24 characters, and all special characters are supported.

   Running the ispasswd script automatically changes the password for all PowerFlow application actions that require credentials for the isadmin user.

3. If you have multiple nodes, copy /etc/iservices/is_pass file, which was just updated by the ispasswd script, to all other manager nodes in the cluster. You need to copy this password file across all nodes in case you deploy from a different node than the node where you changed the password. The need to manually copy the password to all nodes will be removed in a future release of PowerFlow.

If a PowerFlow user makes multiple incorrect login attempts, PowerFlow locks out the user. To unlock the user, run the following command: unlock_user -u <username>

Configuring Security Settings

This topic explains how to change the HTTPS certificate used by PowerFlow, and it also describes password and encryption key security.

Changing the HTTPS Certificate

The PowerFlow user interface only accepts communications over HTTPS. By default, HTTPS is configured using an internal, self-signed certificate.

You can specify the HTTPS certificate to use in your environment by mounting the following two files in the user interface (gui) service:

• /etc/iservices/is_key.pem
• /etc/iservices/is_cert.pem

The SSL certificate for the PowerFlow system only requires the HOST_ADDRESS field to be defined in the certificate. That certificate and key must be identical across all nodes. If needed, you can also add non-HOST_ADDRESS IPs to the Subject Alternative Name field to prevent an insecure warning when visiting the non-HOST_ADDRESS IP.

If you are using a load balancer, the certificates installed on the load balancer should use and provide the hostname for the load balancer, not the PowerFlow nodes. The SSL certificates should always match the IP or hostname that exists in the HOST_ADDRESS setting in /etc/iservices/isconfig.yml. If you are using a load balancer, the HOST_ADDRESS must also be the IP address for the load balancer.

If you are using a clustered configuration for PowerFlow, you will need to copy the key and certificate to the same location on the node.

To specify the HTTPS certificate to use in your environment:

1. Copy the key and certificate to the PowerFlow host.

2. Modify the /opt/iservices/scripts/docker-compose-override.yml file and mount a volume to the gui service. The following code is an example of the volume specification:

   volumes: 
     - "<path to IS key>:/etc/iservices/is_key.pem"
     - "<path to IS certificate>:/etc/iservices/is_cert.pem"

3. Run the following script to validate and apply the change:

   /opt/iservices/scripts/compose_override.sh

4. Re-deploy the gui service by running the following commands:

   docker service rm iservices_gui

   /opt/iservices/scripts/pull_start_iservices.sh

Using Password and Encryption Key Security

When you install the PowerFlow platform, you specified the PowerFlow root password. This root password is also the default isadmin password:

• The root/admin password is saved in a root read-only file here: /etc/iservices/is_pass
• A backup password file is also saved in a root read-only file here: /opt/iservices/backup/is_pass

The user-created root password is also the default PowerFlow password for couchbase (:8091) and all API communications. The PowerFlow platform generates a unique encryption key for every platform installation:

• The encryption key exists in a root read-only file here: /etc/iservices/encryption_key
• A backup encryption key file is also saved in a root read-only file here: /opt/iservices/backup/encryption_key

This encryption key is different from the HTTPS certificate key discussed in the previous topic.

You can use the encryption key to encrypt all internal passwords and user-specified data. You can encrypt any value in a configuration by specifying "encrypted": true, when you POST that configuration setting to the API. There is also an option in the PowerFlow user interface to select encrypted. Encrypted values use the same randomly-generated encryption key.

User-created passwords and encryption keys are securely exposed in the Docker containers using Docker secrets at https://docs.docker.com/engine/swarm/secrets/ to ensure secure handling of information between containers.

The encryption key must be identical between two PowerFlow systems if you plan to migrate from one to another. The encryption key must be identical between High Availability or Disaster Recovery systems as well.

PowerFlow supports all special characters in passwords.

NOTE: For detailed security information about the configuration of Docker Enterprise, see the SL1 PowerFlow: System Security Plan for Docker Enterprise document.

Configuring Additional Elements of PowerFlow

If you have multiple workers running on the same PowerFlow system, you might want to limit the amount of memory allocated for each worker. This helps prevent memory leaks, and also prevents one worker using too many resources and starving other workers. You can apply these limits in two ways:

• Set a hard memory limit in Docker (this is the default)
• Set a soft memory limit in the worker environment

Setting a Hard Memory Limit in Docker

Setting a memory limit for the worker containers in your docker-compose.yml file sets a hard limit. If you set a memory limit for the workers in the docker-compose file and a worker exceeds the limit, the container is terminated via SIGKILL.

If the currently running task caused memory usage to go above the limit, that task might not be completed, and the worker container is terminated in favor of a new worker. This setting helps to prevent a worker from endlessly running and consuming all memory on the PowerFlow system.

You can configure the hard memory limit in the steprunner service of the docker-compose.yml file:

deploy:

resources:

limits:

memory: 2G

Setting a Soft Memory Limit in the Worker Environment

You can set the memory limit for a worker application, and not at the Docker level. Setting the memory limit at the application level differs from the hard memory limit in Docker in that if a worker exceeds the specified memory limit, that worker is not immediately terminated via SIGKILL.

Instead, if a worker exceeds the soft memory limit, the worker waits until the currently running task is completed to recycle itself and start a new process. As a result, tasks will complete if a worker crosses the memory limit, but if a task is running infinitely with a memory leak, that task might consume all memory on the host.

The soft memory limit is less safe from memory leaks than the hard memory limit.

You can configure the soft memory limit with the worker environment variables. The value is in KiB (1024 bytes). Also, each worker instance contains three processes for running tasks. The memory limit applies to each individual instance, and not the container as a whole. For example, a 2 GB memory limit for the container would translate to 2 GB divided by three, or about 700 MB for each worker:

steprunner:

image: repository.auto.sciencelogic.local:5000/is-worker:2.6.0

environment:

additional_worker_args: ' --max-memory-per-child 700000'

PowerFlow Task Processing and Memory Handling

Review the settings in this section to prevent an "Out of Memory" error, also called an "Oomkill" error or exit code 137. These errors occur when a container uses more memory than the container has been allotted.

This section will help you to recognize and diagnose these situations, and determine what additional configurations are available when working with a PowerFlow system that is running out of memory.

Background

By default steprunner containers have a 2 GB memory limit with three process threads by default. Limits for containers are set in the docker-compose file.

Use the docker stats command to see what the current memory usage of containers are in PowerFlow, along with current memory usage for those containers.

CPU and Memory Requirements for PowerFlow

The following table lists the CPU and memory requirements based on the number of synced objects for a PowerFlow system:

Minimum Number of Synced Objects	CPU Cores	Memory RAM (GB)	Hard Disk (GB)
30,000	8	24	100
65,000	8	32	100
100,000	8	64	200

Typical PowerFlow Deployments:

• Standard Single-node Deployment (1 Node): One node, 8 CPU, 24 GB memory minimum, preferably 34 GB to 56 GB memory, depending on workload sizes.

• Standard Three-node Cluster (3 Nodes): Three nodes, 8 CPU, 24 GB memory minimum, preferably 34 GB to 56 GB memory, depending on workload sizes.

• 3+ Node Cluster with Separate Workers (4 or More Nodes): Three nodes, 8 CPU, 24 GB memory minimum, preferably 34 GB to 56 GB memory, depending on workload sizes.

Recommended Memory Allocation of PowerFlow Nodes

The following sizings will automatically be applied if you run powerfcontrol (pfctl) actions such as apply 16GB overrides and apply 32GB overrides. These commands should only be run in a Software-as-a-Service (SaaS) environment. For more information about the pfctl actions, see apply_<n>GB_override, verify_<n>GB_override.

Template Size	Device Load
16 GB	25,000 to 30,000
32 GB	30,000 to 70,000
64 GB	70,0000 to 350,000
128 GB	350,000 and above

SaaS Deployments

The following settings are specifically for Software-as-a-Service (SaaS) environments, and they ensure full replication of all services in any failover scenario.

Example Code: docker-compose for SaaS

16 GB Deployments

The following settings support up to approximately 25,000-30,000 devices, depending on relationship depth.

Example Code: docker-compose for 16 GB Deployments

Allocations (per node):

• Swarm leader: between 2 to 4 GB left over on node
• Couchbase: reserves 1.5 GB memory, uses 1.5 to 4 GB, depending on operation
• Flower, pypiserver, dexserver, scheduler: no limit by default, never use more than 100 MB
• RabbitMQ: no limit, typically low usage (less than 100 MB), might spike with heavy loads
• Contentapi: 1 GB memory limit
• Redis: 3 GB soft limit, 5 GB hard limit; after 5 GB, Redis will automatically eject older data for new (reduced)
• 1x steprunners: 3 GB memory limit each (steprunner count decreased, memory limit increased)

Total limits/max expected memory usage: 4 GB + 4 GB + 500 MB + 1 GB + 3 GB +3 GB = 15.5GB/16GB

32 GB Deployments

The following settings support up to approximately 70,000 devices, depending on relationship depth.

Example Code: docker-compose for 32 GB Deployments

Allocations (per node):

• Swarm leader: between 2-4 GB left over on node
• Couchbase: reserves 1.5 GB memory, uses 1.5 to 4GB, depending on operation
• Flower, pypiserver, dexserver, scheduler: no limit by default, never uses more than 100 MB
• RabbitMQ: should anticipate for 1 GB at larger sizes
• Contentapi: 2 GB memory limit (in a healthy running environment, should be less than 100 MB)
• Redis: 3 GB soft limit, 5 GB hard limit; after 3 GB, Redis will automatically eject older data for new (reduced)
• 2x steprunners: 7 GB memory limit each (steprunner count decreased, memory limit increased)

Total limits/max expected memory usage: 4 GB + 4 GB + 200 MB + 1 GB + 2 GB + 4 GB +7(2) GB = 29.2/32GB

64 GB Deployments

The following settings support over 70,000 devices, depending on relationship depth.

Example Code: docker-compose for 64 GB Deployments

If you use the following format for the names of the custom steprunners, they will display on the PowerFlow Control Tower page: steprunner_<name>.

Other actions needed:

• Increase Couchbase Allocations: increase data bucket allocation by 5 GB, and increase index allocation by 5 GB
• Update the current Device Sync or Interface Sync applications, and specify them to run on the xlsync queue

Allocations (per node):

• Swarm leader: between 2 to 4 GB left over on node
• Couchbase: reserves 1.5 GB memory, uses 1.5 to 4 GB standard; add 10 GB for heavy scale readiness (5 GB to data bucket 5 to index), up to 14 GB
• Flower, pypiserver, dexserver, scheduler: no limit by default; never uses more than 100 MB
• RabbitMQ: anticipate for 4 GB at extremely larger sizes
• Contentapi. 2 GB memory limit (in a healthy running environment, should be less than 100 MB)
• Redis : 3 GB soft limit, 5 GB hard limit
• 4x steprunners: 2 GB memory limit each (steprunner count decreased, memory limit increased), 8 GB total
• 1x steprunner: 15 GB memory limit (xlqueue steprunner), 15 GB total

Total limits/max expected memory usage: 4GB + 14GB +100mb + 4gb + 2GB + 5GB + 8GB + 15GB = 52GB/64

There is still approximately 12 GB to be allocated to needed services. This configuration and allotment may change depending on future assessment of customer systems.

128 GB Deployments

This deployment template is only to be used for customers with a very large number of devices, such as over 350,000 devices. For a deployment this large, you will need append additional customizations and queues to the following template. This is just a baseline; discuss with ScienceLogic if you plan to use a 128 GB deployment.

Example Code: docker-compose for 128 GB Deployments

Differences from 64 GB Deployments

• The default queue worker count was tripled. These additional workers may be dedicated to any other queue as needed by your customizations.
• Increased redis limits to allow for more processing.

Allocations (per node)

• Swarm leader: Between 2-4 GB left over on node.
• Couchbase: Reserves 1.5 GB memory, uses 1.5 – 4 GB standard, +10 GB for heavy scale readiness (5 GB to data bucket, 5 GB to index): up to 14 GB
• Flower, pypiserver, dexserver, scheduler: No limit by default. Never uses more than 100 MB.
• Rabbitmq: Should anticipate 6 GB at extremely larger sizes.
• Contentapi: 2 GB memory limit. Iin a healthy running environment, should be less than 100 MB.
• Redis: 6 GB soft limit, 12 GB hard limit.
• 24x steprunners: 2 GB memory limit each (steprunner count decreased, memory limit increased): 48 GB
• 1x steprunner: 15 GB memory limit (xlqueue steprunner): 15 GB

Total limits/max expected memory usage:

4 GB + 14 GB +100 MB + 6 GB + 2 GB + 12 GB + 48 GB + 15GB = 101 GB/128

Identifying Oomkills

Typically Oomkills occur only on PowerFlow systems with over 8,000 devices with many relationships or 30,000 interfaces that are being synced.

To identify Oomkills:

1. Use the healthcheck action with the powerflowcontrol (pfctl) command-line utitlity to identify the occurrence. Sample feedback that shows an Oomkill situation:

   

2. Log in to the node where the container failed.

3. From the node where the container failed, run the following command:

   journalctl -k | grep -i -e memory -e oom

4. Check the result for any out of memory events that caused the container to stop. Such an event typically looks like the following:

   is-scale-03 kernel: Out of memory: Kill process 5946 (redis-server) score 575 or sacrifice child

Common Causes of High Memory and Oomkills

• Large-scale Device Sync, Attribute Sync, and Interface Syncs can cause out of memory events. The following situations might need more than the default limit allocation:

• Device Sync for about 9,000 to 12,000 devices, with large amounts of relationships
• A large-scale Attribute Sync with large numbers of devices
• An large-scale Interface Sync with about 10,000 or more interfaces

• Python can be very resource-intensive, especially when it comes to loading or dumping large JSON files. JSON dumps and JSON loads can be inefficient and use more memory than expected. To avoid Oomkill in these situations, instead of using JSON for serialization, you can "pickle" the dict() object from Python and then dump or load that.
• A cursor size issue can occur when GraphQL responses contain extremely large cursor sizes, increasing the amount of data returned by SL1 when making API requests. This issue was resolved in SL1 11.1.2 and later.

Questions to Ask when Experiencing Oomkills

• How many devices or interfaces are being synced?
• How often are the devices or interfaces being synced?
• What does the schedule look like? How many scheduled large integrations are running at the same time?
• What is the likelihood that those schedules are hitting double large syncs on one worker?
• If this is a custom SyncPack, should the workload be using this much memory? Can I optimize, or maybe paginate?

Avoiding Oomkills

The following table explains how to configure your PowerFlow system if you are encountering Oomkills:

Configuration	Steps	Requirements	Impact
Update scheduled applications	Review all scheduled application syncs and make sure you do not schedule two very large syncs to run at the same time of day.	None.	Separate timings of large-scale runs.
Increase the memory limit (SaaS only, not on-premises PowerFlow systems)	Increase the memory in the docker-compose file.	Host must have enough additional memory to allocate.	More rooms for tasks to run concurrently on one worker, increased memory allocation to host.
Reduce worker thread count	Set worker_threads=1 for the steprunner environment variable in the docker-compose file.	None.	More room for large tasks to run, but fewer concurrent tasks (throughput).
Dedicated worker nodes, dedicated queues	Create dedicated queues for certain workloads to run on only designated workers.	Additional nodes are needed.	Provides dedicated resources for specific workflows. Generally used for very environment-demanding workloads.

After making any of the above configuration changes, be sure to run the healthcheck and autoheal actions with thepowerflowcontrol (pfctl) command-line utility before you log out of PowerFlow and redeploy the PowerFlow stack. For more information, see healthcheck and autoheal.

Avoiding Node Exhaustion

Node exhaustion occurs when more memory is allocated to containers than is available on the host. If memory is exhausted on the Swarm leader node and the cluster operations cannot process, all containers will restart. You will see "context deadline exceeded" in docker logs if you run journalctl –-no-page |grep docker |grep err.

The following table explains how to configure you PowerFlow system to prevent node exhaustion from occurring again:

Configuration	Steps	Requirements	Impact
Reduce steprunner replica count	Reduce the replica count of the steprunner in the docker-compose file.	None.	Fewer concurrent processes, less memory usage on host.
Reduce redis memory limits	Set the MAXMEMORY environment variable in the docker-compose file for redis (soft limit), reduce memory limit in docker-compose (hard limit)	None.	Less possible room for cached data in very large syncs, less ability for heavy concurrent runs at the same time, less ability to view result data in the user interface.
Dedicated worker nodes, dedicated queues	Create dedicated queues for certain workloads to run on only designated workers	Additional nodes are needed.	Provides dedicated resources for specific workflows. Generally used for very environment-demanding workloads.
Drained manager	Similar to dedicated worker nodes. This offloads swarm management work to another node.	Additional (smaller) nodes are needed.	Eliminates possibility of cluster logic failure due to memory exhaustion. Alternatively, just make sure the existing nodes have enough room.

After making any of the above configuration changes, be sure to run the healthcheck and autoheal actions with thepowerflowcontrol (pfctl) command-line utility before you log out of PowerFlow and redeploy the PowerFlow stack. For more information, see healthcheck and autoheal.

 

Best Practices for Running PowerFlow with Production Workloads

If you are running PowerFlow in a Software-as-a-Service (SaaS) environment in the cloud, consider the following best practices to avoid failed PowerFlow Syncs and memory issues.

Avoid Debug Logging for Large-scale Runs

When you run a large-scale Device Sync or Interface Sync in Debug mode, PowerFlow logs all of the data that is requested, compared, and sen. Using Debug mode in this way moight cause the PowerFlow system to appear to be unresponsive for a period of time, or until the issue is identified and resolved by ScienceLogic Support.

If you need detailed logs for a large number of events, you should use the Info log level instead.

Additional Queues Might be Needed for Large-scale Runs

SaaS environments for PowerFlow are configured by default with a single queue. All Syncs and tasks run in a "first-in, first-out" (FIFO) manner. If an extremely large event spike occurs, or if a backlog of tasks are triggered, PowerFlow will backlog all tasks until the queue is processed. This default is more than sufficient for most PowerFlow environment, and it provides a consistent balance of throughput, scale, and replication for each of your Syncs.

If you want to separate workloads for large-scale environments, such as Device Sync and Incident Sync, you can allocate additional dedicated queues or nodes. To request additional dedicated queues, contact ScienceLogic Support.

Avoid Running Large-scale Syncs Simultaneously

ScienceLogic recommends that you do not simultaneously run multiple Device Syncs or Interface Syncs in large-scale environments (over 15,000 devices). Querying for all devices or interfaces in both ServiceNow and SL1 might have a large performance impact on the PowerFlow system and other systems involved.

If you want to ensure continually optimized performance, run only one large Sync at a time, and schedule the Syncs to run a different times.

For customers of a Managed Service Provider (MSP), ScienceLogic can provide a dedicated node for processing multiple Device Syncs. If you are interested in this deployment, contact ScienceLogic Support.

PowerFlow Management Endpoints

This section provides technical details about managing PowerFlow. The following information is also available in the PowerPacks in Using SL1 to Monitor SL1 PowerFlow.

Flower API

Celery Flower is a web-based tool for monitoring PowerFlow tasks and workers. You can access Flower at https://<IP of PowerFlow>/flower/dashboard.

Flower lets you see task progress, details, and worker status:

The following Flower API endpoints return data about the Flower tasks, queues, and workers. The tasks endpoint returns data about task status, runtime, exceptions, and application names. You can filter this endpoint to retrieve a subset of information, and you can combine filters to return a more specific data set.

/flower/api/tasks. Retrieve a list of all tasks.

/flower/api/tasks?app_id={app_id}. Retrieve a list of tasks filtered by app_id.

/flower/api/tasks?app_name={app_name}. Retrieve a list of tasks filtered by app_name.

/flower/api/tasks?started_start=1539808543&started_end=1539808544. Retrieve a list of all tasks received within a time range.

/flower/api/tasks?state=FAILURE|SUCCESS. Retrieve a list of tasks filtered by state.

/flower/api/workers. Retrieve a list of all queues and workers

For more information, see the Flower API Reference at https://flower.readthedocs.io/en/latest/api.html.

If you use the ScienceLogic: PowerFlow PowerPack to collect this task information, the PowerPack will create events in SL1 if a Flower task fails. For more information, see Using SL1 to Monitor PowerFlow.

Couchbase API

The Couchbase Server is an open-source database software that can be used for building scalable, interactive, and high-performance applications. Built using NoSQL technology, Couchbase Server can be used in either a standalone or cluster configuration.

The following image shows the CouchBase user interface, which you can access at port 8091, such as https://<IP of PowerFlow: 8091:

The following Couchbase API endpoints return data about the Couchbase service. The pools endpoint represents the Couchbase cluster. In the case of PowerFlow, each node is a Docker service, and buckets represent the document-based data containers. These endpoints return configuration and statistical data about each of their corresponding Couchbase components.

<hostname_of_PowerFlow_system>:8091/pools/default. Retrieve a list of pools and nodes.

<hostname_of_PowerFlow_system>:8091/pools/default/buckets. Retrieve a list of buckets.

For more information, see the Couchbase API Reference.

You can also use the Couchbase PowerPack to collect this information. For more information, see Using SL1 to Monitor PowerFlow.

RabbitMQ

RabbitMQ is an open-source message-broker software that originally implemented the Advanced Message Queuing Protocol and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol, Message Queuing Telemetry Transport, and other protocols. 

The following image shows the RabbitMQ user interface, which you can access at port 15672, such as https://<IP of PowerFlow: 15672:

Docker Statistics

You can collect Docker information by using SSH to connect to the Docker socket. You cannot currently retrieve Docker information by using the API.

To collect Docker statistics:

1. Use SSH to connect to the PowerFlow instance.

2. Run the following command:

   curl --unix-socket /var/run/docker.sock http://docker<PATH>

   where <PATH> is one of the following values:

• /info
• /containers/json
• /images/json
• /swarm
• /nodes
• /tasks
• /services

You can also use the Docker PowerPack to collect this information. For more information, see Using SL1 to Monitor PowerFlow.