Updating SL1

This section provides an overview of the System Updates page, detailed steps for performing an SL1 upgrade, and detailed steps on upgrading MariaDB, upgrading PowerPacks, and performing reboots.

As of version 12.2.0, SL1 can be deployed only on Oracle Linux 8 (OL8) operating systems. If you are upgrading from a version of SL1 prior to 12.1.1 that does not already run on OL8, you must first upgrade to SL1 12.1.1 or 12.1.2 and then convert to OL8 before upgrading to SL1 12.2.0 or later.

Depending on the version of SL1 that you are currently running, you might be required to upgrade to one or more earlier versions of SL1 before you can upgrade to 12.2.0 or later.

For more information about supported upgrade paths, see the upgrade notes included in the SL1 release notes for the release you want to upgrade to as well as the OL8 Conversion Resource Center on the ScienceLogic Support portal, which includes links to numerous resources such as the Oracle Linux 8 Conversion Guide. The conversion guide includes prerequisites, full instructions for converting to OL8 for all deployment types, FAQs, and other helpful information to walk you through the OL8 conversion process.

If you take no action before October 31, 2024, all older SL1 systems with OL7 will continue to run, but ScienceLogic will not support them, and the systems might not be secure.

The SL1 system update process supports Python 3.

Use the following menu options to navigate the SL1 user interface:

To view a pop-out list of menu options, click the menu icon ().
To view a page containing all of the menu options, click the Advanced menu icon ().

The System Updates Page

The System Updates page (System > Tools > Updates) allows you to update the software on your SL1 appliances.

You must first download the update file to the local computer. You can then import the software update through the user interface.

After you import a software update to your SL1 system, the SL1 system can automatically stage the software update. Staging is when the software is copied to each ScienceLogic appliance. Staging allows SL1 to simultaneously apply the software changes to each ScienceLogic appliance, regardless of the speed of the connection to each ScienceLogic appliance. You can allow the SL1 system to automatically stage the software or you can manually stage the software.

After the software update is staged, you can run preupgrade checks and then deploy the software.

To apply updates to an existing Data Collector, that Data Collector must be a member of a Collector Group. In some SL1 systems, users might have to create a Collector Group for a single Data Collector before applying updates.

To conserve disk space on Data Collectors and Message Collectors, after an update, SL1 removes previous Docker images.

The System Updates page displays a list of deployed updates and the date and time each update was applied. The following information displays for each update in the list:

EM7 Version. Displays the version number for each SL1 patch that has been imported (and optionally staged and deployed) to the SL1 system. SL1 releases include new features and address issues with previous releases.
OS Version. Displays the version number for each platform patch that has been imported (and optionally staged and deployed) to the SL1 system. Platform patches include updates to the ScienceLogic operating system, including newer versions of industry-standard packages.
Update Signature. Name of the entity that released the patch and type of patch. Usually "ScienceLogic Official Release".
Imported On. Date and time the software update was loaded onto the SL1 system.
Imported By. SL1 user who loaded the software update onto the SL1 system.
Import Status. Status of the import process. Clicking on the log icon displays the log file associated with importing the selected software. Possible values are:
- In Progress. Software is currently being imported by the system.
- Complete. Software has been imported successfully.
- Failed. Software import has failed due to an unexpected condition. Contact ScienceLogic Support for assistance.
Staging Status. Status of the staging process. Clicking on the log icon displays the log file associated with staging the selected software. Possible values are:
- --. No staging request is active and software has not been staged on any SL1 appliances.
- Scheduled. SL1 is aware of the staging request and is preparing for staging.
- In Progress. Staging is in progress but has not completed.
- Complete. Staging has completed, and all appliances are ready to deploy the software.
- Incomplete. Staging has completed, and one or more appliances are ready to deploy the software.
- Cancelled. User manually cancelled the staging process.
- Outdated. The current update is not the latest or has already been installed.
- Failed. An unexpected error occurred in the staging process. Contact ScienceLogic Support for assistance.
Preupgrade Status. Status of the preupgrade checks. Possible values are:
- --. No preupgrade checks are active.
- Complete. Preupgrade checks have completed successfully.
- Incomplete. Preupgrade checks have not yet completed.
- Failed. An unexpected error occurred in the preupgrade checks. Contact ScienceLogic Support for assistance.
Deployment Status. The current deployment state. Possible values are:
- --. No deployment request is active and software has not been deployed on any SL1 appliances.
- Scheduled. SL1 is aware of the deployment request and is preparing for deployment.
- In Progress. Deployment is in progress but has not completed.
- Complete. Deployment has completed, and all appliances are updated.
- Incomplete. Deployment has completed, and one or more appliances are updated.
- Cancelled. User manually canceled the deployment.
- Outdated. The current update is not the latest or has already been installed.
- Failed. An unexpected error occurred in the deployment process. Contact ScienceLogic Support for assistance.
Deployment Status Date. Specifies the date and time the software update was last deployed.

From the System Updates page, you can also click the Appliance List button to view a list of appliances in your SL1 stack. The following information displays for each appliance in the list:

Appliance Name. The name assigned to the SL1 appliance.
IP. The appliance's IP address.
mid. The appliance's module ID.
Module Type. The SL1 appliance type.
Build. The version of SL1 currently running on the appliance.
MariaDB. The version of MariaDB currently running on the appliance.
Platform. The current operating system platform version for the appliance.
Task Manager Paused? Indicates whether the task manager service is paused.
Patch Eligible. Indicates if the appliance is eligible for patching.
Patch Enabled. Indicates if patching is enabled or disabled for the appliance.
To enable or disable patching for an appliance, select the checkbox for that appliance on the Appliance List modal, select Enable Selected Appliances or Disable Selected Appliances from the Select Action menu, and then click Go.

The Workflow for Updating SL1

The following sections describe the steps to plan and deploy an SL1 update. If would like assistance planning an upgrade path that minimizes downtime, please contact your Customer Success Manager.

The workflow for updating SL1 is:

Plan the update.
Schedule maintenance windows.
Review pre-upgrade best practices for SL1.
Back up SSL certificates.
Set the timeout for PhoneHome Watchdog. (Only required if upgrading from SL1 11.1.x or earlier.)
Run the Pre-Upgrade Test for PhoneHome Database Servers. (Only required if upgrading from SL1 11.1.x or earlier.)
Adjust the timeout for slow connections.
Run the system status script on the Database Server or All-In-One before upgrading.
Update the SL1 Distributed Architecture using the System Update tool.
Upgrade MariaDB, if needed.
Reboot SL1 appliances, if needed.
Upgrade to Aurora 3 RDS (MySQL 8.0). (Only required if deployed on AWS and currently on Aurora 2.)
Restore SSL Certificates.
Reset the timeout for PhoneHome Watchdog.
Update default PowerPacks.
Configure Subscription Billing (one time only).

The option to enable a Military Unique Deployment (MUD) configuration is not available for SL1 12.1.x or 12.2.0 installations or upgrades.

Planning the Update

Before upgrading SL1, perform the following steps that are specific to your organization:

Read the release notes to determine:

What is fixed?
What is new?
What has changed?
What has been deprecated?
What are the supported upgrade paths?
What are the important notes that you need to know before upgrading?

Read the Known Issues for the release. These can be found in the release notes and at https://support.sciencelogic.com/s/known-issues.
Identify all integrations and third-party applications that access the SL1 database or manipulate data on SL1. Determine how to disable these integrations during the deployment and re-enable after deployment.
Identify activities and customers that will be affected by maintenance windows and schedule and inform appropriately.
Identify custom work (PowerPacks, Run Book Automations, Event policies, Dashboard widgets) and ensure that it is backed up so you can restore it if necessary.

Ensure that the each SL1 node or appliance has at least 4 GB of free space in the /var partition for pre-upgrade and deployment. Ensure that each SL1 node or appliance has at least 1 GB of free space in / (the root partition) to allow you to deploy the upgrade; however, depending on the appliance type and version you are upgrading from, you might need up to 3 GB of free space in / (the root partition).

If you are upgrading from a version of SL1 prior to version 8.10.0, see Updating SL1 Appliances to Oracle Linux.

Scheduling Maintenance Windows

Upgrading SL1 includes a minimum of two and possibly four maintenance windows:

Import and stage update and run the pre-upgrade script. These steps can take place prior to the day of upgrade and do not affect SL1 functionality. ScienceLogic suggest you perform these steps at least three days before the planned upgrade and ideally a week before the planned upgrade.
Deploy update. On the day of the upgrade, put all SL1 appliances in maintenance mode. The SL1 system will not be available during this procedure. Update the SL1 Distributed systems.
Update MariaDB (if required). The SL1 system will not be available during this procedure. Refer to the release notes for your current release to determine if you must upgrade MariaDB.
Reboot Appliances (if required). Individual SL1 appliances will not be available during these procedures. Refer to the release notes for your current release to determine if you must reboot all SL1 appliances after upgrading.

Identify activities and users that will be affected by these maintenance windows, and schedule the maintenance windows appropriately. Be sure to communicate all downtime with users.

Pre-Upgrade Best Practices for SL1

Before you upgrade, check the following:

Review the hardware specifications of all the appliances in your system to ensure they meet the requirements for the current usage of your system. For more details about sizing and capacity for your specific environment, contact your Customer Success Manager and see https://support.sciencelogic.com/s/system-requirements.
Verify that recent backups are available for your system.
Ensure that each SL1 appliance has a valid license.
Ensure that a Data Collector is a member of a Collector Group if you are applying updates to an existing Data Collector. In some SL1 systems, users might have to create a Collector Group for a single Data Collector.
Ensure that each Data Collector is listed as "Available" to the Database Server. To check, see the Collector Status page (System > Monitor > Collector Status).

Backing Up SSL Certificates

To back up your SSL Certificates:

Log in to the console of the Database Server or SSH to the Database Server.
Open a shell session and type the following commands at the shell prompt:

cp /etc/nginx/silossl.key /etc/nginx/silossl.key.bak

cp /etc/nginx/silossl.pem /etc/nginx/silossl.pem.bak
Repeat these steps on each Database Server in your SL1 system.

Setting the Timeout for PhoneHome Watchdog

This section applies to users who are upgrading from SL1 11.1.x or earlier and have an existing PhoneHome configuration. If you are upgrading from SL1 11.2.0 or later or you do not have a pre-11.2.0 PhoneHome configuration, you can ignore this section.

You can manually adjust the settings for the PhoneHome Watchdog server to reduce CPU consumption during the upgrade process. To do this:

Log in to the console of the Data Collector as the root user or open an SSH session on the Data Collector.
At the command line, type the following:
```
phonehome watchdog view
```

You should see something like the following:

Current settings:
autosync: yes
interval: 20
state: enabled
autoreconnect: yes
timeoutcount: 2
check: default

Note the settings for interval and timeoutcount, so you can restore them after the upgrade.

To change the settings for SL1 upgrade, type the following at the command line:

sudo phonehome watchdog set interval=120;
sudo phonehome watchdog set timeoutcount=2;
systemctl stop em7_ph_watchdog;
systemctl start em7_ph_watchdog;

Repeat the steps in this section on each Data Collector.
Repeat the steps in this section on each Message Collector.
Repeat the steps in this section on each Database Server.

Running the Pre-Upgrade Test for PhoneHome Database Server

When upgrading to SL1 version 11.2.0 or later, a pre-upgrade test in the system update procedure checks for existing PhoneHome Database Servers. Specifically, this test looks for PhoneHome token IDs inside the /home/phonehome0/config.json file, and fails if the value of the id field is less than or equal to "0".

In versions of SL1 prior to version 11.2.0, the primary PhoneHome Database Server was not self-registered with a token, causing it to have an id value of "0".

Therefore, if you are currently using a version of SL1 prior to version 11.2.0 and you are using PhoneHome communication with your SL1 Collectors, you or your SL1 administrator must complete the following one-time manual configuration steps prior to upgrading to version 11.2.0 or later.

Do not attempt to upgrade to SL1 version 11.2.0 or above until all pre-upgrade tests are successful on all PhoneHome Database Servers.

To register a token for your primary PhoneHome Database Server:

Log in to the console of the Database Server or use SSH to access the server.
To determine if your PhoneHome Database Server is registered, type the following command and locate its id value:

cat /home/phonehome0/config.json
If a PhoneHome Database Server has an id value of "0", type the following command and locate the id of the current appliance:

phonehome status
Type the following command and locate the PhoneHome token:

phonehome token <id from step 3>
Type the following command to register the PhoneHome token:

phonehome register <token from step 4>
Repeat steps 2-5 for all PhoneHome Database Servers that have an id value of "0".
Type the following command to ensure that all of your PhoneHome Database Servers are synced:

phonehome sync
Repeat step 2 and confirm that all Database Servers have id values greater than "0".

After you have downloaded but not yet installed the ISO, you can also go to the System Updates page (System > Tools > Updates) and check the Preupgrade Status column to determine if the upgrade has successfully passed the pre-upgrade test. If the Preupgrade Status is Incomplete, click the magnifying glass icon () to determine which appliance is failing during the pre-upgrade test. Do not install the upgrade until the Preupgrade Status status displays as Complete.

Adjusting the Timeout for Slow Connections

If you have slow connections between SL1 appliances, you can adjust the timeout values for staging and deploying upgrades.

To adjust the timeouts:

Log in to the console of the Database Server or SSH to the Database Server.
Open a shell session and type the following at the shell prompt:

sudo pcli set-patcher-param staging_wait_time <timeout_in_seconds>

where:

<timeout_in_seconds> is the timeout value, in seconds, for staging for each SL1 appliance. The default value is 1800 seconds (30 minutes). You can increase this value for slow connections.
Type the following at the shell prompt:

sudo pcli set-patcher-param deploy_wait_time <timeout_in_seconds>

where:

<timeout_in_seconds> is the timeout value, in seconds, for deploying to each SL1 appliance. The default value is 3600 seconds (1 hour). You can increase this value for slow connections.

If you are upgrading from a version of SL1 prior to version 8.14.0, see Adjusting the Timeout for SL1 8.12 and Prior Releases.

Running the System Status Script Before Upgrading

SL1 includes a script, system_status.sh, that provides diagnostic data for each node or appliance in your SL1 system.

On SL1 systems prior to 10.2.0, after running the system status script, you must ensure that the file /var/lob/em7/silo.log has the owner and group "s-em7-core".

Running the System Status Script

If you are running SL1 version SL 8.14.0 or later, SL1 automatically runs the system status script every 15 minutes on each node or appliance in your SL1 system.

ScienceLogic recommends that you view the output from the system status script before upgrading:

In SL1, go to the Appliance Manager page (System > Settings > Appliances).
Locate the SL1 appliance that you want to view diagnostic information about.
Click on its magnifying-glass icon () to view the output of the system status script for that appliance.
If the output includes errors and you need help fixing them, contact ScienceLogic Customer Support to fix the errors before upgrading.
Repeat for each node or appliance in your SL1 system.

To get the very latest status before upgrading, manually run the system status script on each Database Server or All-In-One Appliance.

If you are upgrading from an SL1 8.12.x system, see Running the System Status Script on SL1 8.12.x Releases. If you are upgrading from an SL1 8.10.x or prior release, see Running the System Status Script on SL1 8.10 and Prior Releases.

Updating the SL1 Distributed Architecture

Any SL1 Distributed Architecture system running 8.6.0 or later can be upgraded by importing, staging, and deploying a single update file.

Upgrading the SL1 Distributed Stack includes the following steps:

Download the update.
Import the update.
Stage the Update.
Run the pre-upgrade check.
Place all SL1 appliances in Maintenance Mode.
Deploy the update.
Remove all SL1 appliances from Maintenance Mode

Database Servers and Data Engines in your SL1 stack are deployed first; other appliance types cannot be deployed until the Database Servers and Data Engines are finished deploying.

If you are upgrading from an SL1 system between 8.1.1 and 8.5.0, see Upgrading the SL1 Distributed Architecture on SL1 Versions 8.5.0 and Earlier.

If you are currently running an SL1 version prior to 8.12.0, you must go to the System Updates page and disable automatic staging (System > Tools > Updates > Actions > Disable automatic staging).

If you have previously used manual staging, you must also perform these additional steps:

Select all updates in the EM7 Releases pane and select all updates in the ScienceLogic OS pane.
In the Select Action menu, select Unstage Update (remove staging policy override). Click Go.
For software that was previously staged with automatic staging, Unstage Update (remove staging policy override) does not affect staging.

Downloading the Update

Before you can load a patch or update onto your instance of the SL1 system, you must first download the patch or update to your local computer.

The following steps do not affect the performance of the SL1 system. ScienceLogic recommends that you perform these steps at least three days before upgrading.

To download the patch or update:

Log in to https://support.sciencelogic.com. Use your ScienceLogic customer account and password to access this site.
From the SL1 Product Downloads menu, select SL1 Platform. The Platform Downloads page appears.
Find the release you are interested in and click its name. The Release Version page appears.
Click the specific link for a release, if needed.
Click the link for the release image or release patch you want to download, and click the Download File button. The file is then downloaded to your local computer.

Importing the Update

To import a product update on to your SL1 system:

In the SL1 system, go to the System Updates page (System > Tools > Updates).
In the System Updates page, click the Import button.
In the Import a new update modal page, browse to the product update file and select it.

If you select the Auto Stage button, the SL1 system will begin staging as soon as the import is completed.
If you do not select the Auto Stage button, you must click the staging button() after import is completed. You can do so at any time after import has completed.
For more information on automatic staging and manual staging, see the section on Staging.

Click the Import button.
In the System Updates page, the Import Status column can have one of the following statuses:

In Progress. Software is currently being imported by the SL1 system.
Complete. Software has been imported successfully.
Failed. Software import has failed due to an unexpected condition. Contact ScienceLogic Support for assistance.
Missing Base. The SL1 system cannot import this software until another software package has been imported. The dependency is for compression purposes. Check the log for a message stating which software package needs to be imported.

The update file or patch file is imported to SL1 system and appears in the System Updates page.

NOTE: For details on the import process, go to the System Updates page, find the entry for the software you are interested in, go to its Import Status column, and click the log icon ().

Staging the Update

After you import a software update to your SL1 system, you must stage the software update. During staging, the SL1 system copies the software update to each SL1 appliance. Staging allows SL1 to simultaneously apply the software changes to each SL1 appliance, regardless of the speed of the connection to each SL1 appliance. The SL1 system stages updates per import. You can choose to automatically stage imports or manually stage import.

For easiest troubleshooting, ScienceLogic recommends that you manually stage imports.

A staging mode called "enhanced file upload" enables pre-staging operations and file upload operations to happen in separate worker processes, which does not count file upload time toward the "staging wait time" setting. This mode is enabled by default, and ScienceLogic recommends that you keep it enabled due to the large amount of data that needs to be pushed for SL1 updates. However, if you prefer to update this setting, you can make the following changes in the "master.system_settings_patcher" table:

To change the number of file upload workers (12 by default), insert or update the following parameter:

param="num_file_upload_workers" and value="<number of workers>"
To change the number of pre-staging workers (25 by default), insert or update the following parameter:

param="pool_size" and value="<number of workers>"
To disable enhanced file upload mode, insert or update the following parameter:

param="use_enhanced_file_upload" and value="0"

The Staging Status column on the System Updates page can have one of the following statuses:

--. No staging request is active and software has not been staged on any SL1 appliances.
Scheduled. The SL1 system is aware of the staging request and is preparing for staging.
In Progress. Staging is in progress but has not completed. The page displays the percentage complete as staging progresses.
Complete. Staging has completed, and all appliances are ready to deploy the software.
Incomplete. Staging has completed, and one or more appliances are ready to deploy the software.
Canceled. User manually canceled the staging process.
Outdated. The current update is not the latest or has already been installed.
Failed. An unexpected error occurred in the staging process. Contact ScienceLogic Support.

For details on the staging process, go to the System Updates page, find the entry for the software you are interested in, go to its Staging Status column, and click the log icon ().

After the software update is imported and staged, you can deploy the software.

Automatic Staging

To enable automatic staging:

In SL1, go to the System Updates page (System > Tools > Updates).
In the System Updates page, click the Import button.
In the Import a new update modal page, browse to the product update file and select it. If you select the Auto Stage button, the SL1 system will begin staging as soon as the import is completed.

After import, in the System Updates page, the Staging Status column will display the number of ScienceLogic appliances that have been successfully stage compared to the total number of ScienceLogic appliances

To disable automatic staging:

In SL1, go to the System Updates page (System > Tools > Updates).
In the System Updates page, click the Import button.
In the Import a new update modal page, browse to the product update file and select it.
If you do not select the Auto Stage button, you must click the staging button () after import is completed. You can do so at any time after import has completed.

Manually Staging an Update

You can manually stage a software update:

If you imported an update but do not want to stage it immediately.
If you add another ScienceLogic appliance to your SL1 system and need to apply software updates.
If staging failed on one or more ScienceLogic appliances.
If you want to ensure that a previous staging process was successful.

When you manually stage a software update, SL1 checks the status of the software updated on each ScienceLogic appliance. SL1 then stages the software update only to those SL1 appliances that have not yet been staged for this software update.

To manually stage a software update:

In SL1, go to the System Updates page (System > Tools > Updates).
Locate the software update you want to stage and click its staging icon (). The software update will be copied to each ScienceLogic appliance that has not yet been staged.
The Staging Status column will display the number of ScienceLogic appliances that have been successfully stage compared to the total number of ScienceLogic appliances.

Monitoring Staging

For SL1 versions 8.12.0 and later, you can monitor the staging process:

Either go to the console of the Database Server or use SSH to access the Database Server.
Type the following at the shell prompt:

monitor_stage

You should see something like the following image:
In the monitor_stage results, look for the following information:

System Update Vitals. Displays the current status of the services that are required for System Update.
Staging Process Stats. Displays status of staging on all SL1 appliances.

Running the Pre-Upgrade Check

After importing and staging an update, you can run a pre-upgrade check before deploying. The pre-upgrade check will ensure that all criteria are met before deploying.

If you are upgrading from SL1 8.14 or earlier, see Running the Pre-Upgrade Check for SL1 8.14 or Earlier.

The pre-upgrade check examines the following:

Is each SL1 Appliance eligible to be updated?
Are updates enabled on each SL1 Appliance?
Are any of the SL1 Appliances running CentOS 5?
Is this hostfile on each SL1 Appliance correctly configured?
Is each Data Collector and Message Collector in a Collector Group?
Is there enough free space on the disk to perform the upgrade?
Is the RPM database corrupted?
Are the RPM packages corrupted?
Does the patch hook directory have the correct owner assigned?
Are the CRM templates on High Availability and Disaster Recovery systems out of date?
Does /etc/init.d/mysql exist? (If it does not, it creates the file.)

The pre-upgrade check skips any SL1 appliances that have been deleted since the last upgrade.

Running the Pre-Upgrade Check

To run a pre-upgrade check:

Go to the System Updates page (System > Tools > Updates.
Find the upgrade that you want to deploy.
Click the purple checkmark at the end of the row. The pre-upgrade check will run.
If a pre-upgrade criterion fails, the Deploy button will be disabled for the selected row.
To view the output from the pre-upgrade check, click on the on the magnifying-glass icon () in the selected row.
If the pre-upgrade check finds a failure, see the list below for possible causes.
Fix all failures before deploying the update.

Potential Issues to Address

CentOS 5 Failure

CentOS 5 is no longer supported by System Update. If one or more Data Collectors are running CentOS5, the pre-upgrade check will fail. Contact your Customer Success Manager to determine how to upgrade your Data Collectors.

Collector Group Membership

This test checks that each Data Collector and Message Collector is a member of a Collector Group.

If a Data Collector or Message Collector is not a member of a Collector Group, the pre-upgrade test will define the appliance as "not eligible for patching.

To fix this error, add the Data Collector or Message Collector to a Collector Group.

Eligibility Failure

The most common reasons for eligibility failure are:

The SL1 appliance is not licensed or the license has expired
The SL1 appliance cannot be reached over the network
The Data Collector has failed over
The SL1 appliance is not configured
The Data Collector is waiting to be returned to service
The Data Collector is not assigned to a Collector Group

Enabled Failure

By default, all SL1 appliances are enabled for patching.

However, if you have used a command-line tool to exclude an SL1 appliance from updates, the pre-upgrade check will fail. To fix this error, include the SL1 appliance for updates.

Free Disk-Space Failure

This test checks the root partition and requires 1GB of free disk space. If the root partition does not have 1GB of free disk space, the pre-upgrade check will fail.

If the root partition does not have 1GB of free disk space, you must archive or delete files that are no longer required or add a new empty disk and resize the filesystem.

Host File Failure

This test validates the /etc/hosts file for the presence of an IPv6 entry for localhost, which is required by System Update.

If /etc/hosts does not include an IPv6 entry for localhost, the pre-upgrade test automatically adds the required entry.

Check the following in case of failure:

The /etc/hosts file exists
The /etc/hosts can be edited by root

Patch-Hook Ownership Failure

If the owner of the patch hook directory ((/var/lib/em7/patch_hook) is incorrect, the pre-upgrade test automatically fixes the ownership. However, if this error occurs, check for the following:

The patch hook directory (/var/lib/em7/patch_hook) does not exist
The s-em7-core user or the s-em7-core group does not exist

RPM Database Failure

If the RPM database fails the pre-upgrade test, the RPM database is corrupted.

To recover the RPM database:

Either go to the console of the Database Server or use SSH to access the Database Server. Log in with the credentials you defined when you installed the Database Server.

At the shell prompt, enter the following:

mkdir -p /tmp/rpm.bak
cp /var/lib/rpm/* /tmp/rpm.bak
rm -f /var/lib/rpm/__db*
rpm --rebuilddb -vv
rpm -q kernel

If the last command returns a value, you can delete the backup directory using the following command.
```
rm -Rf /tmp/rpm.bak
```

RPM Package Failure

If one or more RPM packages failed the pre-upgrade test, possible causes are:

Packages are not staged, and hence some files are missing. This can be caused due to a failed staging or a timeout during staging. You can try to stage again. You can also adjust the timeout for staging.
Duplicate packages
Conflicting packages
Unmet dependencies

Duplicate Packages:

Either go to the console of the Database Server or use SSH to access the Database Server. Log in with the credentials you defined when you installed the Database Server.
At the shell prompt, enter the following command:

sudo package-cleanup --dupes
If there are duplicate packages, use the following command to remove them:

sudo package-cleanup --cleandupes --removenewestdupes

Conflicting Packages

Look for conflicting packages in the staging log
Verify that the package is a part of SL1 ISO or patch bundle
If the package is not part of the SL1 ISO or patch bundle, uninstall the package.

Unmet dependencies

You will need to reset the staging status of the appliance and stage it again. Contact ScienceLogic Customer Success for help in resetting the staging status.

Putting All SL1 Appliances into Maintenance Mode

ScienceLogic recommends that you perform these steps during a maintenance window.

Immediately before deploying a software update, ScienceLogic recommends that you put all SL1 appliances in maintenance mode. This will prevent spurious error messages and events during the deployment.

To enable user maintenance mode for all the SL1 appliances in your SL1 system:

Go to the Appliance Manager page (System > Settings > Appliances). Note the list of SL1 appliances in your system.
Go to the Device Manager page (Devices > Classic Devices, or Registry > Devices > Device Manager in the classic SL1 user interface) and select the checkbox for each SL1 appliance in your SL1 system. This includes both primary and secondary Database Servers.
In the Select Action drop-down list, select Change User Maintenance Mode: Enabled without Collection. This option puts the selected devices into user maintenance mode with collection disabled. The devices will remain in this state until you or another user disables user maintenance mode.
Click the Go button.

Deploying the Update

During deployment, avoid the following tasks:

Running integrations and third-party applications that access the SL1 database or manipulate data on SL1
Running discovery sessions
Running nightly discovery
Bringing HA/DR out of maintenance mode
Adding new SL1 Appliances
Importing a new patch
Adding Data Collectors to a Collector Group
Removing Data Collectors from a Collector Group
Rebalancing a Collector Group
Killing processes related to patching and upgrading
Run reporting jobs
Unpausing the proc_mgr process

When you deploy an update, the update is installed on all nodes or appliances that have already been staged.

Database Servers and Data Engines in your SL1 stack are deployed first; other appliance types cannot be deployed until the Database Servers and Data Engines are finished deploying.

When you deploy an update, SL1 checks to ensure that you have already deployed all required updates. If you have not, SL1 will generate an error message specifying the updates you must deploy before continuing with the current update.

During deployment, the Deployment Status column on the System Updates page can have one of the following statuses:

--. No deployment request is active, and software has not been deployed on any SL1 appliances.
Scheduled. The SL1 system is aware of the deployment request and is preparing for deployment.
In Progress. Deployment is in progress but has not completed.
Complete. Deployment has completed, and all appliances are updated.
Incomplete. Deployment has completed, and one or more, but not all, appliances are updated.
Canceled. User manually canceled the deployment.
Outdated. The current update is not the latest or has already been installed.
Failed. An unexpected error occurred in the deployment process. Contact ScienceLogic Support.

To deploy a software update on your nodes or appliances:

Make sure that you have imported and staged the update file.
Go to the System Updates page (System > Tools > Updates).
In the System Updates page, find the software update you want to deploy. Click the lightning bolt icon () to deploy the software. If SL1 is still staging the patch when you click the lightning-bolt icon (), SL1 will wait until staging has completed before deploying the updates to each ScienceLogic appliance.
The software update will be deployed to all appliances in your SL1 system that have already been staged. If one or more appliances in your SL1 system have been successfully staged, SL1 will deploy the update to those appliances.

For details on the deployment process, go to the System Updates page, find the entry for the software you are interested in, go to its Deployment Status column, and click the log icon ().

Troubleshooting System Update

You can use the sysuptb troubleshooting tool to determine issues with System Update and to generate diagnostic information about the update. You can also use the phtb tool to troubleshoot issue with the PhoneHome configuration.

These tools can be useful when System Update does not work as expected, or if you have issues with the PhoneHome configuration or with communication between appliances and the Database Server. These tools are available on all SL1 appliances starting with SL1 version 10.2.0, and the tools are backwards-compatible to SL1 version 8.12.0.

Using the sysuptb Troubleshooting Tool

To use the sysuptb troubleshooting tool:

Either go to the console of any SL1 appliance or use SSH to access the appliance
Enter the following at the shell prompt:

sudo sysuptb -h
For more information about each argument, enter the following at the shell prompt:

sudo sysuptb <argument> -h

Available Commands

The following command executes all troubleshooting tests for System Update:
```
sudo sysuptb all <optionally -x name_of_test_to_exclude>
```
To learn more about a test run, use this command: sudo sysuptb help <test-name>

Example:

sudo sysuptb all

Executing filestore tests
912 / 912 [-----------------------------------------------------] 100.00% 14 p/s
Filestore test summary: [Total: 912, Intact: 912, Incomplete: 0, Corrupt: 0]
Executing test for deleted appliances in patch history
No deleted appliances were found in the patch history
Executing test for invalid file id in patch schedules
No patch schedules were found to have invalid file id
Executing test for RPM database corruption
RPM database is intact
Executing test to check if filestore is empty
Filestore has 1026 files
Executing test for deactivating services
Service test summary: [Total: 1, Active: 1, Inactive: 0, Healed: 0, Skipped: 0, Failed (to heal): 0]
Executing test for free disk space
Free disk space test summary: [Total: 2, Pass: 2, Failed: 0]
Executing test for service errors
Service error test summary: [Total: 2, Without Errors: 2, Restarted: 0, Failed: 0]
Executing hosts file check for IPV6 entry (::1) for localhost
An entry for ::1 is already present in the hosts file
Proxy is not configured for yum.
Executing test for hung yum process
No yum processes found
Yum process summary: [Total: 0, Hung: 0]

The following command searches the logs for errors that match a service name and restarts services if any errors are found.
```
sudo sysuptb check-service-error <optionally -s name_of_service>
```
If you do not provide the name of a service, the command searches the logs for errors for siloupdate-pkgserver.service and siloupdate-spool.service.

Example:

sudo sysuptb check-service-error

Executing test for service errors
Service error test summary: [Total: 2, Without Errors: 2, Restarted: 0, Failed: 0]

The following command removes deleted SL1 appliances from the history of system updates so that they SL1 does not search for them during update.
```
sudo sysuptb clear-mids
```

Example:

sudo sysuptb clear-mids

Executing test for deleted appliances in patch history
No deleted appliances were found in the patch history

The following command cancels al schedule updates that include an invalid ID for the patch file.
```
sudo sysuptb clear-schedule
```

Example:

sudo sysuptb clear-schedule

Executing test for invalid file id in patch schedules
No patch schedules were found to have invalid file id

The following command checks the filestore of downloaded packages for corrupt files and marks the corrupt files as incomplete.

sudo sysuptb filestore

Example:

sudo sysuptb filestore

Executing filestore tests
912 / 912 [-----------------------------------------------------------] 100.00% 14 p/s
Filestore test summary: [Total: 912, Intact: 912, Incomplete: 0, Corrupt: 0]

The following command checks the file system for available free space.
```
sudo sysuptb free-space <optionally, -d path_for_drive = minimum_size> 
```
If you do not provide the path and minimum size of the directory, the command examines /var to make sure it has 300MB of free space and / to make sure it has 1GB of free space.

Example:

sudo sysuptb free-space --disk /var=300MB

Executing test for free disk space
Free disk space test summary: [Total: 1, Pass: 1, Failed: 0]

The command checks for update services that are stuck in a deactivating state and then heals them.
```
sudo sysuptb heal-service <optionally -s service_name>
```
If you do not specify a service, the command examines the service siloupdate-manager.service. Starting with SL1 12.1.0, siloupdate-manager.service replaced em7_patch_manager.service.

Example:

sudo sysuptb heal-service

Executing test for deactivating services

Service test summary: [Total: 1, Active: 1, Inactive: 0, Healed: 0, Skipped: 0, Failed (to heal): 0]

The following command checks the /etc/hosts file for an entry for IPv6 for the current server (like a loopback address). If no entry exists, the command adds ::1 to the /etc/hosts file.
```
sudo sysuptb hosts
```

Example:

sudo sysuptb hosts

Executing hosts file check for IPV6 entry (::1) for localhost
An entry for ::1 is already present in the hosts file

The following command check is the filestore that holds the upgrade packages is empty.

sudo sysuptb is-filestore-empty

Example:

sudo sysuptb is-filestore-empty

Executing test to check if filestore is empty
Filestore has 1026 files

The following command checks the RPM database on /var/lib/rpm for corruption. If the command detects corruption, the output includes steps for remediation.

sudo sysuptb rpmdb

Example:

sudo sysuptb rpmdb

Executing test for RPM database corruption
RPM database is intact

The following command checks for a yum process which is hung.
```
sudo sysuptb yum-proc <optionally, -t timeout_in_minutes>
```
If you do not specify a running time, in minutes, the command searches for yum processes that have been running for more than 120 minutes.

Example:

sudo sysuptb yum-proc

Executing test for hung yum process
No yum processes found
Yum process summary: [Total: 0, Hung: 0]

The following command checks if yum is configured with proxy. If so, the command removes the proxy configuration.
```
sudo sysuptb yum-proxy
```

Example:
```
sudo sysuptb yum-proxy
```
Proxy is not configured for yum.

Using the phtb Troubleshooting Tool

To use the phtb troubleshooting tool:

Either go to the console of an SL1 appliance using PhoneHome communication or use SSH to access the appliance.
Enter the following at the shell prompt:

sudo phtb -h
For more information about each argument, enter the following at the shell prompt:

sudo phtb <argument> -h

To learn more about a test run, use this command: sudo phtb help <test-name>

Available Commands

The following command checks destinations for SSH connectivity issues:
```
sudo phtb destination
```
The following command checks the target host for SSH connectivity issues:
```
sudo phtb probe-host
```
The following command checks connectivity to the proxy host, if configured:
```
sudo phtb proxy
```

Monitoring Deployment

For SL1 versions 8.12.0 and later, you can monitor deployment. To do so:

Either go to the console of the Database Server or use SSH to access the Database Server.
Enter the following command at the shell prompt:
```
monitor_deploy
```
You should see something like the following figure:

System Update Vitals. Displays the current status of the services that are required for System Update.
Deployment Process Stats. Displays status of deployment on all SL1 appliances.
Module Level Status. Displays the status of the three deployment steps.

Installing Additional RPMs on an SL1 Appliance

For certain patch releases, ScienceLogic might require additional RPMs to be installed on specific appliance types. If an RPM install is required, the release notes will indicate the additional RPMs to install on each specific appliance type.

To install additional RPMs on an appliance, perform the following steps:

Download the RPM files provided by ScienceLogic to your local machine.
Log in as root at the appliance console.
Copy each of the downloaded RPM files to the appliance. To copy the downloaded files, perform the following command as root at the console of the appliance:

scp <username-on-local-machine>@<ip-address-of-your-local-machine>:<full-path-to-rpm-on-your-local-machine> <full-path-on-appliance-to-copy-to>

Use the following command to run the RPM installer for each of the RPM files:

rpm -U <name-of-rpm-file>.rpm

If you have not yet done so, apply the latest patch to your SL1 system.

Remove SL1 Appliances from Maintenance Mode

To disable user maintenance mode for all the SL1 appliances in your SL1 system:

Go to the Appliance Manager page. Note the list of SL1 appliances in your system.
Go to the Device Manager page (Devices > Classic Devices, or Registry > Devices > Device Manager in the classic SL1 user interface) and select the checkbox for each SL1 appliance in your SL1 system.
In the Select Action drop-down list, select Change User Maintenance Mode: Disabled. This option disables user maintenance mode for the selected devices.
Click the Go button.

Refer to the release notes for your current release to determine if you must upgrade MariaDB after upgrading.

Also refer to the release notes for your current release to determine if you must reboot all SL1 appliances after upgrading.

Updating SL1 Extended Architecture

New installations of SL1 Extended Architecture are available only on SaaS deployments.

For existing on-premises deployments of SL1 Extended Architecture, see the section on Upgrading SL1 Extended Architecture.

Automatically Upgrading MariaDB

Most SL1 updates require you to upgrade MariaDB after you update SL1. Refer to the release notes for your patch release to determine if you must upgrade MariaDB.

Newer SL1 releases use the command siloupdate upgrade-mariadb to upgrade the MariaDB server. This command also updates MariaDB-client, MariaDB-common, and MariaDB-shared RPMs in addition to the MariaDB Server RPM.

Older versions of SL1 use the module_upgrade_mariadb script to upgrade MariaDB rather than the command siloupdate upgrade-mariadb that is described in this section. For more information, see the section on Using the module_upgrade_mariadb Script in Older Versions of SL1.

You should store all custom configuration settings for each MariaDB database in the file /etc/siteconfig/mysql.siteconfig. If you have added custom settings to the file /etc/my.cnf.d/silo_mysql.cnf, those changes will be overwritten each time you upgrade MariaDB. Before upgrading, copy any custom settings to the file /etc/siteconfig/mysql.siteconfig. SL1 will save these custom settings and apply them after you upgrade MariaDB.

To reduce spurious events, you can put the Database Server in maintenance mode while you upgrade MariaDB. For details, see the chapter on Putting the Database Server into Maintenance Mode

The siloupdate upgrade-mariadb command:

Upgrades the following SL1 appliances:

All Database Servers
All-In-One Appliances
Data Collectors
Message Collectors

Upgrades High Availability (HA) and Disaster Recovery (DR) systems
Includes a "test only" option before executing upgrade
Enforces upgrading the primary Database Server before upgrading secondary Database Server and the Data Collectors.
Will skip SL1 appliances that have already been updated
Logs entire sequence of commands and output for later analysis
Stores log files in /data/logs/module_upgrade_mariadb.log and /data/logs/.upgrade_mariadb.log
Checks for differences between current configuration and version you are about to install and spawns an alert. To skip this check, use the -s -s option

To upgrade MariaDB in newer versions of SL1, perform the following steps:

Either go to the console of the Database Server or use SSH to access the Database Server.
At the shell prompt, enter the following command:

siloupdate upgrade-mariadb

To upgrade all modules, you can use the command siloupdate upgrade-mariadb -m all.
To see all the options for the siloupdate upgrade-mariadb command, enter the following command at the shell prompt:
```
siloupdate upgrade-mariadb -h 
```

Using the module_upgrade_mariadb Script in Older Versions of SL1

SL1 will automatically update MariaDB-client, MariaDB-common, and MariaDB-shared RPMs but will not update the MariaDB Server RPM. You must update the MariaDB Server RPM after you install the SL1 update.

Older SL1 releases include the module_upgrade_mariadb script to automatically upgrade the MariaDB server.

Refer to the release notes for your current release to determine if you must upgrade MariaDB. Not every SL1 update requires an upgrade of for MariaDB.

Newer versions of SL1 use the command siloupdate upgrade-mariadb to upgrade MariaDB rather than the module_upgrade_mariadb script that is described in this section. If you attempt to use the older command on a newer version of SL1 that no longer supports it, you will get a deprecation message telling you to use the newer command.

To reduce spurious events, you can put the Database Server in maintenance mode while you upgrade MariaDB. For details, see the chapter on Putting the Database Server into Maintenance Mode

The module_upgrade_mariadb script:

Upgrades the following SL1 appliances:

All Database Servers
All-In-One Appliances
Data Collectors
Message Collectors

Upgrades High Availability (HA) and Disaster Recovery (DR) systems
Includes a "test only" option before executing upgrade
Enforces upgrading the primary Database Server before upgrading secondary Database Server and the Data Collectors.
Will skip SL1 appliances that have already been updated
Logs entire sequence of commands and output for later analysis
Stores log files in /data/logs/module_upgrade_mariadb.log and /data/logs/.upgrade_mariadb.log
Checks for differences between current configuration and version you are about to install and spawns an alert. To skip this check, use the -s -s option

To upgrade MariaDB in older versions of SL1, perform the following steps:

Either go to the console of the Database Server or use SSH to access the Database Server.
At the shell prompt, enter the following command:
```
sudo /opt/em7/bin/module_upgrade_mariadb  -m all
```
To see all the options for the module_upgrade_mariadb script, enter the following command at the shell prompt:
```
/opt/em7/bin/module_upgrade_mariadb -h
```
Usage:

module_upgrade_mariadb -m <module_id> [-t|--test] [-y|--assumeyes] [-s|--skip_conf_file_error][-p|--pool size <number_of_modules>][-h|--help]
The script includes these options:

-m parameter specifies the SL1 appliances that you want to upgrade. You can specify:

-m <mid1, mid2…midN> provides a comma-separated module IDs.
-m all : upgrade all appliances (Database Servers, All-In-One Appliances, Data Collectors, and Message Collectors).
-m all -db : upgrade all Database Servers.
-m all-cu : upgrade all Data Collectors and Message Collectors.

-t parameter specifies not to upgrade but instead to run a test of the upgrade script.
-y parameter specifies to automatically enter "yes" at all prompts.
-s parameter specifies to ignore errors in the MySQL configuration files and proceed with the upgrade.
-p parameter specifies the number of Data Collectors that you want to upgrade simultaneously. Database Servers will be upgraded one at a time. Possible values are 1 - 20. The default value is 1.

-p <number_of_modules> is the number of Data Collectors to upgrade simultaneously. Values are 1 - 20. The default value is 1.

To view the status of the automatic upgrade, enter the following command:
```
monitor_upgrade_mariadb
```

Additional Steps for MariaDB Upgrades in SL1 10.1.x

SL1 10.1.x included an upgrade to MariaDB. The upgrade did not include a tool, jemalloc, that helps manage memory usage.

This section applies only to the following releases:

10.1.0
10.1.1
10.1.2
10.1.3
10.1.4
10.1.4.1
10.1.4.2
10.1.5
10.1.5.1

For SL1 versions later than 10.1.5.1, jemalloc is included with the platform. For SL1 versions prior to 10.1.0, jemalloc is included with the platform.

To avoid problems with memory usage on Database Servers, perform the following steps after upgrading MariaDB for 10.1.x.

Perform these steps first on the active Database Server and then on each additional Database Server in your SL1 System.

Open an SSH session to the Database Server.
To verify that the Database Server is not currently running jemalloc, enter the following command at the shell prompt:
```
silo_mysql -e 'show global variables like "version_malloc_library"'
```

If the Database Server is not currently running jemalloc, the shell will display the following:

Variable Name	Value
version_malloc_library	system

Search for the file /usr/lib64/libjemalloc.so.1. If the file does not exist, contact ScienceLogic Customer Support to request the file jemalloc-3.6.0-1.el7.x86_64.rpm.

To install the RPM, use a file-transfer utility, copy the file to a directory on the SL1 appliance. Then enter the following commands at the shell prompt:
```
cd /usr/lib64
sudo yum install jemalloc-3.6.0-1.el7.x86_64.rpm
```
Create the file /etc/systemd/system/mariadb.service.d/jemalloc.conf, as follows:
```
vi /etc/systemd/system/mariadb.service.d/jemalloc.conf
```

Add the following lines to the file:

[Service]
Environment="LD_PRELOAD=/usr/lib64/libjemalloc.so.1"

Save and close the file.
Reload the systemd config files with the following command:
```
sudo systemctl daemon-reload
```
Restart the Database Server:

To restart the standalone Database Server or the primary Database Server in a cluster, enter the following:
```
sudo systemctl restart mariadb
```
To restart each secondary Database Server in a cluster:

Open an SSH session to the secondary Database Server. At the shell prompt, enter:
```
coro_config
```
Select 1.
When prompted to put the Database Server into maintenance, select y.
Open an SSH session to the primary Database Server. To pause SL1, enter the following command at the shell prompt:
```
sudo touch /etc/.proc_mgr_pause
```
In the SSH session for the secondary Database Server, restart MariaDB:
```
crm resource restart mysql
```
After MariaDB has restarted successfully on the secondary Database Server, return to the SSH session on the primary Database Server. Remove the pause file for SL1 using the following command:
```
sudo rm /etc/.proc_mgr_pause
```
In the SSH session on the secondary Database Server, take the Database Server out of maintenance. At the shell prompt, enter:
```
coro_config
```
Select 1.
When prompted to take the Database Server out of maintenance, select y.

To verify that jemalloc is running on the Database Server, enter the following command at the shell prompt:
```
silo_mysql -e 'show global variables like "version_malloc_library"'
```

If the Database Server is currently running jemalloc, the shell will display something like the following:

Variable Name	Value
version_malloc_library	jemalloc 3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340

Perform these steps on each Database Server in your SL1 system.

Manually Upgrading MariaDB

Refer to the release notes for your current release to determine if you must upgrade MariaDB. Not every SL1 update requires an upgrade of MariaDB.

ScienceLogic strongly recommends that you upgrade MariaDB using the script described in Automatically Upgrading MariaDB with a Script.

To reduce spurious events, you can put the Database Server in maintenance mode while you upgrade MariaDB. For details, see the chapter on Putting the Database Server into Maintenance Mode

If you prefer to upgrade MariaDB manually, the following sections describe how to upgrade the MariaDB server for different SL1 appliance types and architectures.

When you update MariaDB, you must update the following SL1 appliances:

All Database Servers
All-In-One Appliances
Data Collectors
Message Collectors

Download RPMs to SL1 Appliances

Before upgrading MariaDB, you must copy the RPMs from the primary Database Server to the Database Servers, All-In-One Appliances, Data Collectors, and Message Collectors in your SL1 system. To do this.

Open an SSH session to the Database Server.

To download the latest RPMs from the Database Server, enter the following at the shell prompt:

For SL1 version 10.1.0 to 10.1.5:

wget --output-document /tmp/MariaDB-server-10.4.12-1.el7.centos.x86_64.rpm http://localhost:10080/MariaDB-server.rpm

wget --output-document /tmp/galera-4-26.4.3-1.rhel7.el7.centos.x86_64.rpm http://localhost:10080/galera-4.rpm

For SL1 version 10.1.6 and higher, download all of the packages listed when you enter the command:

cat /opt/em7/share/db_packages

wget --output-document /tmp/MariaDB-server.rpm http://localhost:10080/<mariadb-server-pkg-from-db_packages> 
wget --output-document /tmp/galera-4.rpm http://localhost:10080/<galera-4-pkg-from-db_packages>
wget --output-document /tmp/socat.rpm http://localhost:10080/<socat-pkg-from-db_packages>

Verify if the downloaded packages are valid (not corrupt or incomplete downloads) by entering the following commands:
```
rpm -qip /tmp/MariaDB-server.rpm
rpm -qip /tmp/galera-4.rpm
rpm -qip /tmp/socat.rpm 
```
If any errors are reported, try restarting siloupdate-pkgserver, using the following command, and retry downloading and verifying the RPM files again.
```
systemctl restart siloupdate-pkgserver
```
Use SCP or another secure copy program to copy these files to the /tmp directory on each Database Server, All-In-One Appliance, Data Collector, and Message Collector:

MariaDDB-server.rpm
galera-4.rpm

To conserve disk space, ScienceLogic recommends you delete the RPMs from the /tmp directory the on the Database Servers, All-In-One Appliances, Data Collectors, and Message Collectors in your SL1 system after you successfully upgrade MariaDB.

Manually Upgrade Two Database Servers Configured for High Availability or Disaster Recovery

To upgrade a High Availability or Disaster Recovery cluster, perform the following steps:

The system will be unavailable when performing these steps.

Step 1: On the Secondary Database Server

You must put the secondary Database Server in maintenance mode. To do this:

Open an SSH session to the Database Server.
At the shell prompt, assume root privileges:
```
sudo -s
```
When prompted, enter the administrator password.

At the shell prompt, enter the following command:

coro config

The following menu appears:

1) Enable Maintenance
2) Option Disabled
3) Promote DRBD
4) Stop Pacemaker
5) Resource Status
6) Quit

Enter "1".

Step 2: On the Primary Database Server

To determine the current installed version of the RPMs, enter the following command:
```
sudo rpm -qa ^MariaDB-*
```
To stop SL1 and MariaDB, enter the following commands at the shell prompt:
```
sudo systemctl stop em7
sudo systemctl stop mariadb.service
```
To stop the MySQL resource, enter the following command:
```
sudo crm resource stop mysql
```
To save the current enabled state for mariadb.service, enter the following command:
```
export MSRV=`sudo systemctl is-enabled mariadb.service`
```
Check the version of MariaDB-server that you are running.
```
rpm -q MariaDB-server
```

You must follow the steps below that correspond to your version of MariaDB. Step 6 is specific to MariaDB-server version 10.1.x, while Step 7 is specific to MariaDB-server versions 10.4.12 and higher.

MariaDB-server version 10.1.x. If you are running MariaDB-server version 10.1.x:

Do these steps in order. Doing the steps in any other order will result in unintended consequences.

Remove MariaDB-server by using the following commands:
```
sudo rpm --nodeps -ev MariaDB-server
```

Replace the Galera package and install the new MariaDB-server package by using the following commands:

sudo yum --disablerepo=* swap -- remove galera -- install /tmp/galera-4.rpm
sudo yum --disablerepo=* install /tmp/MariaDB-server.rpm

MariaDB-server version 10.4.12 and higher. If you are running MariaDB-server version 10.4.12 or higher, upgrade the MariaDB-server package and dependent packages (galera-4 and socat) by using the following commands:
```
sudo yum --disablerepo=* install /tmp/galera-4.rpm /tmp/socat.rpm
sudo yum --disablerepo=* upgrade /tmp/MariaDB-server.rpm
```
To remove incompatible backup packages, enter the following command:
```
sudo yum remove percona-xtrabackup
```

NOTE: If the "yum remove" command fails, it means that the package does not exist on the SL1 appliance. You can ignore the error message.

To regenerate the configuration file for MariaDB, enter the following command:

sudo /opt/em7/share/scripts/generate-my-conf.py -f -o /etc/my.cnf.d/silo_mysql.cnf

To re-start MariaDB, enter the following command:

sudo systemctl daemon-reload
sudo systemctl start mariadb

To restart the MySQL resource, enter the following command:
```
sudo crm resource start mysql
```
To restore the mariadb.service enabled state, enter the following command:
```
sudo systemctl ${MSRV::-1} mariadb.service
```
To upgrade the internal configuration for the database, enter the following:
```
sudo mysql_upgrade -u root -p
```
To restart the em7 service, enter the following commands:
```
sudo systemctl start em7
sudo rpm -qa ^MariaDB-*
```

Step 3: On the Secondary Database Server

Determine the current installed version of the RPMs using the following command:
```
sudo rpm -qa ^MariaDB-*
```
To save the current enabled state for mariadb.service, enter the following:
```
export MSRV=`sudo systemctl is-enabled mariadb.service`
```
Check the version of MariaDB-server that you are running.
```
rpm -q MariaDB-server
```

You must follow the steps that correspond to your version of MariaDB. Step 4 is specific to MariaDB-server version 10.1.x, while Step 5 is specific to MariaDB-server versions 10.4.12 and higher.

MariaDB-server 10.1.x: If you are running MariaDB-server version 10.1.x:

Do these steps in order. Doing the steps in any other order will result in unintended consequences.

Remove MariaDB-server by using the following command:
```
sudo rpm --nodeps -ev MariaDB-server
```

Replace the Galera package and install the new MariaDB-server package by using the following commands:

sudo yum --disablerepo=* swap -- remove galera -- install /tmp/galera-4.rpm
sudo yum --disablerepo=* install /tmp/MariaDB-server.rpm

MariaDB-server 10.4.12 or higher: If you are running MariaDB-server version 10.4.12 or higher, upgrade the MariaDB-server package and dependent packages (galera-4 and socat)by using the following commands:
```
sudo yum --disablerepo=* install /tmp/galera-4.rpm /tmp/socat.rpm
sudo yum --disablerepo=* upgrade /tmp/MariaDB-server.rpm
```
To remove incompatible backup packages, enter the following command:
```
sudo yum remove percona-xtrabackup
```

NOTE: If the "yum remove" command fails, it means that the package does not exist on the SL1 appliance. You can ignore the error message.

To regenerate the configuration file for MariaDB, enter the following command:

sudo /opt/em7/share/scripts/generate-my-conf.py -f -o /etc/my.cnf.d/silo_mysql.cnf

To restore the mariadb.service enabled state, enter the following command:
```
sudo systemctl ${MSRV::-1} mariadb.service
```
To take the secondary Database Server out of maintenance mode, enter the following command at the shell prompt:
```
sudo -s
```
When prompted, enter the administrator password.

At the shell prompt, enter the following command:

coro config

The following prompt appears:


1) Disable Maintenance
2) Option Disabled
3) Promote DRBD
4) Stop Pacemaker
5) Resource Status
6) Quit

Enter "1".

Manually Upgrade Three Database Servers Configured for High Availability and Disaster Recovery

To upgrade a High Availability/Disaster Recovery cluster, perform the following steps:

The system will be unavailable when performing these steps.

Step 1: On the Secondary Database Server

You must put the secondary Database Server in maintenance mode. To do this:

Open an SSH session to the Database Server.
At the shell prompt, assume root privileges:
```
sudo -s
```
When prompted, enter the administrator password.
At the shell prompt, enter the following command:
```
coro config
```

The following prompt appears:

1) Disable Maintenance
2) Option Disabled
3) Promote DRBD
4) Stop Pacemaker
5) Resource Status
6) Quit

Enter "1".

Step 2: On the Primary Database Server

Determine the current installed version of the RPMs:
```
sudo rpm -qa ^MariaDB-*
```

Stop SL1 and MariaDB using the following commands:

sudo systemctl stop em7
sudo systemctl stop mariadb.service

Stop the MySQL resource:
```
sudo crm resource stop mysql
```

Save the current enabled state for the mariadb.service:

export MSRV=`sudo systemctl is-enabled mariadb.service`

Check the version of MariaDB-server that you are running:
```
rpm -q MariaDB-server
```

You must follow the steps that correspond to your version of MariaDB.

MariaDB-server version 10.1.x. If you are running MariaDB-server version 10.1.x:

Do these steps in order. Doing the steps in any other order will result in unintended consequences.

Remove MariaDB-server by using the following command:
```
sudo rpm --nodeps -ev MariaDB-server
```

Replace the Galera package and install the new MariaDB-server package by using the following commands:

sudo yum --disablerepo=* swap -- remove galera -- install /tmp/galera-4.rpm
sudo yum --disablerepo=* install /tmp/MariaDB-server.rpm

MariaDB-server version 10.4.12 and higher. If you are running MariaDB-version 10.4.12 or higher, upgrade the MariaDB-server package and dependent packages (galera-4 and socat) by using the following commands:
```
sudo yum --disablerepo=* install /tmp/galera-4.rpm /tmp/socat.rpm
sudo yum --disablerepo=* upgrade /tmp/MariaDB-server.rpm
```
Remove incompatible backup packages:
```
sudo yum remove percona-xtrabackup
```

NOTE: If the "yum remove" command fails, it means that the package does not exist on the SL1 appliance. You can ignore the error message.

Regenerate the configuration file for MariaDB:

sudo /opt/em7/share/scripts/generate-my-conf.py -o -f  /etc/my.cnf.d/silo_mysql.cnf

Restart MariaDB:

sudo systemctl daemon-reload
sudo systemctl start mariadb

Restart the MySQL resource:
```
sudo crm resource start mysql
```

Restore the mariadb.service enabled state:

sudo systemctl ${MSRV::-1} mariadb.service

Upgrade the internal configuration for the database:
```
sudo mysql_upgrade -u root -p
```
Restart the em7 service:
```
sudo systemctl start em7
```

Step 3: On the Secondary Database Server

Save the current enabled state for the mariadb.service:

export MSRV=`sudo systemctl is-enabled mariadb.service`

Check the version of MariaDB-server that you are running:
```
rpm -q MariaDB-server
```

You must follow the steps that correspond to your version of MariaDB.

MariaDB-server version 10.1.x. If you are running MariaDB-server version 10.1.x:

Do these steps in order. Doing the steps in any other order will result in unintended consequences.

Remove MariaDB-server:
```
sudo rpm --nodeps -ev MariaDB-server
```

Replace the Galera package and install the new MariaDB-server package by using the following commands:

sudo yum --disablerepo=* swap -- remove galera -- install /tmp/galera-4.rpm
sudo yum --disablerepo=* install /tmp/MariaDB-server.rpm

MariaDB-server version 10.4.12 and higher. If you are running MariaDB-version 10.4.12 or higher, upgrade the MariaDB-server package and dependent packages (galera-4 and socat) by using the following commands:
```
sudo yum --disablerepo=* install /tmp/galera-4.rpm /tmp/socat.rpm
sudo yum --disablerepo=* upgrade /tmp/MariaDB-server.rpm
```
Remove incompatible backup packages:
```
sudo yum remove percona-xtrabackup
```

NOTE: If the "yum remove" command fails, it means that the package does not exist on the SL1 appliance. You can ignore the error message.

Regenerate the configuration file for MariaDB:

sudo /opt/em7/share/scripts/generate-my-conf.py -o -f  /etc/my.cnf.d/silo_mysql.cnf

Restore the mariadb.service enabled state:

sudo systemctl ${MSRV::-1} mariadb.service

Assume root privileges:
```
sudo -s
```
When prompted, enter the administrator password.
At the shell prompt, enter the following command:
```
coro config
```

The following prompt appears:

1) Disable Maintenance
2) Option Disabled
3) Promote DRBD
4) Stop Pacemaker
5) Resource Status
6) Quit

Enter "1".

Step 4: On the Disaster Recovery Database Server

Open an SSH session to the Disaster Recovery Database Server.
Assume root privileges:
```
sudo -s
```
When prompted, enter the administrator password.

Save the current enabled state for the mariadb.service:

export MSRV=`sudo systemctl is-enabled mariadb.service`

Check the version of MariaDB-server that you are running:
```
rpm -q MariaDB-server
```

You must follow the steps that correspond to your version of MariaDB.

MariaDB-server version 10.1.x. If you are running MariaDB-server version 10.1.x:

Do these steps in order. Doing the steps in any other order will result in unintended consequences.

Remove MariaDB-server:
```
sudo rpm --nodeps -ev MariaDB-server
```

Replace the Galera package and install the new MariaDB-server package by using the following commands:

sudo yum --disablerepo=* swap -- remove galera -- install /tmp/galera-4.rpm
sudo yum --disablerepo=* install /tmp/MariaDB-server.rpm

MariaDB-server version 10.4.12 and higher. If you are running MariaDB-version 10.4.12 or higher, upgrade the MariaDB-server package and dependent packages (galera-4 and socat) by using the following commands:
```
sudo yum --disablerepo=* install /tmp/galera-4.rpm /tmp/socat.rpm
sudo yum --disablerepo=* upgrade /tmp/MariaDB-server.rpm
```
Remove incompatible backup packages:
```
sudo yum remove percona-xtrabackup
```

NOTE: If the "yum remove" command fails, it means that the package does not exist on the SL1 appliance. You can ignore the error message.

Regenerate the configuration file for MariaDB:

sudo /opt/em7/share/scripts/generate-my-conf.py -o -f  /etc/my.cnf.d/silo_mysql.cnf

Restore the mariadb.service enabled state:

sudo systemctl ${MSRV::-1} mariadb.service

Manually Upgrading Standalone Database Servers, All-In-One Appliances, Data Collectors, and Message Collectors

To upgrade MariaDB on one or more Database Servers that are not configured for high availability or disaster recovery, a single All-In-One Appliance, one or more Data Collectors, or one or more Message Collectors, perform the following steps:

The Database Server, All-In-One Appliance, Data Collector, or Message Collector will be unavailable when performing these steps.

Go to the console or open an SSH session to the SL1 appliance.

Stop SL1 and mariadb:

sudo systemctl stop em7
sudo systemctl stop mariadb.service

Save the current enabled state for the mariadb.service:

export MSRV=`sudo systemctl is-enabled mariadb.service`

Check the version of MariaDB-server that you are running:
```
rpm -q MariaDB-server
```

You must follow the steps that correspond to your version of MariaDB.

MariaDB-server version 10.1.x. If you are running MariaDB-server version 10.1.x:

Do these steps in order. Doing the steps in any other order will result in unintended consequences.

Remove MariaDB-server:
```
sudo rpm --nodeps -ev MariaDB-server
```

Replace the Galera package and install the new MariaDB-server package by using the following commands:

sudo yum --disablerepo=* swap -- remove galera -- install /tmp/galera-4.rpm
sudo yum --disablerepo=* install /tmp/MariaDB-server.rpm

MariaDB-server version 10.4.12 and higher. If you are running MariaDB-version 10.4.12 or higher, upgrade the MariaDB-server package and dependent packages (galera-4 and socat) by using the following commands:
```
sudo yum --disablerepo=* install /tmp/galera-4.rpm /tmp/socat.rpm
sudo yum --disablerepo=* upgrade /tmp/MariaDB-server.rpm
```
Remove incompatible backup packages:
```
sudo yum remove percona-xtrabackup
```

NOTE: If the "yum remove" command fails, it means that the package does not exist on the SL1 appliance. You can ignore the error message.

Regenerate the configuration file for MariaDB:

sudo /opt/em7/share/scripts/generate-my-conf.py -o -f  /etc/my.cnf.d/silo_mysql.cnf

Restart MariaDB:

sudo systemctl daemon-reload
sudo systemctl start mariadb

Restore the mariadb.service enabled state:

sudo systemctl ${MSRV::-1} mariadb.service

Upgrade the internal configuration for the database:
```
sudo mysql_upgrade -u root -p
```
Restart the em7 service:
```
sudo systemctl start em7
```
Repeat all the steps in this section on each non-HA/DR Database Server, All-In-One Appliance, Data Collector, and Message Collector.

Additional Steps for MariaDB Upgrades in 10.1.x

SL1 10.1.x included an upgrade to MariaDB. The upgrade did not include a tool, jemalloc, that helps manage memory usage.

This section applies only to the following releases:

10.1.0
10.1.1
10.1.2
10.1.3
10.1.4
10.1.4.1
10.1.4.2
10.1.5
10.1.5.1

For SL1 versions later than 10.1.5.1, jemalloc is included with the platform. For SL1 versions prior to 10.1.0, jemalloc is included with the platform.

To avoid problems with memory usage on Database Servers, perform the following steps after upgrading MariaDB for 10.1.x.

Perform these steps first on the active Database Server and then on each additional Database Server in your SL1 System.

Open an SSH session to the Database Server.
To verify that the Database Server is not currently running jemalloc, enter the following command at the shell prompt:
```
silo_mysql -e 'show global variables like "version_malloc_library"'
```

If the Database Server is not currently running jemalloc, the shell will display the following:

Variable Name	Value
version_malloc_library	system

Search for the file /usr/lib64/libjemalloc.so.1. If the file does not exist, contact ScienceLogic Customer Support to request the file jemalloc-3.6.0-1.el7.x86_64.rpm.

To install the RPM, use a file-transfer utility, copy the file to a directory on the SL1 appliance. Then enter the following commands at the shell prompt:
```
cd /usr/lib64
sudo yum install jemalloc-3.6.0-1.el7.x86_64.rpm
```
Create the file /etc/systemd/system/mariadb.service.d/jemalloc.conf, as follows:
```
vi /etc/systemd/system/mariadb.service.d/jemalloc.conf
```

Add the following lines to the file:

[Service]
Environment="LD_PRELOAD=/usr/lib64/libjemalloc.so.1"

Save and close the file.
Reload the systemd config files with the following command:
```
sudo systemctl daemon-reload
```
Restart the Database Server:

To restart the standalone Database Server or the primary Database Server in a cluster, enter the following:
```
sudo systemctl restart mariadb
```
To restart each secondary Database Server in a cluster:

Open an SSH session to the secondary Database Server. At the shell prompt, enter:
```
coro_config
```
Select 1.
When prompted to put the Database Server into maintenance, select y.
Open an SSH session to the primary Database Server. To pause SL1, enter the following command at the shell prompt:
```
sudo touch /etc/.proc_mgr_pause
```
In the SSH session for the secondary Database Server, restart MariaDB:
```
crm resource restart mysql
```
After MariaDB has restarted successfully on the secondary Database Server, return to the SSH session on the primary Database Server. Remove the pause file for SL1 using the following command:
```
sudo rm /etc/.proc_mgr_pause
```
In the SSH session on the secondary Database Server, take the Database Server out of maintenance. At the shell prompt, enter:
```
coro_config
```
Select 1.
When prompted to take the Database Server out of maintenance, select y.

To verify that jemalloc is running on the Database Server, enter the following command at the shell prompt:
```
silo_mysql -e 'show global variables like "version_malloc_library"'
```

If the Database Server is currently running jemalloc, the shell will display something like the following:

Variable Name	Value
version_malloc_library	jemalloc 3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340

Perform these steps on each Database Server in your SL1 system.

Rebooting Appliances in the SL1 Distributed Stack

Refer to the release notes for your current release to determine if you must reboot all SL1 appliances. Not every SL1 update requires rebooting.

When an upgrade requires a reboot, use the steps listed in this section to reboot all SL1 appliances in the Distributed stack.

Rebooting the Administration Portal

You can reboot Administration Portals either from the user interface or from the command line.

Rebooting Multiple Administration Portals

If your SL1 system includes multiple Administration Portals, you can remotely reboot Administration Portals from another Administration Portal. To do so:

Go to the Appliance Manager page (System > Settings > Appliances).
Select the checkboxes for the SL1 appliances you want to reboot.
In the [Select Action] menu, select Reboot and click the Go button.
Click the OK button when the "Are you sure you want to reboot the selected appliances?" message is displayed.
During the reboot, the user interface for the affected Administration Portal unavailable.
When the reboot has completed, the Audit Logs page (System > Monitor > Audit Logs) will include an entry for each appliance that was rebooted.

Rebooting a Single Administration Portal

If your SL1 system include only a single Administration Portal, perform the following steps to reboot that Administration Portal:

Either go to the console of the Database Server or use SSH to access the Database Server.
Log in as em7admin with the appropriate password.

At the shell prompt, execute the following:

python -m silo_common.admin_toolbox <appliance_ID> "/usr/bin/sudo /usr/sbin/shutdown -r +1"

where:

appliance_ID is the appliance ID for the Data Collector, Message Collector, or Administration Portal.

Rebooting Data Collectors and Message Collectors

You can reboot Data Collectors and Message Collectors either from the user interface or from the command line.

Rebooting Data Collectors and Message Collectors from the Appliance Manager page

From the SL1 user interface, perform the following steps to reboot a Data Collector or a Message Collector:

Go to the Appliance Manager page (Appliance Manager).
Select the checkbox for each SL1 appliance you want to reboot.
In the [Select Action] menu, select Reboot and click the Go button.
Click the OK button when the "Are you sure you want to reboot the selected appliances?" message is displayed.
During the reboot, go to the System Logs page (System > Monitor > System Logs). You should see this message:

Major: Could not connect to module (5) database USING SSL=TRUE: Error attempting to connect to database with SSL enabled True: (2003, 'Can't connect to MySQL server on '10.2.12.77' (113 "No route to host")')

When the reboot has completed, the Audit Logs page (System > Monitor > Audit Logs) will include an entry for each appliance that was rebooted.

Rebooting Data Collectors and Message Collectors from the Command Line

From the console of the Database Server or SSH to the Database Server, perform the following steps to reboot Data Collector or Message Collector:

Either go to the console of a Database Server or SSH to access the Database Server.
Log in as em7admin with the appropriate password.

At the shell prompt, execute the following:

python -m silo_common.admin_toolbox <appliance_ID> "/usr/bin/sudo /usr/sbin/shutdown -r +1"

where:

appliance_ID is the appliance ID for the Data Collector, Message Collector, or Administration Portal.

Rebooting Standalone All-In-One Appliance and Standalone Database Server

Perform the following steps to reboot a standalone All-In-One Appliance or a standalone Database Server:

Either go to the console or use SSH to access the SL1 appliance.
Log in as em7admin with the appropriate password.

On the SL1 appliance, pause the system and shutdown MariaDB.

sudo touch /tmp/.proc_mgr_pause
sudo systemctl stop mariadb

Reboot the SL1 appliance:
```
sudo reboot
```
After the SL1 appliance has rebooted, either go to the console or use SSH to access the SL1 appliance.
Log in as em7admin with the appropriate password.
Un-pause the SL1 Appliance:
```
sudo rm /tmp/.proc_mgr_pause
```

Rebooting Two Database Servers Configured for Disaster Recovery

Perform the following steps to reboot two Database Servers configured for Disaster Recovery:

Either go to the console of the primary Database Server or use SSH to access the primary Database Server.
Log in as em7admin with the appropriate password.

Check the status of both Database Servers. To do this, enter the following at the shell prompt:

cat /proc/drbd

Your output will look like this:

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
ns:17567744 al:0 bm:1072 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12521012

Pause the system and shutdown MariaDB on the primary Database Server. Enter the password for the em7admin user when prompted:
```
sudo touch /tmp/.proc_mgr_pause
sudo systemctl stop pacemaker
sudo systemctl stop mariadb
```
Reboot the primary Database Server:
```
sudo reboot
```
After the primary appliance has rebooted, log in to the console of the primary Database Server again.
Execute the following command on the primary Database Server:
```
coro_config
```
Select the Promote DRBD option.
Execute the following commands on the primary Database Server:
```
sudo rm /tmp/.proc_mgr_pause
```
Enter the password for the em7admin user and confirm the command when prompted.
Log in to the secondary Database Server as the em7admin user using the console or SSH.
Execute the following command on the secondary Database Server to reboot the appliance:
```
sudo reboot
```
Enter the password for the em7admin user when prompted.

Rebooting Two Database Servers in a High Availability Cluster

Perform the following steps to reboot two Database Servers in a high availability cluster:

Either go to the console of the secondary Database Server or use SSH to access the secondary Database Server.
Log in as em7admin with the appropriate password.
Check the status of both Database Servers. To do this, enter the following at the shell prompt:
```
cat /proc/drbd
```
Your output will look like this:

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----

ns:17567744 al:0 bm:1072 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12521012

If your output includes "ro:Secondary/Primary", but does not include "UpToDate/UpToDate", data is being synchronized between the two appliances. You must wait until data synchronization has finished before rebooting.

Stop the cluster service on the secondary Database Server:
```
sudo systemctl stop pacemaker
```
Enter the password for the em7admin user when prompted.
Either go to the console of the primary Database Server or use SSH to access the primary Database Server.
Log in as em7admin with the appropriate password.
Pause the system and stop the cluster service on the primary Database Server. Enter the password for the em7admin user when prompted:
```
sudo touch /tmp/.proc_mgr_pause
sudo systemctl stop pacemaker
sudo systemctl stop mariadb
```
Reboot the primary Database Server:
```
sudo reboot
```
After the primary Database Server has rebooted, either go to the console of the primary Database Server or use SSH to access the primary Database Server.
Log in as em7admin with the appropriate password.
Execute the following command on the primary Database Server:
```
sudo rm /tmp/.proc_mgr_pause
```
Enter the password for the em7admin user and confirm the command when prompted.
Either go to the console of the secondary Database Server or use SSH to access the secondary Database Server.
Log in as em7admin with the appropriate password.
Reboot the secondary Database Server:
```
sudo reboot
```
Enter the password for the em7admin user when prompted.

Rebooting Three Database Servers Configured for High Availability and Disaster Recovery

Perform the following steps to reboot three Database Servers configured for high availability and disaster recovery. In this configuration, two Database Servers are configured as a High Availability cluster and one Database Server is configured for Disaster Recovery.

Either go to the console of the secondary Database Server in the HA cluster or use SSH to access the secondary Database Server in the HA cluster,
Log in as em7admin with the appropriate password.
Check the status of both Database Servers in the HA cluster. To do this, enter the following at the shell prompt:
```
cat /proc/drbd
```
Your output will look like this:

10: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----

ns:17567744 al:0 bm:1072 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12521012

Stop the cluster service with the following command on the secondary Database Server in the HA cluster:
```
sudo systemctl stop pacemaker
```
Enter the password for the em7admin user when prompted.
Either go to the console of the primary Database Server in the HA cluster or use SSH to access the primary Database Server in the HA cluster.
Log in as em7admin with the appropriate password.
Pause the system and stop the cluster service on the primary Database Server in the HA cluster :
```
sudo touch /tmp/.proc_mgr_pause
sudo systemctl stop pacemaker
```
Enter the password for the em7admin user when prompted
Reboot the primary Database Server in the HA cluster:
```
sudo reboot
```
After the primary Database Server in the HA cluster has rebooted, either go to the console of the primary Database Server in the HA cluster or use SSH to access the primary Database Server in the HA cluster.
Execute the following command on the primary Database Server in the HA cluster:
```
sudo rm /tmp/.proc_mgr_pause
```
Enter the password for the em7admin user and confirm the command when prompted.
Either go to the console of the secondary Database Server in the HA cluster or use SSH to access the secondary Database Server in the HA cluster.
Log in as em7admin with the appropriate password.
Reboot the secondary Database Server in the HA cluster:
```
sudo reboot
```
Enter the password for the em7admin user when prompted.
Either go to the console of the Database Server for Disaster Recovery or use SSH to access the Database Server for Disaster Recovery.
Log in as em7admin with the appropriate password.
Reboot the Database Server for Disaster Recovery:
```
sudo reboot
```
Enter the password for the em7admin user when prompted.

Upgrading to Aurora 3 RDS (MySQL 8.0)

This section applies only to users who deploy SL1 on AWS and are currently running on Aurora 2 RDS (MySQL 5.7). If you do not deploy SL1 on AWS or you are already running on Aurora 3, you can skip this section.

If you deploy SL1 using AWS, you are currently running on Aurora 2 RDS (MySQL 5.7), and are upgrading to SL1 12.1.2, you can upgrade to Aurora 3 RDS (MySQL 8.0).

Aurora 3 upgrades are not supported for SL1 12.2.x.

Before You Upgrade to Aurora 3

Before you upgrade to Aurora 3, you must have already done the following:

Ensured that you have administrator access to the AWS data engines, RDS, and IAM from the AWS console.
Already upgraded SL1 to a version that supports Aurora 3.
Created an RDS snapshot.
Created an Ansible AWX instance using the "ScienceLogic SL1 Deployment - AWX V5" AMI.

To obtain this AMI, contact ScienceLogic Support.
Logged in to the AWX instance using credentials provided by ScienceLogic.
Created a jump host using AWX, if you do not already have one.
Updated the deploy, em7admin, and jump host credentials with the correct information.
Updated the DE01 and jump host IP addresses.

Performing the Aurora 3 Upgrade

To upgrade to Aurora 3:

Ensure that your SL1 appliances are in maintenance mode.
Log in to your AWX instance using the credentials provided by ScienceLogic.
Run the "AWS - RDS Aurora3 Upgrade" job with the following input:

---

aws_region:

client:

vpc_id:
After the playbook has executed successfully, verify that your database and application perform the post-application validation check.

Restoring the SSL Certificates

To restore your SSL Certificates:

Login to the console of the Database Server or SSH to the Database Server.
Open a shell session.

Enter the following at the shell prompt:

cp /etc/nginx/silossl.key.bak /etc/nginx/silossl.key
cp /etc/nginx/silossl.pem.bak /etc/nginx/silossl.pem

Repeat these steps on each Database Server in your SL1 system.

Resetting the Timeout for PhoneHome Watchdog

You can manually reset the settings for the PhoneHome Watchdog server back to the settings you used before the upgrade.

To edit the settings for the watchdog service:

Log in to the console of the Data Collector as the root user or open an SSH session on the Data Collector.
View your PhoneHome Watchdog settings:
```
phonehome watchdog view
```
Your output will look like the following:
```
Current settings:
autosync: yes
interval: 120
state: enabled
autoreconnect: yes
timeoutcount: 1
check: default
```
Note the settings for interval and timeoutcount, so you can restore them after the upgrade.

To change the settings for SL1 upgrade, type the following at the command line:

sudo phonehome watchdog set interval=<previous setting>;
sudo phonehome watchdog set timeoutcount=<previous setting>;
systemctl stop em7_ph_watchdog;
systemctl start em7_ph_watchdog;

Repeat these steps on each Data Collector.
Repeat these steps on each Message Collector.
Repeat these steps on each Database Server.

Updating Default PowerPacks

Every time you install a software update on your appliances, ScienceLogic recommends that you also install the updates for all the PowerPacks that were included in the software update.

ScienceLogic includes multiple PowerPacks in the default installation of SL1. When you apply an update to your system, new versions of the default PowerPacks will be automatically imported in to your system. If a PowerPack is included in an update and is not currently installed on your system, SL1 will automatically install the PowerPack. If a PowerPack is included in an update and is currently installed on your system, SL1 will automatically import (but not install) the PowerPack.

If PowerPacks have been imported into your system but have not been installed, the Update column appears in the PowerPack Manager page (System > Manage > PowerPacks). For each PowerPack that has been imported to your system but has not been installed, the lightning bolt icon () appears in the Update field on the PowerPack Manager page.

To install the updates for multiple PowerPacks:

Go to the PowerPack Manager page (System > Manage > PowerPacks) and click the checkbox for each PowerPack you want to install.
In the Select Action drop-down field (in the lower right), choose Update PowerPack(s). SL1 displays a warning message before updating the PowerPack(s).
Click the OK button to continue the installation.
Click the Go button. If you completed the update, updated information about the PowerPack will appear in the PowerPack Manager page. All the items in the PowerPack will be installed in your SL1 system.

NOTE: You can install multiple PowerPacks with the Select Action drop-down list only if each selected PowerPack includes an embedded Installation Key. PowerPacks that do not include embedded Installation Keys will fail to install.

NOTE: If the Enable Selective PowerPack Field Protection checkbox on the Behavior Settings page (System > Settings > Behavior) is selected, certain fields in Event Policies, Dynamic Applications, and Device Classes will not be updated.

Configuring Subscription Billing

If your SL1 system is configured to communicate with the ScienceLogic billing server, usage data will be sent automatically from your SL1 system to the ScienceLogic billing server once a day. After the ScienceLogic billing server receives the usage data, SL1 will automatically mark the license usage file as delivered.

Sending usage data to the ScienceLogic billing server ensures that your bill is accurate and that ScienceLogic can continue making improvements to the SL1 products.

To determine if you have correctly configured Subscription Billing:

Go to the System Usage page (System > Monitor > System Usage) or (Manage > Subscription Usage). Click the Subscription button and choose License Data Delivery Status.
For air-gapped SL1 systems, the value of Summary Date should be within the past 48 hours.
For SL1 systems that connect to ScienceLogic, the value of Summary Date should be within the past 48 hours and the value of Delivery Status is 1.

For details on configuring subscription billing, see the section on configuring subscription billing.