Creating a SyncPack

This section describes how to package your PowerFlow applications, steps, configuration objects, and utilities into a SyncPack.

The label "SyncPack" is used in place of "SyncPack" in the PowerFlow user interface.

What is a SyncPack?

A SyncPack is a Python .whl file that contains PowerFlow applications, steps, configuration objects, and utilities that can be used in PowerFlow. SyncPacks are versioned and can define dependencies on other SyncPacks or on external Python packages.

When applications from a SyncPack are executed, they run within a Python Virtual Environment that contains the SyncPack itself and its dependencies.

Generating the SyncPack Structure

ScienceLogic uses a Cookiecutter repository for generating the standard SyncPack structure. This structure includes all necessary files and the default directory structure that is generated based on your response to the prompts.

For more information about Cookiecutter, see https://cookiecutter.readthedocs.io/en/latest/.

The GitHub repository for the Cookiecutter project can be found at https://github.com/ScienceLogic/is_syncpack_cookiecutter.

To use Cookiecutter to generate the SyncPack structure:

1. On a system with Python 3.6 or later installed, run the following commands to install Cookiecutter (use a virtual environment if possible):

   pip install cookiecutter

   cookiecutter https://github.com/ScienceLogic/is_syncpack_cookiecutter.git

   To use Cookiecutter, you will need to install Visual Studio Code (VSCode) or PyCharm. You will also need to install Docker.

2. Log in to the SL container registry:

   docker login registry.scilo.tools

   You can find the credentials for this step by logging in to the Harbor user interface, clicking your user name in the top right, and selecting User Profile. From the User Profile dialog, copy the username and CLI secret (key) you will need to authenticate with when running the docker login command.

3. Create a root directory where you can start building the SyncPack. For example: /home/username/pf_syncpacks_workspace.

4. In the root directory from step 3, run the Cookiecutter tool to create a SyncPack directory structure.

   For more information about the requirements for the Cookiecutter tool, see the Syncpack Cookie Cutter Readme file.

5. The Cookiecutter tool will ask you for the following information:

• author. Name of the person or organization that created this SyncPack.
• url_project. URL of the repository where this SyncPack is hosted. You can use a placeholder or fake URL in this field if you do not currently plan to push code to a live PowerFlow system.
• syncpack_name. Developer name of the SyncPack, without spaces. For example: test_syncpack.
• syncpack_friendly_name. Friendly name of the SyncPack, which will display in the PowerFlow user interface. For example: Test SyncPack.
• syncpack_description. Short description of the SyncPack, which will display in the PowerFlow user interface and in the readme file.
• version. Number of the SyncPack version.
• requires_minimum_pf_version. The minimum version of the PowerFlow platform required for this SyncPack.
• dev_container_source. Use "SL External" if you are not a ScienceLogic employee.

After you complete this process, Cookiecutter creates a new directory in your current working directory named with the syncpack_name you specified, above. For more information about the contents of the new directory, see Synchronization PowerPack Structure.

To continue the process of creating the SyncPack, see Building a Synchronization PowerPack.

SyncPack Structure

After you use Cookiecutter to generate the SyncPack structure, you can access the following files and folders in the folder created by the previous process.

• MANIFEST.in. Allows the Python .whl file to contain non-Python files.
• README.md. The readme file for this SyncPack, pre-populated with basic information.
• setup.py. Python setup file for building the SyncPack. You should not need to modify this file.
• docker. This directory contains the following Docker information:
  • docker-compose.yml. Docker Compose file that uses the above container.
  • Dockerfile. Sample Dockerfile that can be used for building and testing the SyncPack in a continuous integration and continuous delivery (CI/CD) environment.

• test_syncpack. This directory contains all SyncPack content, including the following:
  • __init__.py. Python init file.
  • apps. This directory contains SyncPack application .json files.
  • configs. This directory contains SyncPack configuration object .json files.
  • meta.json. This file defines the SyncPack properties.
  • steps. This directory contains SyncPack Step Python files.
  • util. This directory contains Python classes or modules that can be used by multiple steps in the SyncPack.
  • tox.ini. Basic tox configuration for running unit tests.

SyncPack Properties

The meta.json file contains all configurable properties for a SyncPack. This file is where you define the name, friendly name, version, description, default schedules, tags, and supported PowerFlow version of the SyncPack.

requires_dist is where you define the Python packages on which this SyncPack depends. For example, if you would like to use a step from the "Base Steps" SyncPack and also use Amazon's boto library, you would use the following:

"requires_dist": ["boto", "base_steps_syncpack"],

Example Code: meta.json

Building a SyncPack

After you complete your SyncPack structure and place your applications, steps, and configuration objects in the right directories, you can build the SyncPack .whl file (wheel).

You can build a SyncPack with Python3 or with the iscli utility that comes with PowerFlow.

Not included is how to build a SyncPack automatically with CI/CD platforms.

Building a SyncPack with Python3

To build the SyncPack wheel with Python3, run the following command from the top-level SyncPack directory:

python3 setup.py sdist bdist_wheel

This command creates build and dist folders within your SyncPack directory. In the dist folder, you will now have a Python wheel that you can upload to PowerFlow.

Building a SyncPack with the iscli Utility

If you build your SyncPack with the iscli utility, the utility builds and uploads the SyncPack to PowerFlow in one step.

To build and upload your SyncPack with the iscli utility, run the following command:

iscli -H <IP_PowerFlow_host> -U <PowerFlow_user> -p <PowerFlow_password> -ukf </path/to/SyncPack/Directory>

where </path/to/SyncPack/Directory> is the SyncPack root that contains the setup.py file.

Extending an Existing SyncPack

Because the content that is included in a SyncPack cannot be overwritten inside PowerFlow, you will need to extend existing SyncPacks with your own SyncPack to add customizations.

Restrictions

• The names for applications, steps, and configuration objects must be unique within your entire PowerFlow platform. For example, you cannot add a custom step named "GetREST", because that step is already included in the "Base Steps" SyncPack. You will need to change the name of your custom step.
• Your new SyncPack must include the SyncPack that you are extending in the requires_dist list in your meta.json. This ensures that your SyncPack will have access to the applications and steps of the parent SyncPack in its execution environment.
• After you install the SyncPack, its environment will never be modified, even if the parent SyncPacks are updated. If you wish to include changes from the parent SyncPack, you will need to re-install your customized SyncPack.

Advanced Method: Extracting and Updating an Existing SyncPack

This method of modifying a SyncPack includes the following steps:

1. Download an existing SyncPack from a PowerFlow system
2. Modify the contents of the SyncPack and save as a new SyncPack
3. Re-upload and install the new SyncPack.

Although this advanced method involves more manual steps, this method is recommended over the basic method described in later sections.

To extract and update an existing SyncPack:

1. Identify the SyncPack that you want to extend and download it from the PowerFlow system or download it from ScienceLogic Support site at https://support.sciencelogic.com/s/powerpacks.

   You can download any SyncPack on a PF system from the devpi interface: https://10.2.11.232:3141/isadmin/syncpacks.

2. You can access the contents of the .whl file for the SyncPack by unzipping the .whl file. Run the following command to unzip the contents into the current working directory:

   unzip <syncpack_whl_file>

   where <syncpack_whl_file> is the name of the .whl file.

   For example:

   unzip base_steps_syncpack-1.0.0-py3-none-any.whl

3. Update the SyncPack directory name to make it easier to recognize than the default version of the SyncPack you are modifying.

4. Update the following elements in the meta.json file:

   • Change the SyncPack name to match the updated directory name chosen in step 3.

   • Update the requires_dist parameter to include the original SyncPack which is being extending. This ensures the modified SyncPack can use the existing code as necessary.

     For example, if you are modifying base_steps_syncpack version 1.0, add the following to the requires_dist parameter:

     "requires_dist":[ "base_steps_syncpack==1.0" ]

5. Delete the contents of the directories for the applications and steps that you are not going to be customizing. As a best practice, start with a single application or step and add required steps or applications as necessary.

6. Modify the existing applications and steps as needed:

   • Be sure to change the file name of the Python file containing the step you are editing, as well as the name of the class contained in the file. The name of the Python file and the class name should match.
   • Be sure to change the file name of the application configuration file in the apps directory, or PowerFlow will reject the installation. Additionally, update the friendly_name parameter within the application JSON file.
   • Make the code changes you need to make to customize the applications and steps. If additional Python packages are required, be sure to add them to the required_dist list mentioned in step 4.

7. Run the following command to build the SyncPack with the new name and version contained in the meta.json file:

   python3 setup.py bdist_wheel

8. To upload the SyncPack using the iscli tool, run the following command:

   iscli -ukf <new_syncpack_whl_file>

   where <new_syncpack_whl_file> is the name from the .whl file created from the previous step.

   To forcefully overwrite an existing version of this SyncPack, use the following command: iscli -ukFf <new_syncpack_whl_file>
   

9. To upload the SyncPack using the PowerFlow user interface, go to the SyncPacks page, click Import SyncPack, and select the .whl file.

Basic Method: Modifying Individual Steps and Applications

This basic method of modifying a SyncPack is less involved than the advanced method, and it is effective for making quick, minor changes to a SyncPack. This approach is limited, however, because the custom steps will not run inside a SyncPack.

Modifying Application Definitions

To change the API endpoint or query used in an existing PowerFlow application:

1. Download the existing application contents from PowerFlow by copying the content of this URL from your browser:

   <URL_for_PowerFlow>/api/v1/applications/<application_name>

   where <application_name> is the name of the application that you want to edit.

   To locate the <application_name> for a PowerFlow application, do a GET /api/v1/applications and use the "id" value for the application in the results.

2. Create a JSON file on your local system and give it a unique name for the new application. The name of this file cannot match the name of the existing application file.

3. Copy the contents of the result of step 1 into the new JSON file.

4. In the JSON code, change the friendly_name, author, description, and version as required.

5. Make any other changes to the application as needed, such as changing step names, adding new steps, and adding or changing parameters.

6. Upload the new json file as a new application to the PowerFlow system using the iscli tool:

   iscli -uaf <my_new_app_name.json> -H <host>

Modifying a Step

You have two methods for modifying a step:

1. Change the code in a step itself.
2. Extend the step class and execute your own code before or after the existing step.

To change the logic performed within a step:

1. Download the existing application contents from PowerFlow by copying the content of this URL from your browser:

   <URL_for_PowerFlow>/api/v1/steps/<step_name>

   where <step_name> is the name of the step that you want to edit.

   To locate the <step_name> for a PowerFlow application, do a GET /api/v1/steps and use the "id" value for the step in the results.

2. Create a Python file on your local system and give it a unique name for the new step. The name of this file cannot match the name of the existing step file.

3. Copy the contents of the result of step 1 into the new Python file.

   echo -e can help with the newlines and make the indents correct when writing to a file.

4. In the step file, change the class name to match the name of the step file (without a .py extension).

5. In the __init__() section of the code, change the friendly_name, description , and version as required.

6. Make any other changes to the step as desired.

To "extend" the behavior of an existing step by performing work before or after the existing logic:

1. Create a new step and give it a unique name.

2. Instead of importing and extending BaseStep, import and extend the step class you would like to modify. For example:

   from base_steps_syncpack.steps.QueryGQL import QueryGQL

   class OptionalQueryGQL(QueryGQL):
   

3. Complete the __init__() as normal, but make sure to call init of the super class to include all of its parameters. For example:

   super(OptionalQueryGQL, self).__init__()

4. Create your additional logic in the execute() function. This logic can be performed either before or after you call the execute of the parent step. For example:

   super(OptionalQueryGQL, self).execute()

   If you would like to perform additional logic after the execute of the parent step runs, you can access the data from that step via self.get_current_saved_data().

5. Upload the new Python file to the PowerFlow system using the iscli tool:

   iscli -usf <new_step.py> -H <host>

Example Code: Full Extended Step

Uploading a SyncPack with the PowerFlow User Interface

After you create your SyncPack wheel, you can upload it to PowerFlow. You can upload this SyncPack to PowerFlow using the PowerFlow user interface or with the iscli utility.

Uploading a SyncPack with the User Interface

To import a SyncPack in the PowerFlow user interface:

1. On the SyncPacks page () of the PowerFlow user interface, click Import SyncPack. The Import SyncPack page appears.
2. Click Browse and select the .whl file for the SyncPack you want to install. You can also drag and drop a .whl file to the Import SyncPack page.
3. Click Import. PowerFlow registers and uploads the SyncPack. The SyncPack is added to the SyncPacks page.
4. You will need to activate and install the SyncPack in PowerFlow. For more information, see Activating and Installing a Synchronization PowerPack.

You cannot edit the content package in a SyncPack published by ScienceLogic. You must make a copy of a ScienceLogic SyncPack and save your changes to the new SyncPack to prevent overwriting any information in the original SyncPack when upgrading.

Uploading a SyncPack with the iscli Utility

To upload a SyncPack with the iscli utility:

1. Using a tool like secure copy (SCP), copy the SyncPack from your local computer to your PowerFlow host.

2. Start an SSH session with your PowerFlow host.

3. Run the following command:

   iscli -ukf /path/to/test_syncpack-0.0.1-py2.py3-none-any.whl

   If you have already uploaded this SyncPack to your PowerFlow system and you want to re-upload with the same version, add the -F flag to the above command.

4. You can now activate the SyncPack in PowerFlow. See the following topic for more information.

Activating and Installing a SyncPack

After you upload a SyncPack to PowerFlow, you need to activate and install the SyncPack to access its applications, steps, and configuration objects.

For the "Activate & Install Syncpacks" application, the retry_max parameter for the "Activate Syncpack" and "Install Syncpack" steps is set to 3 by default. The time between those retries is calculated randomly based on the number of retries. These settings prevent the steps from colliding with each other when the steps are run in a environment with a large number of syncpack_steprunners that are trying to install a SyncPack at the same time in their respective volumes.

As a best practice, you should only edit the retry_max parameter and avoid editing any of the other retry parameters. Only advanced users who understand how the retries work and their side effects when they are not set correctly should change the other retry parameters. For more information, see Defining Retry Options for a Step.

Activating and Installing a SyncPack with the User Interface

To activate and install a SyncPack in the PowerFlow user interface:

1. On the SyncPacks page of the PowerFlow user interface, click the Actions button () for the SyncPack you want to install and select Activate & Install. The Activate & Install SyncPack modal appears.

   If you try to activate and install a SyncPack that is already activated and installed, you can choose to "force" installation across all the nodes in the PowerFlow system.

   If you do not see the PowerPack that you want to install, click the Filter icon () on the SyncPacks page and select Toggle Inactive SyncPacks to see a list of the imported PowerPacks.

2. Click Yes to confirm the activation and installation. When the SyncPack is activated, the SyncPacks page displays a green check mark icon () for that SyncPack. If the activation or installation failed, then a red exclamation mark icon () appears.
3. For more information about the activation and installation process, click the check mark icon () or the exclamation mark icon () in the Activated column for that SyncPack. For a successful installation, the "Activate & Install SyncPack" application appears, and you can view the Step Log for the steps. For a failed installation, the Error Logs window appears.
4. If you have other versions of the same SyncPack on your PowerFlow system, you can click the Actions button () for that SyncPack and select Change active version to activate a different version other than the version that is currently running.

Activating and Installing with the iscli Utility

To activate and install an uploaded SyncPack using the iscli utility, run the following command:

iscli -AIkn <syncpack_name> -V <syncpack version>

where <syncpack_name> is the name from the meta.json file, and <syncpack_version> is the version number of the SyncPack.

For example:

iscli -AIkn test_syncpack -V 0.0.1

To force a re-activation of the SyncPack, add -F to the above command.