Creating a Synchronization PowerPack

Download this manual as a PDF file

This section describes how to package your PowerFlow applications, steps, configuration objects, and utilities into a Synchronization PowerPack.

The label "SyncPack" is used in place of "Synchronization PowerPack" in the PowerFlow user interface.

What is a Synchronization PowerPack?

A Synchronization PowerPack is a Python .whl file that contains PowerFlow applications, steps, configuration objects, and utilities that can be used in PowerFlow. Synchronization PowerPacks are versioned and can define dependencies on other Synchronization PowerPacks or on external Python packages.

When applications from a Synchronization PowerPack are executed, they run within a Python Virtual Environment that contains the Synchronization PowerPack itself and its dependencies.

Generating the Synchronization PowerPack Structure

ScienceLogic uses a Cookiecutter repository for generating the standard Synchronization PowerPack structure. This structure includes all necessary files and the default directory structure that is generated based on your response to the prompts.

For more information about Cookiecutter, see https://cookiecutter.readthedocs.io/en/latest/.

To use Cookiecutter to generate the Synchronization PowerPack structure:

  1. On a system with Python 3.6 or later installed, run the following command:

    pip install cookiecutter

    cookiecutter https://github.com/ScienceLogic/is_syncpack_cookiecutter.git

  2. Complete the following fields:

  • author. Name of the person or organization that created this Synchronization PowerPack.
  • url_project. URL of the repository where this Synchronization PowerPack is hosted.
  • syncpack_name. Developer name of the Synchronization PowerPack, without spaces. For example: test_syncpack.
  • syncpack_friendly_name. Friendly name of the Synchronization PowerPack, which will display in the PowerFlow user interface. For example: Test Synchronization PowerPack.
  • syncpack_description. Short description of the Synchronization PowerPack, which will display in the PowerFlow user interface and in the readme file.
  • version. Number of the Synchronization PowerPack version.
  • requires_minimum_is_version. The minimum version of the PowerFlow platform required for this Synchronization PowerPack.

After you complete this process, Cookiecutter creates a new directory in your current working directory named with the syncpack_name you specified.

Synchronization PowerPack Structure

After you use Cookiecutter to generate the Synchronization PowerPack structure, you can access the following files and folders in the folder created by the previous process.

  • MANIFEST.in. Allows the Python .whl file to contain non-Python files.
  • README.md. The readme file for this Synchronization PowerPack, which is pre-populated with basic information.
  • docker. This directory contains the following Docker information:
  • Dockerfile. Sample Dockerfile that can be used for building and testing the Synchronization PowerPack in a CI/CD environment.
  • docker-compose.yml. Docker Compose file that uses above container.
  • setup.py. Python setup file for building the Synchronization PowerPack. You should not need to modify this file.
  • test_syncpack. This directory contains all Synchronization PowerPack content, including the following:
  • __init__.py. Python init file.
  • apps. This directory contains Synchronization PowerPack application .JSON files.
  • configs. This directory contains Synchronization PowerPack configuration object .JSON files.
  • meta.json. This file defines the Synchronization PowerPack properties.
  • steps. This directory contains Synchronization PowerPack Step Python files.
  • util. This directory contains Python classes or modules that can be used by multiple steps in the Synchronization PowerPack.
  • tox.ini. Basic tox configuration for running unit tests.

Synchronization PowerPack Properties

The meta.json file contains all configurable properties for a Synchronization PowerPack. This file is where you define the name, friendly name, version, description, default schedules, tags, and supported PowerFlow version of the Synchronization PowerPack.

requires_dist is where you define the Python packages on which this Synchronization PowerPack depends. For example, if you would like to use a step from the Base Steps Synchronization PowerPack and also use Amazon's boto library, you would use the following:

"requires_dist": ["boto", "base_steps_syncpack"],

The following code block contains an example of the meta.json file:

{

"name": "test_syncpack",

"version": "0.0.1",

"summary":"Testing Syncpack",

"description":"IS Syncpack for testing",

"author":"John Smith",

"home_page":"https://example.com/test_syncpack",

"requires_dist":[],

"schedules": [

{

"application_id": "dummy_app",

"entry_id": "dummy app",

"params": {},

"schedule": {

"schedule_info": {

"run_every": 3600

},

"schedule_type": "frequency"

}

}

],

"tags":["test_syncpack"],

"critical": [],

"requires_is_version": ">=2.0.0"

}

Building a Synchronization PowerPack

After you complete your Synchronization PowerPack structure and place your applications, steps, and configuration objects in the right directories, you can build the Synchronization PowerPack .whl file (wheel).

You can build a Synchronization PowerPack with Python3 or with the iscli utility that comes with PowerFlow.

Not included is how to build a Synchronization PowerPack automatically with CI/CD platforms.

Building a Synchronization PowerPack with Python3

To build the Synchronization PowerPack wheel with Python3, run the following command from the top-level Synchronization PowerPack directory:

python3 setup.py sdist bdist_wheel

This command creates build and dist folders within your SyncPack directory. In the dist folder, you will now have a Python wheel that you can upload to PowerFlow.

Building a Synchronization PowerPack with the iscli Utility

If you build your Synchronization PowerPack with the iscli utility, the utility builds and uploads the Synchronization PowerPack to PowerFlow in one step.

To build and upload your Synchronization PowerPack with the iscli utility, run the following command:

iscli -H <IS-IP-host> -U <IS User> -p <IS Password> -ukf </path/to/SyncPack/Directory>

where </path/to/SyncPack/Directory> is the Synchronization PowerPack root that contains the setup.py file.

Uploading a Synchronization PowerPack to PowerFlow

After you create your Synchronization PowerPack wheel, you can upload it to PowerFlow. You can upload this Synchronization PowerPack to PowerFlow using the PowerFlow user interface or with the iscli utility.

Uploading a Synchronization PowerPack with the User Interface

To import a Synchronization PowerPack in the PowerFlow user interface:

  1. On the SyncPacks page of the PowerFlow user interface, click Import SyncPack. The Import SyncPack page appears.

  2. Click Browse and select the .whl file for the Synchronization PowerPack you want to install.

    You can also drag and drop a .whl file to the Import SyncPack page.

  3. Click Import. PowerFlow registers and uploads the Synchronization PowerPack. The Synchronization PowerPack is added to the SyncPacks page.

You cannot edit the content package in a Synchronization PowerPack published by ScienceLogic. You must make a copy of a ScienceLogic Synchronization PowerPack and save your changes to the new Synchronization PowerPack to prevent overwriting any information in the original Synchronization PowerPack when upgrading.

Uploading a Synchronization PowerPack with the iscli Utility

To upload a Synchronization PowerPack with the iscli utility:

  1. Using a tool like secure copy (SCP), copy the Synchronization PowerPack from your local computer to your PowerFlow host.

  1. Start an SSH session with your PowerFlow host.

  2. Run the following command:

    iscli -ukf /path/to/test_syncpack-0.0.1-py2.py3-none-any.whl

    If you have already uploaded this Synchronization PowerPack to your PowerFlow system and you want to re-upload with the same version, add the -F flag to the above command.

  1. You can now activate the Synchronization PowerPack in PowerFlow. See the following topic for more information.

Activating and Installing a Synchronization PowerPack

After you upload a Synchronization PowerPack to PowerFlow, you need to activate and install the Synchronization PowerPack to access its applications, steps, and configuration objects.

For the "Activate & Install Syncpacks" application, the retry_max parameter for the "Activate Syncpack" and "Install Syncpack" steps is set to 3 by default. The time between those retries is calculated randomly based on the number of retries. These settings prevent the steps from colliding with each other when the steps are run in a environment with a large number of syncpack_steprunners that are trying to install a SyncPack at the same time in their respective volumes.

As a best practice, you should only edit the retry_max parameter and avoid editing any of the other retry parameters. Only advanced users who understand how the retries work and their side effects when they are not set correctly should change the other retry parameters. For more information, see Defining Retry Options for a Step.

Activating and Installing with the User Interface

To activate and install a Synchronization PowerPack in the PowerFlow user interface:

  1. On the SyncPacks page of the PowerFlow user interface, click the Actions button () for the Synchronization PowerPack you want to install and select Activate & Install. The Activate & Install SyncPack modal appears.

    If you try to activate and install a Synchronization PowerPack that is already activated and installed, you can choose to "force" installation across all the nodes in the PowerFlow system.

    If you do not see the PowerPack that you want to install, click the Filter icon () on the SyncPacks page and select Toggle Inactive SyncPacks to see a list of the imported PowerPacks.

  1. Click Yes to confirm the activation and installation. When the Synchronization PowerPack is activated, the SyncPacks page displays a green check mark icon () for that Synchronization PowerPack. If the activation or installation failed, then a red exclamation mark icon () appears.

  2. For more information about the activation and installation process, click the check mark icon () or the exclamation mark icon () in the Activated column for that Synchronization PowerPack. For a successful installation, the "Activate & Install SyncPack" application appears, and you can view the Step Log for the steps. For a failed installation, the Error Logs window appears.

  3. If you have other versions of the same Synchronization PowerPack on your PowerFlow system, you can click the Actions button () for that Synchronization PowerPack and select Change active version to activate a different version other than the version that is currently running.

Activating and Installing with the iscli Utility

To activate and install an uploaded Synchronization PowerPack using the iscli utility, run the following command:

iscli -AIkn <syncpack_name> -V <syncpack version>

where <syncpack_name> is the name from the meta.json file, and <syncpack_version> is the version number of the Synchronization PowerPack.

For example:

iscli -AIkn test_syncpack -V 0.0.1

To force a re-activation of the Synchronization PowerPack, add -F to the above command.

Extending an Existing Synchronization PowerPack

Because the content that is included in a Synchronization PowerPack cannot be overwritten inside PowerFlow, you will need to extend existing Synchronization PowerPacks with your own Synchronization PowerPack to add customizations.

Restrictions

  • The names for applications, steps, and configuration objects must be unique within the entire PowerFlow platform. For example, you cannot have a step named "QueryREST", because that step is already included in the Base StepsSynchronization PowerPack. You will need to change the name of your customized step.
  • Your new Synchronization PowerPack must include the Synchronization PowerPack you are extending in the requires_dist list in your meta.json. This ensures that your Synchronization PowerPack will have access to the applications and steps of the parent Synchronization PowerPack in its execution environment.
  • After you install the Synchronization PowerPack, its environment will never be modified, even if the parent Synchronization PowerPacks are updated. If you wish to include changes from the parent Synchronization PowerPack, you will need to re-install your customized Synchronization PowerPack.

Modifying Application Definitions

To change the API endpoint or query used in an existing application:

  1. Copy the entire JSON file for the application from the original Synchronization PowerPack into the apps directory of your new Synchronization PowerPack.
  2. Rename the file so it will not conflict with the existing application.
  3. In the JSON code, change the friendly_name, author, description, and version as required.
  4. Make any other changes to the application as desired.

Modifying a Step

You have two methods for modifying a step. The first method is to change the code in a step itself, and the second is to extend the step class and execute your own code before or after the existing step.

To change the logic performed within a step:

  1. Copy the entire step Python file from the original Synchronization PowerPack into the steps directory of your new Synchronization PowerPack.
  2. Rename the file so it will not conflict with the existing step.
  3. In the step file, change the class name to match the name of the step file.
  4. In the __init__() section of the code, change the friendly_name, description , and version as required.
  5. Make any other changes to the step as desired.

To "extend" the behavior of an existing step by performing work before or after the existing logic:

  1. Create a new step and give it a unique name.

  2. Instead of importing and extending BaseStep import and extend the step class you would like to modify. For example:

    from base_steps_syncpack.steps.QueryGQL import QueryGQL

    class OptionalQueryGQL(QueryGQL):

  3. Complete the __init__() as normal, but make sure to call init of the super class to include all of its parameters. For example:

    super(OptionalQueryGQL, self).__init__()

  4. Create your additional logic in the execute() function. This logic can be performed either before or after you call the execute of the parent step. For example:

    super(OptionalQueryGQL, self).execute()

If you would like to perform additional logic after the execute of the parent step runs, you can access the data from that step via self.get_current_saved_data().

Full Extended Step Example

from base_steps_syncpack.steps.QueryGQL import QueryGQL

from ipaascore import parameter_types

from ipaascommon.ipaas_utils import str_to_bool

class OptionalQueryGQL(QueryGQL):

def __init__(self):

super(OptionalQueryGQL, self).__init__()

self.friendly_name = "Optional QueryGQL Call"

self.description = "Optionally Runs GQL call."

self.version = "1.0.0"

 

self.new_step_parameter(

name="enable",

description="Should this step run.",

default_value=True,

required=False,

param_type=parameter_types.BoolParameterToggle()

)

 

def execute(self):

enabled = str_to_bool(self.get_parameter("enable", None))

if enabled:

super(OptionalQueryGQL, self).execute()

 

else:

self.logger.flow(

"Step is Disabled. Returning empty dict."

)

self.save_data_for_next_step(dict())