Performance
The Snippet Framework was written with performance in mind. However there are things you can do as a Snippet Argument developer to help improve the performance of your Dynamic Application. Below are some scenarios how you can instruct the Snippet Framework to be more performant.
Utilizing De-Duplication
The Snippet Framework performs de-duplication between shared step paths. Collections are grouped based on these paths which removed performing the same operation which increases the overall performance of the Snippet Framework. When the step paths diverge, the data is cloned into the divergent collections. This enables the Snippet Framework to have performance closer to writing a pythonic snippet.
In this example, we will use a small data set but this concept
applies to large data sets too. With the provided data set, we want
to extract out 1_1 and 1_2 but allow the framework to reduce the
amount of duplicated calls to key1.
{
"key1": {
"1_1": 11,
"1_2": 12
},
"key2": {
"2_1": 21
}
}
The most performant way to write this utilizing JMESPath would
be to reduce the amount of times you iterate through key1
by having that its own step. This reduces running the lookups
for key1 for each collection which increases the overall
performance and reduces the memory footprint while executing.
low_code:
version: 2
steps:
- gather_data
- json
- jmespath:
value: "'key1'"
- jmespath:
value: "'1_1'"
low_code:
version: 2
steps:
- gather_data
- json
- jmespath:
value: "'key1'"
- jmespath:
value: "'1_2'"
As we can see while using this, the value nested under key1
is only accessed once. If this was a large data set or used a complex
JMESPath expression, this would result in a performance boost
as it will not re-run the same JMESPath expression.
Large Data Sets
The Snippet Framework can process large data sets without issues. However as a Snippet Argument developer you can use more performant steps or arguments. For example, if you are using a parser to convert text you should think about the search and memory implications. Refer to Python Time Complexity when considering result data types.
Caching Across Dynamic Applications
The Snippet Framework can cache information between Dynamic Applications through the use of a Cacher step. This step enables data to be saved and referenced at a later time which can increase performance across Dynamic Applications by reusing previously computed values.
A cache step acts as both a cache writer and reader within your Snippet Arguments. If a valid cache entry is not found, it will run all the steps preceding the cache step and write the result to cache. If a valid cache entry is found, it starts processing at the step following the cache step using the cache entry as the result.
When using a cache step, determine what steps will be required between Dynamic Applications and select where in the Snippet Argument to cache the data. A Snippet Argument can use multiple cacher steps throughout the execution plan.
This functionality should be used when a network request’s result must be transformed, and the transformation time is costly. For example, if an HTTP request’s result is a large json payload and the steps to transform the data takes about 2 minutes. Let’s say the transformed data is needed for 5 Dynamic Applications then without caching, each Dynamic Application would have to spend 2 minutes running through the transformation steps. By using caching, one Dynamic Application will run the 2 minute transformation and the other Dynamic Applications will reuse the newly cached data.
Note
Examples of caching across Dynamic Applications can be found in the following Dynamic Applications that ship with the PowerPack:
Caching: Auto-Key Example 1
Caching: Auto-Key Example 2
Caching: User-Defined Key Example 1
Caching: User-Defined Key Example 2