.. _sf_tools_rest_api: Usage ===== 1) Overview ----------- Endpoints ^^^^^^^^^ - ``POST /lce/api/sf_tools/{tool}`` *(queues a run and returns a receipt, or runs it locally and returns its output)* - ``GET /lce/api/sf_tools/{tool}/{request_id}`` *(reads the log, and the result once finished)* The Snippet Framework Tools API runs the platform's Dynamic Application toolbox tools on a Data Collector and reads their output back. These are the jobs the platform's Dynamic Application editor queues: the same Remote Toolbox tool ids, the same backend scripts and the same command lines, so a run started here appears in the platform's toolbox history as that tool. .. list-table:: :header-rows: 1 :widths: 25 10 65 * - ``{tool}`` - Tool id - What it runs * - ``dynamic_single`` - 13 - Dynamic App Collection: one collection of one Dynamic Application on one device, through ``dynamic_single.py``. Its log is its output; it never has a ``result``. * - ``detailed_collection`` - 20 - Snippet Framework Detailed Collection: every step of every collection object, with its intermediate data, as the editor's step viewer shows it. * - ``collection_object_graph`` - 21 - Snippet Framework Collection Object Graph: the pipeline graph of the collection objects, rendered as Mermaid or Graphviz dot. * - ``dynamic_multi`` - 22 - Snippet Framework Dynamic App Multi Collection: a collection across several devices and applications in one run. .. important:: **A run is two calls, not one.** The run is queued for the Remote Toolbox on the collector and reaches it through a database update, so nothing is available immediately. ``POST`` returns a ``request_id`` with status **202**; ``GET`` returns the log as the collector writes it, and the result once the run has finished. Pickup normally takes tens of seconds. **Unless it runs locally.** With ``"run_local": true`` the run happens on the host this service runs on, and ``POST`` answers with the log and result directly. See :ref:`sf_tools_local`. ---- 2) Authentication & Authorization --------------------------------- Requires a valid Skylar One user session. Every endpoint requires all of these Access Hooks: - ``DYN_APP_VIEW`` - ``DEV_COLLECTIONS_VIEW`` - ``DEV_TOOLS_RUN`` The first two are what the Dynamic Application editor that runs these tools requires. ``DEV_TOOLS_RUN`` is the hook the platform puts on the Remote Toolbox, and is required because a run executes a live collection with the device's credentials. This is stricter than the platform's own toolbox, which accepts ``DEV_COLLECTIONS_VIEW`` alone for these tools. The read endpoints require the same hooks, so a user who loses them loses access to earlier runs too. Each run is audited before it is queued, under the ``Device Tools`` source the platform files its own toolbox runs under, and in the organization of the first device it runs on. A run whose jobs are all device wildcards has no device to take an organization from, so it is filed under the user's. The entry cannot carry the request id, because the id is only assigned once the run is queued. A run that cannot be audited is not queued, and the request fails with a 500. A request that is refused before its collector is resolved is not audited. A local run is audited the same way, before it runs, with an entry that says it ran locally. Ownership is enforced on top. A run is attributed to the user that queued it, and only that user can read its output, and only through the tool it was queued as. The output table is shared by every Remote Toolbox tool, so a read that cannot be shown to belong to the caller returns **404** with the same message whatever the reason. ---- 3) Queueing a run ----------------- ``POST /lce/api/sf_tools/{tool}`` .. code-block:: json { "jobs": [[12, 1730], [13, 1730]], "debug": true } ``jobs`` ``[device id, application id]`` pairs, required, at most 1000. Either half may be ``"*"`` to have the collector expand it against the devices and applications it holds — ``["*", 1730]`` is every device aligned to application 1730, ``[12, "*"]`` every application on device 12 — but not both halves of one pair. Ids are whole numbers or strings of ASCII digits. A pair given twice is sent once. ``dynamic_single`` takes exactly one explicit pair. A malformed pair refuses the whole request with **400**, where the platform's editor would quietly drop it. ``did`` and ``app_id`` A single device and application, instead of ``jobs``, as the platform's editor sends them when it has no list of pairs. Give one form or the other; both, or only one of the two fields, is refused with **400**. ``collector`` The id or name of the collector to run on. Optional, but required when any job uses a device wildcard. See :ref:`sf_tools_collectors`. Ignored when ``run_local`` is true. ``run_local`` ``true`` to run the tool on this host and answer with its output, rather than queue it. Left out, it is ``true`` on an All-In-One (``model_type`` ``1`` in ``silo.conf``), whether or not a ``collector`` is named, and ``false`` otherwise. See :ref:`sf_tools_local`. Flags (``debug``, ``disable_cache``, ``dark_mode``, ``run_local``) take ``true`` or ``false``, or ``1`` or ``0`` as the platform's editor sends them. Any other value is refused rather than read as truthy. ``debug`` Whether the tool logs at debug level. Defaults to what the editor uses: on for every tool but ``collection_object_graph``. Off unsets the debug variables rather than setting them to ``0``, because ``dynamic_single.py`` reads any value, ``0`` included, as on. ``cobj_id`` *(detailed_collection only)* A collection object id, or a list of at most 1000 of them, to restrict the run to. Each is a whole number or a string of digits, as the editor sends it. ``disable_cache`` *(detailed_collection only)* Skip cache fast-forwarding, so every step executes. ``result_format`` *(detailed_collection only)* ``web_ui`` (the default) or ``standard``. ``web_ui`` is the format the platform's editor uses: each step's data and the processed result are pretty printed for display and cut to 4,096 characters. ``standard`` reports them as the values the collection produced, untruncated, which is what a program reading the result usually wants. ``graph_type`` *(collection_object_graph only)* ``mermaid`` or ``dot``. Defaults to ``mermaid``, which is what the editor requests and renders; the CLI's own default is ``dot``. ``dark_mode`` *(collection_object_graph only)* ``true`` for the dark theme, ``false`` for the light theme. Defaults to dark, the CLI's own default. A field the tool does not accept is refused with **400** rather than ignored, so a caller learns that an option had no effect instead of assuming it did. So is a request whose command would be too long for the platform's spool table, which holds 64 KiB after encryption; the platform would otherwise truncate it silently and the run would never start. The receipt: .. code-block:: json { "mode": "remote", "request_id": 9911, "tool": "detailed_collection", "tool_id": 20, "collector": { "id": 4, "name": "cu-01", "ip": "10.64.23.11" }, "jobs": [[12, 1730]], "note": "The output is not available yet. Read it from /api/sf_tools/detailed_collection/ until terminated is true." } ``jobs`` echoes the pairs exactly as they were sent to the collector. A wildcard is sent as it is, so the receipt cannot say which devices it will expand to; the result's ``dev_app_pairs`` does. ``mode`` is ``remote`` for a queued run, and on every read back of one. ---- .. _sf_tools_collectors: 4) Which collector runs it -------------------------- A collector only holds the configuration and credentials of the devices assigned to it. A run sent to any other collector finds nothing to do and finishes with an empty result — a failure that looks like a success. So the collector is never simply taken from the caller: - **Explicit devices only.** The collector is the one the devices are assigned to. All of them must be assigned to the same collector, or the request is refused with **409**. A ``collector`` given as well has to be that collector, or the request is refused with **409** naming the collector the devices are actually on. - **Any device wildcard.** ``collector`` is required (**400** without it), since there is no device to work it out from. It also scopes the wildcard: the collector expands it against the devices it holds, so ``["*", 1730]`` on a Collector Unit is every device aligned to 1730 *on that collector*. On an All-In-One that is every device aligned to 1730. Any explicit devices in the same request must be assigned to the named collector. - A device that is not assigned to any collector is refused with **400**; one that does not exist, with **404**. An application wildcard such as ``[12, "*"]`` still names its device, so it needs no ``collector``. None of this applies to a local run, which always runs on this host, except that a device that does not exist is still refused with **404**. See :ref:`sf_tools_local`. ---- 5) Reading the log and the result --------------------------------- ``GET /lce/api/sf_tools/{tool}/{request_id}`` One endpoint reads every tool's runs, so the same query parameters apply to all four tools: .. list-table:: :header-rows: 1 :widths: 18 82 * - Parameter - Meaning * - ``after_id`` - Return only output written after this cursor. A whole number; defaults to ``0``, the beginning. Pass back the previous response's ``last_id``. * - ``page_size`` - The most output rows to return, from ``1`` to ``5000``. Defaults to ``1000``. * - ``wait_seconds`` - Hold the response for up to this many seconds, capped at ``90``, while the run produces output. The response comes back as soon as the run finishes, the page fills, or the time is up, whichever is first; running out of time returns what was collected with ``terminated`` false, which is not an error. Defaults to no wait. * - ``combine_streams`` - ``true`` to return the output as one stream in the order it was written, rather than split by stream. ``true``, ``false``, ``1`` or ``0``; defaults to ``false``. Poll the same way as the :ref:`Collector Logs API `: 1. Call with no cursor. 2. If ``terminated`` is ``false``, call again with the previous response's ``last_id`` as ``after_id``. An empty page means "not yet", not "nothing there". 3. Stop when ``terminated`` is ``true``. A wait does not hold a database connection between its reads, and it also ends as soon as a run that lost its completion marker settles. .. code-block:: json { "mode": "remote", "request_id": 9911, "tool": "detailed_collection", "terminated": false, "truncated": false, "last_id": 52212, "line_counts": { "started": 1, "stderr": 2 }, "output": { "stdout": "", "stderr": "Running a detailed collection for DID: 12 AID: 1730\nDeploying Execution Environment\n" }, "toolbox": { "phase": "running", "state": 3, "returncode": null, "seconds_since_queued": 24, "seconds_since_last_output": 1 } } The output comes back as text per stream, in ``output``: ``{"stdout": ..., "stderr": ...}``. The CLI writes its log to standard error, so that is where the run's log is. Each record is one line ending in a newline; a record without one, such as one cut to 4,096 characters with ``... [truncated]`` appended, is given one so the next page's first line never runs onto it. The toolbox's own "Process Started" and "Process Completed" rows are left out unless ``combine_streams`` is set; they are bookkeeping, and ``toolbox.phase`` and ``terminated`` carry the same news. ``line_counts`` still counts every row of the page, those included. A page ends on a line, so build the full output by appending each stream's text from each page to what you already have. With ``combine_streams=true``, ``output`` holds only ``combined``: every row of the page joined in the order the collector wrote it, standard output and standard error interleaved and the toolbox's "Process Started" and "Process Completed" rows included, which is how the platform's own toolbox viewer shows a run. The streams are not also returned separately, so a combined page is the same size as a split one, give or take the two marker rows. Blank lines are kept, as they are in the split streams. Use it to show the log as the run wrote it; split it by stream when the stream matters. .. code-block:: json { "output": { "combined": "Process Started on Collector\nRunning a detailed collection for DID: 12 AID: 1730\n" } } The tools also write their JSON result to standard output, in 5 KiB chunks. Those chunks are left out of ``output`` — they are reassembled whole into ``result`` instead. The result ^^^^^^^^^^ The response whose ``terminated`` is ``true`` also carries the result, for every tool but ``dynamic_single``: .. code-block:: json { "request_id": 9911, "tool": "collection_object_graph", "terminated": true, "result": { "success": true, "results": { "py3.11-env": { "graph": "flowchart LR ..." } }, "dev_app_pairs": [[12, 1730]] }, "extracted": { "success": true, "message": null, "dev_app_pairs": [[12, 1730]], "graphs": { "py3.11-env": "flowchart LR ..." } }, "result_error": null } A detailed collection, with the default ``web_ui`` result format, abridged: .. code-block:: json { "request_id": 9912, "tool": "detailed_collection", "terminated": true, "result": { "success": true, "results": { "12_1730": { "5531": { "result": { "result": { "did": 12, "snapshots": [["..."]], "processed_result": "{'cpu': 7}" }, "logs": ["..."], "error": "", "contract_version": 3 } }, "5532": { "result": { "result": { "did": 12, "snapshots": [["..."]], "processed_result": null }, "logs": ["..."], "error": "Step 2 failed", "contract_version": 3 } } } }, "dev_app_pairs": [[12, 1730]] }, "extracted": { "success": true, "message": null, "dev_app_pairs": [[12, 1730]], "collections": { "12_1730": { "5531": { "processed_result": "{'cpu': 7}", "error": null }, "5532": { "processed_result": null, "error": "Step 2 failed" } } } }, "result_error": null } With ``"result_format": "standard"`` the same run reports each processed result as the value itself, so the first entry of ``collections`` would read ``{"processed_result": {"cpu": 7}, "error": null}``. ``result`` The JSON exactly as the tool wrote it. Only each chunk's line terminator is removed in reassembly, never other whitespace, so a string value that straddles a chunk boundary arrives intact. ``extracted`` The parts of the result a caller usually wants, so it need not know the CLI's nesting. For ``detailed_collection`` and ``collection_object_graph``: ============== ========================================================================= Field Holds ============== ========================================================================= success Whether the jobs were scheduled and run. message Why not, when ``success`` is false, such as ``"No jobs scheduled"``. dev_app_pairs The ``[device id, application id]`` pairs the collector ran, wildcards expanded. collections *detailed_collection only.* ``{"_": {"": {"processed_result": ..., "error": ...}}}``. graphs *collection_object_graph only.* ``{"": ""}``. ============== ========================================================================= ``extracted`` is ``null`` for ``dynamic_multi``, whose result is the Snippet Framework's raw run output, and whenever ``result`` is. ``success`` only says the jobs were scheduled and run. What each collection object produced is in ``collections``, keyed ``_`` and then by collection object id, each entry ``{"processed_result": ..., "error": ...}``: the collection object's value, and its error if it reported one. The snapshots of every step and the logs, which are the bulk of a detailed result, stay only in ``result``. With ``result_format`` ``web_ui`` the processed result is the display string the editor shows, cut to 4,096 characters; with ``standard`` it is the value itself. ``graphs`` holds each Execution Environment's graph. ``result_error`` Why ``result`` is ``null``: the tool finished without writing one (its log says why, typically that no jobs were scheduled for the devices and applications requested), the result was larger than 16 MiB, it did not parse, or a chunk arrived at a length the tool could not have written. The tool cuts every chunk but the last to exactly 5 KiB, so any other length means part of the result was lost between the collector and the database. Such a result is withheld even when what remains still parses, as it will when the loss falls inside a long string. A result that is irregular or does not parse is not reported straight away. Until the run has written nothing for 90 seconds, the rest may still be on its way from the collector, so the poll keeps ``terminated`` false; only then is the error final. There is no separate request for the result. Once a run has terminated, every poll carries ``result``, ``extracted`` and ``result_error`` — including a poll from the final ``last_id``, which returns no more log — so a caller that wants the result again polls once more. ---- 6) Knowing when a run is stuck ------------------------------ ``toolbox.phase`` is ``queued``, ``running``, ``finished`` or ``unknown``, derived from the platform's spool row and the output together. There is no "stuck" phase: - A run the collector never picks up — it is offline, or far behind — stays ``queued``. - A run the Remote Toolbox abandons, as it does when it cannot decode the queued command, stays ``running`` without writing anything more. A run that finished but whose completion marker was lost — the Remote Toolbox does not retry a failed write — still terminates, but only once it has been quiet for 90 seconds, so a batch of output still on its way from the collector is not cut off. Neither of the stuck cases ever terminates, so decide from ``seconds_since_queued`` and ``seconds_since_last_output`` when to stop waiting. How long is too long depends on the tool and on the collector: a detailed collection that deploys an Execution Environment can be quiet for a while, and a Collector Unit's output reaches the database later than an All-In-One's. Neither ``phase`` nor ``returncode`` says the run succeeded. The Snippet Framework wrappers exit ``0`` whatever happened; read ``result``, ``result_error`` and the log. ---- .. _sf_tools_local: 7) Running a tool locally ------------------------- ``POST /lce/api/sf_tools/{tool}`` with ``"run_local": true`` .. code-block:: json { "jobs": [["*", 1730]], "cobj_id": 5531, "run_local": true } A local run skips the collector entirely. The tool runs on the host this service runs on, an All-In-One or a Database Server, and the request waits for it to finish. The command is the one a queued run sends, and it is run the way the collector's Remote Toolbox runs it: split with the same quote handling, line buffered with ``stdbuf``, and its output recorded line by line, so the log and the result read the same either way. No collector is resolved, so the rules in :ref:`sf_tools_collectors` do not apply: a device wildcard needs no ``collector``, and it is expanded against the configuration this host holds. A ``collector`` is accepted but ignored, since the run always reads this host. An explicit device that does not exist is still refused with **404**, as it is for a queued run. That configuration is also the run's limit. A run only finds the devices whose configuration and credentials this host holds, so on a Database Server, which collects for nothing, it finds no jobs to run. ``combine_streams`` is accepted as a query parameter on the ``POST``, as it is on the read back. The response has status **200** and holds what the read back of a finished queued run holds, all at once: the whole log, the toolbox progress and, for every tool but ``dynamic_single``, the result. There is no ``request_id``, since there is nothing to read back, and no ``collector``. .. code-block:: json { "mode": "local", "tool": "detailed_collection", "tool_id": 20, "jobs": [["*", 1730]], "terminated": true, "truncated": false, "last_id": 41, "line_counts": { "started": 1, "stderr": 39, "completed": 1 }, "output": { "stdout": "", "stderr": "..." }, "toolbox": { "phase": "finished", "state": 0, "returncode": 0, "seconds_since_queued": 12, "seconds_since_last_output": 0 }, "result": { "success": true, "results": { "...": "..." }, "dev_app_pairs": [[12, 1730]] }, "extracted": { "success": true, "message": null, "dev_app_pairs": [[12, 1730]], "collections": { "12_1730": { "5531": { "processed_result": "{'cpu': 7}", "error": null } } } }, "result_error": null } Each record is still cut to 4,096 characters as on a page of a queued run, but the response is not paged. Instead, a local run keeps at most **32 MiB** of the tool's output. The tool is still read to the end, but every line past the limit is discarded: ``truncated`` is then ``true``, the log is the part written before the limit, and for the tools that write one the result is withheld, with ``result_error`` saying why. Otherwise ``truncated`` is ``false``. ``seconds_since_queued`` is how long the run took. A result that is irregular or does not parse is reported straight away, since the run has finished and nothing more can arrive. A local run is stopped after **90** seconds, to stay inside the reverse proxy's read timeout. A run that is stopped returns **504** with ``"success": false``, ``"error": "Local Run Timed Out"``, a ``message``, and the same fields with the log it wrote before then. ``terminated`` is ``false`` and there is no result. At most **five** local runs execute at once, across every tool and the :ref:`collector log API `. One asked for while five are running is not queued: it returns **429** with ``"success": false``, ``"error": "Too Many Local Runs"`` and a ``message``, and nothing runs. Try it again once one has finished. ``POST /lce/api/run_dynamic_single_detailed`` is a thin wrapper over this: a local ``detailed_collection`` run of one device and application, for one collection object or, with ``cobj_id`` left out, every one of them, answered in the shape it has always used, ``{"cli_logs": ..., "json_output": ...}``, with the log's standard error and the result as the CLI wrote it. An invalid body returns **400**, a run refused for the limit above **429**, and a run that times out **500**.