lce_service.remote_toolbox.retrieve module

class lce_service.remote_toolbox.retrieve.FramedPayload(text: str | None, chunk_count: int, irregular_chunks: List[int], error: str | None)

Bases: NamedTuple

A payload a command wrote to standard output as a series of marked chunks.

Variables:

text – The chunks joined back together, without their markers or line terminators, or None when no chunk was found or the payload could not be read.
chunk_count – How many chunks were found.
irregular_chunks – The 1-based positions of chunks whose length the writer could not have produced: any but the last that is not exactly the chunk size, or a last one longer than it. Any listed here means part of the payload was lost or mangled in transit, whether or not the text that remains still parses.
error – A description of why the payload could not be read, or None.

chunk_count: int: Alias for field number 1

error: str | None: Alias for field number 3

irregular_chunks: List[int]: Alias for field number 2

text: str | None: Alias for field number 0

Bases: NamedTuple

Where a queued Remote Toolbox command is in its life, and how long it has been there.

The phase is derived from the spool row and the output rows together, because neither is reliable on its own. The toolbox swallows a failure to write any of its rows, so a job can finish without its completed row ever being written, and the spool row and the output rows are pruned independently, so either can outlive the other.

No phase means “stuck”. A job the collector never picks up stays queued, and a job the toolbox abandons, as it does when it cannot decode the queued command, stays running, both indefinitely. The two ages are reported so the caller can decide when to give up; how long is too long depends on the tool and on how far the collector is from the database.

Variables:

phase – One of “queued”, “running”, “finished” or “unknown”. Unknown means there is no spool row to read and no output to infer anything from.
state – The spool row’s state as found, or None when there is no spool row.
returncode – The spool row’s return code as found, or None. It is the exit status of the command the toolbox ran, which for some tools is a wrapper that exits 0 whatever happened, so it is reported rather than interpreted.
seconds_since_queued – Seconds since the command was queued, or None.
seconds_since_last_output – Seconds since the job last wrote any output row, or None when it has written none. Every row counts, including any the caller’s reads leave out, since those are output too.
last_output_id – The id of the job’s last output row, or None when it has written none. Used to tell whether a reader has caught up with a finished job.
completed – Whether the job’s last output row is the completed row the toolbox writes after everything else.

as_dict() → Dict[str, Any]

Returns the progress as it is reported to a caller, without the internal cursor.

Returns:: A dictionary of the phase, the raw spool fields and the two ages.

completed: bool: Alias for field number 6

last_output_id: int | None: Alias for field number 5

phase: str: Alias for field number 0

returncode: int | None: Alias for field number 2

seconds_since_last_output: int | None: Alias for field number 4

seconds_since_queued: int | None: Alias for field number 3

state: int | None: Alias for field number 1

class lce_service.remote_toolbox.retrieve.OutputPage(lines: List[Dict[str, Any]], last_id: int, terminated: bool, truncated: bool, line_counts: Dict[str, int], chars: int = 0)

Bases: NamedTuple

One page of output from a queued Remote Toolbox command.

Variables:

lines – The output rows in the order the collector emitted them, each a dictionary holding the row id, the stream it came from and the decoded text.
last_id – The highest row id in this page, to be passed back as the cursor for the next page. Unchanged from the requested cursor when the page is empty.
terminated – True when the collector has finished running the command. This reports termination only and says nothing about whether the command succeeded.
truncated – True when this page did not exhaust the output that was available: more rows were waiting than the row budget allowed, or the records accepted reached the per response character cap with rows still left behind, or both. The caller should request the next page from last_id rather than assume the output ended.
line_counts – A count of rows per stream in this page.
chars – The characters the rows take in the JSON response, which a wait carries on from rather than encoding the rows again.

chars: int: Alias for field number 5

last_id: int: Alias for field number 1

line_counts: Dict[str, int]: Alias for field number 4

lines: List[Dict[str, Any]]: Alias for field number 0

terminated: bool: Alias for field number 2

truncated: bool: Alias for field number 3

lce_service.remote_toolbox.retrieve.assemble_framed_payload(messages: List[Any], marker: str, chunk_size: int, max_chars: int, max_rows: int | None = None) → FramedPayload

Joins the chunks of a framed payload back together, by the rules read_framed_payload sets.

This is the part of reassembly that does not touch the database, so a payload a command wrote locally is reassembled exactly as one read from the output table is.

Parameters:

messages – The raw messages of the chunk rows, in id order, each still behind its marker. Only the first max_rows + 1 are ever needed.
marker – The marker that starts every chunk.
chunk_size – The length, in characters, the writer cuts every chunk but the last to.
max_chars – The most characters of payload to reassemble.
max_rows – The most chunks max_chars allows, or None to derive it from max_chars.

Returns:

A FramedPayload. When no chunk exists, its text and error are both None.

lce_service.remote_toolbox.retrieve.assert_job_readable(dbc: ~sl_mysql.silo_cursor.silo_cursor, proc_id: int, user: ~lce_service.sl1_models.User, tool_ids: ~typing.Set[int], not_accessible: ~typing.Type[~lce_service.remote_toolbox.core.ToolboxJobNotAccessible] = <class 'lce_service.remote_toolbox.core.ToolboxJobNotAccessible'>) → Dict[str, Any]

Verifies that a process id names a Remote Toolbox job of these tools that this user owns.

Two independent checks run here, in this order. The first confirms the job was queued as one of the tools the caller serves rather than some other Remote Toolbox tool, because the output table is shared by every tool and a rejected payload is echoed into it verbatim; without this check a caller walking process ids could read output belonging to a tool it was never granted. The second confirms the job belongs to the requesting user. The cheaper and broader check runs first so a mismatch is rejected before a second query is spent on the narrower one. The ownership row is matched on the job’s own tool, because every toolbox tool writes to that table, so an ownership row recorded for one tool can never authorise another.

Every rejection raises the same error with the same message, so the endpoint cannot be used to learn which process ids exist.

Parameters:

dbc – A silo_cursor instance used to read the spool and ownership rows.
proc_id – The process id returned when the command was queued.
user – The User instance requesting the output.
tool_ids – The Remote Toolbox tool ids the caller is allowed to read.
not_accessible – The ToolboxJobNotAccessible subclass to raise, so each API reports a refusal in its own terms.

Returns:

The job’s spool row, as get_job_status returns it, so the caller does not have to read it again.

Raises:

ToolboxJobNotAccessible – Raised, as the not_accessible type, when no such job exists, when the job belongs to a different Remote Toolbox tool, when the job has no recorded owner, or when the owner is another user.
DatabaseExecutionError – Raised when either check fails to execute.

lce_service.remote_toolbox.retrieve.build_output_page(rows: List[Dict[str, Any]], after_id: int, truncated: bool, max_chars: int | None = 1048576) → OutputPage

Turns output rows, in id order, into a page, decoding each and bounding the whole.

This is the part of a read that does not touch the database, so output that never reached the output table, such as a command run locally, is paged by exactly the same rules.

Parameters:

rows – The output rows, each holding an id, a message type and a raw message, already limited to the row budget.
after_id – The cursor the rows were read after, returned unchanged for no rows.
truncated – Whether the row budget left rows behind.
max_chars – Stop once the records accepted reach this many encoded characters, or None to accept every row.

Returns:

An OutputPage of the rows, as get_output_page describes it.

lce_service.remote_toolbox.retrieve.decode_output_line(message: Any) → str

Decodes one output row’s message for display, bounding its length.

One record longer than MAX_RECORD_CHARS is cut to that length and marked. This bounds a single pathological record, which is a different problem from a large response: a Python traceback arrives as many records and is bounded by the per response cap instead. Rows that carry a framed payload must never be read through this, since cutting a chunk corrupts the payload; they are read with read_framed_payload.

Parameters:: message – The raw message value read from the output table.
Returns:: The message as text, with undecodable bytes replaced and RECORD_TRUNCATION_SUFFIX appended when the record was longer than MAX_RECORD_CHARS.

lce_service.remote_toolbox.retrieve.decode_output_text(message: Any) → str

Decodes one output row’s message into text without ever raising, and without truncating.

Collector output is stored as a binary column and carries whatever the command wrote. A decode failure is not a reason to fail the caller’s read, so undecodable bytes are replaced instead.

Parameters:: message – The raw message value read from the output table.
Returns:: The message as text, with undecodable bytes replaced.

lce_service.remote_toolbox.retrieve.get_job_progress(dbc: silo_cursor, proc_id: int, job: Dict[str, Any] | None = None, exclude_prefix: str | None = None) → JobProgress

Works out where a queued Remote Toolbox command is, from its spool row and its output.

A job is finished when its last output row is the completed row, when its spool row holds a return code, or when its spool row is in a terminal state. Any one signal is enough, because the toolbox writes each of them separately and swallows a failure of any. The return code is the most dependable of the spool row’s two: it is only ever written when the command ends, whereas the state is written by more than one process and the updates can land out of order. A job is running once its spool row says it was launched or is running, or once it has written any output, and queued while its spool row still holds the state it was queued with and nothing has been written.

The last output row is read by id rather than by time, which is both what the index serves and what makes it the row the toolbox wrote last.

Parameters:

dbc – A silo_cursor instance used to read the spool and output rows.
proc_id – The process id returned when the command was queued.
job – The spool row, when the caller has already read it with get_job_status or assert_job_readable, so it is not read twice.
exclude_prefix – Leave out the same rows the caller’s output reads leave out, so the last output id is one the caller’s cursor can actually reach.

Returns:

A JobProgress describing the phase and how long the job has been in it.

Raises:

DatabaseExecutionError – Raised when the spool row or the output rows cannot be read.

lce_service.remote_toolbox.retrieve.get_job_status(dbc: silo_cursor, proc_id: int) → Dict[str, Any]

Reads the platform’s spool row for a queued Remote Toolbox command.

The row is queued with a state of 1, which the toolbox moves to 3 when it starts and to 0 or 4 when the command ends, recording the command’s exit status alongside. How long ago the command was queued is computed by the database, because the spool row’s queue time is written in UTC while the platform’s other timestamps are local.

Parameters:

dbc – A silo_cursor instance used to read the spool row.
proc_id – The process id returned when the command was queued.

Returns:

A dictionary holding the state, return code, process, tool type and the seconds since the command was queued, or an empty dictionary when no spool row exists. The spool row and the output rows are pruned independently, so output can outlive the row that describes it.

Raises:

DatabaseExecutionError – Raised when the spool row cannot be read.

lce_service.remote_toolbox.retrieve.get_output_page(dbc: silo_cursor, proc_id: int, after_id: int = 0, page_size: int | None = None, exclude_prefix: str | None = None) → OutputPage

Reads one page of output from a queued Remote Toolbox command.

Rows are ordered by their own id, which is the order the collector emitted them. The command’s standard output and standard error arrive interleaved within the same id sequence, so reading them as one ordered stream is the only way to preserve the order in which they were produced.

Completion is detected by the presence of a completed row in the stream. That row is written whether the command succeeded, failed, or was refused by the Remote Toolbox before it ran, so a terminated page is not a successful one. A completed row can also go missing, since the toolbox does not retry a write that fails, which is what is_output_complete covers.

An empty page with termination not yet reported means the collector has not produced more output, not that the command produced none. The command has to reach the collector through a database update before anything is written at all.

A page is bounded in size as well as in rows. Accumulation stops once the records accepted reach MAX_RESPONSE_CHARS, and the cursor stops with them, so the rows the cap left behind are returned by the next call rather than skipped.

Parameters:

dbc – A silo_cursor instance used to read the output rows.
proc_id – The process id returned when the command was queued.
after_id – Return only rows with an id greater than this. Pass the previous page’s last_id to continue where it stopped, or zero to read from the beginning.
page_size – The maximum number of rows to return, bounded by MAX_OUTPUT_PAGE_SIZE. Defaults to DEFAULT_OUTPUT_PAGE_SIZE.
exclude_prefix – Leave out standard output rows starting with this marker, for a command that writes a framed payload alongside its log. Those rows are read whole with read_framed_payload instead, so they neither spend the page budget nor get cut by the per record cap here.

Returns:

An OutputPage holding the rows, the cursor to resume from, whether the collector has finished, and whether the page stopped short of the available output because the row budget or the character cap was reached.

Raises:

DatabaseExecutionError – Raised when the output rows cannot be read.

lce_service.remote_toolbox.retrieve.has_been_quiet(progress: JobProgress) → bool

Decides whether a job has written nothing for SETTLE_SECONDS.

A job that has written nothing at all is measured by its queue age instead.

Parameters:: progress – The job’s progress.
Returns:: True when the job has been quiet long enough for a batch in transit to have landed, or when there is nothing to measure from.

lce_service.remote_toolbox.retrieve.is_job_settled(progress: JobProgress) → bool

Decides whether a job is finished and every row it will write has arrived.

A job whose last row is the completed row is settled, since that row is written after all the others. A job that finished by its spool row alone, its completed row lost, is settled once it has written nothing for SETTLE_SECONDS, which gives a batch still in transit from the collector time to land. One that wrote nothing at all is settled by its queue age instead.

Parameters:: progress – The job’s progress.
Returns:: True when the job’s output can be treated as complete.

lce_service.remote_toolbox.retrieve.is_output_complete(page: OutputPage, progress: JobProgress) → bool

Decides whether a reader has seen every row a finished job will ever write.

A page that saw the completed row is complete, which is the signal the platform’s own viewer relies on. That row can be missing, though, since the toolbox does not retry a failed write, and a reader waiting for it would poll a finished job forever. So a settled job, as is_job_settled defines it, is also complete once the page holds everything that was waiting and its cursor has reached the job’s last row. Progress has to be read after the page for the second test to hold, since a row written between the two reads would otherwise be skipped.

Parameters:

page – The page just returned to the reader.
progress – The job’s progress, read after the page.

Returns:

True when the reader can stop polling.

lce_service.remote_toolbox.retrieve.is_stdout_framed(row: Dict[str, Any], prefix: str) → bool

Decides whether an output row is one _exclusion_clause leaves out, without the database.

The comparison is made on the encoded message, as LEFT makes it on the binary column.

Parameters:

row – An output row holding a message type and a raw message.
prefix – The marker that starts every row to leave out.

Returns:

True when the row is standard output starting with the marker.

lce_service.remote_toolbox.retrieve.poll_job_output(connect: Callable[[], ContextManager[silo_cursor]], proc_id: int, authorize: Callable[[silo_cursor], Dict[str, Any] | None], after_id: int = 0, page_size: int | None = None, wait_seconds: float | None = None, exclude_prefix: str | None = None) → Tuple[OutputPage, JobProgress]

Serves one poll of a job: checks access, reads a page, and reads the job’s progress.

A poll with no wait, which is how a caller should normally poll, does all three on one connection, the way one request of the platform’s own viewer does. Only a wait opens more: each read of the wait gets its own connection, closed before the sleep, and progress is read on a last one once the wait ends.

Progress is always read after the page, never before, because is_output_complete relies on it: a row written between the two reads would otherwise be skipped by a reader told it had caught up. The one exception is the spool row authorize returns, which is reused on the first connection rather than read again.

Parameters:

connect – Opens a database connection for one read, such as db_manager.
proc_id – The process id returned when the command was queued.
authorize – Checks that the caller may read the job, given the first connection, and raises if not. Nothing is read before it returns. It returns the job’s spool row if it read one, or None to have it read here.
after_id – Return only rows with an id greater than this.
page_size – The maximum number of rows to return, bounded by MAX_OUTPUT_PAGE_SIZE.
wait_seconds – How long to wait for the collector to finish, bounded by MAX_WAIT_SECONDS. Defaults to no wait.
exclude_prefix – Leave out standard output rows starting with this marker, from the page and from the progress alike.

Returns:

The page and the progress read after it.

Raises:

ToolboxJobNotAccessible – Raised by authorize when the caller may not read the job.
DatabaseExecutionError – Raised when anything cannot be read.

lce_service.remote_toolbox.retrieve.read_framed_payload(dbc: silo_cursor, proc_id: int, marker: str, chunk_size: int, max_chars: int) → FramedPayload

Reassembles a payload a command wrote to standard output as a series of marked chunks.

Some commands write a structured result as fixed length chunks, each on its own line behind a marker, interleaved with their log. Reassembly has three rules the platform’s own viewer breaks. A chunk is cut at an arbitrary character, so only its line terminator is removed and never any other whitespace, which may belong to a string inside the payload. A chunk is longer than a display record, so it is decoded whole rather than through the per record cap. And the whole payload is bounded once, by max_chars, rather than per page.

The terminator removed is a newline, preceded by a carriage return if there is one. A payload whose own text could end a chunk with either would need a different framing; a JSON payload cannot, since JSON escapes both inside strings and the writer emits it on one line.

Reading a payload whole is only meaningful once the job has finished, since a running job may not have written every chunk yet. Callers check progress first.

Parameters:

dbc – A silo_cursor instance used to read the output rows.
proc_id – The process id returned when the command was queued.
marker – The marker that starts every chunk.
chunk_size – The length, in characters, the writer cuts every chunk but the last to. It is used only to report chunks that arrived at a length the writer could not have produced.
max_chars – The most characters of payload to reassemble.

Returns:

A FramedPayload. When no chunk exists, its text and error are both None.

Raises:

DatabaseExecutionError – Raised when the output rows cannot be read.

lce_service.remote_toolbox.retrieve.stitch_output(page: OutputPage, combined: bool = False) → Dict[str, Any]

Joins a page’s rows back into the text the command wrote to each stream.

The toolbox reads the command’s standard output and standard error through separate pipes and stores each line it reads as a row, so the rows of one stream, in id order, join back into that stream’s text.

Each row is one line. A row that does not end in a newline, as a record cut by the per record cap or the last line of a command that ended without one does not, gets one, so that the next page’s first line cannot run onto it. The toolbox’s own marker rows are left out. A row of a stream the toolbox does not normally write is kept under its own name.

A page ends on a row boundary, so a caller that polls page after page appends each page’s text to what it already has.

Combined, every row of the page is joined into one text in the order the collector wrote it, the marker rows included, which is how the platform’s own toolbox viewer shows a job’s output. The streams are then not returned separately, so a combined page is no larger than a split one.

Parameters:

page – A page of output.
combined – Join every row into one text rather than splitting it by stream.

Returns:

A dictionary holding “output”, a dictionary of each stream’s text, which always has stdout and stderr, or, when combined, holding only “combined”.