lce_service.collector_log_api.retrieve_logs module

lce_service.collector_log_api.retrieve_logs.LogOutputPage

alias of OutputPage

lce_service.collector_log_api.retrieve_logs.assert_log_job_is_readable(dbc: silo_cursor, proc_id: int, user: User) → Dict[str, Any]

Verifies that a process id names a collector log command this user may read.

The job has to have been queued as the collector log retrieval tool, and its ownership row for that tool has to name the requesting user. Every rejection raises the same error with the same message, so the endpoint cannot be used to learn which process ids exist.

Parameters:

  • dbc – A silo_cursor instance used to read the spool and ownership rows.

  • proc_id – The process id returned when the command was queued.

  • user – The User instance requesting the output.

Returns:

The job’s spool row.

Raises:

  • CollectorLogJobNotAccessible – Raised when no such job exists, when the job belongs to a different Remote Toolbox tool, when the job has no recorded owner, or when the owner is another user.

  • DatabaseExecutionError – Raised when either check fails to execute.

lce_service.collector_log_api.retrieve_logs.parse_log_inventory(page: OutputPage, directory: LogDirectory) → List[Dict[str, Any]]

Turns the output of a log inventory command into one entry per log file.

The collector runs find(1) with a format of file name, size in bytes, and modification time as an epoch, separated by tabs. Those fields are parsed here rather than by each caller, both so the format lives next to the command that produced it and so the rotation details are derived the same way everywhere.

The whole standard output stream is reassembled before being split into records, instead of treating each output row as one record. That join is defensive rather than required: the collector side toolbox reads a whole line before it writes a row and the batching it applies is to the insert and not to the row, so on this transport a page boundary is always a record boundary. Joining first costs nothing and keeps the parse correct if that ever changes.

Anything that does not parse is skipped rather than raised. A log inventory is a diagnostic aid, and returning the files that were readable is more useful than failing the whole listing because one entry was malformed or a page ended mid record.

Parameters:

  • page – A LogOutputPage from poll_log_output holding the inventory command’s output. Only the standard output rows are read; the started and completed markers and anything the command wrote to standard error are ignored.

  • directory – The log directory the inventory was taken from, recorded on each entry so the result is self describing.

Returns:

One dictionary per log file, sorted by name, each holding:

  • name - the file name within the directory

  • base_name - the log it belongs to, with any rotation suffix removed

  • directory - the key of the directory it was found in, as a caller names it

  • directory_path - the absolute path of that directory on the collector

  • size_bytes - size in bytes

  • modified - modification time as an ISO 8601 timestamp in UTC

  • rotated - whether this is a rotated generation rather than the current log

  • compressed - whether it is gzip compressed

  • generation - the rotation suffix, or None for the current log

lce_service.collector_log_api.retrieve_logs.poll_log_output(connect: Callable[[], ContextManager[silo_cursor]], proc_id: int, user: User, after_id: int = 0, page_size: int | None = None, wait_seconds: float | None = None) → Tuple[OutputPage, JobProgress]

Serves one poll of a collector log command: checks access, reads a page and the progress.

Access is checked before anything is read. A poll with no wait uses a single connection; a wait opens one per read, as poll_job_output describes.

Parameters:

  • connect – Opens a database connection for one read, such as db_manager.

  • proc_id – The process id returned when the command was queued.

  • user – The User instance requesting the output.

  • after_id – Return only rows with an id greater than this.

  • page_size – The maximum number of rows to return, bounded by MAX_OUTPUT_PAGE_SIZE.

  • wait_seconds – How long to wait for the collector to finish, bounded by MAX_WAIT_SECONDS. Defaults to no wait.

Returns:

The page and the job’s progress, read after it.

Raises: