lce_service.collector_log_api package

Collector Log Retrieval Module.

This module provides the functionality needed to list and read Data Collector log files remotely, without shell access to the collector appliance.

A read is queued for the Remote Toolbox on the collector that holds the logs and its output is read back from the platform’s log tables. That makes every read two steps rather than one: the queued command has to reach the collector through a database update before any output exists. Nothing is installed on the collector to support this, so each command is composed from binaries already present in the appliance image, and all filtering runs there rather than here.

exception lce_service.collector_log_api.CollectorLogJobNotAccessible(proc_id: int)

Bases: ToolboxJobNotAccessible

Exception raised when a collector log job cannot be read by the requesting user.

This error is deliberately raised with the same message for every reason a job is not readable: the job id was never issued, the job belongs to a different Remote Toolbox tool, the job was started by another user, or the job has no recorded owner at all. Distinguishing those cases would turn the read endpoint into an oracle for which job ids exist, and the rows it guards are shared with every other toolbox tool.

A job with no recorded owner is unreadable by everyone, including whoever started it. That is the intended outcome rather than a gap: the ownership row is what authorises a read, so a dispatch that could not attribute itself must not leave readable output behind.

exception lce_service.collector_log_api.InvalidLogRequestError(message: str)

Bases: InvalidToolboxRequestError

Exception raised when a collector log request cannot be turned into a safe command.

This error is raised when a caller supplied value fails validation, whether because it names a log directory that is not retrievable, because a log name does not have the shape logrotate produces, or because a pattern or timestamp bound contains a character that could introduce an additional argument into the queued command.

Parameters:

message – A descriptive error message explaining which value was rejected and what shape was expected, suitable for returning to the caller.

lce_service.collector_log_api.assert_log_job_is_readable(dbc: silo_cursor, proc_id: int, user: User) → Dict[str, Any]

Verifies that a process id names a collector log command this user may read.

The job has to have been queued as the collector log retrieval tool, and its ownership row for that tool has to name the requesting user. Every rejection raises the same error with the same message, so the endpoint cannot be used to learn which process ids exist.

Parameters:

• dbc – A silo_cursor instance used to read the spool and ownership rows.

• proc_id – The process id returned when the command was queued.

• user – The User instance requesting the output.

Returns:

The job’s spool row.

Raises:

• CollectorLogJobNotAccessible – Raised when no such job exists, when the job belongs to a different Remote Toolbox tool, when the job has no recorded owner, or when the owner is another user.

• DatabaseExecutionError – Raised when either check fails to execute.

lce_service.collector_log_api.build_log_request(log_name: str | None, directory: str | None = None, lines: Any = None, pattern: str | None = None, since: str | None = None, until: str | None = None, max_bytes: Any = None) → CollectorLogRequest

Validates a caller’s log read parameters and returns a request safe to run.

This is the only supported way to build a CollectorLogRequest. Every caller supplied value is validated here, which keeps the validation rules in one place rather than spread across the endpoints that accept them.

Parameters:

• log_name – Name of the log file to read, within the chosen directory.

• directory – Key of the log directory to read from, or None for the default.

• lines – Number of trailing lines to return, or None for no line bound.

• pattern – An extended regular expression to filter records by, or None.

• since – Inclusive lower bound on the record timestamp, or None.

• until – Inclusive upper bound on the record timestamp, or None.

• max_bytes – Ceiling on the bytes the read may return, or None for MAX_OUTPUT_BYTES.

Returns:

A validated CollectorLogRequest.

Raises:

InvalidLogRequestError – Raised when any supplied value fails validation. The message names the value that was rejected and the shape expected.

lce_service.collector_log_api.build_request_note(request: CollectorLogRequest) → str | None

Describes a limitation of a validated request, for the caller to be told about.

A timestamp bound is compared against the first whitespace separated field of a record, which is the whole timestamp only in the rsyslog written directory. Records in the em7 directory separate the date from the time with a space, so the comparison only ever sees the date and a bound carrying a time component matches nothing at all. A whole day bound works there, so such a request is answered rather than refused, and this is what tells the caller why a narrower bound would come back empty.

Parameters:

request – A validated CollectorLogRequest describing what will be read.

Returns:

A note to include in the dispatch response, or None when there is nothing to add.

lce_service.collector_log_api.dispatch_log_inventory(dbc: silo_cursor, appliance: Dict[str, Any], directory: LogDirectory, user: User) → int

Queues a command that enumerates the log files available on a collector.

Parameters:

• dbc – A silo_cursor instance used to queue the command and record its ownership.

• appliance – The collecting appliance to enumerate logs on, as returned by get_collecting_appliance.

• directory – The log directory to enumerate, already resolved by validate_log_directory.

• user – The User instance the queued command is attributed to.

Returns:

The process id to poll for the inventory.

Raises:

DatabaseExecutionError – Raised when the command cannot be queued or attributed.

lce_service.collector_log_api.dispatch_log_read(dbc: silo_cursor, appliance: Dict[str, Any], request: CollectorLogRequest, user: User) → int

Queues a command that reads one log file on a collector.

Parameters:

• dbc – A silo_cursor instance used to queue the command and record its ownership.

• appliance – The collecting appliance to read from, as returned by get_collecting_appliance.

• request – A validated CollectorLogRequest describing what to read.

• user – The User instance the queued command is attributed to.

Returns:

The process id to poll for the log content.

Raises:

DatabaseExecutionError – Raised when the command cannot be queued or attributed.

lce_service.collector_log_api.get_collecting_appliance(dbc: silo_cursor, appliance_identifier: Any) → Dict[str, Any]

Looks up a collecting appliance by numeric id or by name.

Only appliances that actually collect are returned, because a log read has to be queued against the appliance that holds the logs. The lookup itself is shared with every Remote Toolbox API; this wrapper only reports its refusals in collector log terms.

Parameters:

• dbc – A silo_cursor instance used to query the appliance table.

• appliance_identifier – The numeric appliance id, or the appliance name.

Returns:

A dictionary holding the appliance id, name, ip and function.

Raises:

• InvalidLogRequestError – Raised when the identifier is a value that is neither an id nor a name.

• MissingResourceError – Raised when no collecting appliance matches the identifier.

• DatabaseExecutionError – Raised when the appliance lookup fails to execute.

lce_service.collector_log_api.get_collecting_appliances(dbc: silo_cursor) → List[Dict[str, Any]]

Lists every appliance that collects, and therefore runs Remote Toolbox commands.

A Database Server is excluded even though it is an appliance, because it does not collect and a command queued against one would never run.

Parameters:

dbc – A silo_cursor instance used to query the appliance table.

Returns:

One dictionary per collecting appliance, holding its id, name, ip and function, ordered by name.

Raises:

DatabaseExecutionError – Raised when the appliance lookup fails to execute.

lce_service.collector_log_api.parse_log_inventory(page: OutputPage, directory: LogDirectory) → List[Dict[str, Any]]

Turns the output of a log inventory command into one entry per log file.

The collector runs find(1) with a format of file name, size in bytes, and modification time as an epoch, separated by tabs. Those fields are parsed here rather than by each caller, both so the format lives next to the command that produced it and so the rotation details are derived the same way everywhere.

The whole standard output stream is reassembled before being split into records, instead of treating each output row as one record. That join is defensive rather than required: the collector side toolbox reads a whole line before it writes a row and the batching it applies is to the insert and not to the row, so on this transport a page boundary is always a record boundary. Joining first costs nothing and keeps the parse correct if that ever changes.

Anything that does not parse is skipped rather than raised. A log inventory is a diagnostic aid, and returning the files that were readable is more useful than failing the whole listing because one entry was malformed or a page ended mid record.

Parameters:

• page – A LogOutputPage from poll_log_output holding the inventory command’s output. Only the standard output rows are read; the started and completed markers and anything the command wrote to standard error are ignored.

• directory – The log directory the inventory was taken from, recorded on each entry so the result is self describing.

Returns:

One dictionary per log file, sorted by name, each holding:

• name - the file name within the directory

• base_name - the log it belongs to, with any rotation suffix removed

• directory - the key of the directory it was found in, as a caller names it

• directory_path - the absolute path of that directory on the collector

• size_bytes - size in bytes

• modified - modification time as an ISO 8601 timestamp in UTC

• rotated - whether this is a rotated generation rather than the current log

• compressed - whether it is gzip compressed

• generation - the rotation suffix, or None for the current log

lce_service.collector_log_api.poll_log_output(connect: Callable[[], ContextManager[silo_cursor]], proc_id: int, user: User, after_id: int = 0, page_size: int | None = None, wait_seconds: float | None = None) → Tuple[OutputPage, JobProgress]

Serves one poll of a collector log command: checks access, reads a page and the progress.

Access is checked before anything is read. A poll with no wait uses a single connection; a wait opens one per read, as poll_job_output describes.

Parameters:

• connect – Opens a database connection for one read, such as db_manager.

• proc_id – The process id returned when the command was queued.

• user – The User instance requesting the output.

• after_id – Return only rows with an id greater than this.

• page_size – The maximum number of rows to return, bounded by MAX_OUTPUT_PAGE_SIZE.

• wait_seconds – How long to wait for the collector to finish, bounded by MAX_WAIT_SECONDS. Defaults to no wait.

Returns:

The page and the job’s progress, read after it.

Raises:

• CollectorLogJobNotAccessible – Raised when the user may not read the job.

• DatabaseExecutionError – Raised when anything cannot be read.

lce_service.collector_log_api.run_log_inventory_locally(directory: LogDirectory) → LocalRun

Enumerates the log files on this host with the command dispatch_log_inventory queues.

Parameters:

directory – The log directory to enumerate, already resolved by validate_log_directory.

Returns:

The LocalRun of the inventory command.

Raises:

InvalidToolboxRequestError – Raised when the command cannot be run as given.

lce_service.collector_log_api.run_log_read_locally(request: CollectorLogRequest) → LocalRun

Reads one log file on this host with the command dispatch_log_read queues.

Parameters:

request – A validated CollectorLogRequest describing what to read.

Returns:

The LocalRun of the read command.

Raises:

InvalidToolboxRequestError – Raised when the command cannot be run as given.

lce_service.collector_log_api.validate_log_directory(directory: str | None) → LogDirectory

Resolves a caller supplied log directory key to the directory it names.

The caller never supplies a path. It supplies a key from ALLOWED_LOG_DIRECTORIES, which this function maps to an absolute directory, so a path traversal cannot be expressed in the first place.

Parameters:

directory – The directory key supplied by the caller, or None to default to the rsyslog written directory that holds the current service logs.

Returns:

A LogDirectory holding both the key the caller named and the absolute path it resolves to.

Raises:

InvalidLogRequestError – Raised when the key does not name a retrievable directory.

Submodules

• lce_service.collector_log_api.core module
  • CollectorLogJobNotAccessible
  • CollectorLogRequest
    • CollectorLogRequest.directory
    • CollectorLogRequest.lines
    • CollectorLogRequest.log_name
    • CollectorLogRequest.max_bytes
    • CollectorLogRequest.pattern
    • CollectorLogRequest.since
    • CollectorLogRequest.until
  • InvalidLogRequestError
  • LogDirectory
    • LogDirectory.key
    • LogDirectory.path
  • build_list_command()
  • build_log_request()
  • build_read_command()
  • build_request_note()
  • escape_as_single_token()
  • get_collecting_appliance()
  • get_collecting_appliances()
  • validate_line_count()
  • validate_log_directory()
  • validate_log_name()
  • validate_log_pattern()
  • validate_max_bytes()
  • validate_timestamp_bound()
• lce_service.collector_log_api.dispatch_logs module
  • dispatch_log_inventory()
  • dispatch_log_read()
  • run_log_inventory_locally()
  • run_log_read_locally()
• lce_service.collector_log_api.retrieve_logs module
  • LogOutputPage
  • assert_log_job_is_readable()
  • parse_log_inventory()
  • poll_log_output()