lce_service.collector_log_api.core module

exception lce_service.collector_log_api.core.CollectorLogJobNotAccessible(proc_id: int)

Bases: ToolboxJobNotAccessible

Exception raised when a collector log job cannot be read by the requesting user.

This error is deliberately raised with the same message for every reason a job is not readable: the job id was never issued, the job belongs to a different Remote Toolbox tool, the job was started by another user, or the job has no recorded owner at all. Distinguishing those cases would turn the read endpoint into an oracle for which job ids exist, and the rows it guards are shared with every other toolbox tool.

A job with no recorded owner is unreadable by everyone, including whoever started it. That is the intended outcome rather than a gap: the ownership row is what authorises a read, so a dispatch that could not attribute itself must not leave readable output behind.

class lce_service.collector_log_api.core.CollectorLogRequest(directory: LogDirectory, log_name: str, lines: int | None, pattern: str | None, since: str | None, until: str | None, max_bytes: int)

Bases: NamedTuple

A validated description of one collector log read.

Every field has already been through the validators in this module, so a request instance is safe to turn into a command string. Construct one with build_log_request rather than directly, so that no unvalidated value can reach a command.

Variables:

• directory – The log directory to read from, resolved from ALLOWED_LOG_DIRECTORIES rather than taken from the caller.

• log_name – Name of the log file within that directory, without any path separator.

• lines – Number of trailing lines to return, or None to apply MAX_LINE_COUNT.

• pattern – An extended regular expression to filter records by, or None for no filter.

• since – Inclusive lower bound compared against the record timestamp, or None.

• until – Inclusive upper bound compared against the leading portion of the record timestamp that the bound itself spells out, or None. A day bound therefore includes the whole of that day.

• max_bytes – Ceiling on the bytes the read may return, never above MAX_OUTPUT_BYTES. It is always set, so a read is bounded by bytes whether or not one was asked for, and the bytes kept are the last ones.

A request is always satisfied by a single command run on the collector. Filtering happens there rather than in this service because the output crosses back through a shared platform log table in chunked rows, and an active collector log reaches tens of megabytes. That is also why both caps are unconditional: unbounded output would be written to the platform before this service ever saw it.

directory: LogDirectory

Alias for field number 0

lines: int | None

Alias for field number 2

log_name: str

Alias for field number 1

max_bytes: int

Alias for field number 6

pattern: str | None

Alias for field number 3

since: str | None

Alias for field number 4

until: str | None

Alias for field number 5

exception lce_service.collector_log_api.core.InvalidLogRequestError(message: str)

Bases: InvalidToolboxRequestError

Exception raised when a collector log request cannot be turned into a safe command.

This error is raised when a caller supplied value fails validation, whether because it names a log directory that is not retrievable, because a log name does not have the shape logrotate produces, or because a pattern or timestamp bound contains a character that could introduce an additional argument into the queued command.

Parameters:

message – A descriptive error message explaining which value was rejected and what shape was expected, suitable for returning to the caller.

class lce_service.collector_log_api.core.LogDirectory(key: str, path: str)

Bases: NamedTuple

A log directory, as the caller names it and as it exists on the collector.

Both halves are carried together because each is needed in a different place: a response echoes the key so a caller can send it straight back in the next request, while a command needs the absolute path. Returning only the path would make a response that cannot be fed back in, and returning only the key would put the mapping in every caller.

Variables:

• key – The short name a caller uses to select this directory, such as “sl1”.

• path – The absolute path of the directory on the collector.

key: str

Alias for field number 0

path: str

Alias for field number 1

lce_service.collector_log_api.core.build_list_command(directory: str) → str

Builds the command that enumerates the log files in a directory.

Parameters:

directory – The absolute log directory path, already resolved by validate_log_directory.

Returns:

The command to queue, without the SILOPROC marker the caller prepends. Each output row holds a file name, a size in bytes and a modification time as an epoch, separated by tabs.

lce_service.collector_log_api.core.build_log_request(log_name: str | None, directory: str | None = None, lines: Any = None, pattern: str | None = None, since: str | None = None, until: str | None = None, max_bytes: Any = None) → CollectorLogRequest

Validates a caller’s log read parameters and returns a request safe to run.

This is the only supported way to build a CollectorLogRequest. Every caller supplied value is validated here, which keeps the validation rules in one place rather than spread across the endpoints that accept them.

Parameters:

• log_name – Name of the log file to read, within the chosen directory.

• directory – Key of the log directory to read from, or None for the default.

• lines – Number of trailing lines to return, or None for no line bound.

• pattern – An extended regular expression to filter records by, or None.

• since – Inclusive lower bound on the record timestamp, or None.

• until – Inclusive upper bound on the record timestamp, or None.

• max_bytes – Ceiling on the bytes the read may return, or None for MAX_OUTPUT_BYTES.

Returns:

A validated CollectorLogRequest.

Raises:

InvalidLogRequestError – Raised when any supplied value fails validation. The message names the value that was rejected and the shape expected.

lce_service.collector_log_api.core.build_read_command(request: CollectorLogRequest) → str

Builds the command that reads one log file according to a validated request.

The command is assembled from a fixed template per mode and is never influenced by the caller beyond the validated values in the request. Filtering runs here, on the collector, rather than in this service, because the output travels back through a shared platform log table in chunked rows and an active collector log reaches tens of megabytes.

The command is a pipeline of up to four stages, in a fixed order: a matcher or a decompressor, a timestamp window, a trailing line cap and a trailing byte cap. The first stage is the one that opens the log file and every later stage reads from the one before it, so exactly one stage carries the path. A compressed generation is read with zgrep or zcat rather than by decompressing anything first.

The last two stages are always present, so every command is a pipeline and there is no single stage form. That is deliberate. It makes the bound on a read independent of what the caller asked for, and it means the per stage escaping is applied to every command rather than only to the ones that happen to need more than one filter. A pipeline needs a shell, which the Remote Toolbox does not provide, so it travels as a single escaped argument to /bin/bash -c.

The byte cap is what bounds the response, because a line cap says nothing about how long a record is. It is a tail rather than a head so that an oversized read loses its oldest records rather than its newest, at the cost of the first record returned possibly being cut short.

Parameters:

request – A validated CollectorLogRequest describing what to read.

Returns:

The command to queue, without the SILOPROC marker the caller prepends.

lce_service.collector_log_api.core.build_request_note(request: CollectorLogRequest) → str | None

Describes a limitation of a validated request, for the caller to be told about.

A timestamp bound is compared against the first whitespace separated field of a record, which is the whole timestamp only in the rsyslog written directory. Records in the em7 directory separate the date from the time with a space, so the comparison only ever sees the date and a bound carrying a time component matches nothing at all. A whole day bound works there, so such a request is answered rather than refused, and this is what tells the caller why a narrower bound would come back empty.

Parameters:

request – A validated CollectorLogRequest describing what will be read.

Returns:

A note to include in the dispatch response, or None when there is nothing to add.

lce_service.collector_log_api.core.escape_as_single_token(value: str) → str

Escapes one pipeline stage so it survives transport and reaches its program intact.

A stage is read twice on the way to the program that runs it, and each read needs a different amount of escaping, which is why there are two rules here rather than one.

The first read is by the collector side toolbox, which replaces every quote character with an escaped quote and then splits the command with shlex. That makes grouping quotes useless, so a whole pipeline has to travel as one argument held together by backslash escaped spaces, and shlex consumes one level of escaping as it goes.

The second read is by the shell that runs the pipeline, which would otherwise treat much of an awk program as syntax of its own: a field reference looks like a positional parameter, a comparison looks like a redirection, and a boolean operator looks like a command separator. A metacharacter therefore needs to survive the first read still escaped, which takes two backslashes, while a space only has to survive as a space and takes one.

A space is the only whitespace a stage may carry. The characters in FORBIDDEN_WHITESPACE cannot be escaped into a bash -c script at all, so a stage holding one is refused rather than turned into a command that would fail somewhere the caller cannot see. Every validator in this module already excludes them, which makes this the second line of defence rather than the first.

Parameters:

value – One stage of a pipeline. Callers are responsible for ensuring the stage contains no quote characters, which the validators in this module already guarantee for every caller supplied value.

Returns:

The stage with shell metacharacters neutralised and spaces escaped, ready to be joined to other stages with PIPELINE_SEPARATOR.

Raises:

InvalidLogRequestError – Raised when the stage contains a tab, a newline, a carriage return, a vertical tab or a form feed.

lce_service.collector_log_api.core.get_collecting_appliance(dbc: silo_cursor, appliance_identifier: Any) → Dict[str, Any]

Looks up a collecting appliance by numeric id or by name.

Only appliances that actually collect are returned, because a log read has to be queued against the appliance that holds the logs. The lookup itself is shared with every Remote Toolbox API; this wrapper only reports its refusals in collector log terms.

Parameters:

• dbc – A silo_cursor instance used to query the appliance table.

• appliance_identifier – The numeric appliance id, or the appliance name.

Returns:

A dictionary holding the appliance id, name, ip and function.

Raises:

• InvalidLogRequestError – Raised when the identifier is a value that is neither an id nor a name.

• MissingResourceError – Raised when no collecting appliance matches the identifier.

• DatabaseExecutionError – Raised when the appliance lookup fails to execute.

lce_service.collector_log_api.core.get_collecting_appliances(dbc: silo_cursor) → List[Dict[str, Any]]

Lists every appliance that collects, and therefore runs Remote Toolbox commands.

A Database Server is excluded even though it is an appliance, because it does not collect and a command queued against one would never run.

Parameters:

dbc – A silo_cursor instance used to query the appliance table.

Returns:

One dictionary per collecting appliance, holding its id, name, ip and function, ordered by name.

Raises:

DatabaseExecutionError – Raised when the appliance lookup fails to execute.

lce_service.collector_log_api.core.validate_line_count(lines: Any) → int | None

Validates the number of trailing lines a caller has asked for.

Parameters:

lines – The line count supplied by the caller, or None to leave the line count bounded by MAX_LINE_COUNT alone. A string is accepted so a query parameter can be passed through without conversion at the endpoint layer.

Returns:

The line count as a positive integer, or None when no count was supplied.

Raises:

InvalidLogRequestError – Raised when the value is not a whole number between 1 and MAX_LINE_COUNT. A bool is refused rather than converted, because True would otherwise become a one record read.

lce_service.collector_log_api.core.validate_log_directory(directory: str | None) → LogDirectory

Resolves a caller supplied log directory key to the directory it names.

The caller never supplies a path. It supplies a key from ALLOWED_LOG_DIRECTORIES, which this function maps to an absolute directory, so a path traversal cannot be expressed in the first place.

Parameters:

directory – The directory key supplied by the caller, or None to default to the rsyslog written directory that holds the current service logs.

Returns:

A LogDirectory holding both the key the caller named and the absolute path it resolves to.

Raises:

InvalidLogRequestError – Raised when the key does not name a retrievable directory.

lce_service.collector_log_api.core.validate_log_name(log_name: str | None) → str

Validates that a log file name has the shape the platform actually produces.

The name is matched against the pattern logrotate writes, which covers both a current log and a rotated generation, compressed or not. Because the pattern admits no path separator and no parent directory reference, a validated name cannot escape the directory it is joined to.

Parameters:

log_name – The log file name supplied by the caller.

Returns:

The validated log file name, unchanged.

Raises:

InvalidLogRequestError – Raised when the name is missing, or does not have the shape of a current or rotated log file.

lce_service.collector_log_api.core.validate_log_pattern(pattern: str | None) → str | None

Validates a caller supplied extended regular expression used to filter records.

This is the only free text value that reaches the queued command, so it is the one place a caller could try to influence what the collector runs. The queued command is executed without a shell, so a semicolon, a pipe or a backtick is inert and arrives as an ordinary character. Whitespace is not inert: an unescaped space splits the pattern into two arguments, and the second becomes an additional file for grep to read. The allowlist below is therefore the control that keeps a pattern to exactly one argument.

A pattern carrying the Remote Toolbox marker is refused before the allowlist is applied. The collector takes the last occurrence of the marker and refuses any payload with text in front of it, so a second marker inside a command would turn a valid read into a refused payload whose text is echoed into a table shared with every other toolbox tool.

Unlike the other validators here, this one does not strip surrounding whitespace. A pattern is an exact expression rather than a name, so trimming one would answer a request the caller did not make, and whitespace is the very thing the allowlist exists to refuse: a pattern arriving with a trailing newline is a malformed request and is reported as one.

Parameters:

pattern – The extended regular expression supplied by the caller, or None for no filtering.

Returns:

The validated pattern, or None when no pattern was supplied.

Raises:

InvalidLogRequestError – Raised when the pattern carries the Remote Toolbox marker, is longer than MAX_PATTERN_LENGTH, or contains whitespace, a quote, a backslash or any other character outside the allowlist.

lce_service.collector_log_api.core.validate_max_bytes(max_bytes: Any) → int

Validates the byte ceiling a caller has asked for, and always returns one.

There is no unbounded read. A caller that asks for no ceiling is given MAX_OUTPUT_BYTES, which is also the largest ceiling that can be asked for, so a value above it is clamped rather than refused. The ceiling is enforced on the collector, as the last stage of the command, because output that has already reached the platform’s shared log table is too late to bound.

Parameters:

max_bytes – The byte ceiling supplied by the caller, or None to apply MAX_OUTPUT_BYTES. A string is accepted so a request body value can be passed through without conversion at the endpoint layer.

Returns:

The ceiling to apply, between 1 and MAX_OUTPUT_BYTES.

Raises:

InvalidLogRequestError – Raised when the value is not a positive whole number. A bool is refused rather than converted, because True would otherwise become a one byte read.

lce_service.collector_log_api.core.validate_timestamp_bound(bound: str | None, field_name: str) → str | None

Validates a timestamp bound used to window a log by record time.

Records in the rsyslog written log directory begin with an ISO 8601 timestamp, which sorts correctly as text, so a window is applied by comparing that field against a bound rather than by parsing dates on the collector. A bound may be any leading portion of a timestamp, so a whole day and a single minute are both expressible.

Parameters:

• bound – The timestamp bound supplied by the caller, or None for no bound.

• field_name – Name of the field being validated, used in the error message so the caller can tell which of the two bounds was rejected.

Returns:

The validated bound, or None when no bound was supplied.

Raises:

InvalidLogRequestError – Raised when the bound is not a leading portion of an ISO 8601 timestamp.