Warning

This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.

galaxy.job_metrics.instrumenters package

This module describes the abstract interface for InstrumentPlugin.

These are responsible for collecting and formatting a coherent set of metrics.

class galaxy.job_metrics.instrumenters.ProvidesJobMetricsContext(*args, **kwargs)

Bases: Protocol

The slice of a Galaxy job that plugins may read when metrics are collected.

Declared structurally instead of importing galaxy.model: this package ships to Pulsar compute nodes (see packages/packages_for_pulsar_by_dep_dag.txt) and so depends on galaxy-util alone. The app passes something Job-shaped in; nothing here needs to know it is a Job.

property id: int

property resubmission_count: int

__init__(*args, **kwargs)

class galaxy.job_metrics.instrumenters.InstrumentPlugin

Bases: object

Describes how to instrument job scripts and retrieve collected metrics.

formatter: JobMetricFormatter | None = <galaxy.job_metrics.formatting.JobMetricFormatter object>

default_safety = 4

abstract property plugin_type

Short string providing labelling this plugin

pre_execute_instrument(job_directory: str) → str | list[str] | None

Optionally return one or more commands to instrument job. These commands will be executed on the compute server prior to the job running.

post_execute_instrument(job_directory: str) → str | list[str] | None

Optionally return one or more commands to instrument job. These commands will be executed on the compute server after the tool defined command is ran.

abstract job_properties(job_id, job_directory: str) → dict[str, Any]

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

collect(job: ProvidesJobMetricsContext, job_directory: str) → dict[str, Any]

Collect properties for this plugin, given the job they belong to.

The framework calls this rather than job_properties directly. Override it to read the job itself; the default keeps plugins that only need the directory working unchanged.

classmethod safety(metric_name: str) → Safety

Return safety level of metric.

A classmethod so metrics recorded by a plugin that isn’t configured globally (e.g. only for some destinations) keep their level.

Submodules

galaxy.job_metrics.instrumenters.cgroup module

The module describes the cgroup job metrics plugin.

class galaxy.job_metrics.instrumenters.cgroup.CgroupPlugin(**kwargs)

Bases: InstrumentPlugin

Plugin that collects memory and cpu utilization from within a cgroup.

plugin_type = 'cgroup'

formatter: JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.cgroup.CgroupPluginFormatter object>

__init__(**kwargs)

post_execute_instrument(job_directory: str) → list[str]

Optionally return one or more commands to instrument job. These commands will be executed on the compute server after the tool defined command is ran.

job_properties(job_id, job_directory: str) → dict[str, Any]

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

galaxy.job_metrics.instrumenters.core module

The module describes the core job metrics plugin.

class galaxy.job_metrics.instrumenters.core.CorePlugin(**kwargs)

Bases: InstrumentPlugin

Simple plugin that collects data without external dependencies. In particular it currently collects value set for Galaxy slots.

plugin_type = 'core'

default_safety = 6

__init__(**kwargs)

formatter: formatting.JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.core.CorePluginFormatter object>

pre_execute_instrument(job_directory: str) → list[str]

Optionally return one or more commands to instrument job. These commands will be executed on the compute server prior to the job running.

post_execute_instrument(job_directory: str) → list[str]

Optionally return one or more commands to instrument job. These commands will be executed on the compute server after the tool defined command is ran.

job_properties(job_id, job_directory: str) → dict[str, Any]

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

collect(job: ProvidesJobMetricsContext, job_directory: str) → dict[str, Any]

Collect properties for this plugin, given the job they belong to.

The framework calls this rather than job_properties directly. Override it to read the job itself; the default keeps plugins that only need the directory working unchanged.

get_container_file_path(job_directory)

galaxy.job_metrics.instrumenters.cpuinfo module

The module describes the cpuinfo job metrics plugin.

class galaxy.job_metrics.instrumenters.cpuinfo.CpuInfoPlugin(**kwargs)

Bases: InstrumentPlugin

Gather information about processor configuration from /proc/cpuinfo. Linux only.

plugin_type = 'cpuinfo'

formatter: formatting.JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.cpuinfo.CpuInfoFormatter object>

__init__(**kwargs)

fields: set[str] | None

pre_execute_instrument(job_directory)

Optionally return one or more commands to instrument job. These commands will be executed on the compute server prior to the job running.

job_properties(job_id, job_directory)

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

galaxy.job_metrics.instrumenters.env module

The module describes the env job metrics plugin.

class galaxy.job_metrics.instrumenters.env.EnvPlugin(**kwargs)

Bases: InstrumentPlugin

Instrumentation plugin capable of recording all or specific environment variables for a job at runtime.

plugin_type = 'env'

formatter: formatting.JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.env.EnvFormatter object>

default_safety = 2

__init__(**kwargs)

variables: list[str] | None

pre_execute_instrument(job_directory: str)

Use env to dump all environment variables to a file.

post_execute_instrument(job_directory)

Optionally return one or more commands to instrument job. These commands will be executed on the compute server after the tool defined command is ran.

job_properties(job_id, job_directory)

Recover environment variables dumped out on compute server and filter out specific variables if needed.

galaxy.job_metrics.instrumenters.hostname module

The module describes the hostname job metrics plugin.

class galaxy.job_metrics.instrumenters.hostname.HostnamePlugin(**kwargs)

Bases: InstrumentPlugin

Gather hostname

plugin_type = 'hostname'

__init__(**kwargs)

pre_execute_instrument(job_directory)

Optionally return one or more commands to instrument job. These commands will be executed on the compute server prior to the job running.

job_properties(job_id, job_directory)

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

galaxy.job_metrics.instrumenters.meminfo module

The module describes the meminfo job metrics plugin.

class galaxy.job_metrics.instrumenters.meminfo.MemInfoPlugin(**kwargs)

Bases: InstrumentPlugin

Gather information about processor configuration from /proc/cpuinfo. Linux only.

plugin_type = 'meminfo'

formatter: JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.meminfo.MemInfoFormatter object>

default_safety = 6

__init__(**kwargs)

pre_execute_instrument(job_directory)

Optionally return one or more commands to instrument job. These commands will be executed on the compute server prior to the job running.

job_properties(job_id, job_directory)

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

galaxy.job_metrics.instrumenters.pulsar module

The module describes the pulsar job metrics plugin.

Unlike every other plugin here, this one injects nothing into the job script. What it reports happens either side of the job script, so its files are written by Pulsar and by Galaxy’s Pulsar runner, and this plugin only reads them. Each is a JSON object named by the usual instrumentation convention (__instrument_pulsar_<name>):

• transfer_<phase> - Pulsar staging a job’s inputs in and its outputs back out, keys below. The prefix is also what gets Pulsar to stage it back into Galaxy’s metrics directory.

• version - the version of the Pulsar that staged the job, also written by Pulsar.

• version_target / version_status - written by the Pulsar runner: the version it made its decisions for at submission (and where that came from), and the version the remote reported when the job finished.

Pulsar writes its files in pulsar/managers/staging/metrics.py; the two ends have to keep agreeing on these shapes.

class galaxy.job_metrics.instrumenters.pulsar.PulsarPlugin(**kwargs: Any)

Bases: InstrumentPlugin

Report how long Pulsar spent staging a job’s files and how much it moved, and which Pulsar versions were involved.

plugin_type = 'pulsar'

formatter: formatting.JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.pulsar.PulsarPluginFormatter object>

default_safety = 6

__init__(**kwargs: Any) → None

classmethod safety(metric_name: str) → Safety

Return safety level of metric.

A classmethod so metrics recorded by a plugin that isn’t configured globally (e.g. only for some destinations) keep their level.

job_properties(job_id: int, job_directory: str) → dict[str, Any]

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.

classmethod write_version_target(job_directory: str, client_version: str, target_version: str, source: str) → None

Record the Pulsar version a job was submitted for, and how it was determined.

classmethod write_version_status(job_directory: str, server_version: str) → None

Record the version the remote Pulsar reported when the job finished.

classmethod read_target_version(job_directory: str) → str | None

The version recorded by write_version_target, if the job has one and it’s readable.

galaxy.job_metrics.instrumenters.uname module

The module describes the uname job metrics plugin.

class galaxy.job_metrics.instrumenters.uname.UnamePlugin(**kwargs)

Bases: InstrumentPlugin

Use uname to gather operating system information about remote system job is running on. Linux only.

plugin_type = 'uname'

formatter: JobMetricFormatter | None = <galaxy.job_metrics.instrumenters.uname.UnameFormatter object>

__init__(**kwargs)

pre_execute_instrument(job_directory)

Optionally return one or more commands to instrument job. These commands will be executed on the compute server prior to the job running.

job_properties(job_id, job_directory)

Collect properties for this plugin from specified job directory. This method will run on the Galaxy server and can assume files created in job_directory with pre_execute_instrument and post_execute_instrument are available.