Warning

This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.

galaxy.jobs.runners.util package

This module and its submodules contains utilities for running external processes and interfacing with job managers. This module should contain functionality shared between Galaxy and the Pulsar.

galaxy.jobs.runners.util.kill_pid(pid: int, use_psutil: bool = True)[source]

Subpackages

Submodules

galaxy.jobs.runners.util.env module

galaxy.jobs.runners.util.env.env_to_statement(env: Mapping[str, object]) → str[source]

Return the abstraction description of an environment variable definition into a statement for shell script.

>>> env_to_statement(dict(name='X', value='Y'))
'X="Y"; export X'
>>> env_to_statement(dict(name='X', value='Y', raw=True))
'X=Y; export X'
>>> env_to_statement(dict(name='X', value='"A","B","C"'))
'X="\\"A\\",\\"B\\",\\"C\\""; export X'
>>> env_to_statement(dict(file="Y"))
'. "Y"'
>>> env_to_statement(dict(file="'RAW $FILE'", raw=True))
". 'RAW $FILE'"
>>> # Source file takes precedence
>>> env_to_statement(dict(name='X', value='"A","B","C"', file="S"))
'. "S"'
>>> env_to_statement(dict(execute="module load java/1.5.1"))
'module load java/1.5.1'

galaxy.jobs.runners.util.external module

galaxy.jobs.runners.util.external.parse_external_id(output, type=None)[source]

Attempt to parse the output of job submission commands for an external id.__doc__

>>> parse_external_id("12345.pbsmanager")
'12345.pbsmanager'
>>> parse_external_id('Submitted batch job 185')
'185'
>>> parse_external_id('Submitted batch job 185', type='torque')
'Submitted batch job 185'
>>> parse_external_id('submitted to cluster 125.')
'125'
>>> parse_external_id('submitted to cluster 125.', type='slurm')
>>>

galaxy.jobs.runners.util.kill module

galaxy.jobs.runners.util.kill.kill_pid(pid: int, use_psutil: bool = True)[source]

galaxy.jobs.runners.util.process_groups module

galaxy.jobs.runners.util.process_groups.check_pg(pgid)[source]

Check whether processes in process group pgid are still alive.

galaxy.jobs.runners.util.process_groups.kill_pg(pgid)[source]

Kill all processes in process group pgid.

galaxy.jobs.runners.util.process_groups.kill_pgs(pgids, grace_period=1.0)[source]

Kill all processes in each of the process groups pgids.

Unlike calling kill_pg() for each group, every group gets SIGTERM before waiting, and the groups still alive after grace_period seconds all get SIGKILL, so the wait does not grow with the number of groups.

Liveness is tested by sending signal 0 instead of with check_pg(), so this does not reap the processes and whoever waits on them still gets their exit status.

galaxy.jobs.runners.util.pykube_util module

Interface layer for pykube library shared between Galaxy and Pulsar.

galaxy.jobs.runners.util.pykube_util.ensure_pykube()[source]
galaxy.jobs.runners.util.pykube_util.find_service_object_by_name(pykube_api, service_name, namespace=None)[source]
galaxy.jobs.runners.util.pykube_util.find_ingress_object_by_name(pykube_api, ingress_name, namespace=None)[source]
galaxy.jobs.runners.util.pykube_util.find_job_object_by_name(pykube_api, job_name, namespace=None)[source]
galaxy.jobs.runners.util.pykube_util.find_pod_object_by_name(pykube_api, job_name, namespace=None)[source]
galaxy.jobs.runners.util.pykube_util.galaxy_instance_id(params)[source]

Parse and validate the id of the Galaxy instance from supplied dict.

The optional value from params['k8s_galaxy_instance_id'] is included in the generateName prefix for Jobs and their Pods. Kubernetes appends a unique suffix to every submitted Job, so the instance id distinguishes resources belonging to different Galaxy instances rather than providing uniqueness itself. It must be DNS friendly because it becomes part of the generated resource names.

exception galaxy.jobs.runners.util.pykube_util.HTTPError(code, message)[source]

Bases: PyKubeError

__init__(code, message)[source]
galaxy.jobs.runners.util.pykube_util.is_pod_running(pykube_api, pod, namespace=None)[source]
galaxy.jobs.runners.util.pykube_util.is_pod_unschedulable(pykube_api, pod, namespace=None)[source]
class galaxy.jobs.runners.util.pykube_util.Job(api: HTTPClient, obj: dict)[source]

Bases: NamespacedAPIObject, ScalableMixin

version = 'batch/v1'
endpoint = 'jobs'
kind = 'Job'
scalable_attr = 'parallelism'
property parallelism
class galaxy.jobs.runners.util.pykube_util.Service(api: HTTPClient, obj: dict)[source]

Bases: NamespacedAPIObject

version = 'v1'
endpoint = 'services'
kind = 'Service'
proxy_http_request(method: str, path: str, port: int | None = None, **kwargs: Any) → Response[source]

Issue a HTTP request with specific HTTP method to proxy of a Service. Args:

param method:

The http request method e.g. ‘GET’, ‘POST’ etc.

param path:

The URI path for the request.

param port:

This value can be used to override the

default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object.

Returns:

The requests.Response object.

proxy_http_get(path: str, port: int | None = None, **kwargs) → Response[source]

Issue a HTTP GET request to proxy of a Service. Args:

param path:

The URI path for the request.

param port:

This value can be used to override the

default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.

Returns:

The requests.Response object.

proxy_http_post(path: str, port: int | None = None, **kwargs) → Response[source]

Issue a HTTP POST request to proxy of a Service. Args:

param path:

The URI path for the request.

param port:

This value can be used to override the

default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.

Returns:

The requests.Response object.

proxy_http_put(path: str, port: int | None = None, **kwargs) → Response[source]

Issue a HTTP PUT request to proxy of a Service. Args:

param path:

The URI path for the request.

param port:

This value can be used to override the

default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.

Returns:

The requests.Response object.

proxy_http_delete(path: str, port: int | None = None, **kwargs) → Response[source]

Issue a HTTP DELETE request to proxy of a Service. Args:

param path:

The URI path for the request.

param port:

This value can be used to override the

default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.

Returns:

The requests.Response object.

class galaxy.jobs.runners.util.pykube_util.Ingress(api: HTTPClient, obj: dict)[source]

Bases: NamespacedAPIObject

version = 'networking.k8s.io/v1'
endpoint = 'ingresses'
kind = 'Ingress'
galaxy.jobs.runners.util.pykube_util.job_object_dict(params, job_prefix, spec)[source]
galaxy.jobs.runners.util.pykube_util.service_object_dict(params, service_name, spec)[source]
galaxy.jobs.runners.util.pykube_util.ingress_object_dict(params, ingress_name, spec)[source]
class galaxy.jobs.runners.util.pykube_util.Pod(api: HTTPClient, obj: dict)[source]

Bases: NamespacedAPIObject

version = 'v1'
endpoint = 'pods'
kind = 'Pod'
property ready
logs(container=None, pretty=None, previous=False, since_seconds=None, since_time=None, timestamps=False, tail_lines=None, limit_bytes=None)[source]

Produces the same result as calling kubectl logs pod/<pod-name>. Check parameters meaning at http://kubernetes.io/docs/api-reference/v1/operations/, part ‘read log of the specified Pod’. The result is plain text.

galaxy.jobs.runners.util.pykube_util.produce_k8s_job_prefix(app_prefix=None, instance_id=None)[source]
galaxy.jobs.runners.util.pykube_util.pull_policy(params)[source]
galaxy.jobs.runners.util.pykube_util.pykube_client_from_dict(params)[source]
galaxy.jobs.runners.util.pykube_util.delete_job(job, cleanup='always')[source]
galaxy.jobs.runners.util.pykube_util.delete_service(service, cleanup='always', job_failed=False)[source]
galaxy.jobs.runners.util.pykube_util.delete_ingress(ingress, cleanup='always', job_failed=False)[source]
galaxy.jobs.runners.util.pykube_util.get_volume_mounts_for_job(job_wrapper, data_claim=None, working_claim=None)[source]
galaxy.jobs.runners.util.pykube_util.parse_pvc_param_line(pvc_param)[source]
Parameters:

pvc_param (str) – the pvc mount param in the format pvc-name/subpath/desired:/mountpath/desired[:r]

Return type:

dict

Returns:

a dict like:

{"name": "pvc-name",
 "subPath": "subpath/desired",
 "mountPath": "/mountpath/desired",
 "readOnly": False}

galaxy.jobs.runners.util.sudo module

galaxy.jobs.runners.util.sudo.sudo_popen(*args, **kwargs)[source]

Helper method for building and executing Popen command. This is potentially sensetive code so should probably be centralized.