Warning
This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
galaxy.jobs.runners.util package
This module and its submodules contains utilities for running external processes and interfacing with job managers. This module should contain functionality shared between Galaxy and the Pulsar.
Subpackages
Submodules
galaxy.jobs.runners.util.env module
- galaxy.jobs.runners.util.env.env_to_statement(env: Mapping[str, object]) str[source]
Return the abstraction description of an environment variable definition into a statement for shell script.
>>> env_to_statement(dict(name='X', value='Y')) 'X="Y"; export X' >>> env_to_statement(dict(name='X', value='Y', raw=True)) 'X=Y; export X' >>> env_to_statement(dict(name='X', value='"A","B","C"')) 'X="\\"A\\",\\"B\\",\\"C\\""; export X' >>> env_to_statement(dict(file="Y")) '. "Y"' >>> env_to_statement(dict(file="'RAW $FILE'", raw=True)) ". 'RAW $FILE'" >>> # Source file takes precedence >>> env_to_statement(dict(name='X', value='"A","B","C"', file="S")) '. "S"' >>> env_to_statement(dict(execute="module load java/1.5.1")) 'module load java/1.5.1'
galaxy.jobs.runners.util.external module
- galaxy.jobs.runners.util.external.parse_external_id(output, type=None)[source]
Attempt to parse the output of job submission commands for an external id.__doc__
>>> parse_external_id("12345.pbsmanager") '12345.pbsmanager' >>> parse_external_id('Submitted batch job 185') '185' >>> parse_external_id('Submitted batch job 185', type='torque') 'Submitted batch job 185' >>> parse_external_id('submitted to cluster 125.') '125' >>> parse_external_id('submitted to cluster 125.', type='slurm') >>>
galaxy.jobs.runners.util.kill module
galaxy.jobs.runners.util.process_groups module
- galaxy.jobs.runners.util.process_groups.check_pg(pgid)[source]
Check whether processes in process group pgid are still alive.
- galaxy.jobs.runners.util.process_groups.kill_pg(pgid)[source]
Kill all processes in process group pgid.
- galaxy.jobs.runners.util.process_groups.kill_pgs(pgids, grace_period=1.0)[source]
Kill all processes in each of the process groups pgids.
Unlike calling kill_pg() for each group, every group gets SIGTERM before waiting, and the groups still alive after grace_period seconds all get SIGKILL, so the wait does not grow with the number of groups.
Liveness is tested by sending signal 0 instead of with check_pg(), so this does not reap the processes and whoever waits on them still gets their exit status.
galaxy.jobs.runners.util.pykube_util module
Interface layer for pykube library shared between Galaxy and Pulsar.
- galaxy.jobs.runners.util.pykube_util.find_service_object_by_name(pykube_api, service_name, namespace=None)[source]
- galaxy.jobs.runners.util.pykube_util.find_ingress_object_by_name(pykube_api, ingress_name, namespace=None)[source]
- galaxy.jobs.runners.util.pykube_util.find_job_object_by_name(pykube_api, job_name, namespace=None)[source]
- galaxy.jobs.runners.util.pykube_util.find_pod_object_by_name(pykube_api, job_name, namespace=None)[source]
- galaxy.jobs.runners.util.pykube_util.galaxy_instance_id(params)[source]
Parse and validate the id of the Galaxy instance from supplied dict.
The optional value from
params['k8s_galaxy_instance_id']is included in thegenerateNameprefix for Jobs and their Pods. Kubernetes appends a unique suffix to every submitted Job, so the instance id distinguishes resources belonging to different Galaxy instances rather than providing uniqueness itself. It must be DNS friendly because it becomes part of the generated resource names.
- class galaxy.jobs.runners.util.pykube_util.Job(api: HTTPClient, obj: dict)[source]
Bases:
NamespacedAPIObject,ScalableMixin- version = 'batch/v1'
- endpoint = 'jobs'
- kind = 'Job'
- scalable_attr = 'parallelism'
- property parallelism
- class galaxy.jobs.runners.util.pykube_util.Service(api: HTTPClient, obj: dict)[source]
Bases:
NamespacedAPIObject- version = 'v1'
- endpoint = 'services'
- kind = 'Service'
- proxy_http_request(method: str, path: str, port: int | None = None, **kwargs: Any) Response[source]
Issue a HTTP request with specific HTTP method to proxy of a Service. Args:
- param method:
The http request method e.g. ‘GET’, ‘POST’ etc.
- param path:
The URI path for the request.
- param port:
This value can be used to override the
default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object.
- Returns:
The requests.Response object.
- proxy_http_get(path: str, port: int | None = None, **kwargs) Response[source]
Issue a HTTP GET request to proxy of a Service. Args:
- param path:
The URI path for the request.
- param port:
This value can be used to override the
default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.
- Returns:
The requests.Response object.
- proxy_http_post(path: str, port: int | None = None, **kwargs) Response[source]
Issue a HTTP POST request to proxy of a Service. Args:
- param path:
The URI path for the request.
- param port:
This value can be used to override the
default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.
- Returns:
The requests.Response object.
- proxy_http_put(path: str, port: int | None = None, **kwargs) Response[source]
Issue a HTTP PUT request to proxy of a Service. Args:
- param path:
The URI path for the request.
- param port:
This value can be used to override the
default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.
- Returns:
The requests.Response object.
- proxy_http_delete(path: str, port: int | None = None, **kwargs) Response[source]
Issue a HTTP DELETE request to proxy of a Service. Args:
- param path:
The URI path for the request.
- param port:
This value can be used to override the
default (first defined) port used to connect to the Service. :param kwargs: Keyword arguments for the proxy_http_get function. They are the same as for requests.models.Request object plus the additional ‘port’ kwarg, which can be used to override the default (first defined) port used to connect to the Service.
- Returns:
The requests.Response object.
- class galaxy.jobs.runners.util.pykube_util.Ingress(api: HTTPClient, obj: dict)[source]
Bases:
NamespacedAPIObject- version = 'networking.k8s.io/v1'
- endpoint = 'ingresses'
- kind = 'Ingress'
- class galaxy.jobs.runners.util.pykube_util.Pod(api: HTTPClient, obj: dict)[source]
Bases:
NamespacedAPIObject- version = 'v1'
- endpoint = 'pods'
- kind = 'Pod'
- property ready
- logs(container=None, pretty=None, previous=False, since_seconds=None, since_time=None, timestamps=False, tail_lines=None, limit_bytes=None)[source]
Produces the same result as calling kubectl logs pod/<pod-name>. Check parameters meaning at http://kubernetes.io/docs/api-reference/v1/operations/, part ‘read log of the specified Pod’. The result is plain text.
- galaxy.jobs.runners.util.pykube_util.produce_k8s_job_prefix(app_prefix=None, instance_id=None)[source]
- galaxy.jobs.runners.util.pykube_util.delete_service(service, cleanup='always', job_failed=False)[source]
- galaxy.jobs.runners.util.pykube_util.delete_ingress(ingress, cleanup='always', job_failed=False)[source]
- galaxy.jobs.runners.util.pykube_util.get_volume_mounts_for_job(job_wrapper, data_claim=None, working_claim=None)[source]