Warning
This document is for an old release of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
galaxy.jobs package¶
Support for running a tool in Galaxy via an internal job management system
-
class
galaxy.jobs.
JobDestination
(**kwds)[source]¶ Bases:
galaxy.util.bunch.Bunch
Provides details about where a job runs
-
class
galaxy.jobs.
JobToolConfiguration
(**kwds)[source]¶ Bases:
galaxy.util.bunch.Bunch
Provides details on what handler and destination a tool should use
A JobToolConfiguration will have the required attribute ‘id’ and optional attributes ‘handler’, ‘destination’, and ‘params’
-
class
galaxy.jobs.
JobConfiguration
(app)[source]¶ Bases:
galaxy.web_stack.handlers.ConfiguresHandlers
A parser and interface to advanced job management features.
These features are configured in the job configuration, by default,
job_conf.xml
-
DEFAULT_NWORKERS
= 4¶
-
JOB_RESOURCE_CONDITIONAL_XML
= '<conditional name="__job_resource">\n <param name="__job_resource__select" type="select" label="Job Resource Parameters">\n <option value="no">Use default job resource parameters</option>\n <option value="yes">Specify job resource parameters</option>\n </param>\n <when value="no"/>\n <when value="yes"/>\n </conditional>'¶
-
get_tool_resource_xml
(tool_id, tool_type)[source]¶ Given a tool id, return XML elements describing parameters to insert into job resources.
- Tool id
A tool ID (a string)
- Tool type
A tool type (a string)
- Returns
List of parameter elements.
-
static
get_envs
(parent)[source]¶ Parses any child <env> tags in to a dictionary suitable for persistence.
- Parameters
parent (
lxml.etree._Element
) – Parent element in which to find child <env> tags.- Returns
dict
-
static
get_resubmits
(parent)[source]¶ Parses any child <resubmit> tags in to a dictionary suitable for persistence.
- Parameters
parent (
lxml.etree._Element
) – Parent element in which to find child <resubmit> tags.- Returns
dict
-
property
default_job_tool_configuration
¶ The default JobToolConfiguration, used if a tool does not have an explicit defintion in the configuration. It consists of a reference to the default handler and default destination.
- Returns
JobToolConfiguration – a representation of a <tool> element that uses the default handler and destination
-
get_job_tool_configurations
(ids)[source]¶ Get all configured JobToolConfigurations for a tool ID, or, if given a list of IDs, the JobToolConfigurations for the first id in
ids
matching a tool definition.Note
You should not mix tool shed tool IDs, versionless tool shed IDs, and tool config tool IDs that refer to the same tool.
- Parameters
ids (list or str.) – Tool ID or IDs to fetch the JobToolConfiguration of.
- Returns
list – JobToolConfiguration Bunches representing <tool> elements matching the specified ID(s).
Example tool ID strings include:
Full tool shed id:
toolshed.example.org/repos/nate/filter_tool_repo/filter_tool/1.0.0
Tool shed id less version:
toolshed.example.org/repos/nate/filter_tool_repo/filter_tool
Tool config tool id:
filter_tool
-
get_destination
(id_or_tag)[source]¶ Given a destination ID or tag, return the JobDestination matching the provided ID or tag
- Parameters
id_or_tag (str) – A destination ID or tag.
- Returns
JobDestination – A valid destination
Destinations are deepcopied as they are expected to be passed in to job runners, which will modify them for persisting params set at runtime.
-
get_destinations
(id_or_tag)[source]¶ Given a destination ID or tag, return all JobDestinations matching the provided ID or tag
- Parameters
id_or_tag (str) – A destination ID or tag.
- Returns
list or tuple of JobDestinations
Destinations are not deepcopied, so they should not be passed to anything which might modify them.
-
get_job_runner_plugins
(handler_id)[source]¶ Load all configured job runner plugins
- Returns
list of job runner plugins
-
is_id
(collection)[source]¶ Given a collection of handlers or destinations, indicate whether the collection represents a tag or a real ID
-
-
class
galaxy.jobs.
JobWrapper
(job, queue, use_persisted_destination=False)[source]¶ Bases:
galaxy.jobs.HasResourceParameters
Wraps a ‘model.Job’ with convenience methods for running processes and state management.
-
__init__
(job, queue, use_persisted_destination=False)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
property
external_output_metadata
¶
-
property
dataset_path_rewriter
¶
-
property
outputs_directory
¶ Default location of
outputs_to_working_directory
.
-
property
created_with_galaxy_version
¶
-
property
dependency_shell_commands
¶ Shell fragment to inject dependencies.
-
property
cleanup_job
¶ Remove the job after it is complete, should return “always”, “onsuccess”, or “never”.
-
property
requires_containerization
¶
-
property
use_metadata_binary
¶
-
property
shell
¶
-
disable_commands_in_new_shell
()[source]¶ Provide an extension point to disable this isolation, Pulsar builds its own job script so this is not needed for remote jobs.
-
property
strict_shell
¶
-
property
commands_in_new_shell
¶
-
property
galaxy_lib_dir
¶
-
property
galaxy_virtual_env
¶
-
get_job_runner
()¶
-
property
job_destination
¶ Return the JobDestination that this job will use to run. This will either be a configured destination, a randomly selected destination if the configured destination was a tag, or a dynamically generated destination from the dynamic runner.
Calling this method for the first time causes the dynamic runner to do its calculation, if any.
- Returns
JobDestination
-
property
validate_outputs
¶
-
prepare
(compute_environment=None)[source]¶ Prepare the job to run by creating the working directory and the config files.
-
property
guest_ports
¶
-
property
working_directory
¶
-
property
tool_working_directory
¶
-
fail
(message, exception=False, tool_stdout='', tool_stderr='', exit_code=None, job_stdout=None, job_stderr=None)[source]¶ Indicate job failure by setting state and message on all output datasets.
-
set_job_destination
(job_destination, external_id=None, flush=True, job=None)[source]¶ Persist job destination params in the database for recovery.
self.job_destination is not used because a runner may choose to rewrite parts of the destination (e.g. the params).
-
property
home_target
¶
-
property
tmp_target
¶
-
get_destination_configuration
(key, default=None)[source]¶ Get a destination parameter that can be defaulted back in app.config if it needs to be applied globally.
-
finish
(tool_stdout, tool_stderr, tool_exit_code=None, job_stdout=None, job_stderr=None, check_output_detected_state=None, remote_metadata_directory=None, job_metrics_directory=None)[source]¶ Called to indicate that the associated command has been run. Updates the output datasets based on stderr and stdout from the command, and the contents of the output files.
-
check_tool_output
(tool_stdout, tool_stderr, tool_exit_code, job, job_stdout=None, job_stderr=None)[source]¶
-
property
object_store
¶
-
property
tmp_dir_creation_statement
¶
-
setup_external_metadata
(exec_dir=None, tmp_dir=None, dataset_files_path=None, config_root=None, config_file=None, datatypes_config=None, resolve_metadata_dependencies=False, set_extension=True, **kwds)[source]¶
-
property
user
¶
-
property
user_system_pwent
¶
-
property
galaxy_system_pwent
¶
-
get_output_destination
(output_path)[source]¶ Destination for outputs marked as from_work_dir. This is the normal case, just copy these files directly to the ultimate destination.
-
property
requires_setting_metadata
¶
-
-
class
galaxy.jobs.
TaskWrapper
(task, queue)[source]¶ Bases:
galaxy.jobs.JobWrapper
Extension of JobWrapper intended for running tasks. Should be refactored into a generalized executable unit wrapper parent, then jobs and tasks.
-
property
dataset_path_rewriter
¶
-
prepare
(compute_environment=None)[source]¶ Prepare the job to run by creating the working directory and the config files.
-
fail
(message, exception=False)[source]¶ Indicate job failure by setting state and message on all output datasets.
-
finish
(stdout, stderr, tool_exit_code=None, **kwds)[source]¶ Called to indicate that the associated command has been run. Updates the output datasets based on stderr and stdout from the command, and the contents of the output files.
-
property
-
class
galaxy.jobs.
ComputeEnvironment
[source]¶ Bases:
object
Definition of the job as it will be run on the (potentially) remote compute server.
-
abstract
input_extra_files_rewrite
(dataset)[source]¶ Input extra files path rewrite for specified dataset.
-
abstract
output_extra_files_rewrite
(dataset)[source]¶ Output extra files path rewrite for specified dataset.
-
abstract
Bases:
galaxy.jobs.SimpleComputeEnvironment
Default ComputeEnviornment for job and task wrapper to pass to ToolEvaluator - valid when Galaxy and compute share all the relevant file systems.
Initialize self. See help(type(self)) for accurate signature.
Working directory (possibly as environment variable evaluation).
-
class
galaxy.jobs.
NoopQueue
[source]¶ Bases:
object
Implements the JobQueue / JobStopQueue interface but does nothing
Subpackages¶
- galaxy.jobs.actions package
- galaxy.jobs.rules package
- galaxy.jobs.runners package
- Subpackages
- Submodules
- galaxy.jobs.runners.chronos module
- galaxy.jobs.runners.cli module
- galaxy.jobs.runners.condor module
- galaxy.jobs.runners.drmaa module
- galaxy.jobs.runners.godocker module
- galaxy.jobs.runners.kubernetes module
- galaxy.jobs.runners.local module
- galaxy.jobs.runners.pbs module
- galaxy.jobs.runners.pulsar module
- galaxy.jobs.runners.slurm module
- galaxy.jobs.runners.state_handler_factory module
- galaxy.jobs.runners.tasks module
- galaxy.jobs.runners.univa module
- galaxy.jobs.splitters package
Submodules¶
galaxy.jobs.command_factory module¶
-
galaxy.jobs.command_factory.
build_command
(runner, job_wrapper, container=None, modify_command_for_container=True, include_metadata=False, include_work_dir_outputs=True, create_tool_working_directory=True, remote_command_params=None, remote_job_directory=None, stdout_file=None, stderr_file=None)[source]¶ Compose the sequence of commands necessary to execute a job. This will currently include:
environment settings corresponding to any requirement tags
preparing input files
command line taken from job wrapper
commands to set metadata (if include_metadata is True)
galaxy.jobs.dynamic_tool_destination module¶
-
galaxy.jobs.dynamic_tool_destination.
priority_list
: Set[str] = {}¶ Instantiated to a list of all valid destinations in the job configuration file if run directly to validate configs. Otherwise, remains None. We often check to see if app is None, because if it is then we’ll try using the destination_list instead. -
-
galaxy.jobs.dynamic_tool_destination.
destination_list
: Set[str] = {}¶ The largest the edit distance can be for a word to be considered A correction for another word.
-
galaxy.jobs.dynamic_tool_destination.
max_edit_dist
= 2¶ List of valid categories that can be expected in the configuration.
-
galaxy.jobs.dynamic_tool_destination.
get_keys_from_dict
(dl, keys_list)[source]¶ This function builds a list using the keys from nest dictionaries
-
class
galaxy.jobs.dynamic_tool_destination.
RuleValidator
[source]¶ Bases:
object
This class is the primary facility for validating configs. It’s always called in map_tool_to_destination and it’s called for validating config directly through DynamicToolDestination.py
-
galaxy.jobs.dynamic_tool_destination.
parse_yaml
(path: str = '/config/tool_destinations.yml', job_conf_path: str = '/config/job_conf.xml', app=None, test: bool = False, return_bool: bool = False)[source]¶ Get a yaml file from path and send it to validate_config for validation.
- Parameters
path – the path to the tool destinations config file
job_conf_path – the path to the job config file
test – indicates whether to run in test mode or production mode
return_bool (bool) – True when we are only interested in the result of the validation, and not the validated rule itself.
- Return type
- Returns
validated rule or result of validation (depending on return_bool)
-
galaxy.jobs.dynamic_tool_destination.
validate_destination
(app, destination: str, err_message: str, err_message_contents, return_bool: bool = True)[source]¶ Validate received destination id.
- Parameters
app – Current app
destination – string containing the destination id that is being validated
err_message – Error message to be formatted with the contents of err_message_contents upon the event of invalid destination
err_message_contents (tuple) – A tuple of strings to be placed in
err_message
return_bool – Whether or not the calling function has been told to return a boolean value or not. Determines whether or not to print ‘Ignoring…’ after error messages.
- Return type
- Returns
True if the destination is valid and False otherwise.
-
galaxy.jobs.dynamic_tool_destination.
validate_config
(obj: dict, app=None, return_bool: bool = False)[source]¶ Validate received config.
-
galaxy.jobs.dynamic_tool_destination.
bytes_to_str
(size, unit='YB')[source]¶ Uses the bi convention: 1024 B = 1 KB since this method primarily has inputs of bytes for RAM
@type size: int @param size: the size in int (bytes) to be converted to str
@rtype: str @return return_str: the resulting string
-
galaxy.jobs.dynamic_tool_destination.
str_to_bytes
(size)[source]¶ Uses the bi convention: 1024 B = 1 KB since this method primarily has inputs of bytes for RAM
@type size: str @param size: the size in str to be converted to int (bytes)
@rtype: int @return curr_size: the resulting size converted from str
-
galaxy.jobs.dynamic_tool_destination.
importer
(test)[source]¶ Uses Mock galaxy for testing or real galaxy for production
@type test: bool @param test: True when being run from a test
-
galaxy.jobs.dynamic_tool_destination.
map_tool_to_destination
(job, app, tool, user_email, test=False, path=None, job_conf_path=None)[source]¶ Dynamically allocate resources
@param job: galaxy job @param app: current app @param tool: current tool
@type test: bool @param test: True when running in test mode
@type path: str @param path: path to tool_destinations.yml
@type job_conf_path: str @param job_conf_path: path to job_conf.xml
-
galaxy.jobs.dynamic_tool_destination.
get_destination_list_from_job_config
(job_config_location) → set[source]¶ returns A list of all destination IDs declared in the job configuration
- Parameters
job_config_location (str) – The location of the job config file relative to the galaxy root directory. If NoneType, defaults to galaxy/config/job_conf.xml, galaxy/config/job_conf.xml.sample_advanced, or galaxy/config/job_conf.xml.sample_basic (first one that exists)
- Returns
A list of all of the destination IDs declared in the job configuration file.
-
galaxy.jobs.dynamic_tool_destination.
get_edit_distance
(source, target)[source]¶ returns the edit distance (levenshtein distance) between two strings. code from: en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance
@type str1: str @param str1: The first string
@type str2: str @param str2: The second string
@rtype: int @return: The edit distance between str1 and str2
-
galaxy.jobs.dynamic_tool_destination.
get_typo_correction
(typo_str, word_set, max_dist)[source]¶ returns the string in a set that closest matches the input string, as long as the edit distance between them is equal to or smaller than a value, or the words are the same when case is not considered. If there are no appropriate matches, nothing is returned instead.
- Parameters
- Return type
str or NoneType
- Returns
The closest matching string, or None, if no strings being compared to are within max_dist edit distance.
-
galaxy.jobs.dynamic_tool_destination.
verbose
= True¶ list of all valid priorities, inferred from the global default_desinations section of the config
galaxy.jobs.handler module¶
Galaxy job handler, prepares, runs, tracks, and finishes Galaxy jobs
-
class
galaxy.jobs.handler.
JobHandler
(app)[source]¶ Bases:
object
Handle the preparation, running, tracking, and finishing of jobs
-
class
galaxy.jobs.handler.
ItemGrabber
(app, grab_type='Job', handler_assignment_method=None, max_grab=None, self_handler_tags=None, handler_tags=None)[source]¶ Bases:
object
-
class
galaxy.jobs.handler.
JobHandlerQueue
(app, dispatcher)[source]¶ Bases:
galaxy.util.monitors.Monitors
Job Handler’s Internal Queue, this is what actually implements waiting for jobs to be runnable and dispatching to a JobRunner.
-
STOP_SIGNAL
= <object object>¶
-
-
class
galaxy.jobs.handler.
JobHandlerStopQueue
(app, dispatcher)[source]¶ Bases:
galaxy.util.monitors.Monitors
A queue for jobs which need to be terminated prematurely.
-
STOP_SIGNAL
= <object object>¶
-
-
class
galaxy.jobs.handler.
DefaultJobDispatcher
(app)[source]¶ Bases:
object
-
url_to_destination
(url)[source]¶ This is used by the runner mapper (a.k.a. dynamic runner) and recovery methods to have runners convert URLs to destinations.
New-style runner plugin IDs must match the URL’s scheme for this to work.
-
galaxy.jobs.manager module¶
Top-level Galaxy job manager, moves jobs to handler(s)
-
class
galaxy.jobs.manager.
JobManager
(app)[source]¶ Bases:
object
Highest level interface to job management.
-
enqueue
(job, tool=None, flush=True)[source]¶ Queue a job for execution.
Due to the nature of some handler assignment methods which are wholly DB-based, the enqueue method will flush the job. Callers who create the job typically should not flush the job before handing it off to
enqueue()
. If a job handler cannot be assigned, py:class:ToolExecutionError is raised.- Parameters
job (Instance of
galaxy.model.Job
.) – Job to enqueue.tool (Instance of
galaxy.tools.Tool
.) – Tool that the job will execute.
- Raises
ToolExecutionError – if a handler was unable to be assigned.
- Returns
str or None – Handler ID, tag, or pool assigned to the job.
-
stop
(job, message=None)[source]¶ Stop a job that is currently executing.
This can be safely called on jobs that have already terminated.
- Parameters
job (Instance of
galaxy.model.Job
.) – Job to stop.message (str) – Message (if any) to be set on the job and output dataset(s) to explain the reason for stopping.
-
galaxy.jobs.mapper module¶
-
exception
galaxy.jobs.mapper.
JobNotReadyException
(job_state=None, message=None)[source]¶ Bases:
Exception
-
class
galaxy.jobs.mapper.
JobRunnerMapper
(job_wrapper, url_to_destination, job_config)[source]¶ Bases:
object
This class is responsible to managing the mapping of jobs (in the form of job_wrappers) to job runner url strings.
-
__init__
(job_wrapper, url_to_destination, job_config)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
galaxy.jobs.rule_helper module¶
-
class
galaxy.jobs.rule_helper.
RuleHelper
(app)[source]¶ Bases:
object
Utility to allow job rules to interface cleanly with the rest of Galaxy and shield them from low-level details of models, metrics, etc….
Currently focus is on figuring out job statistics for a given user, but could interface with other stuff as well.
-
supports_docker
(job_or_tool)[source]¶ Job rules can pass this function a job, job_wrapper, or tool and determine if the underlying tool believes it can be containered.
-
should_burst
(destination_ids, num_jobs, job_states=None)[source]¶ Check if the specified destinations
destination_ids
have at leastnum_jobs
assigned to it - send injob_state
asqueued
to limit this check to number of jobs queued.See stock_rules for an simple example of using this function - but to get the most out of it - it should probably be used with custom job rules that can respond to the bursting by allocating resources, launching cloud nodes, etc….
-
choose_one
(lst, hash_value=None)[source]¶ Choose a random value from supplied list. If hash_value is passed in then every request with that same hash_value would produce the same choice from the supplied list.
-
job_hash
(job, hash_by=None)[source]¶ Produce a reproducible hash for the given job on various criteria - for instance if hash_by is “workflow_invocation,history” - all jobs within the same workflow invocation will receive the same hash - for jobs outside of workflows all jobs within the same history will receive the same hash, other jobs will be hashed on job’s id randomly.
Primarily intended for use with
choose_one
above - to consistent route or schedule related jobs.
-
galaxy.jobs.stock_rules module¶
Stock job ‘dynamic’ rules for use in job_conf.xml - these may cover some simple use cases but will just proxy into functions in rule_helper so similar functionality - but more tailored and composable can be utilized in custom rules.
galaxy.jobs.transfer_manager module¶
Manage transfers from arbitrary URLs to temporary files. Socket interface for IPC with multiple process configurations.
-
class
galaxy.jobs.transfer_manager.
TransferManager
(app)[source]¶ Bases:
object
Manage simple data transfers from URLs to temporary locations.