Warning
This document is for an old release of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
galaxy.objectstore package¶
objectstore package, abstraction for storing blobs of data for use in Galaxy.
all providers ensure that data can be accessed on the filesystem for running tools
-
class
galaxy.objectstore.
ObjectStore
[source]¶ Bases:
object
ObjectStore interface.
FIELD DESCRIPTIONS (these apply to all the methods in this class):
- Parameters
obj (StorableObject) – A Galaxy object with an assigned database ID accessible via the .id attribute.
base_dir (string) – A key in self.extra_dirs corresponding to the base directory in which this object should be created, or None to specify the default directory.
dir_only (boolean) – If True, check only the path where the file identified by obj should be located, not the dataset itself. This option applies to extra_dir argument as well.
extra_dir (string) – Append extra_dir to the directory structure where the dataset identified by obj should be located. (e.g., 000/extra_dir/obj.id). Valid values include ‘job_work’ (defaulting to config.jobs_directory = ‘$GALAXY_ROOT/database/jobs_directory’); ‘temp’ (defaulting to config.new_file_path = ‘$GALAXY_ROOT/database/tmp’).
extra_dir_at_root (boolean) – Applicable only if extra_dir is set. If True, the extra_dir argument is placed at root of the created directory structure rather than at the end (e.g., extra_dir/000/obj.id vs. 000/extra_dir/obj.id)
alt_name (string) – Use this name as the alternative name for the created dataset rather than the default.
obj_dir (boolean) – Append a subdirectory named with the object’s ID (e.g. 000/obj.id)
-
abstract
exists
(obj, base_dir=None, dir_only=False, extra_dir=None, extra_dir_at_root=False, alt_name=None)[source]¶ Return True if the object identified by obj exists, False otherwise.
-
abstract
create
(obj, base_dir=None, dir_only=False, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Mark the object (obj) as existing in the store, but with no content.
This method will create a proper directory structure for the file if the directory does not already exist.
-
abstract
empty
(obj, base_dir=None, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Test if the object identified by obj has content.
If the object does not exist raises ObjectNotFound.
-
abstract
size
(obj, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Return size of the object identified by obj.
If the object does not exist, return 0.
-
abstract
delete
(obj, entire_dir=False, base_dir=None, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Delete the object identified by obj.
- Parameters
entire_dir (boolean) – If True, delete the entire directory pointed to by extra_dir. For safety reasons, this option applies only for and in conjunction with the extra_dir or obj_dir options.
-
abstract
get_data
(obj, start=0, count=- 1, base_dir=None, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Fetch count bytes of data offset by start bytes using obj.id.
If the object does not exist raises ObjectNotFound.
-
abstract
get_filename
(obj, base_dir=None, dir_only=False, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Get the expected filename with absolute path for object with id obj.id.
This can be used to access the contents of the object.
-
abstract
update_from_file
(obj, base_dir=None, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False, file_name=None, create=False)[source]¶ Inform the store that the file associated with obj.id has been updated.
If file_name is provided, update from that file instead of the default. If the object does not exist raises ObjectNotFound.
- Parameters
file_name (string) – Use file pointed to by file_name as the source for updating the dataset identified by obj
create (boolean) – If True and the default dataset does not exist, create it first.
-
abstract
get_object_url
(obj, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Return the URL for direct access if supported, otherwise return None.
Note: need to be careful to not bypass dataset security with this.
-
abstract
get_concrete_store_name
(obj)[source]¶ Return a display name or title of the objectstore corresponding to obj.
To accommodate nested objectstores, obj is passed in so this metadata can be returned for the ConcreteObjectStore corresponding to the object.
If the dataset is in a new or discarded state and an object_store_id has not yet been set, this may return
None
.
-
abstract
get_concrete_store_description_markdown
(obj)[source]¶ Return a longer description of how data ‘obj’ is stored.
To accommodate nested objectstores, obj is passed in so this metadata can be returned for the ConcreteObjectStore corresponding to the object.
If the dataset is in a new or discarded state and an object_store_id has not yet been set, this may return
None
.
-
class
galaxy.objectstore.
BaseObjectStore
(config, config_dict=None, **kwargs)[source]¶ Bases:
galaxy.objectstore.ObjectStore
-
__init__
(config, config_dict=None, **kwargs)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
file_ready
(obj, base_dir=None, dir_only=False, extra_dir=None, extra_dir_at_root=False, alt_name=None, obj_dir=False)[source]¶ Check if a file corresponding to a dataset is ready to be used.
Return True if so, False otherwise
-
classmethod
parse_xml
(config_xml)[source]¶ Parse an XML description of a configuration for this object store.
Return a configuration dictionary (such as would correspond to the YAML configuration) for the object store.
-
create
(obj, **kwargs)[source]¶ Mark the object (obj) as existing in the store, but with no content.
This method will create a proper directory structure for the file if the directory does not already exist.
-
empty
(obj, **kwargs)[source]¶ Test if the object identified by obj has content.
If the object does not exist raises ObjectNotFound.
-
size
(obj, **kwargs)[source]¶ Return size of the object identified by obj.
If the object does not exist, return 0.
-
delete
(obj, **kwargs)[source]¶ Delete the object identified by obj.
- Parameters
entire_dir (boolean) – If True, delete the entire directory pointed to by extra_dir. For safety reasons, this option applies only for and in conjunction with the extra_dir or obj_dir options.
-
get_data
(obj, **kwargs)[source]¶ Fetch count bytes of data offset by start bytes using obj.id.
If the object does not exist raises ObjectNotFound.
-
get_filename
(obj, **kwargs)[source]¶ Get the expected filename with absolute path for object with id obj.id.
This can be used to access the contents of the object.
-
update_from_file
(obj, **kwargs)[source]¶ Inform the store that the file associated with obj.id has been updated.
If file_name is provided, update from that file instead of the default. If the object does not exist raises ObjectNotFound.
- Parameters
file_name (string) – Use file pointed to by file_name as the source for updating the dataset identified by obj
create (boolean) – If True and the default dataset does not exist, create it first.
-
get_object_url
(obj, **kwargs)[source]¶ Return the URL for direct access if supported, otherwise return None.
Note: need to be careful to not bypass dataset security with this.
-
get_concrete_store_name
(obj)[source]¶ Return a display name or title of the objectstore corresponding to obj.
To accommodate nested objectstores, obj is passed in so this metadata can be returned for the ConcreteObjectStore corresponding to the object.
If the dataset is in a new or discarded state and an object_store_id has not yet been set, this may return
None
.
-
get_concrete_store_description_markdown
(obj)[source]¶ Return a longer description of how data ‘obj’ is stored.
To accommodate nested objectstores, obj is passed in so this metadata can be returned for the ConcreteObjectStore corresponding to the object.
If the dataset is in a new or discarded state and an object_store_id has not yet been set, this may return
None
.
-
-
class
galaxy.objectstore.
ConcreteObjectStore
(config, config_dict=None, **kwargs)[source]¶ Bases:
galaxy.objectstore.BaseObjectStore
Subclass of ObjectStore for stores that don’t delegate (non-nested).
Currently only adds store_by functionality. Which doesn’t make sense for the delegating object stores.
-
__init__
(config, config_dict=None, **kwargs)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
-
class
galaxy.objectstore.
DiskObjectStore
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.ConcreteObjectStore
Standard Galaxy object store.
Stores objects in files under a specific directory on disk.
>>> from galaxy.util.bunch import Bunch >>> import tempfile >>> file_path=tempfile.mkdtemp() >>> obj = Bunch(id=1) >>> s = DiskObjectStore(Bunch(umask=0o077, jobs_directory=file_path, new_file_path=file_path, object_store_check_old_style=False), dict(files_dir=file_path)) >>> s.create(obj) >>> s.exists(obj) True >>> assert s.get_filename(obj) == file_path + '/000/dataset_1.dat'
-
store_type
= 'disk'¶
-
__init__
(config, config_dict)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the same attributes needed by
ObjectStore
plus:file_path – Default directory to store objects to disk in.
umask – the permission bits for newly created files.
file_path (str) – Override for the config.file_path value.
extra_dirs (dict) – Keys are string, values are directory paths.
-
-
class
galaxy.objectstore.
NestedObjectStore
(config, config_xml=None)[source]¶ Bases:
galaxy.objectstore.BaseObjectStore
Base for ObjectStores that use other ObjectStores.
Example: DistributedObjectStore, HierarchicalObjectStore
-
class
galaxy.objectstore.
DistributedObjectStore
(config, config_dict, fsmon=False)[source]¶ Bases:
galaxy.objectstore.NestedObjectStore
ObjectStore that defers to a list of backends.
When getting objects the first store where the object exists is used. When creating objects they are created in a store selected randomly, but with weighting.
-
store_type
= 'distributed'¶
-
__init__
(config, config_dict, fsmon=False)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the same attributes needed by
NestedObjectStore
plus:distributed_object_store_config_file
fsmon (bool) – If True, monitor the file system for free space, removing backends when they get too full.
-
-
class
galaxy.objectstore.
HierarchicalObjectStore
(config, config_dict, fsmon=False)[source]¶ Bases:
galaxy.objectstore.NestedObjectStore
ObjectStore that defers to a list of backends.
When getting objects the first store where the object exists is used. When creating objects only the first store is used.
-
store_type
= 'hierarchical'¶
-
__init__
(config, config_dict, fsmon=False)[source]¶ The default constructor. Extends NestedObjectStore.
-
-
galaxy.objectstore.
build_object_store_from_config
(config, fsmon=False, config_xml=None, config_dict=None)[source]¶ Invoke the appropriate object store.
Will use the object_store_config_file attribute of the config object to configure a new object store from the specified XML file.
Or you can specify the object store type in the object_store attribute of the config object. Currently ‘disk’, ‘s3’, ‘swift’, ‘distributed’, ‘hierarchical’, ‘irods’, and ‘pulsar’ are supported values.
-
galaxy.objectstore.
local_extra_dirs
(func)[source]¶ Non-local plugin decorator using local directories for the extra_dirs (job_work and temp).
-
galaxy.objectstore.
convert_bytes
(bytes)[source]¶ A helper function used for pretty printing disk usage.
-
galaxy.objectstore.
config_to_dict
(config)[source]¶ Dict-ify the portion of a config object consumed by the ObjectStore class and its subclasses.
-
class
galaxy.objectstore.
ObjectStorePopulator
(app, user)[source]¶ Bases:
object
Small helper for interacting with the object store and making sure all datasets from a job end up with the same object_store_id.
Submodules¶
galaxy.objectstore.azure_blob module¶
Object Store plugin for the Microsoft Azure Block Blob Storage system
-
class
galaxy.objectstore.azure_blob.
AzureBlobObjectStore
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.ConcreteObjectStore
Object store that stores objects as blobs in an Azure Blob Container. A local cache exists that is used as an intermediate location for files between Galaxy and Azure.
-
store_type
= 'azure_blob'¶
-
__init__
(config, config_dict)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
galaxy.objectstore.cloud module¶
Object Store plugin for Cloud storage.
-
class
galaxy.objectstore.cloud.
Cloud
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.ConcreteObjectStore
,galaxy.objectstore.cloud.CloudConfigMixin
Object store that stores objects as items in an cloud storage. A local cache exists that is used as an intermediate location for files between Galaxy and the cloud storage.
-
store_type
= 'cloud'¶
-
__init__
(config, config_dict)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
galaxy.objectstore.irods module¶
Object Store plugin for the Integrated Rule-Oriented Data System (iRODS)
-
class
galaxy.objectstore.irods.
IRODSObjectStore
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.DiskObjectStore
,galaxy.objectstore.irods.CloudConfigMixin
Object store that stores files as data objects in an iRODS Zone. A local cache exists that is used as an intermediate location for files between Galaxy and iRODS.
-
store_type
= 'irods'¶
-
__init__
(config, config_dict)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the same attributes needed by
ObjectStore
plus:file_path – Default directory to store objects to disk in.
umask – the permission bits for newly created files.
file_path (str) – Override for the config.file_path value.
extra_dirs (dict) – Keys are string, values are directory paths.
-
galaxy.objectstore.pithos module¶
-
galaxy.objectstore.pithos.
parse_config_xml
(config_xml)[source]¶ Parse and validate config_xml, return dict for convenience :param config_xml: (lxml.etree.Element) root of XML subtree :returns: (dict) according to syntax :raises: various XML parse errors
-
class
galaxy.objectstore.pithos.
PithosObjectStore
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.ConcreteObjectStore
Object store that stores objects as items in a Pithos+ container. Cache is ignored for the time being.
-
store_type
= 'pithos'¶
-
__init__
(config, config_dict)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
galaxy.objectstore.pulsar module¶
-
class
galaxy.objectstore.pulsar.
PulsarObjectStore
(config, config_xml)[source]¶ Bases:
galaxy.objectstore.BaseObjectStore
Object store implementation that delegates to a remote Pulsar server.
This may be more aspirational than practical for now, it would be good to Galaxy to a point that a handler thread could be setup that doesn’t attempt to access the disk files returned by a (this) object store - just passing them along to the Pulsar unmodified. That modification - along with this implementation and Pulsar job destinations would then allow Galaxy to fully manage jobs on remote servers with completely different mount points.
This implementation should be considered beta and may be dropped from Galaxy at some future point or significantly modified.
-
__init__
(config, config_xml)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
galaxy.objectstore.s3 module¶
Object Store plugin for the Amazon Simple Storage Service (S3)
-
class
galaxy.objectstore.s3.
S3ObjectStore
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.ConcreteObjectStore
,galaxy.objectstore.s3.CloudConfigMixin
Object store that stores objects as items in an AWS S3 bucket. A local cache exists that is used as an intermediate location for files between Galaxy and S3.
-
store_type
= 's3'¶
-
__init__
(config, config_dict)[source]¶ - Parameters
config (object) –
An object, most likely populated from galaxy/config.ini, having the following attributes:
object_store_check_old_style (only used by the
DiskObjectStore
subclass)jobs_directory – Each job is given a unique empty directory as its current working directory. This option defines in what parent directory those directories will be created.
new_file_path – Used to set the ‘temp’ extra_dir.
-
classmethod
parse_xml
(config_xml)[source]¶ Parse an XML description of a configuration for this object store.
Return a configuration dictionary (such as would correspond to the YAML configuration) for the object store.
-
-
class
galaxy.objectstore.s3.
SwiftObjectStore
(config, config_dict)[source]¶ Bases:
galaxy.objectstore.s3.S3ObjectStore
Object store that stores objects as items in a Swift bucket. A local cache exists that is used as an intermediate location for files between Galaxy and Swift.
-
store_type
= 'swift'¶
-
galaxy.objectstore.s3_multipart_upload module¶
Split large file into multiple pieces for upload to S3. This parallelizes the task over available cores using multiprocessing. Code mostly taken form CloudBioLinux.
-
galaxy.objectstore.s3_multipart_upload.
mp_from_ids
(s3server, mp_id, mp_keyname, mp_bucketname)[source]¶ Get the multipart upload from the bucket and multipart IDs.
This allows us to reconstitute a connection to the upload from within multiprocessing functions.