Warning

This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.

galaxy.model.store package

class galaxy.model.store.BagArchiveImportModelStore(bag_archive, **kwd)[source]

Bases: galaxy.model.store.DirectoryImportModelStoreLatest

__init__(bag_archive, **kwd)[source]
class galaxy.model.store.BagArchiveModelExportStore(out_file, bag_archiver='tgz', **kwds)[source]

Bases: galaxy.model.store.BagDirectoryModelExportStore

__init__(out_file, bag_archiver='tgz', **kwds)[source]
class galaxy.model.store.BagDirectoryModelExportStore(out_directory, **kwds)[source]

Bases: galaxy.model.store.DirectoryModelExportStore

__init__(out_directory, **kwds)[source]
class galaxy.model.store.BaseDirectoryImportModelStore(import_options=None, app=None, user=None, object_store=None, tag_handler=None)[source]

Bases: galaxy.model.store.ModelImportStore

collections_properties()[source]
datasets_properties()[source]
defines_new_history()[source]
file_source_root
implicit_collection_jobs_properties()[source]
jobs_properties()[source]
library_properties()[source]
new_history_properties()[source]
class galaxy.model.store.DirectoryImportModelStore1901(archive_dir, **kwd)[source]

Bases: galaxy.model.store.BaseDirectoryImportModelStore

__init__(archive_dir, **kwd)[source]
object_key = 'hid'
trust_hid(obj_attrs)[source]
class galaxy.model.store.DirectoryImportModelStoreLatest(archive_dir, **kwd)[source]

Bases: galaxy.model.store.BaseDirectoryImportModelStore

__init__(archive_dir, **kwd)[source]
object_key = 'encoded_id'
trust_hid(obj_attrs)[source]
class galaxy.model.store.DirectoryModelExportStore(export_directory, app=None, for_edit=False, serialize_dataset_objects=None, export_files=None, strip_metadata_files=True, serialize_jobs=True)[source]

Bases: galaxy.model.store.ModelExportStore

__init__(export_directory, app=None, for_edit=False, serialize_dataset_objects=None, export_files=None, strip_metadata_files=True, serialize_jobs=True)[source]
Parameters:
  • export_directory – path to export directory. Will be created if it does not exist.
  • app – Galaxy App or app-like object. Must be provided if for_edit and/or serialize_dataset_objects are True
  • for_edit – Allow modifying existing HDA and dataset metadata during import.
  • serialize_dataset_objects – If True will encode IDs using the host secret. Defaults for_edit.
  • export_files – How files should be exported, can be ‘symlink’, ‘copy’ or None, in which case files will not be serialized.
  • serialize_jobs – Include job data in model export. Not needed for set_metadata script.
add_dataset(dataset: galaxy.model.DatasetInstance, include_files: bool = True)[source]
add_dataset_collection(collection: typing.Union[galaxy.model.DatasetCollection, galaxy.model.HistoryDatasetCollectionAssociation])[source]
add_job_output_dataset_associations(job_id, name, dataset_instance)[source]
export_collection(collection: typing.Union[galaxy.model.DatasetCollection, galaxy.model.HistoryDatasetCollectionAssociation], include_deleted: bool = False)[source]
export_history(history, include_hidden=False, include_deleted=False)[source]
export_job(job: galaxy.model.Job, tool=None, include_job_data=True)[source]
export_jobs(jobs: typing.List[galaxy.model.Job], jobs_attrs=None, include_job_data=True)[source]

Export jobs.

include_job_data determines whether datasets associated with jobs should be exported as well. This should generally be True, except when re-exporting a job (to store the generated command line) when running the set_meta script.

export_library(library, include_hidden=False, include_deleted=False)[source]
export_library_folder(library_folder, include_hidden=False, include_deleted=False)[source]
exported_key(obj)[source]
serialize_files(dataset, as_dict)[source]
exception galaxy.model.store.FileTracebackException(traceback, *args, **kwargs)[source]

Bases: Exception

__init__(traceback, *args, **kwargs)[source]
class galaxy.model.store.ImportOptions(allow_edit=False, allow_library_creation=False, allow_dataset_object_edit=None)[source]

Bases: object

__init__(allow_edit=False, allow_library_creation=False, allow_dataset_object_edit=None)[source]
class galaxy.model.store.ModelExportStore[source]

Bases: object

add_dataset(dataset: galaxy.model.DatasetInstance, include_files: bool = True)[source]

Add HDA to export store.

include_files controls whether file contents are exported as well.

add_dataset_collection(collection: typing.Union[galaxy.model.DatasetCollection, galaxy.model.HistoryDatasetCollectionAssociation])[source]

Add Dataset Collection or HDCA to export store.

export_history(history: galaxy.model.History, include_hidden: bool = False, include_deleted: bool = False)[source]

Export history to store.

class galaxy.model.store.ModelImportStore(import_options=None, app=None, user=None, object_store=None, tag_handler=None)[source]

Bases: object

__init__(import_options=None, app=None, user=None, object_store=None, tag_handler=None)[source]
collections_properties() → typing.List[typing.Dict[str, typing.Any]][source]

Return a list of HDCA properties.

datasets_properties() → typing.List[typing.Dict[str, typing.Any]][source]

Return a list of HDA properties.

defines_new_history() → bool[source]

Does this store define a new history to create.

file_source_root

Source of valid file data.

jobs_properties() → typing.List[typing.Dict[str, typing.Any]][source]

Return a list of jobs properties.

library_properties() → typing.List[typing.Dict[str, typing.Any]][source]

Return a list of library properties.

new_history_properties() → typing.Dict[str, typing.Any][source]

Dict of history properties if defines_new_history() is truthy.

object_key

Key used to connect objects in metadata.

Legacy exports used ‘hid’ but associated objects may not be from the same history and a history may contain multiple objects with the same ‘hid’.

perform_import(history=None, new_history=False, job=None)[source]
target_history(default_history=None)[source]
trust_hid(obj_attrs) → bool[source]

Trust HID when importing objects into a new History.

class galaxy.model.store.ObjectImportTracker[source]

Bases: object

Keep track of new and existing imported objects.

Needed to re-establish connections and such in multiple passes.

__init__()[source]
find_dce(input_key: typing.Union[str, int]) → typing.Optional[galaxy.model.DatasetCollectionElement][source]
find_hda(input_key: typing.Union[str, int], hda_id: typing.Optional[int] = None) → typing.Optional[galaxy.model.HistoryDatasetAssociation][source]
find_hdca(input_key: typing.Union[str, int]) → typing.Optional[galaxy.model.HistoryDatasetCollectionAssociation][source]
class galaxy.model.store.SessionlessContext[source]

Bases: object

__init__()[source]
add(obj)[source]
flush()[source]
query(model_class)[source]
class galaxy.model.store.TarModelExportStore(out_file, gzip=True, **kwds)[source]

Bases: galaxy.model.store.DirectoryModelExportStore

__init__(out_file, gzip=True, **kwds)[source]
galaxy.model.store.get_export_dataset_filename(name, ext, hid)[source]

Builds a filename for a dataset using its name an extension.

galaxy.model.store.get_import_model_store_for_directory(archive_dir, **kwd)[source]
galaxy.model.store.imported_store_for_metadata(directory, object_store=None)[source]
galaxy.model.store.replace_metadata_file(metadata: typing.Dict[str, typing.Any], dataset_instance: galaxy.model.DatasetInstance)[source]
galaxy.model.store.tar_export_directory(export_directory, out_file, gzip)[source]

Submodules

galaxy.model.store.build_objects module

galaxy.model.store.build_objects.main(argv=None)[source]

galaxy.model.store.discover module

Utilities for discovering files to add to a model store.

Working with input “JSON” format used for Fetch API, galaxy.json imports, etc… High-level utilities in this file can be used during job output discovery or for persisting Galaxy model objects corresponding to files in other contexts.

class galaxy.model.store.discover.DiscoveredFile(path, collector, match)

Bases: tuple

collector

Alias for field number 1

match

Alias for field number 2

path

Alias for field number 0

class galaxy.model.store.discover.DiscoveredFileError(error_message, collector, match, path)[source]

Bases: tuple

collector

Alias for field number 1

error_message

Alias for field number 0

match

Alias for field number 2

path

Alias for field number 3

class galaxy.model.store.discover.JsonCollectedDatasetMatch(as_dict, collector, filename, path=None, parent_identifiers=None)[source]

Bases: object

__init__(as_dict, collector, filename, path=None, parent_identifiers=None)[source]
created_from_basename
dbkey
designation
element_identifiers
ext
extra_files
hashes
name

Return name or None if not defined by the discovery pattern.

object_id
raw_element_identifiers
sources
tag_list
visible
exception galaxy.model.store.discover.MaxDiscoveredFilesExceededError[source]

Bases: ValueError

class galaxy.model.store.discover.MetadataSourceProvider[source]

Bases: object

Interface for working with fetching input dataset metadata with ModelPersistenceContext.

get_metadata_source(input_name)[source]

Get metadata for supplied input_name.

class galaxy.model.store.discover.ModelPersistenceContext[source]

Bases: object

Class for creating datasets while finding files.

This class implement the create_dataset method that takes care of populating metadata required for datasets and other potential model objects.

add_datasets_to_history(datasets, for_output_dataset=None)[source]

Add datasets to the history this context points at.

add_library_dataset_to_folder(library_folder, ld)[source]

Add library dataset to persisted library folder.

add_output_dataset_association(name, dataset)[source]

If discovering outputs for a job, persist output dataset association.

add_tags_to_datasets(datasets, tag_lists)[source]
create_dataset(ext, designation, visible, dbkey, name, filename=None, extra_files=None, metadata_source_name=None, info=None, library_folder=None, link_data=False, primary_data=None, init_from=None, dataset_attributes=None, tag_list=None, sources=None, hashes=None, created_from_basename=None, final_job_state='ok', creating_job_id=None, storage_callbacks=None)[source]
create_library_folder(parent_folder, name, description)[source]

Create a library folder ready from supplied attributes for supplied parent.

finalize_storage(primary_data, dataset_attributes, extra_files, filename, link_data)[source]
flush()[source]

If database bound, flush the persisted objects to ensure IDs.

flush_per_n_datasets
get_implicit_collection_jobs_association_id() → typing.Optional[str][source]

No-op, no job context.

increment_discovered_file_count()[source]
input_dbkey
job

Return associated job object if bound to a job finish context connected to a database.

job_id()[source]
max_discovered_files = inf
metadata_source_provider

Return associated MetadataSourceProvider object.

object_store

Return object store to use for populating discovered dataset contents.

permission_provider

If bound to a database, return the SQL Alchemy session.

Return None otherwise.

persist_library_folder(library_folder)[source]

Add library folder to sessionless export. Noop for session export.

persist_object(obj)[source]

Add the target to the persistence layer.

populate_collection_elements(collection, root_collection_builder, filenames, name=None, metadata_source_name=None, final_job_state='ok')[source]
sa_session

If bound to a database, return the SQL Alchemy session.

Return None otherwise.

static set_datasets_metadata(datasets, datasets_attributes=None)[source]
tag_handler

Return a galaxy.model.tags.TagHandler-like object for persisting tags.

update_object_store_with_datasets(datasets, paths, extra_files)[source]
user

If bound to a database, return the user the datasets should be created for.

Return None otherwise.

class galaxy.model.store.discover.PermissionProvider[source]

Bases: object

Interface for working with permissions while importing datasets with ModelPersistenceContext.

copy_dataset_permissions(init_from, primary_data)[source]

Copy dataset permissions from supplied input dataset.

permissions
set_default_hda_permissions(primary_data)[source]
class galaxy.model.store.discover.RegexCollectedDatasetMatch(re_match, collector, filename, path=None)[source]

Bases: galaxy.model.store.discover.JsonCollectedDatasetMatch

__init__(re_match, collector, filename, path=None)[source]
class galaxy.model.store.discover.SessionlessModelPersistenceContext(object_store, export_store, working_directory)[source]

Bases: galaxy.model.store.discover.ModelPersistenceContext

A variant of ModelPersistenceContext that persists to an export store instead of database directly.

__init__(object_store, export_store, working_directory)[source]
add_datasets_to_history(datasets, for_output_dataset=None)[source]
add_library_dataset_to_folder(library_folder, ld)[source]
add_output_dataset_association(name, dataset)[source]

No-op, no job context to persist this association for.

add_tags_to_datasets(datasets, tag_lists)[source]
create_hdca(name, structure)[source]
create_library_folder(parent_folder, name, description)[source]
flush()[source]

No-op for the sessionless variant of this, no database to flush.

flush_per_n_datasets
get_hdca(object_id)[source]
get_implicit_collection_jobs_association_id()[source]

No-op, no job context.

get_library_folder(destination)[source]
job
metadata_source_provider
object_store
permission_provider
persist_library_folder(library_folder)[source]
persist_object(obj)[source]

No-op right now for the sessionless variant of this.

This works currently because either things are added to a target history with add_datasets_to_history or the parent LibraryFolder was added to the export store in persist_target_to_export_store.

sa_session
tag_handler
user
class galaxy.model.store.discover.UnusedMetadataSourceProvider[source]

Bases: galaxy.model.store.discover.MetadataSourceProvider

get_metadata_source(input_name)[source]

Throws NotImplementedError.

This should only be called as part of job output collection where one can actually collect metadata from inputs, this is unused in the context of SessionlessModelPersistenceContext.

class galaxy.model.store.discover.UnusedPermissionProvider[source]

Bases: galaxy.model.store.discover.PermissionProvider

copy_dataset_permissions(init_from, primary_data)[source]

Throws NotImplementedError.

This should only be called as part of job output collection where there should be a session available to initialize this from.

galaxy.model.store.discover.discover_target_directory(dir_name, job_working_directory)[source]
galaxy.model.store.discover.discovered_file_for_element()[source]
galaxy.model.store.discover.get_required_item(from_dict, key, message)[source]
galaxy.model.store.discover.persist_elements_to_folder(model_persistence_context, elements, library_folder)[source]
galaxy.model.store.discover.persist_elements_to_hdca()[source]
galaxy.model.store.discover.persist_extra_files(object_store, src_extra_files_path, primary_data)[source]
galaxy.model.store.discover.persist_hdas(elements, model_persistence_context, final_job_state='ok')[source]
galaxy.model.store.discover.persist_target_to_export_store(target_dict, export_store, object_store, work_directory)[source]
galaxy.model.store.discover.replace_request_syntax_sugar(obj)[source]
galaxy.model.store.discover.validate_and_normalize_target(obj)[source]