Warning

This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.

galaxy.files.sources package

class galaxy.files.sources.PluginKind(value)[source]

Bases: str, Enum

Enum to distinguish between different kinds or categories of plugins.

rfs = 'rfs'

Remote File System

A remote file system is a file source that is backed by a remote file system that is accessible via a URI. Examples include FTP, SFTP, S3, etc.

drs = 'drs'

Data Repository Service

A data repository service is a file source that is backed by a remote data repository service implementing the (DRS) API which provides a generic interface to data repositories so data consumers, including workflow systems, can access data in a single, standard way regardless of where it’s stored and how it’s managed.

rdm = 'rdm'

Research Data Management

A research data management file source is a file source that is backed by a remote research data management system that can provide DOIs. Examples include InvenioRDM, Zenodo, etc.

stock = 'stock'

Stock Plugin

A stock plugin is a file source that is shipped with Galaxy and covers common use cases. Examples include the UserFTP, LibraryImport, UserLibraryImport, etc.

class galaxy.files.sources.SingleFileSource[source]

Bases: object

Represents a protocol handler for a single remote file that can be read by or written to by Galaxy. A remote file source can typically handle a url like https://galaxyproject.org/myfile.txt or drs://myserver/123456. The filesource abstraction allows programmatic control over the specific source to access, injection of credentials and access control. File sources are typically listed and configured through file_sources_conf.yml or programmatically, as required.

File sources can be contextualized with a user_context, which contains information related to the current user attempting to access that filesource such as the username, preferences, roles etc., which can then be used by the filesource to make authorization decisions or inject credentials.

File sources are loaded through Galaxy’s plugin system in galaxy.util.plugin_config.

abstract get_writable() → bool[source]

Return a boolean indicating whether this target is writable.

abstract user_has_access(user_context: OptionalUserContext) → bool[source]

Return a boolean indicating whether the user can access the FileSource.

abstract realize_to(source_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None)[source]

Realize source path (relative to uri root) to local file system path.

Parameters:

  • source_path (str) – url of the source file to copy from. e.g. https://galaxyproject.org/myfile.txt

  • native_path (str) – local path to write to. e.g. /tmp/myfile.txt

  • user_context (OptionalUserContext, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the realize_to method. Filesource specific, defaults to None

abstract write_from(target_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None) → str[source]

Write file at native path to target_path (relative to uri root).

Parameters:

  • target_path (str) – url of the target file to write to within the filesource. e.g. gxfiles://myftp1/myfile.txt

  • native_path (str) – The local file to read. e.g. /tmp/myfile.txt

  • user_context (_type_, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the write_from method. Filesource specific, defaults to None

Returns:

Actual url of the written file, fixed by the service backing the FileSource. May differ from the target path.

Return type:

str

abstract score_url_match(url: str) → int[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

abstract to_relative_path(url: str) → str[source]

Convert this url to a filesource relative path. For example, given the url gxfiles://mysource1/myfile.txt it will return /myfile.txt. Protocols directly understood by the handler need not be relativized. For example, the url s3://bucket/myfile.txt can be returned unchanged.

abstract to_dict(for_serialization=False, user_context: OptionalUserContext = None) → dict[str, Any][source]

Return a dictified representation of this FileSource instance.

If user_context is supplied, properties should be written so user context doesn’t need to be present after the plugin is re-hydrated.

Prefer linking to files.

class galaxy.files.sources.SupportsBrowsing[source]

Bases: object

An interface indicating that this filesource is browsable.

Browsable file sources will typically have a root uri from which to start browsing. e.g. In an s3 bucket, the root uri may be gxfiles://bucket1/

They will also have a list method to list files in a specific path within the filesource.

abstract get_uri_root() → str[source]

Return a prefix for the root (e.g. gxfiles://prefix/).

abstract list(path='/', recursive=False, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None, limit: int | None = None, offset: int | None = None, query: str | None = None, sort_by: str | None = None) → tuple[list[RemoteDirectory | RemoteFile], int][source]

Return a list of ‘Directory’s and ‘File’s and the total count in a tuple.

class galaxy.files.sources.FilesSource[source]

Bases: SingleFileSource, SupportsBrowsing

Represents a combined interface for single or browsable file sources. The get_browsable method can be used to determine whether the filesource is browsable and implements the SupportsBrowsing interface.

plugin_type: ClassVar[str]
abstract get_browsable() → bool[source]

Return true if the filesource implements the SupportsBrowsing interface.

galaxy.files.sources.file_source_type_is_browsable(target_type: type[BaseFilesSource]) → bool[source]
class galaxy.files.sources.BaseFilesSource(template_config: TTemplateConfig)[source]

Bases: FilesSource, Generic[TTemplateConfig, TResolvedConfig]

A base class for file sources that can resolve a template configuration to a specific configuration.

Implementations of this class should define 2 configuration models and assign them to the template_config_class and resolved_config_class class variables.

The template_config_class should be a subclass of BaseFileSourceTemplateConfiguration and the resolved_config_class should be a subclass of BaseFileSourceConfiguration.

Assuming a File Source named MyFileSource: - MyFileSourceTemplateConfiguration: A template configuration class that defines the template variables. - MyFileSourceConfiguration: A resolved configuration class that defines the final configuration after template resolution.

plugin_kind: ClassVar[PluginKind] = 'rfs'
supports_pagination: ClassVar[bool] = False
supports_sorting: ClassVar[bool] = False
template_config_class: type[TTemplateConfig]
resolved_config_class: type[TResolvedConfig]
__init__(template_config: TTemplateConfig)[source]
classmethod build_template_config(**kwargs) → TTemplateConfig[source]

Build a template configuration instance from the provided keyword arguments.

get_browsable() → bool[source]

Return true if the filesource implements the SupportsBrowsing interface.

get_prefix() → str | None[source]
get_scheme() → str[source]
get_writable() → bool[source]

Return a boolean indicating whether this target is writable.

user_has_access(user_context: OptionalUserContext) → bool[source]

Return a boolean indicating whether the user can access the FileSource.

Prefer linking to files.

property user_context_required: bool
get_uri_root() → str[source]

Return a prefix for the root (e.g. gxfiles://prefix/).

get_url() → str | None[source]

Returns a URL that can be used to link to the remote source.

to_relative_path(url: str) → str[source]

Convert this url to a filesource relative path. For example, given the url gxfiles://mysource1/myfile.txt it will return /myfile.txt. Protocols directly understood by the handler need not be relativized. For example, the url s3://bucket/myfile.txt can be returned unchanged.

score_url_match(url: str) → int[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

uri_from_path(path: str) → str[source]
to_dict(for_serialization=False, user_context: OptionalUserContext = None) → dict[str, Any][source]

Return a dictified representation of this FileSource instance.

If user_context is supplied, properties should be written so user context doesn’t need to be present after the plugin is re-hydrated.

to_dict_time(ctime) → str | None[source]
list(path='/', recursive=False, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None, limit: int | None = None, offset: int | None = None, query: str | None = None, sort_by: str | None = None) → tuple[list[RemoteDirectory | RemoteFile], int][source]

Return a list of ‘Directory’s and ‘File’s and the total count in a tuple.

create_entry(entry_data: EntryData, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None) → Entry[source]
write_from(target_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None) → str[source]

Write file at native path to target_path (relative to uri root).

Parameters:

  • target_path (str) – url of the target file to write to within the filesource. e.g. gxfiles://myftp1/myfile.txt

  • native_path (str) – The local file to read. e.g. /tmp/myfile.txt

  • user_context (_type_, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the write_from method. Filesource specific, defaults to None

Returns:

Actual url of the written file, fixed by the service backing the FileSource. May differ from the target path.

Return type:

str

realize_to(source_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None)[source]

Realize source path (relative to uri root) to local file system path.

Parameters:

  • source_path (str) – url of the source file to copy from. e.g. https://galaxyproject.org/myfile.txt

  • native_path (str) – local path to write to. e.g. /tmp/myfile.txt

  • user_context (OptionalUserContext, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the realize_to method. Filesource specific, defaults to None

galaxy.files.sources.uri_join(*args: str) → str[source]
galaxy.files.sources.slash_join(*args: str) → str[source]
class galaxy.files.sources.DefaultBaseFilesSource(template_config: TTemplateConfig)[source]

Bases: BaseFilesSource[BaseFileSourceTemplateConfiguration, BaseFileSourceConfiguration]

A default implementation of BaseFilesSource that uses the base configuration models without any custom configuration.

template_config_class

alias of BaseFileSourceTemplateConfiguration

resolved_config_class

alias of BaseFileSourceConfiguration

Submodules

galaxy.files.sources.anvil module

class galaxy.files.sources.anvil.AnVILFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[AnVILFileSourceTemplateConfiguration, AnVILFileSourceConfiguration]

plugin_type: ClassVar[str] = 'anvil'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs.anvilfs'
template_config_class

alias of AnVILFileSourceTemplateConfiguration

resolved_config_class

alias of AnVILFileSourceConfiguration

galaxy.files.sources.basespace module

class galaxy.files.sources.basespace.BaseSpaceFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[BaseSpaceFileSourceTemplateConfiguration, BaseSpaceFileSourceConfiguration]

plugin_type: ClassVar[str] = 'basespace'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs-basespace'
template_config_class

alias of BaseSpaceFileSourceTemplateConfiguration

resolved_config_class

alias of BaseSpaceFileSourceConfiguration

galaxy.files.sources.dropbox module

class galaxy.files.sources.dropbox.DropboxFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[DropboxFileSourceTemplateConfiguration, DropboxFilesSourceConfiguration]

plugin_type: ClassVar[str] = 'dropbox'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs.dropboxfs'
template_config_class

alias of DropboxFileSourceTemplateConfiguration

resolved_config_class

alias of DropboxFilesSourceConfiguration

galaxy.files.sources.ftp module

class galaxy.files.sources.ftp.FtpFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[FTPFileSourceTemplateConfiguration, FTPFileSourceConfiguration]

plugin_type: ClassVar[str] = 'ftp'
required_module

alias of FTPFS

required_package: ClassVar[str] = 'fs.ftpfs'
template_config_class

alias of FTPFileSourceTemplateConfiguration

resolved_config_class

alias of FTPFileSourceConfiguration

score_url_match(url: str)[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

galaxy.files.sources.galaxy module

Static Galaxy file sources - ftp and libraries.

class galaxy.files.sources.galaxy.UserFtpFilesSource(template_config: PosixTemplateConfiguration)[source]

Bases: PosixFilesSource

plugin_type: ClassVar[str] = 'gxftp'
plugin_kind: ClassVar[PluginKind] = 'stock'
__init__(template_config: PosixTemplateConfiguration)[source]
get_prefix() → str | None[source]
get_scheme() → str[source]
class galaxy.files.sources.galaxy.LibraryImportFilesSource(template_config: PosixTemplateConfiguration)[source]

Bases: PosixFilesSource

plugin_type: ClassVar[str] = 'gximport'
plugin_kind: ClassVar[PluginKind] = 'stock'
__init__(template_config: PosixTemplateConfiguration)[source]
get_prefix() → str | None[source]
get_scheme() → str[source]
class galaxy.files.sources.galaxy.UserLibraryImportFilesSource(template_config: PosixTemplateConfiguration)[source]

Bases: PosixFilesSource

plugin_type: ClassVar[str] = 'gxuserimport'
plugin_kind: ClassVar[PluginKind] = 'stock'
__init__(template_config: PosixTemplateConfiguration)[source]
get_prefix() → str | None[source]
get_scheme() → str[source]

galaxy.files.sources.googlecloudstorage module

class galaxy.files.sources.googlecloudstorage.GoogleCloudStorageFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[GoogleCloudStorageFileSourceTemplateConfiguration, GoogleCloudStorageFileSourceConfiguration]

plugin_type: ClassVar[str] = 'googlecloudstorage'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs-gcsfs'
template_config_class

alias of GoogleCloudStorageFileSourceTemplateConfiguration

resolved_config_class

alias of GoogleCloudStorageFileSourceConfiguration

galaxy.files.sources.googledrive module

class galaxy.files.sources.googledrive.GoogleDriveFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[GoogleDriveFileSourceTemplateConfiguration, GoogleDriveFilesSourceConfiguration]

plugin_type: ClassVar[str] = 'googledrive'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs.googledrivefs'
template_config_class

alias of GoogleDriveFileSourceTemplateConfiguration

resolved_config_class

alias of GoogleDriveFilesSourceConfiguration

galaxy.files.sources.onedata module

class galaxy.files.sources.onedata.OnedataFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[OnedataFileSourceTemplateConfiguration, OnedataFileSourceConfiguration]

plugin_type: ClassVar[str] = 'onedata'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs.onedatarestfs'
template_config_class

alias of OnedataFileSourceTemplateConfiguration

resolved_config_class

alias of OnedataFileSourceConfiguration

galaxy.files.sources.posix module

class galaxy.files.sources.posix.PosixFilesSource(template_config: PosixTemplateConfiguration)[source]

Bases: BaseFilesSource[PosixTemplateConfiguration, PosixConfiguration]

plugin_type: ClassVar[str] = 'posix'
template_config_class

alias of PosixTemplateConfiguration

resolved_config_class

alias of PosixConfiguration

__init__(template_config: PosixTemplateConfiguration)[source]

Prefer linking to files.

property root: str | None

Return the root directory for backward compatibility.

score_url_match(url: str)[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

to_relative_path(url: str) → str[source]

Convert this url to a filesource relative path. For example, given the url gxfiles://mysource1/myfile.txt it will return /myfile.txt. Protocols directly understood by the handler need not be relativized. For example, the url s3://bucket/myfile.txt can be returned unchanged.

galaxy.files.sources.s3 module

galaxy.files.sources.s3fs module

class galaxy.files.sources.s3fs.S3FsFilesSource(template_config: FsspecTemplateConfigType)[source]

Bases: FsspecFilesSource[S3FSFileSourceTemplateConfiguration, S3FSFileSourceConfiguration]

plugin_type: ClassVar[str] = 's3fs'
required_module

alias of S3FileSystem

required_package: ClassVar[str] = 's3fs'
template_config_class

alias of S3FSFileSourceTemplateConfiguration

resolved_config_class

alias of S3FSFileSourceConfiguration

score_url_match(url: str)[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

galaxy.files.sources.ssh module

class galaxy.files.sources.ssh.SshFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[SshFileSourceTemplateConfiguration, SshFileSourceConfiguration]

plugin_type: ClassVar[str] = 'ssh'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs.sshfs'
template_config_class

alias of SshFileSourceTemplateConfiguration

resolved_config_class

alias of SshFileSourceConfiguration

galaxy.files.sources.webdav module

class galaxy.files.sources.webdav.WebDavFilesSource(template_config: TTemplateConfig)[source]

Bases: PyFilesystem2FilesSource[WebDavFileSourceTemplateConfiguration, WebDavFileSourceConfiguration]

plugin_type: ClassVar[str] = 'webdav'
required_module: ClassVar[type[FS] | None] = None
required_package: ClassVar[str] = 'fs.webdavfs'
allow_key_error_on_empty_directories = True
template_config_class

alias of WebDavFileSourceTemplateConfiguration

resolved_config_class

alias of WebDavFileSourceConfiguration