galaxy.files.sources package

class galaxy.files.sources.PluginKind(value)[source]

Bases: str, Enum

Enum to distinguish between different kinds or categories of plugins.

rfs = 'rfs'

Remote File System

A remote file system is a file source that is backed by a remote file system that is accessible via a URI. Examples include FTP, SFTP, S3, etc.

drs = 'drs'

Data Repository Service

A data repository service is a file source that is backed by a remote data repository service implementing the (DRS) API which provides a generic interface to data repositories so data consumers, including workflow systems, can access data in a single, standard way regardless of where it’s stored and how it’s managed.

rdm = 'rdm'

Research Data Management

A research data management file source is a file source that is backed by a remote research data management system that can provide DOIs. Examples include InvenioRDM, Zenodo, etc.

stock = 'stock'

Stock Plugin

A stock plugin is a file source that is shipped with Galaxy and covers common use cases. Examples include the UserFTP, LibraryImport, UserLibraryImport, etc.

class galaxy.files.sources.FileSourceSupports[source]

Bases: TypedDict

Feature support flags for a file source plugin

pagination: typing_extensions.NotRequired[bool]
search: typing_extensions.NotRequired[bool]
sorting: typing_extensions.NotRequired[bool]
class galaxy.files.sources.FilesSourceProperties[source]

Bases: TypedDict

Initial set of properties used to initialize a filesource.

Filesources can extend this typed dict to define any additional filesource specific properties.

file_sources_config: typing_extensions.NotRequired[FileSourcePluginsConfig]
id: typing_extensions.NotRequired[str]
label: typing_extensions.NotRequired[str]
doc: typing_extensions.NotRequired[str | None]
scheme: typing_extensions.NotRequired[str]
writable: typing_extensions.NotRequired[bool]
requires_roles: typing_extensions.NotRequired[str | None]
requires_groups: typing_extensions.NotRequired[str | None]
disable_templating: typing_extensions.NotRequired[bool | None]
uri_root: typing_extensions.NotRequired[str]
type: typing_extensions.NotRequired[str]
browsable: typing_extensions.NotRequired[bool]
url: typing_extensions.NotRequired[str | None]
supports: typing_extensions.NotRequired[FileSourceSupports]
class galaxy.files.sources.FilesSourceOptions(writeable: bool | None = False, extra_props: ~galaxy.files.sources.FilesSourceProperties | None = <factory>)[source]

Bases: object

Options to control behavior of file source operations, such as realize_to, write_from and list.

writeable: bool | None = False
extra_props: FilesSourceProperties | None
__init__(writeable: bool | None = False, extra_props: ~galaxy.files.sources.FilesSourceProperties | None = <factory>) → None
class galaxy.files.sources.EntryData[source]

Bases: TypedDict

Provides data to create a new entry in a file source.

name: str
class galaxy.files.sources.Entry[source]

Bases: TypedDict

Represents the result of creating a new entry in a file source.

name: str
uri: str
class galaxy.files.sources.RemoteEntry[source]

Bases: TypedDict

name: str
uri: str
path: str
class galaxy.files.sources.TDirectoryClass

Bases: TypedDict

class: typing_extensions.Literal[Directory]
class galaxy.files.sources.TFileClass

Bases: TypedDict

class: typing_extensions.Literal[File]
class galaxy.files.sources.RemoteDirectory[source]

Bases: RemoteEntry, TDirectoryClass

name: str
uri: str
path: str
class: typing_extensions.Literal[Directory]
class galaxy.files.sources.RemoteFile[source]

Bases: RemoteEntry, TFileClass

size: int
ctime: str
name: str
uri: str
path: str
class: typing_extensions.Literal[File]
class galaxy.files.sources.SingleFileSource[source]

Bases: object

Represents a protocol handler for a single remote file that can be read by or written to by Galaxy. A remote file source can typically handle a url like https://galaxyproject.org/myfile.txt or drs://myserver/123456. The filesource abstraction allows programmatic control over the specific source to access, injection of credentials and access control. Filesources are typically listed and configured through file_sources_conf.yml or programmatically, as required.

Filesources can be contextualized with a user_context, which contains information related to the current user attempting to access that filesource such as the username, preferences, roles etc., which can then be used by the filesource to make authorization decisions or inject credentials.

Filesources are loaded through Galaxy’s plugin system in galaxy.util.plugin_config.

abstract get_writable() → bool[source]

Return a boolean indicating whether this target is writable.

abstract user_has_access(user_context: OptionalUserContext) → bool[source]

Return a boolean indicating whether the user can access the FileSource.

abstract realize_to(source_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None)[source]

Realize source path (relative to uri root) to local file system path.

Parameters:

  • source_path (str) – url of the source file to copy from. e.g. https://galaxyproject.org/myfile.txt

  • native_path (str) – local path to write to. e.g. /tmp/myfile.txt

  • user_context (OptionalUserContext, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the realize_to method. Filesource specific, defaults to None

abstract write_from(target_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None) → str[source]

Write file at native path to target_path (relative to uri root).

Parameters:

  • target_path (str) – url of the target file to write to within the filesource. e.g. gxfiles://myftp1/myfile.txt

  • native_path (str) – The local file to read. e.g. /tmp/myfile.txt

  • user_context (_type_, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the write_from method. Filesource specific, defaults to None

Returns:

Actual url of the written file, fixed by the service backing the FileSource. May differ from the target path.

Return type:

str

abstract score_url_match(url: str) → int[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

abstract to_relative_path(url: str) → str[source]

Convert this url to a filesource relative path. For example, given the url gxfiles://mysource1/myfile.txt it will return /myfile.txt. Protocols directly understood by the handler need not be relativized. For example, the url s3://bucket/myfile.txt can be returned unchanged.

abstract to_dict(for_serialization=False, user_context: OptionalUserContext = None) → FilesSourceProperties[source]

Return a dictified representation of this FileSource instance.

If user_context is supplied, properties should be written so user context doesn’t need to be present after the plugin is re-hydrated.

Prefer linking to files.

class galaxy.files.sources.SupportsBrowsing[source]

Bases: object

An interface indicating that this filesource is browsable.

Browsable filesources will typically have a root uri from which to start browsing. e.g. In an s3 bucket, the root uri may be gxfiles://bucket1/

They will also have a list method to list files in a specific path within the filesource.

abstract get_uri_root() → str[source]

Return a prefix for the root (e.g. gxfiles://prefix/).

abstract list(path='/', recursive=False, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None, limit: int | None = None, offset: int | None = None, query: str | None = None, sort_by: str | None = None) → Tuple[List[RemoteDirectory | RemoteFile], int][source]

Return a list of ‘Directory’s and ‘File’s and the total count in a tuple.

class galaxy.files.sources.FilesSource[source]

Bases: SingleFileSource, SupportsBrowsing

Represents a combined interface for single or browsable filesources. The get_browsable method can be used to determine whether the filesource is browsable and implements the SupportsBrowsing interface.

plugin_type: ClassVar[str]
abstract get_browsable() → bool[source]

Return true if the filesource implements the SupportsBrowsing interface.

galaxy.files.sources.file_source_type_is_browsable(target_type: Type[BaseFilesSource]) → bool[source]
class galaxy.files.sources.BaseFilesSource[source]

Bases: FilesSource

plugin_kind: ClassVar[PluginKind] = 'rfs'
supports_pagination: ClassVar[bool] = False
supports_sorting: ClassVar[bool] = False
get_browsable() → bool[source]

Return true if the filesource implements the SupportsBrowsing interface.

get_prefix() → str | None[source]
get_scheme() → str[source]
get_writable() → bool[source]

Return a boolean indicating whether this target is writable.

user_has_access(user_context: OptionalUserContext) → bool[source]

Return a boolean indicating whether the user can access the FileSource.

Prefer linking to files.

property user_context_required: bool
get_uri_root() → str[source]

Return a prefix for the root (e.g. gxfiles://prefix/).

get_url() → str | None[source]

Returns a URL that can be used to link to the remote source.

to_relative_path(url: str) → str[source]

Convert this url to a filesource relative path. For example, given the url gxfiles://mysource1/myfile.txt it will return /myfile.txt. Protocols directly understood by the handler need not be relativized. For example, the url s3://bucket/myfile.txt can be returned unchanged.

score_url_match(url: str) → int[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

uri_from_path(path: str) → str[source]
to_dict(for_serialization=False, user_context: OptionalUserContext = None) → FilesSourceProperties[source]

Return a dictified representation of this FileSource instance.

If user_context is supplied, properties should be written so user context doesn’t need to be present after the plugin is re-hydrated.

to_dict_time(ctime)[source]
list(path='/', recursive=False, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None, limit: int | None = None, offset: int | None = None, query: str | None = None, sort_by: str | None = None) → Tuple[List[RemoteDirectory | RemoteFile], int][source]

Return a list of ‘Directory’s and ‘File’s and the total count in a tuple.

create_entry(entry_data: EntryData, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None) → Entry[source]
write_from(target_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None) → str[source]

Write file at native path to target_path (relative to uri root).

Parameters:

  • target_path (str) – url of the target file to write to within the filesource. e.g. gxfiles://myftp1/myfile.txt

  • native_path (str) – The local file to read. e.g. /tmp/myfile.txt

  • user_context (_type_, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the write_from method. Filesource specific, defaults to None

Returns:

Actual url of the written file, fixed by the service backing the FileSource. May differ from the target path.

Return type:

str

realize_to(source_path: str, native_path: str, user_context: OptionalUserContext = None, opts: FilesSourceOptions | None = None)[source]

Realize source path (relative to uri root) to local file system path.

Parameters:

  • source_path (str) – url of the source file to copy from. e.g. https://galaxyproject.org/myfile.txt

  • native_path (str) – local path to write to. e.g. /tmp/myfile.txt

  • user_context (OptionalUserContext, optional) – A user context , defaults to None

  • opts (Optional[FilesSourceOptions], optional) – A set of options to exercise additional control over the realize_to method. Filesource specific, defaults to None

galaxy.files.sources.uri_join(*args: str) → str[source]
galaxy.files.sources.slash_join(*args: str) → str[source]

Submodules

galaxy.files.sources.anvil module

class galaxy.files.sources.anvil.AnVILFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'anvil'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs.anvilfs'

galaxy.files.sources.basespace module

class galaxy.files.sources.basespace.BaseSpaceFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'basespace'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs-basespace'

galaxy.files.sources.dropbox module

class galaxy.files.sources.dropbox.DropboxFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'dropbox'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs.dropboxfs'

galaxy.files.sources.ftp module

class galaxy.files.sources.ftp.FtpFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'ftp'
required_module

alias of FTPFS

required_package: ClassVar[str] = 'fs.ftpfs'
score_url_match(url: str)[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

galaxy.files.sources.galaxy module

Static Galaxy file sources - ftp and libraries.

class galaxy.files.sources.galaxy.UserFtpFilesSource(**kwd: Unpack[PosixFilesSourceProperties])[source]

Bases: PosixFilesSource

plugin_type: ClassVar[str] = 'gxftp'
plugin_kind: ClassVar[PluginKind] = 'stock'
__init__(**kwd: Unpack[PosixFilesSourceProperties])[source]
get_prefix() → str | None[source]
get_scheme() → str[source]
class galaxy.files.sources.galaxy.LibraryImportFilesSource(**kwd: Unpack[PosixFilesSourceProperties])[source]

Bases: PosixFilesSource

plugin_type: ClassVar[str] = 'gximport'
plugin_kind: ClassVar[PluginKind] = 'stock'
__init__(**kwd: Unpack[PosixFilesSourceProperties])[source]
get_prefix() → str | None[source]
get_scheme() → str[source]
class galaxy.files.sources.galaxy.UserLibraryImportFilesSource(**kwd: Unpack[PosixFilesSourceProperties])[source]

Bases: PosixFilesSource

plugin_type: ClassVar[str] = 'gxuserimport'
plugin_kind: ClassVar[PluginKind] = 'stock'
__init__(**kwd: Unpack[PosixFilesSourceProperties])[source]
get_prefix() → str | None[source]
get_scheme() → str[source]

galaxy.files.sources.googlecloudstorage module

class galaxy.files.sources.googlecloudstorage.GoogleCloudStorageFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'googlecloudstorage'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs-gcsfs'

galaxy.files.sources.googledrive module

class galaxy.files.sources.googledrive.GoogleDriveFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'googledrive'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs.googledrivefs'

galaxy.files.sources.onedata module

class galaxy.files.sources.onedata.OnedataFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'onedata'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs.onedatarestfs'

galaxy.files.sources.posix module

class galaxy.files.sources.posix.PosixFilesSource(**kwd: Unpack[PosixFilesSourceProperties])[source]

Bases: BaseFilesSource

plugin_type: ClassVar[str] = 'posix'
__init__(**kwd: Unpack[PosixFilesSourceProperties])[source]

Prefer linking to files.

score_url_match(url: str)[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

to_relative_path(url: str) → str[source]

Convert this url to a filesource relative path. For example, given the url gxfiles://mysource1/myfile.txt it will return /myfile.txt. Protocols directly understood by the handler need not be relativized. For example, the url s3://bucket/myfile.txt can be returned unchanged.

galaxy.files.sources.s3 module

galaxy.files.sources.s3fs module

class galaxy.files.sources.s3fs.S3FsFilesSource(**kwd: Unpack[S3FsFilesSourceProperties])[source]

Bases: BaseFilesSource

plugin_type: ClassVar[str] = 's3fs'
__init__(**kwd: Unpack[S3FsFilesSourceProperties])[source]
score_url_match(url: str)[source]

Return how well a given url matches this filesource. A score greater than zero indicates that this filesource is capable of processing the given url.

Scoring is based on the following rules: a. The maximum score will be the length of the url. b. The minimum score will be the length of the scheme if the filesource can handle the file. c. The score will be zero if the filesource cannot handle the file.

For example, given the following file source config: - type: webdav

id: cloudstor url: “https://cloudstor.aarnet.edu.au” root: “/plus/remote.php/webdav”

  • type: http id: generic_http

the generic http handler may return a score of 8 for the url https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/myfolder/myfile.txt, as it can handle only the scheme part of the url. A webdav handler may return a score of 55 for the same url, as both the webdav url and root combined are a specific match.

Parameters:

url (str) – The url to score for a match against this filesource.

Returns:

A score based on the aforementioned rules.

Return type:

int

galaxy.files.sources.ssh module

class galaxy.files.sources.ssh.SshFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'ssh'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs.sshfs'

galaxy.files.sources.webdav module

class galaxy.files.sources.webdav.WebDavFilesSource(**kwd: Unpack[FilesSourceProperties])[source]

Bases: PyFilesystem2FilesSource

plugin_type: ClassVar[str] = 'webdav'
required_module: ClassVar[Type[FS] | None] = None
required_package: ClassVar[str] = 'fs.webdavfs'
allow_key_error_on_empty_directories = True