Tool data

Galaxy stores tool data in the path defined by tool_data_path (by default tool-data/). It’s possible to separate tool data of toolshed-installed tools by setting shed_tool_data_path.

Tool data consists of:

  1. the actual data

  2. one or more loc files

  3. entries in a tool data table (config) file

History

In order to understand the naming and structure of these three components it might be of advantage to look in the history. Tool data was organized in tabular loc that contained metadata and paths of the data. Those files were installed with the tool and could be accessed with the from_file mechanism from tools. Since each tool version had it’s own loc file the maintenance was difficult. With tool data tables an additional abstraction layer was introduced that is used from tools via from_datatable.

Tool data

This is the actual data that is stored by default in tool_data_path. It may be favorable to store the actual tool data in a separate folder. For manually managed tool data this can be achieved by simply storing the data in another folder. For data that is added by data managers this can be achieved by setting galaxy_data_manager_data_path.

loc files

In order to make tool data usable from Galaxy tools loc files are used. Those are tabular (by default tab separated) files with the extension .loc. Besides the actual paths, the entries can contain IDs, names, or other metadata that can be used in tools to select reference data. The paths should be given as absolute paths, but can also be given relative to the Galaxy root dir. By default loc files are installed in tool_data_path (where also built-in loc files are stored). By setting shed_tool_data_path this can be separated.

Tool data tables

The tool data tables that should be used in a Galaxy instance are listed in tool data table config files. In addition these contain metadata.

Tool data table config files are XML files listing tool data table configurations:

<tables>
    ....
</tables>

An entry for a tool data looks like this

<table name="bwa_indexes_color" comment_char="#" allow_duplicate_entries="False">
    <columns>value, dbkey, name, path</columns>
    <file path="bwa_index_color.loc" />
</table>

  • table: name, comment_char (default "#"), separator (default "\t"), allow_duplicate_entries (default True), empty_field_value (default "")

  • columns: a comma separated list of column names

  • file: path (alternatively url, from_config)

Tool data table entries for tools installed from a toolshed have an additional element tool_shed_repository and sub-tags tool_shed repository_name, repository_owner, installed_changeset_revision, e.g.:

<table name="plasmidfinder_database" comment_char="#">
    <columns>value, name, date, path</columns>
    <file path="/home/berntm/projects/galaxy/tool-data/toolshed.g2.bx.psu.edu/repos/iuc/plasmidfinder/7075b7a5441b/plasmidfinder_database.loc.sample"/>
    <tool_shed_repository>
        <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
        <repository_name>plasmidfinder</repository_name>
        <repository_owner>iuc</repository_owner>
        <installed_changeset_revision>7075b7a5441b</installed_changeset_revision>
    </tool_shed_repository>
</table>

The file path points to a loc file and can be given relative (to the tool_data_path) or absolute. If a given relative path does not exist also the base name is checked (many tools use something like tool-data/xyz.loc and store example loc files in a tool-data/ directory in the tool repository). Currently also .loc.sample may be used in case the specified .loc is absent.

Configuration order

tool_data_table_config_path lists the files that describe the reference data available to tools. Put preferred files first: when several files provide the same reference, earlier files take precedence by default.

shed_tool_data_table_config is the file Galaxy updates when you install tools from a Tool Shed. Include it in the list to choose its priority; otherwise Galaxy reads it last. The other files can be read-only:

galaxy:
  shed_tool_data_table_config: /srv/galaxy/var/shed_tool_data_table_conf.xml
  tool_data_table_config_path:
    - /srv/galaxy/config/tool_data_table_conf.xml
    - /srv/galaxy/var/shed_tool_data_table_conf.xml
    - /mnt/idc/read_only_data_table_conf.xml

Relative paths in tool_data_table_config_path resolve against config_dir; shed_tool_data_table_config resolves against managed_config_dir. Equivalent paths, including symlinks, load the managed file once at its first position. A missing default managed file keeps its place when later created; explicitly configured missing files are errors.

Tables with the same name must have matching columns. Rows retain source, XML, and file order, skipping identical rows when merging definitions.

This order applies to tool installation, data-manager updates, and reloads across Galaxy processes. Changing the configured source order requires a restart.

When a new tool is installed that uses a data table a new entry is added to shed_tool_data_table_config and a loc file is placed in a versioned subdirectory in tool_data_path (in a subdirectory that has the name of the toolshed). By default this is tool-data/toolshed.g2.bx.psu.edu/. Note that these directories will also contain tool data table config files, but they are unused.

Duplicate values

Rows that share a value are all kept, so tools can filter them, for example by index version. For a tool select, the tool’s filters, including sorting filters, run first. If several rows still match the submitted value, Galaxy uses the first one and logs a warning naming it. The submitted value comes from the select’s value column, which may differ from the table’s. Table-backed dropdowns show one choice per submitted value, labelled from the first remaining row; dataset choices stay separate. The data-table field API applies no tool filters and returns the first matching row.

Apart from the tool’s filters, only source and row order decide which copy is used; local, CVMFS and IDC files are treated alike. This lets references be deployed to IDC while local copies still exist. With the IDC file listed after the local file, a local /local/hg38.fa row keeps precedence over an IDC /idc/hg38.fa row when both survive the tool’s filters, so adding the IDC row leaves existing selections on the local file.

Check the effective order on each server. Unless shed_tool_data_table_config is listed in tool_data_table_config_path, Galaxy reads it last, after an IDC file appended to that list, so IDC rows take precedence over rows from installed tools. The reference used for an existing submitted value, for example when rerunning a job, changes when you:

  • reorder config files or rows,

  • remove the first matching row, or

  • add a row for that value to an earlier file, for example through a data manager.

The huggingface shared data table

Galaxy tools that consume pre-downloaded Hugging Face models share a single data table named huggingface. Using one shared table means admins maintain one .loc file and all tools benefit from every registered model entry.

Declaring the table

Add the following block to tool_data_table_conf.xml:

<!-- Hugging Face models -->
<table name="huggingface" comment_char="#" allow_duplicate_entries="False">
    <columns>value, name, pipeline_tag, domain, free_tag, version, path</columns>
    <file path="/opt/galaxy/tool-data/huggingface.loc" />
</table>

Each tool ships a tool-data/huggingface.loc.sample that uses the same 7-column layout.

Column reference

#

Column

Purpose

0

value

Unique row ID across the whole table

1

name

Human-readable label shown in the Galaxy select widget

2

pipeline_tag

Model role — see controlled vocabulary below

3

domain

Coarse data domain — see controlled vocabulary below

4

free_tag

Optional narrowing tag; fallback filter when pipeline_tag/domain alone are not specific enough

5

version

Model version string

6

path

Path to the model data, a directory or a specific file, depending on the model structure

value (column 0)

Must be globally unique across every row in huggingface.loc, regardless of which tool added it. Use the Hugging Face model ID (<owner>/<model-name>) directly — it is stable and unambiguous. If the same model is registered at more than one version, append the version:

black-forest-labs/FLUX.1-dev
sentence-transformers/all-MiniLM-L6-v2
openai/whisper-large-v3_3.0

pipeline_tag (column 2)

Use the official Hugging Face pipeline tag. Common values:

Value

When to use

text-to-image

Image generation models

automatic-speech-recognition

ASR / transcription models

feature-extraction

Sentence / document embedding models

tabular-classification

Tabular ML classifiers

tabular-regression

Tabular ML regressors

text-generation

Causal / instruction-tuned LLMs

Do not invent synonyms for existing Hugging Face tags.

domain (column 3)

A broad category for the data type the model works with: image · text · audio · tabular · sequence · video · multimodal

free_tag (column 4)

An optional short identifier used as a fallback narrowing filter when pipeline_tag and domain alone are not specific enough. Because a model can be consumed by multiple tools, free_tag must not encode a specific tool name. Choose a short, lowercase, descriptive value and document it alongside the tool that introduces it.

version (column 5)

The model version string. A tool declares in its XML which version(s) it accepts, allowing multiple versions of the same model to coexist. Where possible, rows are only added, never removed or edited.

path (column 6)

The path to the model data on the production server (maintained by admins). Can be a directory (when the tool reads the whole Hugging Face cache layout) or a specific file (e.g. a .ckpt checkpoint).

XML filter convention

Filter primarily by pipeline_tag (column 2) and/or domain (column 3) so only relevant model types are shown to the user. Add a version or free_tag filter only when you need to narrow the selection further:

<param name="model" type="select" label="Model">
    <options from_data_table="huggingface">
        <filter type="static_value" column="2" value="<pipeline_tag>"/>
        <filter type="static_value" column="3" value="<domain>"/>
        <!-- optional: narrow further by version or free_tag -->
        <!-- <filter type="static_value" column="5" value="<version>"/> -->
        <!-- <filter type="static_value" column="4" value="<free_tag>"/> -->
    </options>
</param>

Example from the Flux tool (filters by free_tag to restrict to Flux-specific model variants):

<options from_data_table="huggingface">
    <filter type="static_value" column="4" value="flux"/>
    <filter type="static_value" column="5" value="1"/>
</options>

Example .loc entry

Each row is TAB-separated (7 columns):

# Columns: value <TAB> name <TAB> pipeline_tag <TAB> domain <TAB> free_tag <TAB> version <TAB> path
#
# Flux
black-forest-labs/FLUX.1-dev	FLUX.1 [dev]	text-to-image	image	flux	1	/data/hf_models