Warning

This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.

Tool Source Storage

Overview

By default, Galaxy parses every tool at startup and keeps all of them in memory. For installations with many tools this:

  • Slows down Galaxy startup significantly

  • Consumes large amounts of memory in every Galaxy process

Tool source storage addresses this by doing the parsing work once, ahead of time:

  1. Tool sources are pre-parsed (with macros expanded) and stored in a configurable database backend

  2. A lightweight index over the stored tools supports fast tool listings and search without touching tool files

A toolbox that consumes this store to load tools on demand is planned as follow-up work; this document covers the store, the populator, and the index that it will build on.

Configuration

Tool source storage is configured in galaxy.yml. The following options are available:

Default Store

galaxy:
  # SQLAlchemy URI for storing tool sources.
  tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite

The store lives in a standalone database - a SQLite file under <data_dir>/tool_sources.sqlite by default - separate from Galaxy’s main database. It is a rebuildable cache: it can be deleted at any time and recreated by re-running the population script.

Multi-host deployments must point every Galaxy process (web workers and job handlers) at the same store — typically a SQLite file on a shared filesystem:

galaxy:
  tool_source_database_connection: sqlite:////shared/galaxy/tool_sources.sqlite

Any other SQLAlchemy-supported database (e.g. PostgreSQL) works as well.

Per-conf Store Routing (CVMFS Recipe)

Individual tool_conf files can opt into a named tool source store, distinct from the global default. The typical use case is shipping a read-only SQLite bundle on CVMFS alongside a tool_conf, so worker processes can resolve every tool in that conf with local-cached lookups instead of one network round-trip per JSON file.

Declare the named stores under the top-level tool_source_stores key in galaxy.yml. Each entry takes a SQLAlchemy url and optional read_only flag. SQLite is the typical choice for CVMFS bundles (single self-contained file), but any SQLAlchemy-supported database works:

galaxy:
  tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite
  tool_source_stores:
    cvmfs_main:
      url: sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true
      read_only: true
    site_shared:
      url: sqlite:///file:/shared/galaxy/tool_sources.sqlite?mode=ro&uri=true
      read_only: true

Then point the tool_conf at it via the root element’s store attribute (XML) or top-level key (YAML):

<?xml version="1.0"?>
<toolbox store="cvmfs_main">
  <section id="cvmfs_tools" name="CVMFS Tools">
    <tool file="bwa/bwa.xml"/>
    ...
  </section>
</toolbox>

At startup, Galaxy inspects every tool_conf for the attribute, builds the referenced stores, and wraps them with the writable default in a composite store. Reads are tried in declared order (first hit wins) and writes always go to the default store. If no tool_conf opts in, the default store is used directly with zero overhead.

Building the bundle

Build the SQLite file from a writable host before shipping it:

$ python scripts/tool_source/populate_store.py -c galaxy.yml --target cvmfs_main

Use --target to restrict population to a single named store; without it, populate_store.py populates every writable store referenced from a tool_conf in the same run.

Once the bundle is in place on CVMFS (or any read-only mount), restart Galaxy. The read_only: true flag prevents Galaxy from writing through that store. For SQLite connection-level read-only, use mode=ro&uri=true in the SQLite URI as shown above.

Populating the Tool Source Store

After configuring tool source storage, you need to populate it with your tools. Use the populate_store.py script:

Basic Usage

$ python scripts/tool_source/populate_store.py --config /path/to/galaxy.yml

Deployments that install Galaxy from packages get the same command as the galaxy-populate-tool-source-store console script (shipped with the galaxy-app package), so no Galaxy source checkout is needed:

$ galaxy-populate-tool-source-store --config /path/to/galaxy.yml

This will:

  1. Discover tools from your tool configs (uses the same logic as Galaxy startup)

  2. Parse each tool (with macro expansion) and compute a content hash

  3. Store the tool sources in the configured backend (skipping unchanged tools)

Note: --config is required; the script does not assume a default path.

Command Line Options

$ python scripts/tool_source/populate_store.py --help

Options:
  --config, -c PATH      Galaxy configuration file (required)
  --dry-run              Show what would be stored without storing
  --incremental          Only store new/changed tools (default)
  --full                 Force re-store of all tools
  --tool-id PATTERN      Only process tools whose ID contains PATTERN
  --parallel, -j N       Number of parallel workers (default: 4)
  --rebuild-index        Rebuild the tool index after population
  --target NAME          Restrict to a single named store from
                         tool_source_stores (or '__default__'). Without
                         this, every writable store is populated.
  --verbose, -v          Verbose output
  --watch, -w            Watch tool directories and send reload notifications
  --watch-polling        Use polling observer (for NFS/CVMFS/network FS)

Examples

Initial population:

$ python scripts/tool_source/populate_store.py -c /path/to/galaxy.yml

Force re-store everything (e.g., after a parser change):

$ python scripts/tool_source/populate_store.py -c galaxy.yml --full

Process only a subset of tools:

$ python scripts/tool_source/populate_store.py -c galaxy.yml --tool-id samtools

Automation with Cron

For installations where tools are frequently updated, you can run the population script on a schedule:

# Update tool source store every hour (incremental is the default)
0 * * * * /path/to/galaxy/.venv/bin/python /path/to/galaxy/scripts/tool_source/populate_store.py -c /path/to/galaxy.yml >> /var/log/galaxy/tool_source_update.log 2>&1

Watch Mode (Live Updates)

As an alternative to cron, you can run the population script in watch mode to keep the store continuously up to date. This uses watchdog to monitor tool directories for changes and automatically updates the store, then sends a notification via Kombu to trigger cache reloads in all Galaxy processes.

$ python scripts/tool_source/populate_store.py --config galaxy.yml --watch

Watch mode options:

  • --watch, -w - Enable watch mode

  • --watch-polling - Use polling observer (required for network filesystems like NFS/CVMFS)

Example with polling for network filesystem:

$ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling

Requirements:

  • The watchdog library must be installed: pip install watchdog

  • Galaxy must have amqp_internal_connection configured for Kombu notifications

  • All Galaxy processes must be connected to the same AMQP broker

When a tool XML file changes, watch mode will:

  1. Detect the file change (with debouncing to handle rapid edits)

  2. Re-parse the tool and update the store

  3. Send a reload_tool_source_cache control message via Kombu

  4. All Galaxy processes will invalidate their local caches

This is useful for:

  • Installations using shared storage where tools may be updated externally

  • CI/CD pipelines that deploy tool updates

  • Development environments where tools are being actively edited

Troubleshooting

Tools not appearing in the index

  1. Re-run the population script with verbose output:

    $ python scripts/tool_source/populate_store.py -c galaxy.yml -v

  2. Check for parsing errors in the Galaxy log

Populating an existing installation

To set up tool source storage on an existing Galaxy installation:

  1. Add the configuration to galaxy.yml:

    galaxy:
  tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite

  2. Run the population script:

    $ python scripts/tool_source/populate_store.py -c /path/to/galaxy.yml

  3. Restart Galaxy