Warning
This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
September 2017 Galaxy Release (v 17.09)
Highlights
- Singularity
Tool execution using the HPC-friendly container technology Singularity is now supported. Custom containers can be specified by the Galaxy admin on a per job destination basis or standardized containers corresponding to Conda requirements can be built or downloaded automatically using the mulled toolkit built into Galaxy (just like is possible for Docker). For more information checkout this presentation from the 2017 Galaxy Community Conference. Pull Request 4175
- Download entire collection
Downloading whole colections is now possible from the history interface. Pull Request 4098 Thanks to @mvdbeek.
- Switch tool versions in workflows
You can now select exactly what version of tool you want to use when building workflows. Pull Request 4373 Thanks to @mvdbeek.
Get Galaxy
The code lives at Github and you should have Git to obtain it.
- To get a new Galaxy repository run:
$ git clone -b release_17.09 https://github.com/galaxyproject/galaxy.git
- To update an existing Galaxy repository run:
$ git checkout release_17.09 && git pull --ff-only origin release_17.09
See the community hub for additional details regarding the source code locations.
Security
The 17.09 Galaxy version includes many security patches. Per our new Security Policy some of these have been applied to Galaxy releases going back 12 months.
Details of the vulnerabilities that have been backported can be found in the Security patch details section of these release notes. Some issues have only been addressed in 17.09, for this reason if security is important to your Galaxy instance we strongly recommend upgrading to this latest release as soon as possible.
If you maintain a publicly accessible Galaxy please consider signing up for this mailing list to receive the future security patches in advance of the public disclosure.
Deprecation Notices
The Galaxy Sample Tracking and External Services functionality is now considered deprecated. In the next releases we will remove it completely. Related PRs:#4526 #4872 .
The deprecated admin-only interface for Galaxy Data Libraries is staged to be removed in the next release.
Workflows API: When exposing WorkflowInvocationSteps
state
will no longer be available.The
refresh_on_change
attribute of a<param>
tag in the tool syntax can no longer be set to a value of another parameter. Use boolean instead (e.g.refresh_on_change="True"
). DetailsThe endpoint
/api/configuration/toolbox
is now deprecated and will be removed in the future. All tools are now watched for changes and this feature became obsolete.
Release Notes
Enhancements
Galaxy Workflow support for GenomeSpace (thanks to @nuwang and @gvlproject). Pull Request 1814
Support tags (including new “name” tags) in data libraries. (thanks to @bwlang). Pull Request 4262
HiCBrowser as Galaxy IE (thanks to @bgruening). Pull Request 3330
Synnefo/Pithos+ object store backend (thanks to @saxtouri). Pull Request 3611
Add job runner for Chronos (thanks to @theosotr). Pull Request 3946, Pull Request 4120
Many improvements to the shell job runner (including a new Paramiko based runner with retry support) (thanks to @mvdbeek). Pull Request 4358, Pull Request 4599, Pull Request 4343
Add entry for
empty_extra_files_path
validator (thanks to @gregvonkuster). Pull Request 3994Extend async data sources to support dataset collections (thanks to @fabio-cumbo). Pull Request 4198
Add a repository uninstall endpoint to the tool dependency API (thanks to @mvdbeek). Pull Request 4248
Display preview feature for Bam datatype (thanks to @ashvark). Pull Request 4279
Add Unipept taxonomy viewer plugin (thanks to @caleb-easterly). Pull Request 4310
Created alphabetic and numerical sort tool for collection operations (thanks to @glormph). Pull Request 4329
Prevent loading tools that require a newer galaxy (thanks to @mvdbeek). Pull Request 4382
Add explicit support for gzipped VCF files (thanks to @ffinfo). Pull Request 4254
Add ptkscmp datatype (thanks to @gregvonkuster). Pull Request 4259
Add imzML datatype (thanks to @bgruening). Pull Request 4370
Add deepTools datatypes to Galaxy (thanks to @bgruening). Pull Request 4392
Add support for the MEME psp format (thanks to @gregvonkuster). Pull Request 4430
Add biom2 datatype (thanks to @shiltemann). Pull Request 4519
Add Excel97 Datatype Pull Request 4410
Add mz5 datatype (thanks to @bgruening). Pull Request 4466
Refactor tool error reporting into plugins and implement a new plugin for influxdb (thanks to @hexylena). Pull Request 4305, Pull Request 4533
Implement second toolshed installation UX targetting the API. Pull Request 3626
Tool-Describing-Tours (thanks to @anatskiy). Pull Request 4019
Revamp configuration and default server for tool shed and Galaxy reports application (these now use a uwsgi server by default configured with a YAML configuration file). Pull Request 3179
Speed up and optimize the process of identifying public libraries. Pull Request 4640
Numerous data library fixes and improvements - both to the frontend and backend. Pull Request 4568, Pull Request 4595, Pull Request 4594, Pull Request 4579, Pull Request 4560 Pull Request 4512, Pull Request 4621, Pull Request 4752
Enhance workflow import API endpoint to allow admins to install repositories corresponding to a supplied workflow (thanks to @manabuishii). Pull Request 3064
GFF3 support in filter by attribute (thanks to @peterjc). Pull Request 3076
Add extension point for user preferences (thanks to @bgruening). Pull Request 3383
Respect
expose_user_email
configuration in data libraries permissions list. Pull Request 4611Explicitly include PyPI as an
--extra-index-url
to pip. Pull Request 4674Improved Kubernetes support (non-privileged filesystem access, updated k8s Job API target) (thanks to @phnmnl). Pull Request 3972
New Dependency resolver for the LMOD environment modules system (thanks to @arbernard). Pull Request 4489
Nicer tool click targets for toolbox (thanks to @hexylena). Pull Request 4470
Implement a “plain” bootstrap theme for the UX (thanks to @hexylena). Pull Request 4470
Add Selenium tests for the “saved histories”, “published histories”, and “custom builds” (thanks to @anatskiy). Pull Request 4587, Pull Request 4502, Pull Request 4641,
Dozens of Selenium functional test enhancements and fixes Pull Request 3993, Pull Request 4523, Pull Request 4524, Pull Request 4549, Pull Request 4588, Pull Request 4589, Pull Request 4647, Pull Request 3992, Pull Request 4003, Pull Request 4053, Pull Request 4576, Pull Request 4561, Pull Request 4531, Pull Request 4564, Pull Request 4582, Pull Request 4586, Pull Request 4592, Pull Request 4042
Prevent graphite from dying loudly (thanks to @hexylena). Pull Request 4036
Updated parameters check for OpenID (thanks to @VJalili). Pull Request 4039
Tooltip translation . Add Deutsch,Spanish (thanks to @ValentinChCloud). Pull Request 4045
New Workflow Management Screen UI using JavaScript (thanks to @bgruening). Pull Request 4047
Improve release checklist Pull Request 4077
Fix French translation errors (thanks to @yvanlebras). Pull Request 4080
remove the content alert from TS response Pull Request 4082
Touchup .gitignore to not ignore new bundles. Pull Request 4091
New Configure Workflow Menu UI (thanks to @bgruening). Pull Request 4092
get real user name from username and email (for submission as real user with drmaa) (thanks to @bernt-matthias). Pull Request 4096
Security policy (thanks to @hexylena). Pull Request 4113
Improved styling of workflow list. Pull Request 4138
Swap over all URLs to new wiki location (thanks to @hexylena). Pull Request 4141
Documentation fixes and improvements (thanks to @nsoranzo). Pull Request 4146
Slider enhancement, shared event listeners Pull Request 4149
Add the ability to control job conf plugin loading from environment variables. Pull Request 4154
Replace various mako templates with client side JavaScript rendering. Pull Request 4157, Pull Request 4167, Pull Request 4169, Pull Request 4201, Pull Request 4311, Pull Request 4377, Pull Request 4411, Pull Request 4334, Pull Request 4305, Pull Request 4067, Pull Request 4327
Related to the above mako transition, rework various “grid” rendering code to do more on the client side as well. Pull Request 4302, Pull Request 4291, Pull Request 4293, Pull Request 4385, Pull Request 4405, Pull Request 4449, Pull Request 4263, Pull Request 4341, Pull Request 4163, Pull Request 4101
Set content type to application/json for JSON text types. Pull Request 4160
Assorted enhancements for container support (both for mulled container generation and fetching as well as to new Singularity support). Pull Request 4173, Pull Request 4179, Pull Request 4180, Pull Request 4185
Make updateucsc.sh.sample usable without any change (thanks to @nsoranzo). Pull Request 4192
deals with .xls older file extension (thanks to @FredericBGA). Pull Request 4200
add dev requirement Pull Request 4008
Improve search plugin (thanks to @bgruening). Pull Request 4213
Upgrading Neo4j IE to 3.1 (thanks to @thobalose). Pull Request 4216
Running a word count on a compressed fastq file now runs wc on the un… (thanks to @dpryan79). Pull Request 4226
“Join two Datasets” (join1) preserve column headers (thanks to @lecorguille). Pull Request 4229
Add public-only filter for data libraries (thanks to @pvanheus). Pull Request 4232
Fix import order and Python3 compatibility for 51 files (thanks to @nsoranzo). Pull Request 4236
Sniff fastqsanger and prefer it over fastq if the quality scores match (thanks to @dpryan79). Pull Request 4237
Enable skipping comment lines while sniffing files (thanks to @dpryan79). Pull Request 4239
Expose 4 more rows from the galaxy_user table to help out sys admins (thanks to @XDtim). Pull Request 4255
Change ‘register or login’ to just ‘login’ if ‘allow_user_creation=False’ (thanks to @scholtalbers). Pull Request 4258
Keep tool parameters when rerunning a job (thanks to @mvdbeek). Pull Request 4271
More uniform backend code for generating page titles (thanks to @hexylena). Pull Request 4272, Pull Request 4578
Fix blockquote>p sizing (thanks to @hexylena). Pull Request 4273
Expose more user properties to admins via the API. Pull Request 4274
Ensure tags are copied and presented when moving collections (thanks to @pvanheus). Pull Request 4277
Add
test_errors
to .gitignore. Pull Request 4278Implement package locking for node modules. Pull Request 4283
Enhancement for Phylocanvas plugin (thanks to @bgruening). Pull Request 4284
Improve security via local network access restrictions (thanks to @hexylena). Pull Request 4289
Enhancement for Phyloviz viz plugin (thanks to @bgruening). Pull Request 4292
Allow submitting the galaxy jobs as a predefined system user using
real_system_username
(thanks to @ashvark). Pull Request 4294Update Japanese translation (thanks to @manabuishii). Pull Request 4304
Allow localization of more UI strings (thanks to @manabuishii). Pull Request 4306, Pull Request 4307
Allow import of history archives via simple file upload (thanks to @mvdbeek). Pull Request 4326
Fix some French errors (thanks to @loraine-gueguen). Pull Request 4337
Updated the version requirements (thanks to @VJalili). Pull Request 4342
CSRF protection for login, logout, and user registration (most parts of the Galaxy UI still don’t have explicit CSRF protection). Pull Request 4365, Pull Request 4710
Implement
default_identifier_source
for tool collection outputs and sort input keys by default (thanks to @mvdbeek). Pull Request 4368, Pull Request 4380Galactic Radio Telescope Update (thanks to @hexylena). Pull Request 4376
Show minimum required galaxy version in tool shed and galaxy installation process (thanks to @mvdbeek). Pull Request 4386
Cleanup of Unit Test Code for Loading Tools Pull Request 4387
Refactor ToolOutputCollectionStructure… Pull Request 4389
Slightly improve workflow warnings/errors logging. Pull Request 4396
Better Error Summaries for API Tests Pull Request 4397
Fix import order and Python3 compatibility for lib/galaxy/web/base/ (thanks to @nsoranzo). Pull Request 4403
Generalize allow_library_path_paste to allow_path_paste. Pull Request 4404
Small GRT bugfixes (thanks to @hexylena). Pull Request 4408
Update conda channel order to sync with Bioconda (thanks to @nsoranzo). Pull Request 4409
Fixes and enhancements for the upload API Pull Request 4417
Avoid adding (imported from API) to workflows (thanks to @chambm). Pull Request 4434
Enhanced tool options for dataset discovery Pull Request 4437
Augment form module separation Pull Request 4438
Fix E201 and E202 style errors (thanks to @nsoranzo). Pull Request 4440
Add autopep8 script to help rebasing branches after #4440. Pull Request 4447
Uniform indentation of multiline if conditionals (thanks to @nsoranzo). Pull Request 4455
Small logging improvements (thanks to @nsoranzo). Pull Request 4456
Improvements to Sentry integration (thanks to @hexylena). Pull Request 4457, Pull Request 4471
Fix for GIEs to not automatically uppercase environment variables (thanks to @xgaia). Pull Request 4458, Pull Request 4454
Fix security checking of WorkflowInvocation for published workflows (thanks to @mvdbeek). Pull Request 4465
Show more items per page in saved history view and directly show tags (thanks to @mvdbeek). Pull Request 4467
A variety of usability improvements to the workflow index page (show tags, allow import by drag-and-drop, and avoid unneeded page refreshs) (thanks to @mvdbeek). Pull Request 4476, Pull Request 4369
Use pysam instead of samtools binary when discovering BAM metadata in one place (samtools is still required by Galaxy) (thanks to @nsoranzo). Pull Request 4479
Replace exclamation with a clock icon for “new” datasets in the history panel. Pull Request 4485
Minor copy datasets dialog improvement. Pull Request 4486
41% reduction in freiburg galaxy startup time (thanks to @hexylena). Pull Request 4495
Allow paramiko cli to recover from ssh timeout (thanks to @mvdbeek). Pull Request 4503
Tests and clarifications for composite uploads. Pull Request 4505
Load tools with collection type source attributes into the workflow editor Pull Request 4514
Display d3 hierarchy datatype in upload interface Pull Request 4522
Sync non-CWL galaxy-lib changes. Pull Request 4535
Add a small howto on profiling galaxy code (thanks to @mvdbeek). Pull Request 4542
allow linking, posix and spaces conversions in datalib imports Pull Request 4547
Allow exporting workflows when tool is uninstalled (thanks to @mvdbeek). Pull Request 4553
Lower threshold for auto-matching dataset pairs in the collection builder Pull Request 4555
Allow working with pbzip2 compressed files (thanks to @mvdbeek). Pull Request 4559
add logging to ts index building Pull Request 4567
Add show/hide deleted/hidden links when history is filtered… (thanks to @chambm). Pull Request 4581
Python 3 fixes for set_metadata.py Pull Request 4607
Allow adding new data tables without restart (thanks to @mvdbeek). Pull Request 4617
remove .git and .hg from recursive file search (thanks to @bgruening). Pull Request 4636
Better handling of long id secrets when generating per-kind encryption keys. Pull Request 4713
Add test case clarifying datasets do get “renamed” by rename PJA in collections. Pull Request 3983
Add some test tools for collection job failures. Pull Request 4044
Merge ShedToolLineage and StockLineage to ToolLineage (thanks to @mvdbeek). Pull Request 4119
Move PlantTribes datatypes to a separate file with enhancements and fixes (thanks to @gregvonkuster). Pull Request 4137
Move
galaxy.tools.toolbox.cache
togalaxy.tools.cache
. Pull Request 4155Small code style and error message fixes for data library code (thanks to @bwlang). Pull Request 4250
Refactors tag manager to not consume the unneeded
app
argument (thanks to @bwlang). Pull Request 4253Use contextmanager when using
compression_util.get_fileobj()
(thanks to @mvdbeek). Pull Request 4270Swap the deprecated
unescape()
fordecodeURIComponent
. Pull Request 4321More consistent formatting for model mapping code (thanks to @VJalili). Pull Request 4333
Merge configure menu with main workflow view (thanks to @bgruening). Pull Request 4353
Delete
conda_exec_home
instead ofenv['HOME']
(thanks to @mvdbeek). Pull Request 4360Replace list with generator when iterating headers in datatypes code (thanks to @mvdbeek). Pull Request 4388
Get tools with
tool_version
in more places (thanks to @mvdbeek). Pull Request 4391Fix issues identified by the JavaScript linting application lgtm. (thanks to @xiemaisi). Pull Request 4421
Remove /mobile and associated templates (the code was unused and broken). Pull Request 4494
Workflow loading speedup (thanks to @mvdbeek). Pull Request 4500
Prevent transient job state API test failures from failing the build. Pull Request 4510
More robust workflow invocation testing. Pull Request 4530
Refactor history import/export tests to allow collection testing. Pull Request 4534
Refactor test modules toward cleaner dependencies Pull Request 4536
Prevent in-place editing of uploaded files if files are imported from the FTP folder (thanks to @mvdbeek). Pull Request 4539
Speedup toolform building (thanks to @mvdbeek). Pull Request 4541
Check user roles only once if user has no roles (thanks to @mvdbeek). Pull Request 4546
Remove unused bam to bai converter (thanks to @mvdbeek). Pull Request 4598
Remove various unreachable statements (thanks to @hexylena). Pull Request 4603
Refactored a dataset variable to HDA (thanks to @VJalili). Pull Request 4630
Disable MacOS tests on TravisCI. Pull Request 4631
Fixes
Various hashtag (or name tag) fixes (also backported to 17.05). Pull Request 4139, Pull Request 4188
Simplify RC creation in Makefile. Pull Request 4011
Do not recurse in
ensure_installed()
(thanks to @bernt-matthias). Pull Request 4049Minor fixes for new locales, rebuild of client. Pull Request 4050
French translation error (thanks to @yvanlebras). Pull Request 4051
Minor history tour tweaks. Pull Request 4061
Improve tour button logic (thanks to @bagnacan). Pull Request 4062
Bug fix to prevent fetching the file path of purged files (thanks to @dpryan79). Pull Request 4066
ToolShed tool dependency install fixes (thanks to @nsoranzo). Pull Request 4105
Fix legacy Python path for genome diversity tools from miller lab. Pull Request 4117
Avoid locale specific string.letters for job_name (thanks to @peterjc). Pull Request 4121
Fix typo in job_conf.xml.sample (thanks to @manabuishii). Pull Request 4126
Hide error highlighting if user interacts with highlighted field Pull Request 4147
Fix webhooks loading URL. Pull Request 4158
Fix older GIE config sample description. Pull Request 4164
Fix #3990, don’t chown non-galaxy files (thanks to @dpryan79). Pull Request 4186
Debug of script library_upload_dir.py (thanks to @FredericBGA). Pull Request 4199
Correct name of neo4j datatype class (thanks to @pvanheus). Pull Request 4223
Fix
Registry.get_datatype_by_extension()
to returnNone
ifext
is unknown (thanks to @nsoranzo). Pull Request 4224Avoid broken version of Mercurial in conda (thanks to @bwlang). Pull Request 4227
Fix empty tabular output error when using discover_datasets. (thanks to @pkrog). Pull Request 4240
Fix importing old exported histories. (thanks to @cche). Pull Request 4268
Update
tool_conf.xml.main
to reflect usegalaxy.org’s current state. Pull Request 4295Fix
TypeError
when uploading large files from FTP to S3. (thanks to @jlhg). Pull Request 4315Fix for loading tools when
tool.lineage is None
. Pull Request 4317Fix for the caching of location filenames when they are broken symlinks. Pull Request 4318
Remove print debug option from toolshed. Pull Request 4332
Fix default output labels for subworkflows (thanks to @mvdbeek). Pull Request 4346
Search overlay plugin bug fix (thanks to @bgruening). Pull Request 4348
Tool version and lineage fixes (thanks to @mvdbeek). Pull Request 4375
Fix bug in
scripts/extract_dataset_part.py
during task splitting. Pull Request 4383Fix
get_tool
returning list when it shouldn’t. Pull Request 4390Bug fix for loading subworkflows from workflow descriptions. Pull Request 4394
Bug fix
__str__
method on tool parsers that previously would throw anException
. Pull Request 4395Add missing chemical formats to
datatypes_conf.xml.sample
(thanks to @nsoranzo). Pull Request 4413Don’t cast
tool_version
to string iftool_version
is None-type (thanks to @mvdbeek). Pull Request 4420Some unicoding for local runner (thanks to @hexylena). Pull Request 4426
Break connection in workflow editor if necessary (thanks to @mvdbeek). Pull Request 4431
Fix private role validation. Pull Request 4432
Remove print of non-existent attribute ‘content’ (thanks to @chambm). Pull Request 4439
Fix quota function name Pull Request 4469
Fixes for VCF/BCF datatypes (thanks to @nsoranzo). Pull Request 4477
Validate workflow step after step argument injection. Pull Request 4483
Fix drag and drop from history for Firefox Pull Request 4496
2017-08 Security Patch (thanks to @hexylena). Pull Request 4501
Fix history import if using symlink in database directory (thanks to @FredericBGA). Pull Request 4511
Fix missing tools in the workflow editor (thanks to @mvdbeek). Pull Request 4552
Prevent unbound variable error history controller. Pull Request 4557
Backport uwsgi fix from #2836. Pull Request 4565
Bug fix startup of galaxy when webhooks dir is empty (thanks to @mvdbeek). Pull Request 4570
Fix delete option in history menu. Pull Request 4574
Cancel workflow invocations when histories are deleted. Pull Request 4580
Multiview missing histories fix Pull Request 4610
Fix virtualenv activation for some scripts (thanks to @nsoranzo). Pull Request 4616
Pulsar remote metadata fixes Pull Request 4622
Swap sanitize whitelist form to use a post. Pull Request 4625
Fix various spelling and grammatical error fixes. Pull Request 4626
Typo fix (thanks to @VJalili). Pull Request 4628
Updated function description and fix some typos (thanks to @VJalili). Pull Request 4629
Fix display of data in “custom builds” page (thanks to @anatskiy). Pull Request 4634
Do not wrap lines in the upload URL fetch. Pull Request 4639
Explicitly write registry.xml when creating a job for the upload tool (thanks to @mvdbeek). Pull Request 4644, Pull Request 4600
Fix links on workflow, history items. Pull Request 4656
Fix for modules resolver. Pull Request 4663
Remove overly chatty debug statement. Pull Request 4671
Client side fixes for GIEs. Pull Request 4680
Update target versions of conda and conda-build (thanks to @nsoranzo). Pull Request 4701
Correct base route for workflows, allowing proxy-prefix to work. Pull Request 4705
Cachebust IE require’d files (jupyter.js, etc). Pull Request 4714
Add message (error and info) display to workflows display list. Pull Request 4716
If the newest version of a tool is hidden, load the newest older version, if any, into the tool panel. Pull Request 4726
Fix missing support for command_inject when using containers lib in GIEs. Pull Request 4740
Fix t, a, g, s returned in to_dict() method (thanks to @mvdbeek). Pull Request 4742
Fix joiner tool to inherit datatype from the input format Pull Request 4745
Fix auth with
ldaps://
(thanks to @abretaud). Pull Request 4748Force onload webhooks to wait for Galaxy object (and root) resolution. Pull Request 4750
GIEs: Grandfather automatic uppercasing of some env_override variables Pull Request 4760
Security patch details
Arbitrary code execution for Galaxy servers with Galaxy Interactive Environments enabled
Tracked as GX-2017-0002
A high severity security vulnerability was recently discovered in Galaxy Interactive Environments (GIEs) by the Galaxy Committers Team. Anyone with a Galaxy account can exploit this vulnerability to execute arbitrary code on the Galaxy server as the user running the Galaxy server process.
The vulnerability only affects Galaxy servers on which Galaxy Interactive
Environments are enabled (by setting the
interactive_environment_plugins_directory
option in galaxy.ini
). Because the vulnerability can be exploited to
execute arbitrary code, the impact for affected servers is severe.
Administrators of Galaxy servers where GIEs are enabled should update immediately.
The fix for this issue has been applied to Galaxy releases back to 17.05 and can be found in this commit
Cross site scripting and session fixation
Disclosed on the mailing list in August 2017.
Vulnerabilities were found by Helena Rasche and Manabu Ishii respectively. Detailed descriptions of these categories of vulnerabilities can be found at:
The fix for these issues has been applied to Galaxy releases back to 16.10 and can be found in this diff
To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.
You can always chat with us on Matrix.
Thanks for using Galaxy!