October 2016 Galaxy Release (v 16.10)
Highlights
- Galaxu UI plugins - Webhooks
We introduce Galaxy Webhooks - optional plugins for the web UI that allow for better customization of your instance. See the documentation. Includes work from @bgruening, @anatskiy, and Joachim Wolff @joachimwolff. Implemented in Pull Request 3040.
- Workflow run form replaced
The workflow run form has been replaced by one backed by the new tool form and the API. Nicer, faster, standardized. Implemented in Pull Request 2669, Pull Request 2720, Pull Request 2766, Pull Request 2776, Pull Request 2770, Pull Request 2895, Pull Request 2935, Pull Request 3007, Pull Request 2969, Pull Request 2799, Pull Request 3082, Pull Request 2672, Pull Request 2920, Pull Request 2777, Pull Request 2902.
- Automatic tool reload after installation
Galaxy does not need to be restarted after tool installation anymore. This provides a smoother experience for the users. Yay! Thanks to @mvdbeek. Implemented in Pull Request 2840, Pull Request 3025.
Get Galaxy
The code lives at Github and you should have Git to obtain it.
- To get a new Galaxy repository run:
$ git clone -b release_16.10 https://github.com/galaxyproject/galaxy.git
- To update an existing Galaxy repository run:
$ git checkout release_16.10 && git pull --ff-only origin release_16.10
See our wiki for additional details regarding the source code location.
Deprecation Notices
API-like endpoints (returning JSON) that start with
<galaxy>/root/
in the URL are now deprecated and will be removed in future releases. Please use proper API (e.g./api/histories
or/api/tools
) to replace your implementations.The
jsonp
parameter for search API endpoints in Tool Shed is deprecated and will be removed in the future. Instead the presence of acallback
argument will trigger the JSONP format to be returned.
Security
An arbitrary code execution vulnerability in two tools and an XSS vulnerability with the upload tool were identified this release cycle and have been fixed concurrently with the release. In addition, the fixes have been backported to older releases.
The Galaxy Committers would like to thank David Wyde for disclosing these vulnerabilities. Details follow:
The vulnerable tools are “Filter GFF data by attribute” and “Filter GFF data by feature count”, both of which are provided with and enabled by default in the Galaxy server. These two tools share code with each other and the more general “Filter data on any column using simple expressions” tool. The latter was fixed in a previous security disclosure but these GFF variants of the tool were missed when updating the Filter tool. These tools use the Python
eval
andexec
functions and do not properly sanitize input to these functions. The fix for this issue has been applied to Galaxy releases back to v14.10 and can be found in Commit c1e3087An uploaded file’s name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. The fix for this issue has been applied to Galaxy releases back to v16.07 and can be found in Pull Request 3278.
Release Notes
Highlighted Enhancements
Overhaul of charts visualization - more visualizations, more options, and better user interface. Pull Request 2875, Pull Request 2820
Paginate contents of large histories. Pull Request 2588
Implement a collection operation tool for merging collections (thanks to @Takadonet). Pull Request 2771
Replace reference documentation for tool XML files with automatically generated documentation from a now official Galaxy XSD documentation (with help from many). Pull Request 2923, Pull Request 2936, Pull Request 3086, Pull Request 2932, Pull Request 2903, Pull Request 3020, Pull Request 3072
Add a password strength evaluation bar (thanks to @benfulton). Pull Request 2687
Implement a GoDocker job runner (thanks to @varunshankar). Pull Request 2791, Pull Request 2653
Support for API batch requests. Pull Request 1768
Allow JSONP to be returned from API endpoints. Pull Request 2937
Add “Save as” as an option in the workflow editor (thanks to @tmcgowan). Pull Request 3035
Allow naming input datasets and collections during workflow extraction. Pull Request 2943
Various enhancements for API driven installation of tool shed repositories.
Security
Fix an ACE vulnerability with gff filter tools (thanks to David Wyde). Commit c1e3087
Escape filename in upload dialog, default and composite (thanks to David Wyde). Pull Request 3278
Enhancements
Add “Save as” as an option in the workflow editor (thanks to @tmcgowan). Pull Request 3035
Allow ‘.’ and ‘_’ for public names (thanks to @nsoranzo). Pull Request 2947
Allow naming input datasets and collections during workflow extraction. Pull Request 2943
Various enhancements for API driven installation of Tool Shed repositories. Pull Request 2769
Implement beta support for discovering biocontainer Docker containers automatically for tool dependencies. Pull Request 2986
Improved API support for downloading datasets and relevant files. Pull Request 2741, Pull Request 2885
Make tool search field stay at top of window when tool list scrolls. (thanks to @sszakony). Pull Request 2730, Issue 2375
Scratchbook enhancement to include info and view details links for datasets. Pull Request 2733
Add implicit conversions target extension to HDA name when listed as selection option in the tool form. Pull Request 2734
Add column names to BED datatype so that column names are displayed in the dataset viewer. Pull Request 2784
Add “All” and “None” buttons to copy dataset view. Pull Request 2957
Lazy load workflow invocation steps to vastly improve scheduling performance of large collection workflows (thanks to @nsoranzo). Pull Request 2978
Add CEL datatype (thanks to @nturaga). Pull Request 3073
Add a netcdf datatype for metabolomics spectre data (thanks to @lecorguille). Pull Request 2706
Returning Blast datatypes to Galaxy core (thanks to @peterjc). Pull Request 2696
Add HDT format (with turtle format sniffer improvements) (thanks to @jjkoehorst). Pull Request 2787
Allow more configurability for containerized jobs. Pull Request 2790
Functionality to allow admins to switch on/off the line endings conversion on library dataset import (thanks to @SANBI-SA). Pull Request 2819
Add a tutorial on building job runners to the documentation (thanks to @varunshankar). Pull Request 2700
Allow viewing tool help from history datasets (thanks to @tmcgowan). Pull Request 3051
Prevent TS from creating sessions for repository status checks. Pull Request 2732
Shorten Conda environment names (thanks to @bgruening). Pull Request 2794
Various release process improvements (including an enhancement from @nsoranzo). Pull Request 2857, Pull Request 2858, Pull Request 2689, Pull Request 2865, Pull Request 2952, Pull Request 2695, Pull Request 2991, Pull Request 3232
Improve samtools usage with
--version-only
check (thanks to @remimarenco). Pull Request 2705All-around improvements to toolbox loading/reloading (thanks to @mvdbeek). Pull Request 2901
Refactor shed tool lineage registration (thanks to @mvdbeek). Pull Request 2909
Log the resolved dependency type (thanks to @mvdbeek). Pull Request 2917, Pull Request 3036
Fixes for Python 3 and new import linting for dozens of packages (largely thanks to @nsoranzo). Pull Request 2731, Pull Request 3177, Pull Request 2983, Pull Request 2889, Pull Request 3031, Pull Request 3170, Pull Request 3154
Change every occurrence in the code of
__all__
from list to a tuple (thanks to @nsoranzo). Pull Request 3041, Pull Request 3071Docstring linting and fixes. Pull Request 3003, Pull Request 3004
Update python-ldap module (thanks to @abretaud). Pull Request 2738
Update sqlalchemy to 1.0.15 (from 1.0.8). Pull Request 2979
Slightly revise auto configuration of loggers. Pull Request 2735
Various
Makefile
improvements and fixes. Pull Request 2757, Pull Request 2688, Pull Request 3167Improve startup scripts (thanks to @nsoranzo). Pull Request 2765
Remove
enable_beta_tool_command_isolation
config option (thanks to @nsoranzo). Pull Request 2830Regularize comments configuration samples. Pull Request 2824, Pull Request 2841, Pull Request 2814
Skip whoami check for LDAP servers not supporting it (thanks to @nsoranzo). Pull Request 2832
Filter disposable mail blacklist on the first domain level only. Pull Request 2839
Refactor duplicated back-end code to create a new
trans
method for getting the most recently touched history. Pull Request 2870Document setting column names in Galaxy tool XSD. Pull Request 2965
Functional test case for testing column names as meta-data (thanks to @yhoogstrate). Pull Request 2970
Improve
version_command
tag. Pull Request 2961Add
rgb
option for color picker tool parameter. Pull Request 3006Improved docs and testing for color parameters. Pull Request 3016
Add documentation on optional
multiple="true"
data parameters. Pull Request 3024In Trackster, read numerical data directly from bigWig files. Pull Request 2880
Improve wording of tool installation forms (thanks to @nturaga). Pull Request 2893
Add database/dependencies to .gitignore (thanks to @nturaga). Pull Request 2904
Allow admin to serialize/deserialize dataset permissions via the API. Pull Request 2912
Allow API access to genome fasta index and sequence (thanks to @jj-umn). Pull Request 2939
Extend
PUT /api/workflows/{id}
with additional workflow properties (thanks to @tmcgowan). Pull Request 3080Remove unnecessary use of binascii (thanks to @nsoranzo). Pull Request 2946
Change wording to be more clear about how workflow output cleanup works. Pull Request 2966
Optimized workflow invocation step update. Pull Request 2973
Update issues.rst to bring inline with actual usage on Github. Pull Request 2976
Improve Conda FAQ documentation. Pull Request 2998, Pull Request 2891, Pull Request 2871
Add documentation for
conda_copy_dependencies
option (thanks to @abretaud). Pull Request 3105Slight clarification of run.sh messaging when using Conda. Pull Request 3188
Display tool requirements for conda-only tools (thanks to @mvdbeek). Pull Request 3262
Pass dependency shell command errors to log (thanks to @mvdbeek). Pull Request 3129
Increase min number of columns for pileup sniffing to 5 (thanks to @nsoranzo). Pull Request 3033
Remove outdated warning on ID_SECRET (thanks to @hexylena). Pull Request 3046
Add Galaxy architecture slides documentation to project. Pull Request 3060, Pull Request 3091
Add tool XML reload to tool run page - for admins only (thanks to @tmcgowan). Pull Request 3070, Issue 943
Move thumbnail list view to shared UI elements, revise tabs, add test cases Pull Request 3079
Include filenames in .loc file DEBUG lines (thanks to @peterjc). Pull Request 3095
Log loading of resources from .loc files (thanks to @peterjc). Pull Request 3099
Eager load tags and annotations in history display. Pull Request 3110
Display an error message when toolbox filtering fails. Pull Request 3123
Bring in various updates from galaxy-lib. Pull Request 3169, Pull Request 2888
Add links to local tours to sample welcome. Pull Request 3176
Fix dataset selector update when deleting history items. Pull Request 2677
Refactor upload dialog to use local CSS classes and avoid model listeners. Pull Request 2711
Rework Tool Shed middleware Pull Request 2782, Pull Request 2753
Refactor - move filelock.py to galaxy.util. Pull Request 2879
Use requests in driver_util for better proxy handling (thanks to @mvdbeek). Pull Request 3013
Add
download_to_file
function togalaxy.util
(thanks to @nsoranzo). Pull Request 3100Quote all paths in converter tool XML files (thanks to @mvdbeek). Pull Request 3165
Add routes to get genomes indexes and sequences (thanks to @jj-umn). Pull Request 3109
Admin installation UX wording fixes. Pull Request 3226
Fixes
Fix for bioconda recipes depending on conda-forge. Pull Request 3023
Fix history deletion bug when using impersonation. Pull Request 2654
Fix workflow extraction API tests on PostgreSQL (thanks to @nsoranzo). Pull Request 3039
Fix link target in admin view of tool lineages. Pull Request 2862
Avoid pointless lambda function (thanks to @peterjc). Pull Request 3097
Remove unnecessary
set_output_history
parameter (thanks to @nsoranzo). Pull Request 3155Fix BLAST database
*.loc
files inconsistency (thanks to @peterjc). Pull Request 3098Log invalid XML filename (thanks to @mvdbeek). Pull Request 3124
Various code fixes for object store and docs (thanks to @nsoranzo). Pull Request 3119
Use default job history on
build_for_rerun
if current history is unavailable. Pull Request 2668Always display text Pull Request 3052
Fixing button width on composite upload dialog (thanks to @sszakony). Pull Request 2703, Issue 2591
Fix for splitting dataset collections (thanks to @gregvonkuster). Pull Request 2708
Various datatypes fixes and enhancements (thanks to @nsoranzo). Pull Request 2690
Various fixes and improvements for galaxy.json. Pull Request 2697
Fix tours pasting content into text area. Pull Request 2715
Slight styling tweak to prevent overflow on the right edge in composite upload dialog. Pull Request 2716
Workflow toolbox style updates. Pull Request 2740
Fix docker command_list (thanks to @manabuishii). Pull Request 2748
Fix
list:list
history display in the GUI. Pull Request 2758Parse internal parameters such as job resource selections when data converter tools are executed implicitly. Pull Request 2761
Fix broken check when creating Docker volumes default for jobs (thanks to @manabuishii). Pull Request 2763
Fix to prevent login form from appearing in scratchbook window (thanks to @sszakony). Pull Request 2808
Allow NodeJS proxy to do reverse proxying (thanks to @hexylena). Pull Request 2817
Fixes for the RStudio interactive environment (thanks to @hexylena). Pull Request 2818
Acquire lock before attempting to install conda (thanks to @mvdbeek). Pull Request 2826
Fix Galaxy for if pyuwsgi is install in Galaxy’s virtual environment. Pull Request 2836
Fix tabular renderer display when
total_cols > line_cols > 1
. Pull Request 2845Remove empty file in toolbox module. Pull Request 2877
Fix for workflow execution when using collections. Pull Request 2898, Issue 2806
Implement
split
on a validator. Pull Request 2921Catch failure to import grp in util (for Pulsar on Windows) Pull Request 2928
Bump pysam, remove
already_compressed
usage due to new wheel. Pull Request 2953Re-build DOM on render for modal in the front-end. Pull Request 2955
Fix optional column form values when editing dataset properties. Pull Request 2960
Fix optional flag for library datasets in legacy library UI. Pull Request 2962
Reload metadata and lib tools on toolbox reload (thanks to @mvdbeek). Pull Request 2992
Update some old information in API guideline docs. Pull Request 3011
Fix for rerunning tool with workflow resume that is part of a collection. Pull Request 3019
Missing double quotation in tool XSD example (thanks to @ramezrawas). Pull Request 3027
Include base classes when searching datatypes by name. Pull Request 3028
Image detection fixes. Pull Request 3042
Fix tabular filtering tool broken in #2983 Pull Request 3043
Fix GFF attribute filtering tool broken in #2983 Pull Request 3044
Fix for displaying parameters to the user that have a
False
value. Pull Request 3045Fix downloading tool tarball with boolean test input values. Pull Request 3047
Sort bedtools output in
bam_to_bigwig
conversion (thanks to @lparsons). Pull Request 3049Remove incorrect communication server check. Pull Request 3053
Fix tool XSD to accept a help attribute for
section
s (thanks to @joachimwolff). Pull Request 3131Fix import orders for updates to flake8_import_order. Pull Request 3059
Dataset metadata should not be cached when building chart visualizations. Pull Request 3062
Fix libraries prefix. Pull Request 3074
Fix qunit local testing install – now runs on modern node (v6.9.1). Pull Request 3089
Escape section name when writing to XML (thanks to @mvdbeek). Pull Request 3122
Fix tools placed outside of panel section (thanks to @mvdbeek). Pull Request 3125
Fix XSD for
exclude_min
andexclude_max
(thanks to @gregvonkuster). Pull Request 3108Update galaxy.xsd (thanks to @lecorguille). Pull Request 3132
Fix to treat the value of
message_box_visible
ingalaxy.ini
as a boolean. Pull Request 3139Ensure a From: address is set for user activation emails. Pull Request 3140
Explicitly specify attributes which should be copied to converted datasets. Pull Request 3149
Prefer existing .venv over conda, when both are available. Pull Request 3180
Validate cycles and step size before building model for workflow execution. Pull Request 3183
Fix a typo in the intro tour. Pull Request 3184
Make SAM to bam converter tool compatible with samtools >= 1.3 (thanks to @mvdbeek). Pull Request 3187
Remove unused (and unintended) config override. Pull Request 3198
Fixes for deleting histories. Pull Request 3203
Change ‘History Actions’ to ‘Current History’ in history options menu. Pull Request 3205
Preserve custom event handler for form inputs. Pull Request 3210
Admin installation UX wording fixes. Pull Request 3226
Backport #3106 and #3222: Cached conda environments and API to manage them (thanks to @mvdbeek). Pull Request 3227
Update the copyright year in the LICENSE.txt. Pull Request 2926
Fix for ToolShed install when copied sample data target exists, but is broken symlink. Pull Request 3279
To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.
You can always chat with us on Matrix.
Thanks for using Galaxy!