Warning
This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
October 2016 Galaxy Release (v 16.10)¶
Highlights¶
- Galaxu UI plugins - Webhooks
- We introduce Galaxy Webhooks - optional plugins for the web UI that allow for better customization of your instance. See the documentation. Includes work from @bgruening, @anatskiy, and Joachim Wolff @joachimwolff. Implemented in Pull Request 3040.
- Workflow run form replaced
- The workflow run form has been replaced by one backed by the new tool form and the API. Nicer, faster, standardized. Implemented in Pull Request 2669, Pull Request 2720, Pull Request 2766, Pull Request 2776, Pull Request 2770, Pull Request 2895, Pull Request 2935, Pull Request 3007, Pull Request 2969, Pull Request 2799, Pull Request 3082, Pull Request 2672, Pull Request 2920, Pull Request 2777, Pull Request 2902.
- Automatic tool reload after installation
- Galaxy does not need to be restarted after tool installation anymore. This provides a smoother experience for the users. Yay! Thanks to @mvdbeek. Implemented in Pull Request 2840, Pull Request 3025.
Get Galaxy¶
The code lives at Github and you should have Git to obtain it.
- To get a new Galaxy repository run:
$ git clone -b release_16.10 https://github.com/galaxyproject/galaxy.git
- To update an existing Galaxy repository run:
$ git checkout release_16.10 && git pull --ff-only origin release_16.10
See our wiki for additional details regarding the source code location.
Deprecation Notices¶
- API-like endpoints (returning JSON) that start with
<galaxy>/root/
in the URL are now deprecated and will be removed in future releases. Please use proper API (e.g./api/histories
or/api/tools
) to replace your implementations. - The
jsonp
parameter for search API endpoints in Tool Shed is deprecated and will be removed in the future. Instead the presence of acallback
argument will trigger the JSONP format to be returned.
Security¶
An arbitrary code execution vulnerability in two tools and an XSS vulnerability with the upload tool were identified this release cycle and have been fixed concurrently with the release. In addition, the fixes have been backported to older releases.
The Galaxy Committers would like to thank David Wyde for disclosing these vulnerabilities. Details follow:
- The vulnerable tools are “Filter GFF data by attribute” and “Filter GFF data
by feature count”, both of which are provided with and enabled by default in
the Galaxy server. These two tools share code with each other and the more
general “Filter data on any column using simple expressions” tool. The
latter was fixed in a previous security disclosure but these GFF variants of
the tool were missed when updating the Filter tool. These tools use the
Python
eval
andexec
functions and do not properly sanitize input to these functions. The fix for this issue has been applied to Galaxy releases back to v14.10 and can be found in Commit c1e3087 - An uploaded file’s name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. The fix for this issue has been applied to Galaxy releases back to v16.07 and can be found in Pull Request 3278.
Release Notes¶
Highlighted Enhancements¶
- Overhaul of charts visualization - more visualizations, more options, and better user interface. Pull Request 2875, Pull Request 2820
- Paginate contents of large histories. Pull Request 2588
- Implement a collection operation tool for merging collections (thanks to @Takadonet). Pull Request 2771
- Replace reference documentation for tool XML files with automatically generated documentation from a now official Galaxy XSD documentation (with help from many). Pull Request 2923, Pull Request 2936, Pull Request 3086, Pull Request 2932, Pull Request 2903, Pull Request 3020, Pull Request 3072
- Add a password strength evaluation bar (thanks to @benfulton). Pull Request 2687
- Implement a GoDocker job runner (thanks to @varunshankar). Pull Request 2791, Pull Request 2653
- Support for API batch requests. Pull Request 1768
- Allow JSONP to be returned from API endpoints. Pull Request 2937
- Add “Save as” as an option in the workflow editor (thanks to @tmcgowan). Pull Request 3035
- Allow naming input datasets and collections during workflow extraction. Pull Request 2943
- Various enhancements for API driven installation of tool shed repositories.
Security¶
- Fix an ACE vulnerability with gff filter tools (thanks to David Wyde). Commit c1e3087
- Escape filename in upload dialog, default and composite (thanks to David Wyde). Pull Request 3278
Enhancements¶
- Add “Save as” as an option in the workflow editor (thanks to @tmcgowan). Pull Request 3035
- Allow ‘.’ and ‘_’ for public names (thanks to @nsoranzo). Pull Request 2947
- Allow naming input datasets and collections during workflow extraction. Pull Request 2943
- Various enhancements for API driven installation of Tool Shed repositories. Pull Request 2769
- Implement beta support for discovering biocontainer Docker containers automatically for tool dependencies. Pull Request 2986
- Improved API support for downloading datasets and relevant files. Pull Request 2741, Pull Request 2885
- Make tool search field stay at top of window when tool list scrolls. (thanks to @sszakony). Pull Request 2730, Issue 2375
- Scratchbook enhancement to include info and view details links for datasets. Pull Request 2733
- Add implicit conversions target extension to HDA name when listed as selection option in the tool form. Pull Request 2734
- Add column names to BED datatype so that column names are displayed in the dataset viewer. Pull Request 2784
- Add “All” and “None” buttons to copy dataset view. Pull Request 2957
- Lazy load workflow invocation steps to vastly improve scheduling performance of large collection workflows (thanks to @nsoranzo). Pull Request 2978
- Add CEL datatype (thanks to @nturaga). Pull Request 3073
- Add a netcdf datatype for metabolomics spectre data (thanks to @lecorguille). Pull Request 2706
- Returning Blast datatypes to Galaxy core (thanks to @peterjc). Pull Request 2696
- Add HDT format (with turtle format sniffer improvements) (thanks to @jjkoehorst). Pull Request 2787
- Allow more configurability for containerized jobs. Pull Request 2790
- Functionality to allow admins to switch on/off the line endings conversion on library dataset import (thanks to @SANBI-SA). Pull Request 2819
- Add a tutorial on building job runners to the documentation (thanks to @varunshankar). Pull Request 2700
- Allow viewing tool help from history datasets (thanks to @tmcgowan). Pull Request 3051
- Prevent TS from creating sessions for repository status checks. Pull Request 2732
- Shorten Conda environment names (thanks to @bgruening). Pull Request 2794
- Various release process improvements (including an enhancement from @nsoranzo). Pull Request 2857, Pull Request 2858, Pull Request 2689, Pull Request 2865, Pull Request 2952, Pull Request 2695, Pull Request 2991, Pull Request 3232
- Improve samtools usage with
--version-only
check (thanks to @remimarenco). Pull Request 2705 - All-around improvements to toolbox loading/reloading (thanks to @mvdbeek). Pull Request 2901
- Refactor shed tool lineage registration (thanks to @mvdbeek). Pull Request 2909
- Log the resolved dependency type (thanks to @mvdbeek). Pull Request 2917, Pull Request 3036
- Fixes for Python 3 and new import linting for dozens of packages (largely thanks to @nsoranzo). Pull Request 2731, Pull Request 3177, Pull Request 2983, Pull Request 2889, Pull Request 3031, Pull Request 3170, Pull Request 3154
- Change every occurrence in the code of
__all__
from list to a tuple (thanks to @nsoranzo). Pull Request 3041, Pull Request 3071 - Docstring linting and fixes. Pull Request 3003, Pull Request 3004
- Update python-ldap module (thanks to @abretaud). Pull Request 2738
- Update sqlalchemy to 1.0.15 (from 1.0.8). Pull Request 2979
- Slightly revise auto configuration of loggers. Pull Request 2735
- Various
Makefile
improvements and fixes. Pull Request 2757, Pull Request 2688, Pull Request 3167 - Improve startup scripts (thanks to @nsoranzo). Pull Request 2765
- Remove
enable_beta_tool_command_isolation
config option (thanks to @nsoranzo). Pull Request 2830 - Regularize comments configuration samples. Pull Request 2824, Pull Request 2841, Pull Request 2814
- Skip whoami check for LDAP servers not supporting it (thanks to @nsoranzo). Pull Request 2832
- Filter disposable mail blacklist on the first domain level only. Pull Request 2839
- Refactor duplicated back-end code to create a new
trans
method for getting the most recently touched history. Pull Request 2870 - Document setting column names in Galaxy tool XSD. Pull Request 2965
- Functional test case for testing column names as meta-data (thanks to @yhoogstrate). Pull Request 2970
- Improve
version_command
tag. Pull Request 2961 - Add
rgb
option for color picker tool parameter. Pull Request 3006 - Improved docs and testing for color parameters. Pull Request 3016
- Add documentation on optional
multiple="true"
data parameters. Pull Request 3024 - In Trackster, read numerical data directly from bigWig files. Pull Request 2880
- Improve wording of tool installation forms (thanks to @nturaga). Pull Request 2893
- Add database/dependencies to .gitignore (thanks to @nturaga). Pull Request 2904
- Allow admin to serialize/deserialize dataset permissions via the API. Pull Request 2912
- Allow API access to genome fasta index and sequence (thanks to @jj-umn). Pull Request 2939
- Extend
PUT /api/workflows/{id}
with additional workflow properties (thanks to @tmcgowan). Pull Request 3080 - Remove unnecessary use of binascii (thanks to @nsoranzo). Pull Request 2946
- Change wording to be more clear about how workflow output cleanup works. Pull Request 2966
- Optimized workflow invocation step update. Pull Request 2973
- Update issues.rst to bring inline with actual usage on Github. Pull Request 2976
- Improve Conda FAQ documentation. Pull Request 2998, Pull Request 2891, Pull Request 2871
- Add documentation for
conda_copy_dependencies
option (thanks to @abretaud). Pull Request 3105 - Slight clarification of run.sh messaging when using Conda. Pull Request 3188
- Display tool requirements for conda-only tools (thanks to @mvdbeek). Pull Request 3262
- Pass dependency shell command errors to log (thanks to @mvdbeek). Pull Request 3129
- Increase min number of columns for pileup sniffing to 5 (thanks to @nsoranzo). Pull Request 3033
- Remove outdated warning on ID_SECRET (thanks to @erasche). Pull Request 3046
- Add Galaxy architecture slides documentation to project. Pull Request 3060, Pull Request 3091
- Add tool XML reload to tool run page - for admins only (thanks to @tmcgowan). Pull Request 3070, Issue 943
- Move thumbnail list view to shared UI elements, revise tabs, add test cases Pull Request 3079
- Include filenames in .loc file DEBUG lines (thanks to @peterjc). Pull Request 3095
- Log loading of resources from .loc files (thanks to @peterjc). Pull Request 3099
- Eager load tags and annotations in history display. Pull Request 3110
- Display an error message when toolbox filtering fails. Pull Request 3123
- Bring in various updates from galaxy-lib. Pull Request 3169, Pull Request 2888
- Add links to local tours to sample welcome. Pull Request 3176
- Fix dataset selector update when deleting history items. Pull Request 2677
- Refactor upload dialog to use local CSS classes and avoid model listeners. Pull Request 2711
- Rework Tool Shed middleware Pull Request 2782, Pull Request 2753
- Refactor - move filelock.py to galaxy.util. Pull Request 2879
- Use requests in driver_util for better proxy handling (thanks to @mvdbeek). Pull Request 3013
- Add
download_to_file
function togalaxy.util
(thanks to @nsoranzo). Pull Request 3100 - Quote all paths in converter tool XML files (thanks to @mvdbeek). Pull Request 3165
- Add routes to get genomes indexes and sequences (thanks to @jj-umn). Pull Request 3109
- Admin installation UX wording fixes. Pull Request 3226
Fixes¶
- Fix for bioconda recipes depending on conda-forge. Pull Request 3023
- Fix history deletion bug when using impersonation. Pull Request 2654
- Fix workflow extraction API tests on PostgreSQL (thanks to @nsoranzo). Pull Request 3039
- Fix link target in admin view of tool lineages. Pull Request 2862
- Avoid pointless lambda function (thanks to @peterjc). Pull Request 3097
- Remove unnecessary
set_output_history
parameter (thanks to @nsoranzo). Pull Request 3155 - Fix BLAST database
*.loc
files inconsistency (thanks to @peterjc). Pull Request 3098 - Log invalid XML filename (thanks to @mvdbeek). Pull Request 3124
- Various code fixes for object store and docs (thanks to @nsoranzo). Pull Request 3119
- Use default job history on
build_for_rerun
if current history is unavailable. Pull Request 2668 - Always display text Pull Request 3052
- Fixing button width on composite upload dialog (thanks to @sszakony). Pull Request 2703, Issue 2591
- Fix for splitting dataset collections (thanks to @gregvonkuster). Pull Request 2708
- Various datatypes fixes and enhancements (thanks to @nsoranzo). Pull Request 2690
- Various fixes and improvements for galaxy.json. Pull Request 2697
- Fix tours pasting content into text area. Pull Request 2715
- Slight styling tweak to prevent overflow on the right edge in composite upload dialog. Pull Request 2716
- Workflow toolbox style updates. Pull Request 2740
- Fix docker command_list (thanks to @manabuishii). Pull Request 2748
- Fix
list:list
history display in the GUI. Pull Request 2758 - Parse internal parameters such as job resource selections when data converter tools are executed implicitly. Pull Request 2761
- Fix broken check when creating Docker volumes default for jobs (thanks to @manabuishii). Pull Request 2763
- Fix to prevent login form from appearing in scratchbook window (thanks to @sszakony). Pull Request 2808
- Allow NodeJS proxy to do reverse proxying (thanks to @erasche). Pull Request 2817
- Fixes for the RStudio interactive environment (thanks to @erasche). Pull Request 2818
- Acquire lock before attempting to install conda (thanks to @mvdbeek). Pull Request 2826
- Fix Galaxy for if pyuwsgi is install in Galaxy’s virtual environment. Pull Request 2836
- Fix tabular renderer display when
total_cols > line_cols > 1
. Pull Request 2845 - Remove empty file in toolbox module. Pull Request 2877
- Fix for workflow execution when using collections. Pull Request 2898, Issue 2806
- Implement
split
on a validator. Pull Request 2921 - Catch failure to import grp in util (for Pulsar on Windows) Pull Request 2928
- Bump pysam, remove
already_compressed
usage due to new wheel. Pull Request 2953 - Re-build DOM on render for modal in the front-end. Pull Request 2955
- Fix optional column form values when editing dataset properties. Pull Request 2960
- Fix optional flag for library datasets in legacy library UI. Pull Request 2962
- Reload metadata and lib tools on toolbox reload (thanks to @mvdbeek). Pull Request 2992
- Update some old information in API guideline docs. Pull Request 3011
- Fix for rerunning tool with workflow resume that is part of a collection. Pull Request 3019
- Missing double quotation in tool XSD example (thanks to @ramezrawas). Pull Request 3027
- Include base classes when searching datatypes by name. Pull Request 3028
- Image detection fixes. Pull Request 3042
- Fix tabular filtering tool broken in #2983 Pull Request 3043
- Fix GFF attribute filtering tool broken in #2983 Pull Request 3044
- Fix for displaying parameters to the user that have a
False
value. Pull Request 3045 - Fix downloading tool tarball with boolean test input values. Pull Request 3047
- Sort bedtools output in
bam_to_bigwig
conversion (thanks to @lparsons). Pull Request 3049 - Remove incorrect communication server check. Pull Request 3053
- Fix tool XSD to accept a help attribute for
section
s (thanks to @joachimwolff). Pull Request 3131 - Fix import orders for updates to flake8_import_order. Pull Request 3059
- Dataset metadata should not be cached when building chart visualizations. Pull Request 3062
- Fix libraries prefix. Pull Request 3074
- Fix qunit local testing install – now runs on modern node (v6.9.1). Pull Request 3089
- Escape section name when writing to XML (thanks to @mvdbeek). Pull Request 3122
- Fix tools placed outside of panel section (thanks to @mvdbeek). Pull Request 3125
- Fix XSD for
exclude_min
andexclude_max
(thanks to @gregvonkuster). Pull Request 3108 - Update galaxy.xsd (thanks to @lecorguille). Pull Request 3132
- Fix to treat the value of
message_box_visible
ingalaxy.ini
as a boolean. Pull Request 3139 - Ensure a From: address is set for user activation emails. Pull Request 3140
- Explicitly specify attributes which should be copied to converted datasets. Pull Request 3149
- Prefer existing .venv over conda, when both are available. Pull Request 3180
- Validate cycles and step size before building model for workflow execution. Pull Request 3183
- Fix a typo in the intro tour. Pull Request 3184
- Make SAM to bam converter tool compatible with samtools >= 1.3 (thanks to @mvdbeek). Pull Request 3187
- Remove unused (and unintended) config override. Pull Request 3198
- Fixes for deleting histories. Pull Request 3203
- Change ‘History Actions’ to ‘Current History’ in history options menu. Pull Request 3205
- Preserve custom event handler for form inputs. Pull Request 3210
- Admin installation UX wording fixes. Pull Request 3226
- Backport #3106 and #3222: Cached conda environments and API to manage them (thanks to @mvdbeek). Pull Request 3227
- Update the copyright year in the LICENSE.txt. Pull Request 2926
- Fix for ToolShed install when copied sample data target exists, but is broken symlink. Pull Request 3279
To stay up to date with Galaxy’s progress watch our screencasts, visit our community hub, and follow @galaxyproject on Twitter.
You can always reach us on Gitter or IRC.
Thanks for using Galaxy!