January 2016 Galaxy Release (v 16.01)
Highlights
- Interactive Tours
The interactive tours framework allows developers and deployers to build interactive tutorials for users superimposed on the actual Galaxy web front end. Unlike video tutorials, these will not become stale and are truly interactive (allowing users to actually navigate and interact with Galaxy). Galaxy 16.01 ships with two example tours and new ones can easily be added by creating a small YAML file describing the tour. Try the Galaxy UI tour on Main.
- Wheels
Galaxy’s Python dependencies have traditionally been distributed as eggs using custom dependency management code to enable Galaxy to distribute binary dependencies (enabling quick downloads and minimal system requirements). With this release all of that infrastructure has been replaced with a modern Python infrastructure based on pip and wheels. Work done as part of this to enable binary dependencies on Linux has been included with the recently released pip 8.
Detailed documentation on these changes and their impact under a variety of Galaxy deployment scenarios can be found in the Framework Dependencies section of the Admin documentation.
- Nested Workflows
Workflows may now run other workflows as a single abstract step in the parent workflow. This allows for reusing or subworkflows in your analyses.
Github
- New
% git clone -b master https://github.com/galaxyproject/galaxy.git
- Update to latest stable release
% git checkout master && pull --ff-only origin master
- Update to exact version
% git checkout v16.01
BitBucket
- Upgrade
% hg pull % hg update latest_16.01
See our wiki for additional details regarding the source code locations.
Deprecation Notices
Barring a strong outcry from deployers, 16.01 will be the last release of Galaxy to support Python 2.6. For more information, see Galaxy Github Issue #1596.
Security
Multiple security vulnerabilities were identified during this release cycle and fixed concurrently with the release. In addition, the fixes have been backported to older releases.
The Galaxy Committers would like to thank Youri Hoogstrate at the Erasmus MC, Rotterdam, who initially
reported the hg push
vulnerability. Through additional auditing based on
this attack vector, we discovered the other vulnerabilities.
- Galaxy
Multiple security vulnerabilities were discovered in Galaxy that allow malicious actors to read and write files on the Galaxy server. Additionally, Galaxy servers on which a rarely used feature has been enabled are vulnerable to an arbitrary code execution exploit.
A write vulnerability exists in the history import mechanism. It is possible to create a history tar archive that contains files with parent directory components in the file path (e.g.
foo/../../bar
would extract to../bar
), and these archive members would be written if the user running the Galaxy server had write permission to the given path.A read vulnerability exists in the object store path composition code. Galaxy allows clients to add elements to the end of a path to “extra” files associated with a dataset (as is the case with composite datatypes). These elements were not being checked to ensure they did not contain relative parent references (
..
) or did not start with an absolute path character (/
). Because of this, the dataset display methods could be manipulated to return the contents of any files for which the Galaxy server user had read permission.An arbitrary code execution vulnerability exists in the Galaxy sample tracking system. The sample tracking system included a feature which allowed administrators to browse remote “external services” (such as sequencers) to choose files to transfer to the Galaxy server. This browsing code used a shell invocation which did not sanitize user input. However, this code is only reachable if at least one external service has ever been defined.
Fixes for all three issues have been applied to Galaxy releases back to v14.10.
- Tool Shed
Multiple security vulnerabilities were discovered in the Tool Shed that allow malicious actors to read and write files on the Tool Shed server outside of normal Tool Shed repository directories.
A write vulnerability exists in the Tool Shed tarball and capsule upload functionality. It is possible to create a tar archive that contains files with parent directory components in the file path (e.g.
foo/../../bar
would extract to../bar
), and these archive members would be written if the user running the Tool Shed had write permission to the given path. The Tool Shed tarball handling code checked for invalid characters (/
or..
) at the beginning of the path but not for..
in the middle of a path.A read vulnerability exists in multiple places. The first is in the (now deprecated)
hg push
functionality for updating Tool Shed repositories. This method allows malicious actors to push symlinks whose targets are outside the repository (either via an absolute or relative path). The contents of the targets would then be visible in the Tool Shed repository contents viewer, if the Tool Shed user has read permission on the target.A second read vulnerability exists in the Tool Shed repository contents viewer. The viewer would allow a malicious actor to specify a path outside the repository, and if the Tool Shed system user had read permissions on that path, it would be displayed. The viewer also did not check to ensure that the targets of symlinks in a repository did not point outside the repository.
The repository contents viewer read vulnerability also exists in Galaxy, but is only reachable/exploitable by admin users. Fixes for vulnerability #3 have been applied to Galaxy/Tool Shed releases back to v14.10, and issues #1 and #2 have been applied to releases back to v15.01.
Release Notes
Enhancements
Interactive tours (with significant help from @bgruening.) Pull Request 1422, Pull Request 1464
Replace Galaxy eggs dependency management with wheels. Pull Request 428, Pull Request 989, Pull Request 988, Pull Request 1389, Pull Request 1485, Pull Request 995, Pull Request 996, Pull Request 1006, Pull Request 1017, Pull Request 1037, Pull Request 1495
Implement nested workflows. Pull Request 1306
Use Webpack to greatly optimize JavaScript and other assets served by Galaxy. Pull Request 1144
Overhaul Galaxy analysis view toward a more modular JavaScript architecture. Pull Request 706, Pull Request 1184
Add BAM iobio visualization as a Galaxy Interactive Environment (thanks to @bgruening.) Pull Request 1131
Add Jupyter Interactive Environment (thanks to @bgruening.) Pull Request 1400
Procedures for bug and issue handling as well as roadmap management. Pull Request 902, Pull Request 1020
Implement a conda dependency resolver (with significant help from @bgruening.) Pull Request 1345, Pull Request 1527, Pull Request 1421, Pull Request 1409
Many ToolBox improvements including allowing specifying labels on individual tools, monitoring and automatic reloading of
tool_conf.xml
files, and allowing specification of such files in JSON/YAML. Pull Request 1012, Pull Request 1398Configurable client side logging. Pull Request 1011
Allow input collections to specify multiple
collection_type
s. Pull Request 1308Allow multiple collections to be supplied to a multiple data parameter. Pull Request 805
Implement
type_source
on output collections. Pull Request 1153Allow tools to request a special configuration file containing tool parameters dumped to json. Pull Request 1405
Add search on username/any for API calls to user and use Select2 for sharing workflows/pages/histories (thanks to @hexylena.) Pull Request 1111
Improve masthead and scratchbook UI and architecture. Pull Request 1407, Pull Request 1424
Add support for constructive solid geometry datatypes
ply
andvtk
. (Thanks to @gregvonkuster.) Pull Request 905, Pull Request 1211Add searchgui datatype (thanks to @jj-umn.) Pull Request 914
Add BIOM v1 datatype (thanks to @fescudie.) Pull Request 950, Pull Request 1198
Add CRAM datatype (thanks to @yhoogstrate.) Pull Request 1108, Pull Request 1182
Add UniProtXML datatype (thanks to @jj-umn.) Pull Request 1004
Avoid confusion by adding the PDF datatype to the list of uploadable file types. Pull Request 901
Add idpDB and HDF5 sniffers and fix MzSQlite sniffer (thanks to @chambm.) Pull Request 1209
Improve tool shed installation API for installation of repositories if they are already installed (thanks to @bgruening.) Pull Request 1095
Prevent GIEs from being closed without user confirmation (thanks to @bgruening.) Pull Request 1178
Add the constructive solid geometery (CSG) viewer visualization plugin (thanks to @gregvonkuster.) Pull Request 1254
Add Google analytics tracking to new data libraries. Pull Request 959
Extend users API with delete action (thanks to @nsoranzo.) Pull Request 1063
Finish swapping unencoded ids with order_index in workflows API. Pull Request 1137
Add documentation and extra options to Environment Modules resolver (thanks to @pvanheus.) Pull Request 1221
Documentation for dependency resolvers (thanks to @pvanheus.) Pull Request 1296
Populate env.sh files created via tool shed installations with
X_ROOT_DIR
environment variables. Pull Request 564, Pull Request 889Reorganize history menu (thanks to @hexylena.) Pull Request 985
Look up installed tools from different toolshed(s) (thanks to @mvdbeek.) Pull Request 818, Pull Request 1286
Documentation updates for ObjectStore (thanks to @mr-c.) Pull Request 853
Update JavaScript dependency Dynatree. Pull Request 856
Eliminate LWR from Galaxy (you will need to upgrade to Pulsar). Pull Request 857
Isolate tool commands by default. Pull Request 1412, Pull Request 1494
Allow override of job shell (for conda resolver). Pull Request 1473
Slight tweak to sidebar section naming from ‘security’ to ‘user management’. Pull Request 877
Fix the issue where too many datasets are linked to a certain role (thanks to @scholtalbers.) Pull Request 881
Fix R dependency installation to be more portable (thanks to @bgruening.) Pull Request 893
Reorder workflow inputs to top (thanks to @scholtalbers.) Pull Request 912
Add an API script to upload directory as a data library (thanks to @hexylena.) Pull Request 918
Enable dockerized toolshed tests. Pull Request 942
Improvements and fixes for abstract tool interface. Pull Request 955
Remove remaining references to galaxy.eggs and flake8 some tools (thanks to @nsoranzo.) Pull Request 990
Apply stricter flake8 rules for the directories shared with Pulsar (thanks to @nsoranzo.) Pull Request 1010
Updates to default welcome page. Pull Request 1013
Revise deferred queue, use jQuery promises. Pull Request 1018
Keep select field open in tool form multi select fields to ease selecting many options quickly. Pull Request 1019
Allow resizing of regular multi-select boxes. Pull Request 1025
Add error message for wrongly formatted data in wig_to_bigwig (thanks to @scholtalbers.) Pull Request 1033
Verify len files during download from UCSC (thanks to @nsoranzo.) Pull Request 1066
Add biocrusoe to contributors Pull Request 1067
Outline “Format 2” workflow definitions. Pull Request 1096
Improve the API attributes display on parameters page. Pull Request 1098
Update run_tests.sh help for recent changes (thanks to @nsoranzo.) Pull Request 1101
Show welcome page with required login. Pull Request 1105
Allow accessing collection elements in format_source. (with significant help from @nsoranzo.) Pull Request 1113, Pull Request 1155
Add nginx config for GIE proxy to documentation. Pull Request 1123
Enforce client build deps are up-to-date. Pull Request 1130
Menu onclick addition. Pull Request 1142
Ensure confirmation when leaving GIE windows. Pull Request 1157
Add local grunt-cli dependency for qunit tests. Pull Request 1159
Tighten up GG sniffing Pull Request 864
Use common exceptions in tools. Pull Request 874
Refactor the remote user middleware for complexity (thanks to @hexylena.) Pull Request 875
Allowed for capitalized package names such as R in unlinked tool shed dependency resolver (thanks to @Christian-B.) Pull Request 1160
A slew of tool execution performance optimizations including a huge fix from @ericenns. Pull Request 1166, Pull Request 1163, Pull Request 1199, Pull Request 1212
Do not check for tool migrations when running tests. Pull Request 1176
Lazy load HDA/LDDA metadata to speed up history loading (thanks to @abretaud.) Pull Request 1179
Add debug statement in output checker for why job is failing. Pull Request 1213
Small enhancements in workflow inputs and outputs. Pull Request 1214
More small workflow tweaks. Pull Request 1216
Add Workflow editor UI for step labels. Pull Request 1251
Ease the grunt uglify process (Thanks to @hexylena.) Pull Request 1238
Add beta run workflow form based on the newer tool form code. Pull Request 1249
During tool migrations, do not mess with galaxy_config_file if provided (thanks to @openlangrid and @nsoranzo.) Pull Request 1230, Pull Request 1253
In the users API, properly return a boolean from has_requests Pull Request 1262
Extend the logging of tool dependency status changes (thanks to @nsoranzo.) Pull Request 1263
Remove history_options and options.mako (history options as a page). Pull Request 1271
Minor mail config cleanup Pull Request 1299
Replace uses of history.imp and history.copy web methods for API create Pull Request 1303
More tightening up of parameter validation during workflow stuff. Pull Request 1319
Show a more user-friendly error when the webserver port is blocked (thanks to @shano.) Pull Request 1320
Upgrade Paste to 2.0.2. Pull Request 1344
A series of small refactoring enabling use of certain Galaxy modules a stand-alone Python library with minimal dependencies and Python 3 compatibility. Pull Request 1350, Pull Request 1351, Pull Request 1352, Pull Request 1359, Pull Request 1362, Pull Request 1376, Pull Request 1413, Pull Request 1427, Pull Request 1363, Pull Request 1367, Pull Request 1377, Pull Request 1388, Pull Request 1448
Adding UUID support to directory_hash_id (thanks to @kellrott.) Pull Request 1397
Unify and abstract code for checking if file looks like a tool definition. Pull Request 1368
Added hashes to pip requirements file (thanks to @hexylena.) Pull Request 1393
Interactive environment updates (thanks to @bgruening.) Pull Request 1399
Change scratchbook close icon. Pull Request 1425
Rename reports_wsgi.ini to reports.ini (thanks to @souravsingh.) Pull Request 1181
Attempt to fix more transiently failing API tests. Pull Request 859, Pull Request 865
Fix data library test case. Pull Request 898
Remove
test_map_over_two_collections_legacy
test case as it is obsolete. Pull Request 924Version the testing-base docker image. Pull Request 938
Update casperjs functional tests. Pull Request 944
Fix the docker db client_encoding to not be ascii (default). Pull Request 952
Use the database temp directory to store the Mako template cache when running framework tests. Pull Request 956
Rename
lib/galaxy/main.py
to scripts/galaxy-main Pull Request 994Fix deferred dom removal. Pull Request 997
Options to more easily test esoteric tooling options. Pull Request 1014
Fix log statements and link to logger. Pull Request 1040
Add more data libraries API tests. Pull Request 1074
Remove upload unused function parameter (thanks to @einon.) Pull Request 1078
Update testing docker image. Pull Request 1083
Small tool and workflow refactorings. Pull Request 1097
Revise and fix waiting for tool tests. Pull Request 1119
Small Tool and Workflow Refactoring and Fixes Pull Request 1202
Wait on jobs and history in certain API test cases. Pull Request 1226
Fix qunit shim to match base_panels.mako shim. Pull Request 1233
Improved logging related tool test timeouts. Pull Request 1243
Refactor generic side workflow editor panel toward backbone. Pull Request 1247
Attempt to fix transiently failing tool test on Jenkins. Pull Request 1248
Set client_encoding for TS dockerized test db Pull Request 1276
Check for sessionStorage using a more cross-browser way. Pull Request 1279
Small API test improvements. Pull Request 1285
Enforce metrics related to moving toward modern client infrastructure. Pull Request 1292
Refactor tool stuff for generic model actions. Pull Request 1307
Remove workflow casperjs API test duplicating test coverage of API tests. Pull Request 1316
Refactor tool parsing handling toward reuse outside Galaxy. Pull Request 1349, Pull Request 1353
Lint with Python 3 several modules. Pull Request 1354
Fix and potential fix for transiently failing tests. Pull Request 1401
Fix CasperJS tests. Pull Request 1438, Pull Request 1439
Add a log.warn() if extracted file does not exist when changing permissions. (thanks to @nsoranzo.) Pull Request 939
Add a safe_relpath util function for ensuring a path does not reference an absolute or parent directory. Commit f540a16
Security
Security fixes for history imports. Commit bf1c77d
Security fixes for object store paths. Commit 5da91bd
Remove sample tracking manual external service transfer due to security concerns. Commit cd8b965
Security fixes for tool shed repository browsing. Commit e4a1d57
Security fixes for tool shed hg push and capsule/tarball uploads. Commit e845d64
Fixes
Add check for
HTTP_GX_SECRET
(thanks to @golharam.) Pull Request 827Test if parameter present in value to avoid index error in tool state handling (thanks to @mvdbeek.) Pull Request 863
Bump version of sqlite3 in nodejs proxy (thanks to @hexylena.) Pull Request 866
Expose API keys to admins under
REMOTE_USER
(thanks to @hexylena.) Pull Request 872Bug fix for tool shed repository API (thanks to @mvdbeek.) Pull Request 891
Add a workaround to return a proper error code during R package installation (thanks to @bgruening.) Pull Request 892
Various Quota bug fixes. Pull Request 907
Always set ‘tests’ for a visualization plugin to avoid an attribute error. Pull Request 908
Bug fix for condor runner changes made this release cycle (thanks to @hexylena.) Pull Request 909
Change user preference datatype to text. Pull Request 916
Fix syntax of error parameter of
tryCatch
insetup_r_environment action
(thanks to @nsoranzo.) Pull Request 929Revise tool URL building. Pull Request 947
Fix for code checking if a file seems to a tool definition file (thanks to @hexylena.) Pull Request 949
Fix base panels to include scripts as the last element of the body. Pull Request 969
Grid batch operation fixes. Pull Request 971
Fix extra files path URL problem. (thanks to @hexylena.) Pull Request 991
Fixes broken env-var declarations for tools with weird chars in their names (thanks to @yhoogstrate.) Pull Request 1028
Added gitignore rules to ignore custom tool-data (thanks to @yhoogstrate.) Pull Request 1048
Fix import of history datasets into library. (thanks to @mvdbeek.) Pull Request 1049
Fix for reloading tools that have non-standard tool_ids/versions. Pull Request 1050
Improved encoding handling for Jobs. Pull Request 1052
Fix lped report output from converter. Pull Request 1069, Pull Request 1070, Pull Request 1072
Trivial: Replace unnecessary duplicated var check with ‘else if’ (thanks to @einon.) Pull Request 1073
Fix a bug in IEs when proxying the proxy. Pull Request 1076
Fix 500 error when attempting to update installed repository. Pull Request 1082
Resolve conflicting label CSS class for trackster. Pull Request 1086
Fix bug with referrer attribute type change in WebOb. Pull Request 1091
Fix API TS installation (thanks to @bgruening.) Pull Request 1094
Better error when tool shed repository directory is missing (thanks to @lparsons.) Pull Request 1107
Don’t let
$input
hang cheetah evaluation. Pull Request 1117Fix for re-installing an uninstalled TS repository with a dependency (Thanks to @gregvonkuster.) Pull Request 1154
Fix interface and usage of
WorkflowModule.get_runtime_inputs
. Pull Request 1174Add enhancements to the Galaxy repository install process (thanks to @mvdbeek.) Pull Request 1193
Tool shed fixes (thanks to @nsoranzo.) Pull Request 1200
Fix for updating tool parameter dicts when a new parameter has been added to a section. Pull Request 1215
Replace the defunct readthedocs badge. Pull Request 1229
Export
GALAXY_TEST_DBURI
asGALAXY_CONFIG_OVERRIDE_DATABASE_CONNECTION
before installing wheels. (thanks to @nsoranzo.) Pull Request 1231Fix passing of nginx_upload_path and ftp_upload_site. Pull Request 1250
Fixed indentation errors for reports app (thanks to @markiskander.) Pull Request 1259
Open select2 drop down on caret click. Pull Request 1298
Improved validation of tools during workflow execution. Pull Request 1302
Properly remove datasets from the filtered lists when pairing datasets for the paired dataset list creator. Pull Request 1310
Update Kombu and AMQP wheels to fix problems with El Capitan’s System Integrity Protection. Pull Request 1327
Fix for creating workflow outputs on initial workflow upload. Pull Request 1330
Don’t query on unencoded IDs for error form (thanks to @hexylena.) Pull Request 1340
If
GALAXY_SLOTS
is defined in the environment, use it for the local runner. Pull Request 1346Use both
SLURM_NTASKS
andSLURM_CPUS_PER_TASK
to setGALAXY_SLOTS
(thanks to @lparsons.) Pull Request 1347Fix for loading workflows that have tool version / step upgrade messages. Pull Request 1348
Allow installation of different repositories with the same name in a single request (thanks to @mvdbeek.) Pull Request 1366
Copy workflow objects when importing them. Pull Request 1474
Undo user icon in masthead. Pull Request 1493
Fix mime type when previewing certain tabular data. Pull Request 1498
Fix disabled CSS. Pull Request 1501
catch Exception and properly log errors Pull Request 1511
Fix for workflow validation problem introduced in 15.10. Pull Request 1536, Issue #1514
Keep track of hidden datasets. Pull Request 1551
Force
--skip-venv
if we can detect that Python is Conda Python. Pull Request 1554Fix installation of Tool Shed repositories containing non-ASCII characters in the description. (thanks to @nsoranzo.) Pull Request 1759
Fix pretty_print_time_interval for MySQL. (thanks to @nsoranzo.) Pull Request 1761
To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.
You can always chat with us on Matrix.
Thanks for using Galaxy!