Warning
This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
January 2016 Galaxy Release (v 16.01)¶
Highlights¶
- Interactive Tours
- The interactive tours framework allows developers and deployers to build interactive tutorials for users superimposed on the actual Galaxy web front end. Unlike video tutorials, these will not become stale and are truly interactive (allowing users to actually navigate and interact with Galaxy). Galaxy 16.01 ships with two example tours and new ones can easily be added by creating a small YAML file describing the tour. Try the Galaxy UI tour on Main.
- Wheels
Galaxy’s Python dependencies have traditionally been distributed as eggs using custom dependency management code to enable Galaxy to distribute binary dependencies (enabling quick downloads and minimal system requirements). With this release all of that infrastructure has been replaced with a modern Python infrastructure based on pip and wheels. Work done as part of this to enable binary dependencies on Linux has been included with the recently released pip 8.
Detailed documentation on these changes and their impact under a variety of Galaxy deployment scenarios can be found in the Framework Dependencies section of the Admin documentation.
- Nested Workflows
- Workflows may now run other workflows as a single abstract step in the parent workflow. This allows for reusing or subworkflows in your analyses.
Github¶
- New
% git clone -b master https://github.com/galaxyproject/galaxy.git
- Update to latest stable release
% git checkout master && pull --ff-only origin master
- Update to exact version
% git checkout v16.01
BitBucket¶
- Upgrade
% hg pull % hg update latest_16.01
See our wiki for additional details regarding the source code locations.
Deprecation Notices¶
Barring a strong outcry from deployers, 16.01 will be the last release of Galaxy to support Python 2.6. For more information, see Galaxy Github Issue #1596.
Security¶
Multiple security vulnerabilities were identified during this release cycle and fixed concurrently with the release. In addition, the fixes have been backported to older releases.
The Galaxy Committers would like to thank Youri Hoogstrate at the Erasmus MC, Rotterdam, who initially
reported the hg push
vulnerability. Through additional auditing based on
this attack vector, we discovered the other vulnerabilities.
- Galaxy
Multiple security vulnerabilities were discovered in Galaxy that allow malicious actors to read and write files on the Galaxy server. Additionally, Galaxy servers on which a rarely used feature has been enabled are vulnerable to an arbitrary code execution exploit.
- A write vulnerability exists in the history import mechanism. It is
possible to create a history tar archive that contains files with parent
directory components in the file path (e.g.
foo/../../bar
would extract to../bar
), and these archive members would be written if the user running the Galaxy server had write permission to the given path. - A read vulnerability exists in the object store path composition code.
Galaxy allows clients to add elements to the end of a path to “extra”
files associated with a dataset (as is the case with composite datatypes).
These elements were not being checked to ensure they did not contain
relative parent references (
..
) or did not start with an absolute path character (/
). Because of this, the dataset display methods could be manipulated to return the contents of any files for which the Galaxy server user had read permission. - An arbitrary code execution vulnerability exists in the Galaxy sample tracking system. The sample tracking system included a feature which allowed administrators to browse remote “external services” (such as sequencers) to choose files to transfer to the Galaxy server. This browsing code used a shell invocation which did not sanitize user input. However, this code is only reachable if at least one external service has ever been defined.
Fixes for all three issues have been applied to Galaxy releases back to v14.10.
- A write vulnerability exists in the history import mechanism. It is
possible to create a history tar archive that contains files with parent
directory components in the file path (e.g.
- Tool Shed
Multiple security vulnerabilities were discovered in the Tool Shed that allow malicious actors to read and write files on the Tool Shed server outside of normal Tool Shed repository directories.
- A write vulnerability exists in the Tool Shed tarball and capsule upload
functionality. It is possible to create a tar archive that contains files
with parent directory components in the file path (e.g.
foo/../../bar
would extract to../bar
), and these archive members would be written if the user running the Tool Shed had write permission to the given path. The Tool Shed tarball handling code checked for invalid characters (/
or..
) at the beginning of the path but not for..
in the middle of a path. - A read vulnerability exists in multiple places. The first is in the (now
deprecated)
hg push
functionality for updating Tool Shed repositories. This method allows malicious actors to push symlinks whose targets are outside the repository (either via an absolute or relative path). The contents of the targets would then be visible in the Tool Shed repository contents viewer, if the Tool Shed user has read permission on the target. - A second read vulnerability exists in the Tool Shed repository contents viewer. The viewer would allow a malicious actor to specify a path outside the repository, and if the Tool Shed system user had read permissions on that path, it would be displayed. The viewer also did not check to ensure that the targets of symlinks in a repository did not point outside the repository.
The repository contents viewer read vulnerability also exists in Galaxy, but is only reachable/exploitable by admin users. Fixes for vulnerability #3 have been applied to Galaxy/Tool Shed releases back to v14.10, and issues #1 and #2 have been applied to releases back to v15.01.
- A write vulnerability exists in the Tool Shed tarball and capsule upload
functionality. It is possible to create a tar archive that contains files
with parent directory components in the file path (e.g.
Release Notes¶
Enhancements¶
- Interactive tours (with significant help from @bgruening.) Pull Request 1422, Pull Request 1464
- Replace Galaxy eggs dependency management with wheels. Pull Request 428, Pull Request 989, Pull Request 988, Pull Request 1389, Pull Request 1485, Pull Request 995, Pull Request 996, Pull Request 1006, Pull Request 1017, Pull Request 1037, Pull Request 1495
- Implement nested workflows. Pull Request 1306
- Use Webpack to greatly optimize JavaScript and other assets served by Galaxy. Pull Request 1144
- Overhaul Galaxy analysis view toward a more modular JavaScript architecture. Pull Request 706, Pull Request 1184
- Add BAM iobio visualization as a Galaxy Interactive Environment (thanks to @bgruening.) Pull Request 1131
- Add Jupyter Interactive Environment (thanks to @bgruening.) Pull Request 1400
- Procedures for bug and issue handling as well as roadmap management. Pull Request 902, Pull Request 1020
- Implement a conda dependency resolver (with significant help from @bgruening.) Pull Request 1345, Pull Request 1527, Pull Request 1421, Pull Request 1409
- Many ToolBox improvements including allowing specifying labels on individual
tools, monitoring and automatic reloading of
tool_conf.xml
files, and allowing specification of such files in JSON/YAML. Pull Request 1012, Pull Request 1398 - Configurable client side logging. Pull Request 1011
- Allow input collections to specify multiple
collection_type
s. Pull Request 1308 - Allow multiple collections to be supplied to a multiple data parameter. Pull Request 805
- Implement
type_source
on output collections. Pull Request 1153 - Allow tools to request a special configuration file containing tool parameters dumped to json. Pull Request 1405
- Add search on username/any for API calls to user and use Select2 for sharing workflows/pages/histories (thanks to @hexylena.) Pull Request 1111
- Improve masthead and scratchbook UI and architecture. Pull Request 1407, Pull Request 1424
- Add support for constructive solid geometry datatypes
ply
andvtk
. (Thanks to @gregvonkuster.) Pull Request 905, Pull Request 1211 - Add searchgui datatype (thanks to @jj-umn.) Pull Request 914
- Add BIOM v1 datatype (thanks to @fescudie.) Pull Request 950, Pull Request 1198
- Add CRAM datatype (thanks to @yhoogstrate.) Pull Request 1108, Pull Request 1182
- Add UniProtXML datatype (thanks to @jj-umn.) Pull Request 1004
- Avoid confusion by adding the PDF datatype to the list of uploadable file types. Pull Request 901
- Add idpDB and HDF5 sniffers and fix MzSQlite sniffer (thanks to @chambm.) Pull Request 1209
- Improve tool shed installation API for installation of repositories if they are already installed (thanks to @bgruening.) Pull Request 1095
- Prevent GIEs from being closed without user confirmation (thanks to @bgruening.) Pull Request 1178
- Add the constructive solid geometery (CSG) viewer visualization plugin (thanks to @gregvonkuster.) Pull Request 1254
- Add Google analytics tracking to new data libraries. Pull Request 959
- Extend users API with delete action (thanks to @nsoranzo.) Pull Request 1063
- Finish swapping unencoded ids with order_index in workflows API. Pull Request 1137
- Add documentation and extra options to Environment Modules resolver (thanks to @pvanheus.) Pull Request 1221
- Documentation for dependency resolvers (thanks to @pvanheus.) Pull Request 1296
- Populate env.sh files created via tool shed installations with
X_ROOT_DIR
environment variables. Pull Request 564, Pull Request 889 - Reorganize history menu (thanks to @hexylena.) Pull Request 985
- Look up installed tools from different toolshed(s) (thanks to @mvdbeek.) Pull Request 818, Pull Request 1286
- Documentation updates for ObjectStore (thanks to @mr-c.) Pull Request 853
- Update JavaScript dependency Dynatree. Pull Request 856
- Eliminate LWR from Galaxy (you will need to upgrade to Pulsar). Pull Request 857
- Isolate tool commands by default. Pull Request 1412, Pull Request 1494
- Allow override of job shell (for conda resolver). Pull Request 1473
- Slight tweak to sidebar section naming from ‘security’ to ‘user management’. Pull Request 877
- Fix the issue where too many datasets are linked to a certain role (thanks to @scholtalbers.) Pull Request 881
- Fix R dependency installation to be more portable (thanks to @bgruening.) Pull Request 893
- Reorder workflow inputs to top (thanks to @scholtalbers.) Pull Request 912
- Add an API script to upload directory as a data library (thanks to @hexylena.) Pull Request 918
- Enable dockerized toolshed tests. Pull Request 942
- Improvements and fixes for abstract tool interface. Pull Request 955
- Remove remaining references to galaxy.eggs and flake8 some tools (thanks to @nsoranzo.) Pull Request 990
- Apply stricter flake8 rules for the directories shared with Pulsar (thanks to @nsoranzo.) Pull Request 1010
- Updates to default welcome page. Pull Request 1013
- Revise deferred queue, use jQuery promises. Pull Request 1018
- Keep select field open in tool form multi select fields to ease selecting many options quickly. Pull Request 1019
- Allow resizing of regular multi-select boxes. Pull Request 1025
- Add error message for wrongly formatted data in wig_to_bigwig (thanks to @scholtalbers.) Pull Request 1033
- Verify len files during download from UCSC (thanks to @nsoranzo.) Pull Request 1066
- Add biocrusoe to contributors Pull Request 1067
- Outline “Format 2” workflow definitions. Pull Request 1096
- Improve the API attributes display on parameters page. Pull Request 1098
- Update run_tests.sh help for recent changes (thanks to @nsoranzo.) Pull Request 1101
- Show welcome page with required login. Pull Request 1105
- Allow accessing collection elements in format_source. (with significant help from @nsoranzo.) Pull Request 1113, Pull Request 1155
- Add nginx config for GIE proxy to documentation. Pull Request 1123
- Enforce client build deps are up-to-date. Pull Request 1130
- Menu onclick addition. Pull Request 1142
- Ensure confirmation when leaving GIE windows. Pull Request 1157
- Add local grunt-cli dependency for qunit tests. Pull Request 1159
- Tighten up GG sniffing Pull Request 864
- Use common exceptions in tools. Pull Request 874
- Refactor the remote user middleware for complexity (thanks to @hexylena.) Pull Request 875
- Allowed for capitalized package names such as R in unlinked tool shed dependency resolver (thanks to @Christian-B.) Pull Request 1160
- A slew of tool execution performance optimizations including a huge fix from @ericenns. Pull Request 1166, Pull Request 1163, Pull Request 1199, Pull Request 1212
- Do not check for tool migrations when running tests. Pull Request 1176
- Lazy load HDA/LDDA metadata to speed up history loading (thanks to @abretaud.) Pull Request 1179
- Add debug statement in output checker for why job is failing. Pull Request 1213
- Small enhancements in workflow inputs and outputs. Pull Request 1214
- More small workflow tweaks. Pull Request 1216
- Add Workflow editor UI for step labels. Pull Request 1251
- Ease the grunt uglify process (Thanks to @hexylena.) Pull Request 1238
- Add beta run workflow form based on the newer tool form code. Pull Request 1249
- During tool migrations, do not mess with galaxy_config_file if provided (thanks to @openlangrid and @nsoranzo.) Pull Request 1230, Pull Request 1253
- In the users API, properly return a boolean from has_requests Pull Request 1262
- Extend the logging of tool dependency status changes (thanks to @nsoranzo.) Pull Request 1263
- Remove history_options and options.mako (history options as a page). Pull Request 1271
- Minor mail config cleanup Pull Request 1299
- Replace uses of history.imp and history.copy web methods for API create Pull Request 1303
- More tightening up of parameter validation during workflow stuff. Pull Request 1319
- Show a more user-friendly error when the webserver port is blocked (thanks to @shano.) Pull Request 1320
- Upgrade Paste to 2.0.2. Pull Request 1344
- A series of small refactoring enabling use of certain Galaxy modules a stand-alone Python library with minimal dependencies and Python 3 compatibility. Pull Request 1350, Pull Request 1351, Pull Request 1352, Pull Request 1359, Pull Request 1362, Pull Request 1376, Pull Request 1413, Pull Request 1427, Pull Request 1363, Pull Request 1367, Pull Request 1377, Pull Request 1388, Pull Request 1448
- Adding UUID support to directory_hash_id (thanks to @kellrott.) Pull Request 1397
- Unify and abstract code for checking if file looks like a tool definition. Pull Request 1368
- Added hashes to pip requirements file (thanks to @hexylena.) Pull Request 1393
- Interactive environment updates (thanks to @bgruening.) Pull Request 1399
- Change scratchbook close icon. Pull Request 1425
- Rename reports_wsgi.ini to reports.ini (thanks to @souravsingh.) Pull Request 1181
- Attempt to fix more transiently failing API tests. Pull Request 859, Pull Request 865
- Fix data library test case. Pull Request 898
- Remove
test_map_over_two_collections_legacy
test case as it is obsolete. Pull Request 924 - Version the testing-base docker image. Pull Request 938
- Update casperjs functional tests. Pull Request 944
- Fix the docker db client_encoding to not be ascii (default). Pull Request 952
- Use the database temp directory to store the Mako template cache when running framework tests. Pull Request 956
- Rename
lib/galaxy/main.py
to scripts/galaxy-main Pull Request 994 - Fix deferred dom removal. Pull Request 997
- Options to more easily test esoteric tooling options. Pull Request 1014
- Fix log statements and link to logger. Pull Request 1040
- Add more data libraries API tests. Pull Request 1074
- Remove upload unused function parameter (thanks to @einon.) Pull Request 1078
- Update testing docker image. Pull Request 1083
- Small tool and workflow refactorings. Pull Request 1097
- Revise and fix waiting for tool tests. Pull Request 1119
- Small Tool and Workflow Refactoring and Fixes Pull Request 1202
- Wait on jobs and history in certain API test cases. Pull Request 1226
- Fix qunit shim to match base_panels.mako shim. Pull Request 1233
- Improved logging related tool test timeouts. Pull Request 1243
- Refactor generic side workflow editor panel toward backbone. Pull Request 1247
- Attempt to fix transiently failing tool test on Jenkins. Pull Request 1248
- Set client_encoding for TS dockerized test db Pull Request 1276
- Check for sessionStorage using a more cross-browser way. Pull Request 1279
- Small API test improvements. Pull Request 1285
- Enforce metrics related to moving toward modern client infrastructure. Pull Request 1292
- Refactor tool stuff for generic model actions. Pull Request 1307
- Remove workflow casperjs API test duplicating test coverage of API tests. Pull Request 1316
- Refactor tool parsing handling toward reuse outside Galaxy. Pull Request 1349, Pull Request 1353
- Lint with Python 3 several modules. Pull Request 1354
- Fix and potential fix for transiently failing tests. Pull Request 1401
- Fix CasperJS tests. Pull Request 1438, Pull Request 1439
- Add a log.warn() if extracted file does not exist when changing permissions. (thanks to @nsoranzo.) Pull Request 939
- Add a safe_relpath util function for ensuring a path does not reference an absolute or parent directory. Commit f540a16
Security¶
- Security fixes for history imports. Commit bf1c77d
- Security fixes for object store paths. Commit 5da91bd
- Remove sample tracking manual external service transfer due to security concerns. Commit cd8b965
- Security fixes for tool shed repository browsing. Commit e4a1d57
- Security fixes for tool shed hg push and capsule/tarball uploads. Commit e845d64
Fixes¶
- Add check for
HTTP_GX_SECRET
(thanks to @golharam.) Pull Request 827 - Test if parameter present in value to avoid index error in tool state handling (thanks to @mvdbeek.) Pull Request 863
- Bump version of sqlite3 in nodejs proxy (thanks to @hexylena.) Pull Request 866
- Expose API keys to admins under
REMOTE_USER
(thanks to @hexylena.) Pull Request 872 - Bug fix for tool shed repository API (thanks to @mvdbeek.) Pull Request 891
- Add a workaround to return a proper error code during R package installation (thanks to @bgruening.) Pull Request 892
- Various Quota bug fixes. Pull Request 907
- Always set ‘tests’ for a visualization plugin to avoid an attribute error. Pull Request 908
- Bug fix for condor runner changes made this release cycle (thanks to @hexylena.) Pull Request 909
- Change user preference datatype to text. Pull Request 916
- Fix syntax of error parameter of
tryCatch
insetup_r_environment action
(thanks to @nsoranzo.) Pull Request 929 - Revise tool URL building. Pull Request 947
- Fix for code checking if a file seems to a tool definition file (thanks to @hexylena.) Pull Request 949
- Fix base panels to include scripts as the last element of the body. Pull Request 969
- Grid batch operation fixes. Pull Request 971
- Fix extra files path URL problem. (thanks to @hexylena.) Pull Request 991
- Fixes broken env-var declarations for tools with weird chars in their names (thanks to @yhoogstrate.) Pull Request 1028
- Added gitignore rules to ignore custom tool-data (thanks to @yhoogstrate.) Pull Request 1048
- Fix import of history datasets into library. (thanks to @mvdbeek.) Pull Request 1049
- Fix for reloading tools that have non-standard tool_ids/versions. Pull Request 1050
- Improved encoding handling for Jobs. Pull Request 1052
- Fix lped report output from converter. Pull Request 1069, Pull Request 1070, Pull Request 1072
- Trivial: Replace unnecessary duplicated var check with ‘else if’ (thanks to @einon.) Pull Request 1073
- Fix a bug in IEs when proxying the proxy. Pull Request 1076
- Fix 500 error when attempting to update installed repository. Pull Request 1082
- Resolve conflicting label CSS class for trackster. Pull Request 1086
- Fix bug with referrer attribute type change in WebOb. Pull Request 1091
- Fix API TS installation (thanks to @bgruening.) Pull Request 1094
- Better error when tool shed repository directory is missing (thanks to @lparsons.) Pull Request 1107
- Don’t let
$input
hang cheetah evaluation. Pull Request 1117 - Fix for re-installing an uninstalled TS repository with a dependency (Thanks to @gregvonkuster.) Pull Request 1154
- Fix interface and usage of
WorkflowModule.get_runtime_inputs
. Pull Request 1174 - Add enhancements to the Galaxy repository install process (thanks to @mvdbeek.) Pull Request 1193
- Tool shed fixes (thanks to @nsoranzo.) Pull Request 1200
- Fix for updating tool parameter dicts when a new parameter has been added to a section. Pull Request 1215
- Replace the defunct readthedocs badge. Pull Request 1229
- Export
GALAXY_TEST_DBURI
asGALAXY_CONFIG_OVERRIDE_DATABASE_CONNECTION
before installing wheels. (thanks to @nsoranzo.) Pull Request 1231 - Fix passing of nginx_upload_path and ftp_upload_site. Pull Request 1250
- Fixed indentation errors for reports app (thanks to @markiskander.) Pull Request 1259
- Open select2 drop down on caret click. Pull Request 1298
- Improved validation of tools during workflow execution. Pull Request 1302
- Properly remove datasets from the filtered lists when pairing datasets for the paired dataset list creator. Pull Request 1310
- Update Kombu and AMQP wheels to fix problems with El Capitan’s System Integrity Protection. Pull Request 1327
- Fix for creating workflow outputs on initial workflow upload. Pull Request 1330
- Don’t query on unencoded IDs for error form (thanks to @hexylena.) Pull Request 1340
- If
GALAXY_SLOTS
is defined in the environment, use it for the local runner. Pull Request 1346 - Use both
SLURM_NTASKS
andSLURM_CPUS_PER_TASK
to setGALAXY_SLOTS
(thanks to @lparsons.) Pull Request 1347 - Fix for loading workflows that have tool version / step upgrade messages. Pull Request 1348
- Allow installation of different repositories with the same name in a single request (thanks to @mvdbeek.) Pull Request 1366
- Copy workflow objects when importing them. Pull Request 1474
- Undo user icon in masthead. Pull Request 1493
- Fix mime type when previewing certain tabular data. Pull Request 1498
- Fix disabled CSS. Pull Request 1501
- catch Exception and properly log errors Pull Request 1511
- Fix for workflow validation problem introduced in 15.10. Pull Request 1536, Issue #1514
- Keep track of hidden datasets. Pull Request 1551
- Force
--skip-venv
if we can detect that Python is Conda Python. Pull Request 1554 - Fix installation of Tool Shed repositories containing non-ASCII characters in the description. (thanks to @nsoranzo.) Pull Request 1759
- Fix pretty_print_time_interval for MySQL. (thanks to @nsoranzo.) Pull Request 1761
To stay up to date with Galaxy’s progress watch our screencasts, visit our community hub, and follow @galaxyproject on Twitter.
You can always reach us on Gitter or IRC.
Thanks for using Galaxy!