Warning
This document is for an in-development version of Galaxy. You can alternatively view this page in the latest release if it exists or view the top of the latest release's documentation.
April 2016 Galaxy Release (v 16.04)¶
Highlights¶
- Tool Profile Versions
- Tools may now declare which version of Galaxy they require. Tools requiring 16.04 or newer will have new default behaviors (such as using exit code for error detection) that should simplify tool development. See PR #1688.
- Embedded Pulsar Job Runner
- Galaxy can now start a Pulsar application embedded within the Galaxy process itself. This allows using Pulsar’s job staging and isolation without requiring a RESTful web service or a message queue. This is enabling usegalaxy.org to run jobs to on the new JetStream cloud. See PR #2057.
- New chemical datatypes
- Galaxy now detects and supports many molecular datatypes. See Pull Request 1941. Thanks to Björn Grüning (@bgruening).
Github¶
- New
% git clone -b master https://github.com/galaxyproject/galaxy.git
- Update to latest stable release
% git checkout master && pull --ff-only origin master
- Update to exact version
% git checkout v16.04
BitBucket¶
Note: Version 16.04 will be the final release to be pushed to Bitbucket. More details can be found in the Deprecation Notices below.
- Upgrade
% hg pull % hg update latest_16.04
See our wiki for additional details regarding the source code locations.
Security¶
TL;DR Only Tool Sheds newer than 16.01 should be deployed from now on. (with commit 449098d8b14b45269be106f6410c0b9145c51d50 from Mar 30 present)
Due to the security fixes on the Mercurial side we had to update the hg version that
both Galaxy and TS depend on because the fixes have not been backported to older versions.
However this has broken the TS’s hg push
functionality as Mercurial changed their bundle
format in a non-compatible manner. Given that we deprecated the hg push
API
functionality back in the 15.10 we decided to disable it fully from 16.01 (retroactively).
Deprecation Notices¶
API deprecations¶
- API for history contents, index:
- types: is no longer a valid parameter but accessible using
?q=history_content_type&qv=[dataset | dataset_collection]
- ids: is no longer a valid url parameter but is accessible using
?q=type_id-in&qv=<e.g. dataset-abcdef123,dataset_collection-987fedcba, ...>
- deleted and visible: are no longer parameters but are still accessible using
q=deleted&qv=[True | False]&q=visible&qv=[True | False]
- types: is no longer a valid parameter but accessible using
- API for history contents, show:
- api_type: removed and unavailable (was
file
and constant across HDAs) - display_apps, display_types, visualization: removed from the default
detailed
view but still available by calling url with?keys=display_apps,display_types,visualization
- api_type: removed and unavailable (was
- API histories (removed from the available serialized data on all calls):
- state, state_details, state_ids - can be replaced by specifically requesting a single array of contents, each containing
{ id, state, deleted, visible }
- state, state_details, state_ids - can be replaced by specifically requesting a single array of contents, each containing
Galaxy no longer on Bitbucket¶
Galaxy moved its code and development activities from Bitbucket to GitHub in early 2015. Since this time, releases have been mirrored back to Bitbucket. However, after this release, no new changes will be pushed to Bitbucket. Anyone still receiving updates to a Galaxy server via Bitbucket and Mercurial should switch to GitHub and Git. This can be done using the following process:
- Backup everything.
- Find what branch and commit your Mercurial Galaxy is at using
hg log -b $(hg branch)
git clone https://github.com/galaxyproject/galaxy
in a temporary directory.- Find the corresponding commit in the cloned Git repository on the corresponding branch (Bitbucket default->GitHub dev; Bitbucket stable->GitHub master).
- Checkout the GitHub repository at the commit you found in the previous step.
- Backup your .hg/ folder.
- Replace your .hg/ folder with the .git/ folder from the new checkout.
- Your Galaxy should be switched to Git. Unless you have local changes, git status should show none.
- You can now update to the latest Git revision using
git pull
Release Notes¶
Enhancements¶
- Overhaul of Tools and Jobs Pull Request 1688
- Implement an Embedded Pulsar Job Runner Pull Request 2057
- Use the API to install repositories instead of loading the Tool Shed in an iframe. (Beta) Pull Request 1392
- Phinch interactive environment (thanks to @shiltemann.) Pull Request 1647
- Add iobio external display applications for BAM and VCF. Pull Request 1926
- add chemical datatypes (thanks to @bgruening.) Pull Request 1941
- Scratchbook tour Pull Request 1463
- Work toward automating release management. Pull Request 1613
- Basic tool error Sentry reporting. Pull Request 1900
- disable ‘hg push’ to TS repositories Pull Request 2033
- Generic GIE Launcher, GIE Image Chooser, multiple datasets (thanks to @erasche.) Pull Request 1403
- The entire Python source code is now continuously checked with flake8 for PEP-8 style consistency and common errors. The last 3500 errors were fixed in this release (thanks to @nsoranzo.) Pull Request 1489, Pull Request 1713, Pull Request 1755
- Implement new GIE proxy (thanks to @erasche.) Pull Request 1807
- Workflow editor overview click navigation Pull Request 1843
- API, history contents: allow filters, limit/offset, and ordering Pull Request 1602
- datalibs: introduce folder management; fix various glitches; refactor Pull Request 1562
- Add makefile target for release process (thanks to @erasche.) Pull Request 1433
- libraries: provide select all/none options when importing Pull Request 1970
- libraries: add ‘create history &import to it’ feature Pull Request 2017
- Bcf.gz to bcf (and reverse) converters + bcf_bgzip data type (thanks to @markiskander.) Pull Request 1148
- Update SnpEffDb and SnpSiftDbNSFP for SnpEff v4.1 (thanks to @jj-umn.) Pull Request 1280
- Docker Testing - Newer versions of phantomjs and casperjs, in container wheels. Pull Request 1449
- Add run_toolshed_tests.html to .gitignore . (thanks to @nsoranzo.) Pull Request 1456
- Allow sorting of discover_datasets elements. Pull Request 1512
- Cleanup common ui elements, add test cases Pull Request 1519
- add minor styling to tour popover Pull Request 1539
- Code design and legacy cleanup of datatypes and metadata. Pull Request 1556
- show_params input walking Pull Request 1561
- Add API endpoint for fetching all metadata for a repository. Pull Request 1565
- Fix parameter validation Pull Request 1572
- Remove recovery hack for REALLY old jobs. Pull Request 1574
- Clarify area github tags. Pull Request 1579
- update pinned-requirements.txt (thanks to @matthdsm.) Pull Request 1605
- Improve dbkey handling for discovered datasets. Pull Request 1610
- Various improvements to the heartbeat Pull Request 1614
- Visualization plugins: add routes that more closely match potential static assets. Pull Request 1615
- Disable individual file metadata changes in upload Pull Request 1625
- added resolver dependency display to the tool view page (thanks to @zipho.) Pull Request 1632
- Cherry-picked changes from pull request #1392. Pull Request 1821
- Export toolshed repository information for workflow portability (thanks to @mvdbeek.) Pull Request 1636
- Lint optional handling on conditional test parameters. Pull Request 1639
- Updated hashes for bioblend v0.7.0 (thanks to @matthdsm.) Pull Request 1646
- Add tox startup (run.sh) test and test on OS X and linux (thanks to @mvdbeek.) Pull Request 1649
- Optionally pass dataset lists as a single file within tool wrappers (thanks to @einon.) Pull Request 1658
- add a zip datatype which won’t be extracted (thanks to @lecorguille.) Pull Request 1665
- Tour improvements and bugfixes Pull Request 1671
- Add Marius van den Beek to Galaxy committer group. Pull Request 1699
- Revise tool parameter handling, remove redundancies Pull Request 1711
- Python 2/3 string handling (thanks to @mvdbeek.) Pull Request 1731
- workaround for error message on tool lookup in case original toolshed had been disabled (thanks to @mvdbeek.) Pull Request 1738
- Make api/tools return tool_shed_repository information. (thanks to @mvdbeek.) Pull Request 1752
- Use visit inputs for check_and_update_param_values Pull Request 1764
- Update makefile and add the script I used to apply security patches for 16.01 Pull Request 1793
- Implement dataset empty input validator. Pull Request 1808
- Makefile help target + formatting (thanks to @erasche.) Pull Request 1824
- Add license and repository information to package.json. Pull Request 1829
- Improved logging during tool executions. Pull Request 1832
- Do not access the history when loading the workflow editor Pull Request 1861
- Tones down the warning message about tool changes (thanks to @bwlang.) Pull Request 1893
- Use shared value serialization for tool model Pull Request 1901
- adding contrib init for reports and modifying status check to have output by process in case multiprocess (thanks to @miloaec.) Pull Request 1911
- Retrieve authentification also for long urls (thanks to @mvdbeek.) Pull Request 1912
- Modify run.sh to add –restart option, for a more consistent use of parameters. (thanks to @remimarenco.) Pull Request 1914
- Rename tool_shed_get to url_get. Pull Request 1918
- Allow testing element counts in tool output collections. Pull Request 1942
- Use database query rather than dataset iteration to compute dataset counts in saved histories grid. Pull Request 1948
- Add “new” state to list of dataset states in saved history list. Pull Request 1949
- Add IUC as a new default bioconda channel (thanks to @bgruening.) Pull Request 1950
- More job related timing logging. Pull Request 1956
- Improvements to workflow scaling performance testing scripts. Pull Request 1958
- Optimization for discovering datasets for dynamic output collection. Pull Request 1959
- Improve tabular datatypes. (thanks to @brenninc and @nsoranzo.) Pull Request 1968, Pull Request 2317
- Optimize adding datasets to large histories. Pull Request 1983
- Much more friendly route mapping. Pull Request 2002
- Revise content selection, add test cases Pull Request 2011
- Don’t setup a default tool_data_table_config_path in the shed. Pull Request 2025
- data_column to json config file (thanks to @bgruening.) Pull Request 2026
- Revise input wrapper, add test cases Pull Request 2027
- enable obfs datatype during upload (thanks to @bgruening.) Pull Request 2028
- Better element_identifier handling in repeat sections (thanks to @mvdbeek.) Pull Request 2029
- Pulsar-As-A-Dependency Pull Request 2052
- Simplify Workflow sharing menu (thanks to @nturaga.) Pull Request 2060
- Add Phinch as an external display application. Pull Request 2069
- For non-shed tool data - read loc from sample if available. Pull Request 2070
- tool lineage for versionless toolshed tool_ids (thanks to @mvdbeek.) Pull Request 2082
- make new beta TS tools installation via API configurable Pull Request 2088
- Patch in visual separation of section parameters (thanks to @erasche.) Pull Request 2091
- Add sample supervisor config (thanks to @mvdbeek.) Pull Request 2096
- Fix cli runner: use embed_metadata_in_job parameter (thanks to @ThomasWollmann.) Pull Request 2107
- Switch Dockerized commands to use sh instead of bash. Pull Request 2282
- Remove IPython IE, which was replaced by Jupyter IE. (thanks to @bgruening.) Pull Request 1402
- Added py34-lint tox target but forgot to update .travis.yml. Pull Request 1457
- fix old jobparam hack for importing files to library Pull Request 1488
- An attempt to add limit/offset to history contents Pull Request 1490
- Revise label handling in form Pull Request 1496
- History UI: clean up. Pull Request 1522
- Client-build, Webpack: add tasks to grunt for the common webpack tasks, update readme Pull Request 1523
- Remove handlebars and rely solely on underscore templates Pull Request 1537
- disable email notifications from travis Pull Request 1592
- Remove unnecessary variable and assignment of job command (thanks to @einon.) Pull Request 1616
- Implement the ratable mixin Pull Request 1618
- Tour cleanup, remove a few globals. Pull Request 1621
- Refactor workflow loading. Pull Request 1735
- Swapping from svgfig to svgwrite Pull Request 1747
- various libraries refactoring and bugfixes Pull Request 1751
- Refactor the Html Datatype Class into text.py instead of images.py (thanks to @remimarenco.) Pull Request 1760
- Drop python2.6 (deprecated, will not be supported in 16.04) from testing. Pull Request 1785
- Rework history updating Pull Request 1788
- Tests for some of the 16.01 security vulnerabilities Pull Request 1794
- Reduce the use of mutable types in toolshed test framework’s method definitions. Pull Request 1813
- Remove decryption/encryption of tool states Pull Request 1838
- Do not import dumps and loads from galaxy.util.json . (thanks to @nsoranzo.) Pull Request 1840
- Replace get_tool_shed_repository_by_tool_shed_name_owner_changeset_revision with get_repository_for_dependency_relationship. (thanks to @mvdbeek.) Pull Request 1868
- Cosmetic comma fixes :) (thanks to @remimarenco.) Pull Request 1874
- More debugging for transiently failing tool shed test. Pull Request 1939
- Improve logging and retry for another TS test… Pull Request 1961
- Add page layout test cases Pull Request 1991
- Fix inconsistency in update state handling for tool modules Pull Request 1993
- Functional Test Drivers Overhaul Pull Request 2016
- Remove install and test code. Pull Request 2018
- Remove twill functional tests we don’t actively run. Pull Request 2019
- Fix transiently failing tool shed tests. Pull Request 2030
- Libraries: faster, refactored, cleaned Pull Request 2031
- Adjust testing directories so no python root is ever “tool_shed”. Pull Request 2067
- Fix pbs runner file touch. Pull Request 2074
- Move scripts out of lib/tool_shed. Pull Request 2093
- Consolidate duplicated method in tool_shed/model. Pull Request 2099
- Remove redundant ui divs, move masthead into #everything Pull Request 2182
- Remove SLURM memory limit warning from stderr if the job was successful (thanks to @nsoranzo). Pull Request 2309
Fixes¶
- upgrade mercurial wheel to latest (fixing CVE issues) Pull Request 2045
- Add changeset_revision to tool attributes, avoid self.tool_shed_repository. (thanks to @mvdbeek.) Pull Request 1802
- Do not convert spaces to tabs when importing datasets into a library (thanks to @nsoranzo). Pull Request 1985
- Fix that typo in relation_builder. Pull Request 1453
- URL generation tweaks for utils.js Pull Request 1478
- Fix project linting for new pep8 Pull Request 1483
- Fix Python 3 problem causing Travis failure of dev. Pull Request 1505
- catch Exception and properly log errors Pull Request 1510
- Change python print() format to be backward compatible with older versions. (thanks to @einon.) Pull Request 1520
- Add js for mako based masthead Pull Request 1533
- restrict blue popover to tours Pull Request 1577
- small spelling error (thanks to @matthdsm.) Pull Request 1582
- Fix installation of repository suites outside of tool panel section (thanks to @mvdbeek.) Pull Request 1601
- Fixes <param argument=”–set” /> not working when help=”“ is not set (thanks to @yhoogstrate.) Pull Request 1650
- Fix validation for data source tools Pull Request 1654
- Better fix for missing element identifier (thanks to @mvdbeek.) Pull Request 1693
- Update david identifier types (thanks to @pavanvidem.) Pull Request 1696
- Drop Ross from the committers group. Pull Request 1698
- Wrap conditional test parameters Pull Request 1714
- Remove len(stderr), breaks on recent docker versions (thanks to @mvdbeek.) Pull Request 1769
- Strip URL of download_file and download_by_url install actions. (thanks to @nsoranzo.) Pull Request 1775
- Fix tool form rendering of sections Pull Request 1783
- Fix unused import. (thanks to @nsoranzo.) Pull Request 1796
- Fix tool shed tests for commit e4a1d5727805168a9fd15aca1cdd21630ada2bbc (thanks to @nsoranzo). Pull Request 1798
- Checks for api_key before checking for header from SSO. (thanks to @MatthewRalston.) Pull Request 1801
- Ensure tool.changeset_revision is set… Pull Request 1806
- Change many job mapped properties to lazy loads Pull Request 1809
- Whitelist logging tweaks Pull Request 1819
- Fix upload tool routing Pull Request 1827
- Using node 5.7, ‘grunt style’ fails with the error: Pull Request 1841
- Do not create text values for failed inputs Pull Request 1844
- Prevent tours from kicking off within iframes Pull Request 1846
- Fix repeat prefix Pull Request 1848
- also update rrda when repairing or updating a repository (thanks to @mvdbeek.) Pull Request 1850
- Fix regex validator Pull Request 1862
- Tour routing overhaul Pull Request 1870
- Add dedicated client endpoint to the root controller. Pull Request 1879
- Update check_python.py now dropping Python 2.6 (thanks to @peterjc.) Pull Request 1883
- Fix citation-model to fail silently/gracefully Pull Request 1884
- Change to sentry middleware to work with modern raven clients. Pull Request 1895
- svgfig->svgwrite in unpinned requirements Pull Request 1896
- Fix icon sizes Pull Request 1934
- Fix tool downloads in tool form Pull Request 1935
- Fixing error in run.sh script (thanks to @kellrott.) Pull Request 1954
- Fix typo: send–>sent Pull Request 1965
- Fix farbtastic’s use of deprecated jquery fns by loading jq-migrate in galaxy.panels.mako . Pull Request 1972
- Remove duplicate help target (thanks to @erasche.) Pull Request 1980
- Fix booleans in workflow editor Pull Request 1990
- libraries: change default size of fa in iconspans Pull Request 2000
- libraries: move text out of icon spans for library dataset view Pull Request 2008
- Fix unbound error that is possible if using tool bursting. Pull Request 2009
- Fix multiple flag for workflow dataset inputs Pull Request 2021
- fix duplication of select2 entries Pull Request 2022
- Fix fluent query log serialization when datetime types are in use. Pull Request 2039
- Workflow import fix when tools are missing. Pull Request 2048
- Managers: remove UTC zone when parsing dates (thanks also to @nsoranzo). Pull Request 2042, Pull Request 2062, Pull Request 2065
- Change user disk usage pgcalc function up a bit to make a slightly safer dry-run. Pull Request 2063
- Allow tool confs with a tool_path to not be interpreted as shed confs. Pull Request 2066
- Fix deps.command.download_command on Mac OS X. Pull Request 2075
- Show sections in workflow run Pull Request 2087
- Workflow section fix backport (thanks to @erasche.) Pull Request 2092
- Run external local set_metadata jobs in the job’s working directory Pull Request 2094
- make dependencies browsable again Pull Request 2101
- Convert the DRMAA runner to use Pulsar’s DRMAA session wrapper Pull Request 2102
- Updated to Dependency change in b167a741a444c3988447b0d63a1ba3dc5e4e62f5 (thanks to @Christian-B.) Pull Request 2104
- Fix datatype list in workflow editor Pull Request 2105
- Workaround for the toolshed’s hgweb. Pull Request 2106
- Update debian init script (thanks to @mvdbeek.) Pull Request 2109
- Rev Pulsar to 0.7.0.dev3. Pull Request 2122
- change run_tool_shed.py to see the file in the correct lcoation (thanks to @nturaga.) Pull Request 2131
- Fixes due to #2093 and #2018 (thanks to @nsoranzo.) Pull Request 2133
- markupsafe.escape() in Python2 does not work on str containing non-ASCII characters (thanks to @nsoranzo.) Pull Request 2138
- load options from config, not from options Pull Request 2139
- fix/replace biomart data source Pull Request 2149
- Add tool parameters back to parameters and expose for certain tools Pull Request 2156
- Fix wrapper issue, overload __ne__ Pull Request 2161
- Fix grouping tool and enhance performance when removing lines. Pull Request 2166
- libraries: always fetch new permissions as we are reusing the view Pull Request 2176
- Fix for redirecting a non-user when a tool has require_login=True. Pull Request 2180
- Fix.published history long titles have to be truncated Pull Request 2189
- fix a bug with missing history_id in libraries dataset import Pull Request 2190
- Fix message in legacy panel mako Pull Request 2191
- Prepass on remote user header based on maildomain and normalize config. Pull Request 2195
- Extend to_json for dataset tool parameters Pull Request 2196
- Pages parser fix Pull Request 2197
- Translate data source tool parameters on parameter expansion Pull Request 2201
- Sync job_script module with Pulsar to fix doctest. Pull Request 2203
- Improve error message for data source tools, executed through the tool_runner controller Pull Request 2204
- Add fixed validation check for data_source tools URL parameter Pull Request 2208
- Fix tool action redirect url for non-default tools Pull Request 2211
- Browse library date handling for non-ascii month abbreviations Pull Request 2214
- Avoid reset of cursor position for input fields during manual entry Pull Request 2218
- Add conditional statsd requirement Pull Request 2227
- Fix js value validator Pull Request 2234
- Encode collection reduce in serializable fashion Pull Request 2238
- Update _condarc automatically (thanks to @bgruening.) Pull Request 2255
- Truncate job name in DRMAA runner by default for PBSPro. Pull Request 2265
- Cherrypick of encoding fix Pull Request 2266
- If a Slurm post-mortem determines that a job is in a non-terminal state, return the job to the monitor queue Pull Request 2311
- Unicodify pretty datetimes which can contain non-ASCII chars (thanks to @nsoranzo). Pull Request 2312
- Relax default value acceptance condition Pull Request 2316
- Fixes the proxy naming (thanks to @bgruening.) Pull Request 2333
- Re-fix the bug with multiple preceding ‘/’ characters in the GIE proxy prefix Pull Request 2339
- Do not pollute param_dict with a non JSONifiable dict (thanks to @nsoranzo.) Pull Request 2345
To stay up to date with Galaxy’s progress watch our screencasts, visit our community hub, and follow @galaxyproject on Twitter.
You can always reach us on Gitter or IRC.
Thanks for using Galaxy!