May 2017 Galaxy Release (v 17.05)
Highlights
- Tag your data with propagating hashtags
Large Galaxy histories used to be messy. Hashtags make it easy to track dataset (and collection) relationships. These two movies (both under a minute) explain how to use this with datasets and with collections.
Learn more about Galaxy histories in our updated tutorial
- Drag & Drop datasets into tool inputs
Interface now allows dragging datasets from history panel into the content selectors of the tool form. Implemented in Pull Request 3871.
- Upload directly to a collection
You can now bypass the history manipulation and upload your data straight into a collection for convenience.
Learn more about collections and how to use them in a new tutorial.
We extend special thanks to the 64 New Contributors to Galaxy in the past year.
Get Galaxy
The code lives at Github and you should have Git to obtain it.
- To get a new Galaxy repository run:
$ git clone -b release_17.05 https://github.com/galaxyproject/galaxy.git
- To update an existing Galaxy repository run:
$ git checkout release_17.05 && git pull --ff-only origin release_17.05
See the community hub for additional details regarding the source code locations.
Additionaly there are New Configuration Options and New Datatypes sections.
Release Notes
Enhancements
Recalculate user disk usage on logout. Pull Request 3600
Overlay search for tools, workflows, libraries and histories (thanks to @anuprulez and @bgruening). Pull Request 3460
Webhook plugin: tool_list (thanks to @blankclemens). Pull Request 3753
Add Cytoscape as the Galaxy charts plugin (thanks to @anuprulez and @bgruening). Pull Request 3817
Ethercalc Interactive Environment (thanks to @shiltemann). Pull Request 3699
Add a simple UI to manage tool dependencies (thanks to @mvdbeek). Pull Request 3479
Add nrrd image datatype (thanks to @ThomasWollmann). Pull Request 3385
NGL Viewer for molecule representation (thanks to @anuprulez and @bgruening). Pull Request 3601
Add script to encode or decode the IDs that Galaxy exposes to users and admins (thanks to @hexylena). Pull Request 3622
Support Docker Engine swarm mode for Interactive Environments. Pull Request 3666
Interactive Environment container waiting and swarm management. Pull Request 3740
Phylocanvas as a BioJS component for the Galaxy charts (thanks to @anuprulez @bgruening). Pull Request 3850
Container interface abstraction. Pull Request 3863
Graphite support (thanks to @hexylena). Pull Request 3964
Normalize the handling of steps and missing tools for the workflow editor. Pull Request 3181
Add possibility to load collection into data library (thanks to @mvdbeek). Pull Request 3559
Prefer versioned Conda dependencies to unversioned Galaxy ones. Pull Request 3708
Speedup startup and toolbox operations (thanks to @mvdbeek). Pull Request 3909
Add missing imports for documentation builds (thanks to @athos-ribeiro). Pull Request 3104
Workflow editor UI touch up. Pull Request 3407
Add an example group filter for displaying tool panel sections (thanks to @gregvonkuster). Pull Request 3484
Python3: use @six.add_metaclass() decorator (thanks to @nsoranzo). Pull Request 3491
Improvements to build sites configuration. Pull Request 3503
Add peek and preview for compressed FASTQ datatype (thanks to @mvdbeek). Pull Request 3512
Add
refresh_on_change
to tool XSD. Pull Request 3515Store module step states as nested dictionaries, fix export for backward compatibility. Pull Request 3522
Add Jenkins documentation tasks to the release issue. Pull Request 3523
Allow decompressing tool test outputs. Pull Request 3550
Add a task to protect the release branch in GitHub. Pull Request 3562
Remove duplicated/unused display_migration_details() functions (thanks to @nsoranzo). Pull Request 3564
Display the genbank datatype in upload. Pull Request 3569
Database migrations: improvements and fixes (thanks to @nsoranzo). Pull Request 3570
Add d3_hierarchy subclass (thanks to @jj-umn). Pull Request 3580
Clean up docker-compose resources better for Selenium tests. Pull Request 3583
Add option to send API key in header (thanks to @jvolkening). Pull Request 3584
Increase patience of Galaxy tests throughout. Pull Request 3589
Introduce label and annotation attributes to subworkflows, fix annotation display, unify module form update handler. Pull Request 3593
Align architecture slides with training-material remarkjs template. Pull Request 3594
Revision of tool XML reloading feature. Pull Request 3596
Implement collection operation to relabel list identifiers from contents of a file. Pull Request 3603
Slow query logging. Pull Request 3616
Remove un-needed check in histories manager code. Pull Request 3618
Improve interaction between job state and purged datasets. Pull Request 3619
Usable slow query logging. Pull Request 3625
Allow composite datatypes of type auto_primary_file to automatically generate dataset. Pull Request 3643
Introduce labels for all workflow modules, remove name handling field for data inputs. Pull Request 3644
When using the new slow query logger, include parameters. Pull Request 3649
Allow nested parameters in the dynamic_tool rule -> argument (thanks to @lecorguille). Pull Request 3657
By default, do not allow workflow invocations to schedule indefinitely. Pull Request 3659
Add the mzData datatype (thanks to @lecorguille). Pull Request 3661
Display an error message when GIE loading raises an exception. Pull Request 3662
Set LDAP options only once (thanks to @nsoranzo). Pull Request 3663
Bump Jupyter container version to 16.01.1 . Pull Request 3665
Improve wording of getgalaxy in bootstrapping script for release notes. Pull Request 3671
Filter hda tags for tool parameters. Pull Request 3688
Add datatypes for soon coming PlantTribes Galaxy tools (thanks to @gregvonkuster). Pull Request 3698
Touch up Galaxy testing documentation. Pull Request 3715
Add API and UI to remove requirements installed by Conda (thanks to @mvdbeek). Pull Request 3725
Quota recalculation logic consolidation. Pull Request 3734
Percolator out datatypes (thanks to @glormph). Pull Request 3737
Modify our docs in preparation for restructuring. Pull Request 3738
Update codefile documentation in tool XSD with info from old Wiki. Pull Request 3739
Mirror latest galaxy-lib changes for Conda and CWL improvements. Pull Request 3742
Add the possibility to uninstall Conda environments that are not in use by Galaxy (thanks to @mvdbeek). Pull Request 3750
New Authentication section of the docs. Pull Request 3751
Bump routes requirement to 2.4.1 . Pull Request 3752
Abstract uWSGI handling into new galaxy.web.stack package. Pull Request 3759
i18n improvements and French localization. Pull Request 3762
Should exit immediately to avoid index error (thanks to @hexylena). Pull Request 3770
Made collection operations help sections slightly more readable. Pull Request 3784
Avoid potential inconsistencies between differently stored labels. Pull Request 3793
Remove custom builds mako. Pull Request 3794
Small enhancements for collection builders. Pull Request 3806
Small improvements for driving Qunit tests. Pull Request 3810
Implement more upload testing in Selenium. Pull Request 3811
API & reports documentation improvements. Pull Request 3812
Make ToolConfWatcher watch ToolCache (thanks to @mvdbeek). Pull Request 3814
Disable tool shed test function that fails frequently on Jenkins. Pull Request 3819
Restrict workflow scheduling within a history to a fixed, random handler. Pull Request 3820
Order processing evaluation of workflow invocations when scheduling. Pull Request 3830
Stricter checks for GFF/GFF3/GTF datatypes sniffing (thanks to @nsoranzo). Pull Request 3832
Add root redirect for backbone router. Pull Request 3837
Allow explicit configuration of handlers for workflow scheduling. Pull Request 3844
Log reason when workflow scheduling of a step is delayed. Pull Request 3849
Update datatypes_conf.xml.sample (thanks to @lecorguille). Pull Request 3852
Clean up old TODO into history tests. Pull Request 3855
Small changes to links and add Gitter (thanks to @bgruening). Pull Request 3861
Use detect_errors=”exit_code” in Trim tool (thanks to @nsoranzo). Pull Request 3869
Synchronise against latest UCSC versions (thanks to @bgruening). Pull Request 3874
Hardklor and kronik datatypes for proteomics (thanks to @glormph). Pull Request 3884
History drag-and-drop feature with iframe support for run workflow form. Pull Request 3886
Update to a new Docker test image to fix dependency bugs. Pull Request 3890
Python3: Use shlex_quote() from six.moves instead of pipes.quote() (thanks to @nsoranzo). Pull Request 3891
Add a validator that ensures a dataset’s extra_files_path (thanks to @gregvonkuster). Pull Request 3918
Update text.py Add interaction annotations to SnpEffDb metadata (thanks to @jj-umn). Pull Request 3938
Collection operation: Filtering from a file. Pull Request 3940
Remove Google from OpenID providers in the sample. Pull Request 3953
Change ‘User’ menu to ‘Login or Register’ for anonymous users. Pull Request 3955
Stats docs (thanks to @hexylena). Pull Request 3968
Enhance filenames with collection context when downloading datasets from GUI. Pull Request 3985
Displaying a message instead of just redirecting. Pull Request 4000
Implicitly set use_volumes = False if using Docker swarm. Pull Request 4006
Update Jupyter image to 17.05 . Pull Request 4007
Inline CSS in error report email (thanks to @hexylena). Pull Request 3445
Improve routing for user preference views. Pull Request 3446
Tool documentation and testing improvements. Pull Request 3457
Update readme badges. Pull Request 3465
Lint some tools (thanks to @nsoranzo). Pull Request 3468
Improve release issue docs. Pull Request 3496
Watch tool data table
*.loc
files, and reload if modified (thanks to @mvdbeek). Pull Request 3533
Attempt work around for race-y sort of condition in Selenium sizzle checkers. Pull Request 3451
Webhooks: add tests (thanks to @anatskiy). Pull Request 3469
Disable transiently failing tour test. Pull Request 3599
Selenium testing improvements. Pull Request 3716
Merge ConfiguresGalaxyMixin.reload_toolbox() in _configure_toolbox() (thanks to @nsoranzo). Pull Request 3732
Abstraction for re-use between collection creators. Pull Request 3781
Merge ToolConfWatchers and separate from Toolbox (thanks to @mvdbeek). Pull Request 3821
Attempt to improve the robustness and verboseness of the test_sharing_valid Selenium test. Pull Request 3841
Routes cleanup. Pull Request 3895
Standardize usage of log.exception . Pull Request 3933
Followup to exception refactoring. Pull Request 3959
Use container-based infrastructure (sudo: false) in TravisCI (thanks to @nsoranzo). Pull Request 3977
Improve implicit workflow connection test case. Pull Request 3991
Fixes
Fix tag escaping. Pull Request 3720
Fix kombu documentation link in galaxy.ini.sample . Pull Request 3629
Fix for EtherCalc IE image specification. Pull Request 3825
Fix margin style values for color picker element. Pull Request 3965
17.05 rstudio fixes / one small GIE fix (thanks to @hexylena). Pull Request 3975
Fix transiently failing Selenium upload test. Pull Request 3587
Fix webhook search overlay interfering with masthead load. Pull Request 3731
Fix collection operations in workflows. Pull Request 3848
Remove old, unused search controller. Pull Request 3894
Deduplicate obsolete API readme. Pull Request 3777
Assorted fixes for dockerized test setup. Pull Request 3755
Another fix for Selenium testing on Jenkins. Pull Request 3422
Fix most broken Selenium tests. Pull Request 3440
Fix for transiently failing workflow test. Pull Request 3668
Fix selenium workflow tests broken with #3181. Pull Request 3504
Fix integration test for default Conda mapping. Pull Request 3508
Various fixes for test image Docker user switching. Pull Request 3443
Fix Travix builds which use flake8-docstrings (thanks to @nsoranzo). Pull Request 3981
Azure - Leasing container sometimes breaks things in Docker Galaxy (thanks to @zfrenchee). Pull Request 3263
run_tests.sh: a small typo (thanks to @anatskiy). Pull Request 3322
Fix cycle detection for single node workflows. Pull Request 3374
Small selenium hack to fix occasionally failing collection upload tests. Pull Request 3967
Bug fix in mulled.py (thanks to @abdulrahmanazab). Pull Request 3434
Do not select first value as default if select value is optional. Pull Request 3472
jsonify initial value when returned through to_dict caller. Pull Request 3513
Fix <is_valid_xml> output test assert check (thanks to @jvolkening). Pull Request 3530
Fix output tests for checksum example tools. Pull Request 3545
Update Nagios check. Pull Request 3568
Set ‘deleted’ to True when purging a history dataset association (thanks to @nsoranzo). Pull Request 3572
Update tool XSD for (hopefully deprecated) change_format syntax. Pull Request 3579
Remove broken tour step. Pull Request 3582
Avoid step duplication when ordering steps on the fly. Pull Request 3590
Minor fix of core.galaxy-ui tour. Pull Request 3608
Fix RELEASE_ISSUE_TEMPLATE in bootstrap_history.py (thanks to @nsoranzo). Pull Request 3653
Update dynamic_tool_destination.py (thanks to @lecorguille). Pull Request 3655
Ignore Node modules installed for the GIE proxy when linting. Pull Request 3664
Logout recalc improvements. Pull Request 3669
Strip broken autocomplete feature, webbrowsers console error is an artifact. Pull Request 3676
Generate primary composite file after extra files collection. Pull Request 3678
Fix tool search index building for tools with slash in id and not from ToolShed. Pull Request 3680
Set $HOME to a temp directory rather than conda_prefix because conda_prefix may not be writable either. Pull Request 3701
Properly handle boolean values when parsed to a string-based/default tool parameter. Pull Request 3702
Mothur datatypes: fix metadata failure for large count tables (thanks to @shiltemann). Pull Request 3706
Filename handling for FTP upload. Pull Request 3722
Do not require stored workflow id for subworkflows. Pull Request 3723
Remove outdated items from tool cache when reloading the toolbox (thanks to @mvdbeek). Pull Request 3729
Change API path in docstrings to correct /api/dependency_resolvers/* (thanks to @mvdbeek). Pull Request 3749
Wiki to hub link update. Pull Request 3754
Fix KeyError when getting request and response content-type headers (thanks to @nsoranzo). Pull Request 3757
Fix for uploading rdata datasets when explicitly declaring datatype. Pull Request 3766
Fixes for users API. Pull Request 3771
Fix XSS vulnerability in grids. Pull Request 3776
Layered routing fix for when controller route fails and client route does not exist. Pull Request 3783
Fix collection operations not having a history. Pull Request 3786
Fix problem with cleaning up jobs in local runner. Pull Request 3803
Have tool-search listen to change as well as keyup. Pull Request 3808
Fix BED dataset display in Trackster (thanks to @hrhotz). Pull Request 3822
More CONVERTER fixes for restricting tools’ Python environment. Pull Request 3824
Have IGV bigWig display application make use of build_sites/ site_type config instead of direct filename. Pull Request 3828
Properly handle multiple hidden datasets when populating data select options. Pull Request 3842
Fix for direct message routing using a ‘real’ exchange like RabbitMQ. Pull Request 3846
Fix setting UUID on steps when copying steps from another workflow. Pull Request 3856
Fix certain aspects of collection reductions in conditionals/repeats. Pull Request 3864
Fix multiple collection reductions on multiple data parameters. Pull Request 3875
Fix inability for viz to raise loading with non-ascii messages (thanks to @hexylena). Pull Request 3881
Fix new history endpoint providing custom builds metadata. Pull Request 3887
Remove unused model import from users API. Pull Request 3888
Cytoscape bugfix for SIF files (thanks to @bgruening). Pull Request 3896
Add missing trigger event to drop handler in content selector. Pull Request 3897
Fix Instance is not bound to a Session (thanks to @mvdbeek). Pull Request 3904
Fix library parameter style width to avoid second line. Pull Request 3926
More careful inspection of fixes for MAF tools requiring Galaxy. Pull Request 3935
Containers interface and GIE bugfixes. Pull Request 3941
Show default value for runtime values. Pull Request 3943
Fix JJ’s column_join tool depending on Galaxy at runtime. Pull Request 3961
Fix edge case for workflow mapping handling when tools not available. Pull Request 3962
Tour fixes (thanks to @nsoranzo). Pull Request 3970
Fix some typos in config/galaxy.ini.sample (thanks to @nsoranzo). Pull Request 3971
Add two tools from str_fm repository to galaxy module whitelist. Pull Request 3973
Remote user logout fixes, rework header handling a bit. Pull Request 3976
Fixes to get GIEs working on Mac OS X. Pull Request 3978
Rework Jupyter IE for additional_ids handling. Pull Request 3980
Update Babel to a version compat. with sphinx. Pull Request 3986
Use the configured tool_path when loading data managers. Pull Request 3989
Typo in remote user / user preferences exception (thanks to @hexylena). Pull Request 4015
Fix setting up explicit container resolution from a container resolution config file. Pull Request 4020
Fix spelling error in error message (thanks to @pvanheus). Pull Request 4026
Fixes for containerized job executions. Pull Request 4033
Try activating Conda environment again if previous activation failed (thanks to @mvdbeek). Pull Request 4035
HTCondor fix for running containers (thanks to @bgruening). Pull Request 4038
Fix destructive tag rerender to be a simpler, more precise html replacement. Pull Request 4052
Fix for S3ObjectStore _clean_cache method. Pull Request 4055
Correct webhooks to use Galaxy.root and be compatible with proxy_prefix (thanks to @hexylena). Pull Request 4065
Fix global grid action event handler. Pull Request 4068
Fix various transiently failing Selenium tests. Pull Request 3450
Do not modify self.app.config.tool_data_path and self.app.config.tool_data_table_config_path during metadata setting (thanks to @mvdbeek). Pull Request 4089
Open Babel update for datatype converters (thanks to @bgruening). Pull Request 4094
Change to only inheriting ‘name’ tags. Pull Request 4104
Fixes repeat handling in workflow editor, store values as passed from UI. Pull Request 4107
New Contributors
We thank these new contributors that helped with the Galaxy codebase since 16.04:
Evgeny Anatskiy
Abdulrahman Azab
balto
Bérénice Batut
Marius van den Beek
Maria Bernard
Jean-Frédéric Berthelot
Léo Biscassi
Clemens Blank
Jorrit Boekel
Christian Y. Brenninkmeijer
Freek de Bruijn
Matt Chambers
Peter Cock
Matthias Desmet
Shane Dowling
Ignacio Eguinoa
Mark Einon
fescudie
Anne Fouilloux
Jean-Frédéric
Maximilian Friedersdorff
Ben Fulton
Carrie Ganote
Ryan Golhar
Peter van Heusden
Saskia Hiltemann
Hans-Rudolf Hotz
Manabu Ishii
Xiaoqian Jiang
Iyad Kandalaft
Edward Kirton
Jasper Koehorst
Anup Kumar
Delphine Lariviere
Yvan Le Bras
Gildas Le Corguillé
Alexander Lenail
Pierre Lindenbaum
Mikael Loaec
Thoba Lose
Zipho Mashologu
Hervé Ménager
Pablo Moreno
Takao Nakaguchi
Balthazar Pavot
Matt Ralston
ramezrawas
Athos Ribeiro
Devon Ryan
Michael Sauria
Varun Shankar
Timur Shtatland
Sourav Singh
Eteri Sokhoyan
Nick Stoler
Scott Szakonyi
Jesse c j van Dam
Ashok Varadharajan
Jeremy Volkening
Pavan Videm
Junzhou Wang
Joachim Wolff
Thomas Wollmann
Jay Young
New Configuration Options
Below are the newly added options you can configure in galaxy.ini
.
To see the whole sample file visit our GitHub repository.
# Set to True to enable monitoring of the tool_data and shed_tool_data_path
# directories. If changes in tool data table files are found, the tool data
# tables for that data manager are automatically reloaded.
# Watchdog ( https://pypi.python.org/pypi/watchdog ) must be installed and
# available to Galaxy to use this option. Other options include 'auto'
# which will attempt to use the watchdog library if it is available but won't
# fail to load Galaxy if it is not and 'polling' which will use a less
# efficient monitoring scheme that may work in wider range of scenarios
# than the watchdog default.
#watch_tool_data_dir = False
# To run interactive environment containers in Docker Swarm mode (on an
# existing swarm), set this option to True and set `docker_connect_port` in the
# IE plugin config (ini) file(s) of any IE plugins you have enabled and ensure
# that you are not using any `docker run`-specific options in your plugins'
# `command_inject` options (swarm mode services run using `docker service
# create`, which has a different and more limited set of options). This option
# can be overridden on a per-plugin basis by using the `swarm_mode` option in
# the plugin's ini config file.
#interactive_environment_swarm_mode = False
# Galaxy can run a "swarm manager" service that will monitor utilization of the
# swarm and provision/deprovision worker nodes as necessary. The service has
# its own configuration file.
#swarm_manager_config_file = config/swarm_manager_conf.yml
# Log to graphite
# Graphite is an external statistics aggregator (https://github.com/graphite-project/carbon)
# Enabling the following options will cause galaxy to log request timing and
# other statistics to the configured graphite instance. The graphite_prefix is
# useful if you are running multiple Galaxy instances and want to segment
# statistics between them within the same aggregator.
#graphite_host=
#graphite_port=2003
#graphite_prefix=galaxy
# Enable the new container interface for Interactive Environments
#enable_beta_containers_interface = False
New Datatypes
Below are the newly added datatypes and sniffers you can configure in datatypes_conf.xml
.
To see the whole sample file visit our GitHub repository.
<datatype extension="d3_hierarchy" type="galaxy.datatypes.text:Json" mimetype="application/json" subclass="true" display_in_upload="false"/>
<datatype extension="mzdata" type="galaxy.datatypes.proteomics:MzData" mimetype="application/xml" display_in_upload="true" />
<datatype extension="percin" type="galaxy.datatypes.tabular:Tabular" subclass="true" />
<datatype extension="percout" type="galaxy.datatypes.xml:GenericXml" subclass="true" />
<datatype extension="hardklor" type="galaxy.datatypes.tabular:Tabular" subclass="true" />
<datatype extension="kronik" type="galaxy.datatypes.tabular:Tabular" subclass="true" />
<datatype extension="ptalign" type="galaxy.datatypes.text:PlantTribesMultipleSequenceAlignment" />
<datatype extension="ptortho" type="galaxy.datatypes.text:PlantTribesOrtho" />
<datatype extension="ptorthocs" type="galaxy.datatypes.text:PlantTribesOrthoCodingSequence" />
<datatype extension="pttree" type="galaxy.datatypes.text:PlantTribesPhylogeneticTree" />
<sniffer type="galaxy.datatypes.proteomics:MzData"/>
To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.
You can always chat with us on Matrix.
Thanks for using Galaxy!