January 2019 Galaxy Release (v 19.01)¶

Highlights¶

Revised UI Style

The Galaxy stylesheet has been substantially reworked resulting in the largest visual refresh to Galaxy in years.

Pull Request 6422, Pull Request 6790, Pull Request 6868, Pull Request 6870, Pull Request 6881, Pull Request 6905, Pull Request 6910, Pull Request 6967, Pull Request 6985, Pull Request 7017

More Colorful Tags

Use consistently hashed randomized colors for history dataset name tags to increase visual distinction between tags, making it easier to track data flow in complex histories. Thanks to @hexylena. Pull Request 7072, Pull Request 7081

Extensive Workflow Enhancements (continued)

Galaxy 18.09 featured a bunch of important and diverse fixes and enhancements to workflows on both the frontend and backend. That progress continues in release 19.01.

The workflow editor now features explicit step parameter nodes for non-file data (integers, strings, booleans, etc) and the ability to connect these to all tool inputs - not just files (Pull Request 6925, Pull Request 7150). The YAML-based Galaxy Format 2 workflow language is now included in Galaxy as a beta option and allows import and export of human readable workflows (Pull Request 6776, Pull Request 6807, Pull Request 6811, Pull Request 6746) as well as extensions to allow Galaxy to serve as a file editor for such workflows (Pull Request 7019). The workflow editor features more helpful labels on inputs and outputs (Pull Request 6770) and important fixes including logic for connecting collections (Pull Request 6732) and dealing with missing tools (Pull Request 7227, Pull Request 7258). The workflow run form now deals with default values more correctly (Pull Request 7157).

Enhanced Support for Singularity

Singularity container support in Galaxy has been brought to parity with Docker container support. Specifically, Galaxy can now be configured to fetch Biocontainers automatically for enabled job destinations. Moreover, explicitly annotated Docker containers (either in tools or job destinations) can be used with Singularity. Thanks to @mvdbeek. Pull Request 7125

Get Galaxy¶

The code lives at GitHub and you should have Git to obtain it.

To get a new Galaxy repository run:

$ git clone -b release_19.01 https://github.com/galaxyproject/galaxy.git

To update an existing Galaxy repository run:

$ git fetch origin && git checkout release_19.01 && git pull --ff-only origin release_19.01

See the community hub for additional details regarding the source code locations.

Security¶

Permissions Not Checked to Modify Dataset Metadata¶

Jelle Scholtalbers reported an issue where permissions were not being checked when modifying dataset metadata. This has been resolved with Pull Request 7322 and the fix backported through 18.01.

Release Notes¶

Enhancements¶

Adds Aequatus visualisation plugin (thanks to @anilthanki). Pull Request 6484
Integrate framework code for interfacing with cloud authorization providers. Pull Request 5903
Added support for chaining dynamic destinations (thanks to @nuwang). Pull Request 7006
Upgrade web client building to use webpack 4 Pull Request 6917
Vue.js rewrite of admin data manager user interface (thanks to @scrathat). Pull Request 6729
Implement ‘make all datasets private’ functionality. Pull Request 6952
Add datatype sniff button to edit dataset attributes page (thanks to @qiagu). Pull Request 7018
Overhaul handler assignment methods. Pull Request 7129, Pull Request 7091
Add tool for sending dataset to the cloud. Pull Request 6583
Infrastructure/API Support for composite uploads/downloads. Pull Request 6844
Improvements to tool testing including allowing remote tests and tests without admin API keys. (thanks to @mvdbeek). Pull Request 7160, Pull Request 7167, Pull Request 7199
Add ISA-TAB datatype (thanks to @phnmnl). Pull Request 5787
Adding itp and mdp datatypes (for gromacs) (thanks to @simonbray). Pull Request 6812
Add pqr datatype (thanks to @jj-umn). Pull Request 6816
Add nmrML datatype (thanks to @phnmnl). Pull Request 6903
Add RMA6 datatype (MEGAN6 read-match archive) (thanks to @bernt-matthias). Pull Request 6962
Add fastg datatype (thanks to @jj-umn). Pull Request 6987
Add datatype for Mash Sketch files. (thanks to @dfornika). Pull Request 7087
Add JavaScript client library utility function to create history entries. Pull Request 6693
Give all model objects with an id a __repr__ that includes the id Pull Request 6757
Add styled loader vue component (thanks to @inkuzmin). Pull Request 7052
Restore the ability to track jobs for a WF invocation step. Pull Request 6559
Improvements to the grouping tool (thanks to @bernt-matthias). Pull Request 6577
Trim whitespace from data manager conf paths (thanks to @pvanheus). Pull Request 6682
Track workflow step input definitions in our model. Pull Request 6850, Pull Request 7103
Deprecate size attribute of <param/> and remove it from tools (thanks to @nsoranzo). Pull Request 6684
Create container_resolvers_conf.xml.sample (thanks to @pcm32). Pull Request 6700
Update to latest galaxy-lib. Pull Request 6709
Updates to OIDC backends default configuration. Pull Request 6727
Small local testing improvements (thanks to @nsoranzo). Pull Request 6738
Add backrefs to mapped objects pointing at Dataset. Pull Request 6754
Fixes for the upload row template. Pull Request 6758
Disable file name text field after upload. Pull Request 6762
Remove empty log messages. Pull Request 6763
Release issue scaffold tweaks. Pull Request 6778
Warn job runner plugin writers not to emulate condor.py. Pull Request 6783
Add PB and EB to util.size_to_bytes Pull Request 6786
Add a pgcleanup.py action to purge HDAs of purged histories. Pull Request 6801
Various pgcleanup.py enhancements. Pull Request 6736, Pull Request 6877
Fix typos on “reserve_memory” / “reserve_cpu” (thanks to @pvanheus). Pull Request 6817
Add more detail about container tags (thanks to @pvanheus). Pull Request 6818
Add docs about running jobs in containers (thanks to @pvanheus). Pull Request 6820
Remove trans from OIDC-related types. Pull Request 6826
Clarify code used in output <filter> tags (thanks to @pvanheus). Pull Request 6827
Convert user_is_admin and user_can_do_run_as to properties. Pull Request 6835
Don’t create jobs for inactive users. Pull Request 6837
Replace deprecated assertEquals() method (thanks to @nsoranzo). Pull Request 6841
Improvements to GIE docs (thanks to @thobalose). Pull Request 6846
Indicate language in client docs to enable syntax highlighting.. Pull Request 6848
Fix panel header margins in history and tool panel Pull Request 6854
Small fixes (thanks to @nsoranzo). Pull Request 6861
Datatype upload integration test and fixes (thanks to @mvdbeek). Pull Request 6869
Use small button for editable text fields in history annotation Pull Request 6878
Improve Kubernetes job error handling (thanks to @ilveroluca). Pull Request 6886
Fixes and updates to the Neo4j GIE (thanks to @thobalose). Pull Request 6897
Allow remove_value filter on data tables (thanks to @nsoranzo). Pull Request 6914
Update all dependencies (thanks to @nsoranzo). Pull Request 6921
Be more precise with the job runner worker thread join timeout and improve documentation. Pull Request 6924
Selenium test enhancements and fixes. Pull Request 6927, Pull Request 6982
Small testing tweaks from the CWL branch. Pull Request 6929
Tests need to count data sets and collections (thanks to @bernt-matthias). Pull Request 6931
Eliminate some mutable defaults in populators.py. Pull Request 6933
Default shed tool installation to database/shed_tools. Pull Request 6937
Add option to strip url prefix from pair names to collection builder (thanks to @shiltemann). Pull Request 6943
Small API workflow test improvements. Pull Request 6954
Switch from PhantomJS to Chrome Headless for unit tests. Pull Request 6959
Added a metadata element to the biom1 datatype (thanks to @Slugger70). Pull Request 6972
Update locale.js (thanks to @istevenshen). Pull Request 6974
Use virtualenv when conda is on PATH (thanks to @mvdbeek). Pull Request 6975
Improvements to exit code and regex parsing (thanks to @bernt-matthias). Pull Request 6977
Optimize images with optipng (thanks to @bgruening). Pull Request 6980
Fix tool XSD to list tags as attribute of TestCollection (thanks to @mvdbeek). Pull Request 6983
Use detect_errors="exit_code" in __SET_METADATA__ tool (thanks to @nsoranzo). Pull Request 6989
Don’t mount external /tmp for tests. Pull Request 6990
Upgrade nodeenv’s nodejs to 10.13.0 LTS. Pull Request 6991
Fix broken links on Job Configuration page. Pull Request 6992
Allow rules module override and refactor submodule imports (thanks to @nuwang). Pull Request 6993
Fix JSON loading permission problem with run-as-real-user DRMAA job runner (thanks to @bernt-matthias). Pull Request 6995
CLI plugin for LSF (thanks to @pcm32), Pull Request 7007 Pull Request 6788
A new UNIVA extension to the DRMAA job runner (thanks to @bernt-matthias). Pull Request 7004, Pull Request 7161
DRMAA fix to soft catch of external runner fails (thanks to @bernt-matthias). Pull Request 6997
Replace global Galaxy references with galaxy singleton accessor/setter. Pull Request 7005
Workflow test case - test marking inputs as outputs. Pull Request 7016
Expose published workflows through API (thanks to @frederikcoppens). Pull Request 7020
Pass a JobWrapper instead of a Job to stop_job() methods (thanks to @nsoranzo). Pull Request 7023
Fix element_identifier access (thanks to @mvdbeek). Pull Request 7031
Avoid replacing data-referenced input values with default values Pull Request 7043
Update error parsing docs (thanks to @bernt-matthias). Pull Request 7049
Add resubmit handlers on Pulsar connection failures (thanks to @nuwang). Pull Request 7053
Add new “Help” item to the help menu. Pull Request 7054
Eliminate dependency on make for automatic client build. Pull Request 7060
Remove legacy job handling hacks. Pull Request 7082
De-couple objectstore abstraction from XML. Pull Request 7085
Localize version strings to the working directory w/predictable path. Pull Request 7086
Replace display applications mako with vue component Pull Request 7097
Run singularity in silent mode (thanks to @mvdbeek). Pull Request 7104
Eliminate unused code block in output_collect. Pull Request 7109
Add to_dict() to ObjectStores. Pull Request 7110
Enable tagging and annotation of jobs regardless of state. Pull Request 7115
Remove unused __collected_datasets__ from tool hook dict. Pull Request 7123
Add a Vue.js component to display tool errors in the admin panel. Pull Request 7124
Update help text in filter_from_file tool. (thanks to @pvanheus). Pull Request 7127
Remove unneeded session flushes and adds when discovering outputs. Pull Request 7134
Drop tools without installable dependencies (thanks to @mvdbeek). Pull Request 7136
Small de-duplication cleanups to galaxy.jobs Pull Request 7155
Improve tool form client to add error listener for input elements. Pull Request 7183, Pull Request 7229
Stop tool migrations checking by default (was broken in some cases). Pull Request 7275
Replace trans with session and request in PSA Storage. Pull Request 6838
Share ssh key generation across tests (thanks to @mvdbeek). Pull Request 6865
Specify POST and GET actions for cloudauthz API actions. Pull Request 6876
Prevent sanitized history names being saved in database. Pull Request 6898
Rework language in cloud data fetch tool from “Upload” to “Get” Pull Request 6899
Refactor collection mapping workflows toward independence from tools. Pull Request 6911
Normalized jQuery bundle. Pull Request 6970
Replaced jStorage with store. Pull Request 6994
Remove sudo: false from .travis.yml, it will stop being supported soon Pull Request 7035
Fixes state-change animations causing ‘block’ display. Pull Request 7122

Fixes¶

Fixes to allow config schema to validate (thanks to @hexylena). Pull Request 5919
Patch update.sh to address dependency update issues on a macOS. Pull Request 6694
Fix wrong version being returned if requested version doesn’t exist (thanks to @mvdbeek). Pull Request 6702
Small mixed fixes (thanks to @nsoranzo). Pull Request 6722
Prevent errors when a library dataset has been purged from disk (thanks to @dpryan79). Pull Request 6784
Fix apply rules bug in filter first/last N rows. (thanks to @shiltemann). Pull Request 6814
Updated volume kwds. (thanks to @thobalose). Pull Request 6840
Remove some model objects added in #5013 but unused before the PR was merged, and fix a model object typo Pull Request 6843
Revert selective metadata flushing - breaks multiple uploads/outputs. Pull Request 6858
prevent double sanitizing history name when renaming (thanks to @almahmoud). Pull Request 6871
Fix rule editor element alignment. Pull Request 6874
Fix styleguide; change permissions on jstree.scss. Pull Request 6879
Fix username filtering for published workflows (thanks to @almahmoud). Pull Request 6882
Align toolshed makos used in the admin panel, fix slider height. Pull Request 6883
Improve resilience on handling sharing attributes. Pull Request 6884
Update Bug Reports docs to use YAML syntax (thanks to @nsoranzo). Pull Request 6909
Fix expr: non-integer argument error when running make (thanks to @nsoranzo). Pull Request 6913
Do not log exception info twice (thanks to @nsoranzo). Pull Request 6916
Fix typos in scripts/bootstrap_history.py (thanks to @nsoranzo). Pull Request 6941
Make BAM index only available for coordinate sorted BAM files (thanks to @mvdbeek). Pull Request 6946
Use recently introduced overflow utility instead of overwriting card defaults Pull Request 6948
Fix sniffing of readname sorted BAM files (thanks to @mvdbeek). Pull Request 6950
Fix for workflow parameter collection in workflows view. Pull Request 6955
Topic/empty data table (thanks to @bernt-matthias). Pull Request 6957
Allow deferring ‘to_ext’ extension to the server Pull Request 6960
Allow upload for rdata.sce format (thanks to @nsoranzo). Pull Request 6963
Fix traceback when starting under Python3 with galaxy.ini (thanks to @nsoranzo). Pull Request 6978
Fix charts entrypoint for compatibility with newer chunk changes. Pull Request 6984
reclaim ownership for failed jobs (thanks to @bernt-matthias). Pull Request 6996
Change join two dataset output format to tabular (thanks to @qiagu). Pull Request 6998
Fixes for Python 3.7 (thanks to @nsoranzo). Pull Request 7008
Fix PID file handling for gunicorn (thanks to @nsoranzo). Pull Request 7013
Fix display_in_tool_panel status for shared workflows (thanks to @mvdbeek). Pull Request 7014
Use solid background for text field autocomplete options Pull Request 7027
Follow-up on #6850 (thanks to @nsoranzo). Pull Request 7029
Fix error when building images (thanks to @scholtalbers). Pull Request 7036
Fix involucro_auto_init default value (thanks to @nsoranzo). Pull Request 7038
Fix resubmission on tool error detection. Pull Request 7040
GRT: Rip out all parameters, round file sizes (thanks to @hexylena). Pull Request 7041
Pass through --daemon option to gunicorn (thanks to @nsoranzo). Pull Request 7042
Escape dataset names in job submission success message Pull Request 7062
Update galaxy.yml.sample and galaxy_options.rst (thanks to @nsoranzo). Pull Request 7064
Fix link to requirements.txt. Pull Request 7065
Improved parsing of sizes (bytes) (thanks to @bernt-matthias). Pull Request 7067
Downgrade bdbag - 1.5.0 requires python:>=2.7.9 Pull Request 7070, Pull Request 7079
Fix process key (thanks to @mvdbeek). Pull Request 7071
Fix welcome on redirect from facebook auth Pull Request 7074
Fix set_peek method for copied datasets (thanks to @mvdbeek). Pull Request 7078
Fix toolshed repository error display (thanks to @jvanbraekel). Pull Request 7080
Fix via refactor set_metadata so more output metadata pruned properly. Pull Request 7084
Replace credentials with authz_id Pull Request 7088
TravisCI configuration workaround for problems with boto configuration. Pull Request 7092
Fix size_to_bytes for some float values (thanks to @nsoranzo). Pull Request 7096
Client readme updates. Pull Request 7102
Don’t write outputs to object store twice. Pull Request 7111
Fix make client-watch extra error message. Pull Request 7113
Properly escape history names in scratchbook titles. Pull Request 7114
Fix activation of _galaxy_ conda env (thanks to @nsoranzo). Pull Request 7116
Increase timeout on flakey unit test. Pull Request 7120
Fixes and tests for collecting extra files for discovered datasets. Pull Request 7131
Address CORS issue in toolshed iframe handlers. Pull Request 7135
Use regular js-event handling instead of jquery for uploader Pull Request 7145
Fixes toolshed and reports styling that was previously applied at the… Pull Request 7146
Correct code error in ‘build job runner’ tutorial. Pull Request 7148
Added store library to exported libs in bundleEntries. Pull Request 7149
Fixes for library support of YAML/dictified tools. Pull Request 7152, Pull Request 7153
Fix for authnz managers (thanks to @machristie). Pull Request 7166
Misc fixes (thanks to @nsoranzo). Pull Request 7168
Fix process_key test and deal with unordered repeat elements (thanks to @mvdbeek). Pull Request 7169
Fix test_test_data_download (thanks to @mvdbeek). Pull Request 7171
Document message_box_class option (thanks to @nsoranzo). Pull Request 7174
Prevent XSS on toolshed when browsing repo changeset Pull Request 7180
Fix HDA accessibility check in Page’s renderForSave name fetching Pull Request 7181
Fix for workflow parameter test case. Pull Request 7188
Fixes for library dataset permission API. Pull Request 7190
Properly document handler assignment method configuration and fix logging configuration docs. Pull Request 7191
Re-enable script staging. Pull Request 7196
Fix error in view_tool_metadata (thanks to @mvdbeek). Pull Request 7201
Data library: fix tag_using_filenames not properly converted to bool (thanks to @abretaud). Pull Request 7204
Fixes to event handling re-enabling multi-history panel drag and drop functionality. Pull Request 7218
Fix displaying tool help images on python 3 (thanks to @mvdbeek). Pull Request 7228
Fix data_source tools on python 3 (thanks to @mvdbeek). Pull Request 7236
Interval datatype fixes (thanks to @mvdbeek). Pull Request 7237
strip() text when parsing xml (thanks to @mvdbeek). Pull Request 7252
Fix install operation in installed_repository_grid (thanks to @mvdbeek). Pull Request 7253
Update sqlalchemy-migrate requirement to 0.12.0 (thanks to @nsoranzo). Pull Request 7265
Fix _.escape typo in toolshed group detail view. Pull Request 7267
Fix installing tools via tool_id (thanks to @mvdbeek). Pull Request 7277
Remove errant space in intermine_simple_display.loc.sample. Pull Request 7298

To stay up to date with Galaxy’s progress, watch our screencasts, visit our community Hub, and follow @galaxyproject@mstdn.science on Mastodon or @galaxyproject on Twitter.

You can always chat with us on Matrix.

Thanks for using Galaxy!

The Galaxy Team