May 2020 Galaxy Release (v 20.05)

Highlights

Many new Interactive Tools Galaxy interactive tools allow a greater depth of analysis via access to GUI-based tools inside an instance. 10 new interactive tools (some previously available only on the UseGalaxy.eu server) join the standard base toolset. Improvements to the Higlass interactive tool now also allow for a more in-depth visualization of multiple datasets in various formats simultaneously (Pull Request 9559).

Data Tables can now be backed by refgenie Refgenie manages storage, access, and transfer of reference genome resources. Galaxy can now fill data tables that were received from a refgenie installation (Pull Request 9600).

Tool Shed is now Python3 ready As the last component in the Galaxy Codebase the Tool Shed has been ported to Python 3. This concludes a 4 year effort to port the Galaxy codebase to Python 3 (Pull Request 9528).

Workflow Editor and Workflow Run Form in Vue.js The Workflow Editor and Workflow Run Form have been re-written in Vue.js. While the functionality has been preserved this lays the groundwork for creating beautiful and reactive custom variants of these components in the future (Pull Request 9346, Pull Request 9332, Pull Request 9324, Pull Request 9249, Pull Request 9201, Pull Request 9147, Pull Request 9138, Pull Request 9471).

Accelerated Galaxy Startup Galaxy now caches expanded tool documents, delays creating the tool search index until after startup and creates search indexes incrementally (Pull Request 9610).

Also check out the 20.05 user release notes

Get Galaxy

The code lives at GitHub and you should have Git to obtain it.

To get a new Galaxy repository run:

$ git clone -b release_20.05 https://github.com/galaxyproject/galaxy.git

To update an existing Galaxy repository run:

$ git fetch origin && git checkout release_20.05 && git pull --ff-only origin release_20.05

See the community hub for additional details regarding the source code locations.

Upgrade Warning for Admins

We have been simplifying the organization of our static and client content. As a result, the /static directory now only requires a single rule in your proxy (nginx, Apache, etc.) configuration. Previously, additional rules for /static/scripts and /static/style were required. Both of these additional rules should be removed from your proxy server configuration when upgrading your Galaxy server to release 20.05.

More information about proxy server configuration for Galaxy can be found in Proxying Galaxy with NGINX and Proxying Galaxy with Apache.

Release Notes

Enhancements

New data types for OpenMS (thanks to @bernt-matthias). Pull Request 8109
Refactor Workflow Run Form Toward Vue (part 1) Pull Request 9138
Refactor Workflow Run Form Toward Vue (part 2) Pull Request 9147
Replace Workflow Editor Tool Panel with Vue component Pull Request 9201
Replace right panel in Workflow Editor with Vue component Pull Request 9249
Replacing Workflow Editor Node with Vue-component Pull Request 9332
Render workflow node connections using d3 Pull Request 9471
Make Toolshed Python 3 compatible & tests Pull Request 9528
Migrate interactive tools from EU server (thanks to @astrovsky01). Pull Request 9559
Allow Tool Data Tables to be backed by refgenie Pull Request 9600
Startup speedups (XML doc cache, delay non-essential tool param parsing & indexing) Pull Request 9610

Enable admin to resend activation email Pull Request 9154
Create Python3 virtualenv by default (thanks to @nsoranzo). Pull Request 9344
iRODS object store rewrite (thanks to @kxk302). Pull Request 9438
Tool recommendation (thanks to @anuprulez). Pull Request 9440
Tool recommendations in workflow editor (thanks to @anuprulez). Pull Request 9569
Added whitelisting option for user registration (thanks to @MiguelJulia). Pull Request 9501
Editor as visualization (thanks to @OlegZharkov). Pull Request 9531
Chira visualization for Chimeras (thanks to @OlegZharkov). Pull Request 9562
Datasource tool for SRA manifest Pull Request 9613, Pull Request 9892

Skip visualization build, if no changes were registered (thanks to @OlegZharkov). Pull Request 9634

Add tpr datatype (thanks to @simonbray). Pull Request 9244
Add datatypes for xcms and camera (thanks to @computational-metabolomics). Pull Request 9470
index with -c flag for large bam (one region larger than 2**29 - 1) (thanks to @FredericBGA). Pull Request 9570
Sniffer for multi-member zip archives (thanks to @lecorguille). Pull Request 9595

Followup to shed tool static fixes Pull Request 9450
Remove deprecated interpreter attribute from tools <command> (thanks to @nsoranzo). Pull Request 9490
Resize D3 tree and make it responsive (thanks to @anuprulez). Pull Request 9547
Fix ITs to work in remote Kubernetes Pull Request 9614

Fix galaxy_tool_util for composite file input testing in workflows. Pull Request 9312
Refactor toward testing workflow collection outputs. Pull Request 9340
Remove Backbone and Globals dependency from Workflow client code Pull Request 9346
Allow anonymous downloading of public workflows Pull Request 9736

MS Edge Fixes Pull Request 9289

Refactor more of jobs admin component toward API. Pull Request 9093
make more tool parsing errors contain id (thanks to @bernt-matthias). Pull Request 9139
“Normalize” more custom functions in admin jobs index. Pull Request 9157
Job Lock VueJS component. Pull Request 9181
add a new function to get the local ip address (thanks to @gmauro). Pull Request 9353
Integration test for GxITs against remote proxies. Pull Request 9504
paramiko: load system host keys (thanks to @eancelet). Pull Request 9553
Optimize setting external ID for jobs. Pull Request 9691

Enable manual user activation Pull Request 9143
Skip migration if uninitialized database exists Pull Request 9491

Add Okta as OIDC authentication option (thanks to @selten). Pull Request 7497
Added trigger to dataset table to update history.update_time Pull Request 8187
Notifications on job completion (thanks to @astrovsky01). Pull Request 8427
Issue #8445: Clicking on user/email in User menu triggers a page refresh - fix (thanks to @janani-prasad). Pull Request 8644
Third Party Identity Management (thanks to @juleengraham). Pull Request 9060
Disable the admin_toolshed controller (except for tests). Pull Request 9072
Upgrade to Fontawesome 5 Pull Request 9101
Add a test for a tool with from_working_dir, outputs_to_working_directory and pulsar (thanks to @bgruening). Pull Request 9108
Client build work (HMR, build upgrades, refactoring asset placement) Pull Request 9123
Correct typos, update location of api tests Pull Request 9158
Add more repository installation test cases Pull Request 9160
Add support for logging out of OIDC Identity Provider (thanks to @machristie). Pull Request 9162
Vue-ify Upload Dialog w/small fixes Pull Request 9167
Update galaxy.tool_util.cwl for latest changes in CWL branch. Pull Request 9175
More progress on Tool shed cleanup Pull Request 9186
Full Python dependencies update (thanks to @nsoranzo). Pull Request 9188, Pull Request 9402, Pull Request 9529, Pull Request 9697
Allow mixed store_by objectstore configurations. Pull Request 9197
implement workflow bookmark selenium test (thanks to @OlegZharkov). Pull Request 9208
Use a Galaxy root path with a space for integration tests (thanks to @nsoranzo). Pull Request 9223
implementing a global tools-view component + schema.org support (thanks to @OlegZharkov). Pull Request 9261
Add icons to dropdowns in dataset and workflow list Pull Request 9273
extend config file inputs tests (thanks to @bernt-matthias). Pull Request 9279
Do not strip tab characters from data_column reference and clarify documentation for use_header_names (thanks to @bernt-matthias). Pull Request 9291
Update CONTRIBUTING.md (thanks to @nsoranzo). Pull Request 9299
Add more info to debug logging for loading datatypes. Pull Request 9301
Add do_not_cache decorator, apply to codebase. Pull Request 9311
Use Bootstrap components to build the Workflow Editor Toolbar Pull Request 9324
implement selenium test User->Preferences->Manage information (thanks to @OlegZharkov). Pull Request 9355
Fix success message in user preferences, add help text to toolbox filter page Pull Request 9368
CSS changes for a modern look and feel (thanks to @bedroesb). Pull Request 9386
Added conda python update (thanks to @astrovsky01). Pull Request 9390
Replace editor grid image with css, add editor client unit tests Pull Request 9392
Vuefy interactive tools (thanks to @OlegZharkov). Pull Request 9409
Add delta_frac for sim_size to specify relative file size differences (thanks to @bernt-matthias). Pull Request 9425
Make mutable_config_dir configurable + rename Pull Request 9444
Remove virtualenv<20 requirement for mulled tests (thanks to @nsoranzo). Pull Request 9447
Remove mapping related abstraction classes from workflow client, increase node header contrast Pull Request 9474
Add eslint rule to check for invalid slot specifications Pull Request 9481
Another undocumented option: job metrics path (thanks to @hexylena). Pull Request 9482
Document maxpctfull objecto store option (thanks to @hexylena). Pull Request 9498
Selenium Multi-History View (thanks to @OlegZharkov). Pull Request 9505
Adjust dataset error reporting page, remove email option, remove long info text Pull Request 9508
Drop remaining Python 2.7 CircleCI tests (thanks to @nsoranzo). Pull Request 9518
Implementation selenium test for user change password (thanks to @Realjuan). Pull Request 9521
Rename and modify current_history_contents method Pull Request 9526
Improvement to the selenium test to change password from user preferences (thanks to @Realjuan). Pull Request 9545
Implementation of Selenium test for signout from user preferences (thanks to @Realjuan). Pull Request 9546
Move OSX startup tests to github CI, drop travis tests Pull Request 9563
Check if path is set before joining Pull Request 9568
Sort Tool Search Result by ranking (thanks to @juleengraham). Pull Request 9579
TS performance fixes Pull Request 9580
Vuefy Job Destination Parameters (thanks to @OlegZharkov). Pull Request 9583
Fix kubernetes test failures again Pull Request 9585
Enabled various object store types to store data objects by uuid. (thanks to @kxk302). Pull Request 9609
Add IDs to admin links and selenium navigation. Pull Request 9616
Feature/refactor rule builder (thanks to @assuntad23). Pull Request 9619
Add selenium test for admin toolshed operations. Pull Request 9628
Add unit tests for all schema-defined config properties Pull Request 9631
Officially allow adding new datatypes to the latest tagged release (thanks to @nsoranzo). Pull Request 9632
Update the main readme with new Python version Pull Request 9635
Minor IT doc clarification Pull Request 9668
Enhance toolshed selenium test. Pull Request 9678
Optimize history_content union and filtering query Pull Request 9680
Add CSS class to messages in error report pages to make errors more visible (thanks to @hexylena). Pull Request 9690
Feature/3937 advanced search tooltip (thanks to @assuntad23). Pull Request 9696

Rename function, params in ConfigSerializer Pull Request 9247
Make str.join()s more efficient (thanks to @nsoranzo). Pull Request 9258
Faster and cleaner set creation (thanks to @nsoranzo). Pull Request 9300
Remove broken script (thanks to @nsoranzo). Pull Request 9302
Clean up old ‘blue’ style path, update documentation and samples Pull Request 9306
Consistent “import” of maketrans (thanks to @nsoranzo). Pull Request 9351
Move schema loading/validating code out of config Pull Request 9371
Remove more duplication from TS, galaxy config Pull Request 9379
Unify default values used for delta (sim_size) (thanks to @bernt-matthias). Pull Request 9439
Vue slot refactoring Pull Request 9443
Move sanitize_whitelist into mutable_config_dir Pull Request 9455
Use auto_create functionality of get_private_user_role() (thanks to @nsoranzo). Pull Request 9509
Combine selenium tests for creating and switching histories Pull Request 9527
Eslint one-var spec / airbnbstyle Pull Request 9532
unify usage of Popen (thanks to @bernt-matthias). Pull Request 9544
Prettier 2.0 Pull Request 9552
Abstract ObjectStore interface implementation Pull Request 9554
Misc. refactoring of galaxy+TS config Pull Request 9582
Use PyGithub for issue creation Pull Request 9637
clean up code to use helper function instead (thanks to @lawrence14701). Pull Request 9643
Respect retry-after header, raise on unexpected status codes Pull Request 9659
restore tool_util.deps.commands (thanks to @bernt-matthias). Pull Request 9707

Fixes

Update charts entrypoint inclusion of dist artifacts Pull Request 9511

Python3 compatible SnpEff and SnpSift metadata detection (thanks to @wm75). Pull Request 9282
Add PAF, GFA1 and odgi datatypes (thanks to @nsoranzo). Pull Request 9592
Add xg, vg and protobuf datatypes (thanks to @bgruening). Pull Request 9593
Fix gff3.gz and gff3.bz2 datatypes Pull Request 9731
Use right extension in bz2_to_uncompressed.xml Pull Request 9734
Fix disappearing datatypes after restart Pull Request 9835

fix error in bundled tool: shrimp_color_wrapper (thanks to @bernt-matthias). Pull Request 9665
extract_genomic_dna.py: import Comment, Header from bx.tabular.io instead of bx.intervals.io (thanks to @cat-bro). Pull Request 9748
Move some tools that no longer need galaxy on the PATH (thanks to @bernt-matthias). Pull Request 9850

Fix workflow list filtering dropdown Pull Request 9468
Fix workfow parameter connections and default values Pull Request 9648
Fix ChangeOutputDatatype being dropped on node update Pull Request 9676
Trigger onchange event, fixes tool version changes being lost Pull Request 9695
Input dataset module format filtering fix; Pull Request 9884

fix for python3 and GEDA, was getting “Connection closed at byte 0” Pull Request 9507
Checkbox bugfix Pull Request 9667

Fix none type exception in provided metadata handling. Pull Request 9173
Handles floats correctly in ByteSize (thanks to @pcm32). Pull Request 9418
unicodify message (thanks to @ieguinoa). Pull Request 9494
Fixes for embedded Pulsar with MQ (with tests) Pull Request 9519
Fix outputs written to storage despite outputs_to_working_directory Pull Request 9683
Fix collection mapping in extended metadata mode. Pull Request 9709
Force container image names to lower case to satisfy Docker (thanks to @nuwang). Pull Request 9733
Make watcher more resilient to missing files (thanks to @nuwang). Pull Request 9738
Improved exception handling for kubernetes runner (thanks to @nuwang). Pull Request 9751
Check dataset is accessible and use payload in job error API (thanks to @nsoranzo). Pull Request 9753
Fix the cleanup condition when job state is OK and cleanup_job is set as “onsuccess’ (thanks to @gmauro). Pull Request 9761
Migrate condor runner to jwd for job-files (thanks to @bgruening). Pull Request 9833
Fix incorrect variable assignment Pull Request 9907

Fix vue helper in admin router, fixes toolshed installation, avoids console errors Pull Request 9220
Fix default quota creation message. Pull Request 9479
Fix data manager table headers Pull Request 9480
Save integrated tool panel once on toolbox reload Pull Request 9487
Cannot find migrate directory for tools database (thanks to @phac-nml). Pull Request 9699

Fix S3 issue discovered during admin training (thanks to @hexylena). Pull Request 7294
Make failing to load tools from a tool_dir non-fatal. Pull Request 9106
Fix incorrect reference to app (thanks to @nuwang). Pull Request 9159
Missing env in AskOmics IT (thanks to @xgaia). Pull Request 9177
Check return value of quick_load Pull Request 9185
Fix sending email error on python-3.7 (thanks to @shenweiyan). Pull Request 9204
Properly quote paths in upload tool (thanks to @bernt-matthias). Pull Request 9222
Fix reference in workflows API docs (thanks to @nsoranzo). Pull Request 9259
Fix target_str_to_targets Pull Request 9268
Do not test config_dir default value Pull Request 9288
Remove a dead symlink (thanks to @abretaud). Pull Request 9292
Fixes reports app under py3 Pull Request 9305
Allow setting container path for docker_volumes (thanks to @AndreasSko). Pull Request 9310
Skip test_config_option[build_sites_config_file] (thanks to @nsoranzo). Pull Request 9357
fix special characters in citation field (thanks to @OlegZharkov). Pull Request 9373
Use ES6 function declaration to handle error response in workflow form Pull Request 9412
Fix first_startup Travis tests (thanks to @nsoranzo). Pull Request 9423
Use a Galaxy root path with a space for converters and mulled tests (thanks to @nsoranzo). Pull Request 9429
Fix test_run_simple() test broken on Python 2 by virtualenv 20.0.5 (thanks to @nsoranzo). Pull Request 9458
Use browser locale over global default Pull Request 9460
Fix login handling w/ redirects and user creation disabled. Pull Request 9462
Add missing pygithub dev-requirement (thanks to @nsoranzo). Pull Request 9463
Drop manage_dependency_relationships option Pull Request 9465
fix options_filter_attribute: needs to be str (thanks to @galaxyproject). Pull Request 9466
History Dropdown Fix Pull Request 9469
Print error message if no action is selected in pgcleanup script (thanks to @nsoranzo). Pull Request 9489
Drop custom json encoding for tool test API endpoint (thanks to @nvk747). Pull Request 9496
Keep status code on message exception and history import/export fixes Pull Request 9500
Fixed tool locations (thanks to @astrovsky01). Pull Request 9512
Update the current_history method in navigates_galaxy.py Pull Request 9514
Free space before running tests Pull Request 9533
History import/export fixes Pull Request 9535
Retry reading tool conf on IOError Pull Request 9537
Fix citation ids and vue unit test in ToolsView (thanks to @OlegZharkov). Pull Request 9538
Bump tensorflow from 1.12.2 to 1.15.2 in /lib/galaxy/dependencies (thanks to @galaxyproject). Pull Request 9540
Incremental indexing for tool shed whoosh indexes Pull Request 9541
fix documentation for data checksum attribute (thanks to @bernt-matthias). Pull Request 9543
Prevent exception in toolshed if userid is ‘current’ Pull Request 9548
Fix setting annotation that contains only integers Pull Request 9549
Fix docs to update conda from conda-forge (thanks to @nsoranzo). Pull Request 9550
Disable workflow parameter switch for data inputs, augment form handling Pull Request 9551
Hide LDDA when creating collection in library interface Pull Request 9561
Fix indexing script Pull Request 9565
Assign random password in user manager if none specified (thanks to @nuwang, @nsoranzo). Pull Request 9587, Pull Request 9851
Fix removal of workflow outputs from workflow nodes in editor Pull Request 9606
Fix ELIXIR AAI (thanks to @lecorguille). Pull Request 9620
Fix tensorflow bug (thanks to @anuprulez). Pull Request 9622
Change session timeout redirect URL to login entrypoint. Pull Request 9626
Changed CSS overflow property from scroll to hidden to make table scrollable (thanks to @assuntad23). Pull Request 9627
fix links in Galaxy xsd docs (thanks to @bernt-matthias). Pull Request 9633
Fix TypeError: OSError object is not subscriptable under Python 3 (thanks to @nsoranzo). Pull Request 9640
Fix vcf_bgzip mimetype, allow cors, tbi from metadata element Pull Request 9653
Hide “python: not found” error when Python 2 is not installed (thanks to @nsoranzo). Pull Request 9657
Fix for time.clock() removal in Python 3.8 (thanks to @nsoranzo). Pull Request 9666
Fixed an issue in object store integration tests (thanks to @kxk302). Pull Request 9677
Plugin build hash bugfix Pull Request 9679
Backport auth process lock fix Pull Request 9681
Update documentation (thanks to @nsoranzo). Pull Request 9693
Rename galaxy.util.logging to galaxy.util.custom_logging Pull Request 9705
Make collection import from library import into correct history Pull Request 9715
Fix workflow steps import feature in workflow editor Pull Request 9724
Fix custos credential fetching in python3 Pull Request 9726
oidc-based authnz provider overhaul; breaking changes for keycloak Pull Request 9730
Sort Workflow list by last updated time instead of created time Pull Request 9740
Remove dummy tool sequence (thanks to @anuprulez). Pull Request 9752
Switch to lxml for xml parsing Pull Request 9759
Set UTF-8 for parsing tool xml (thanks to @AndreasSko). Pull Request 9760
inputs config file generation depend on profile (thanks to @bernt-matthias). Pull Request 9776
Pass through IOError/OSError in tool_shed parse_xml Pull Request 9781
Maintain whitespace when parsing token from xml Pull Request 9815
Log exception if reading tool file failed unexpectedly Pull Request 9816
Drop custom in-process lock class Pull Request 9817
Update NoseHTML to 0.4.6 for Python 3.8 compatibility (thanks to @nsoranzo). Pull Request 9821
Tolerate exception in container resolution Pull Request 9824
Invalidate tool cache when tool modified outside of Galaxy Pull Request 9866
Custos spec fixes Pull Request 9889
Fixes for flake8 3.8 and bugbear (thanks to @nsoranzo). Pull Request 9896
Fix webpack import proxy prefix Pull Request 9898
Fix extra preference values leaking between requests (thanks to @nuwang). Pull Request 9913
Rewrite check_html() and Text.count_data_lines() to read a chunk of each line (thanks to @nsoranzo). Pull Request 9920
Make cookie handling more robust Pull Request 9921
Enable regex validator to validate lists of values. Pull Request 9925
oidc: fix dict_keys does not support indexing (thanks to @almahmoud). Pull Request 9931
Move the --debug flag after the conda subcommand (thanks to @nsoranzo). Pull Request 9932

To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.

You can always chat with us on Matrix.

Thanks for using Galaxy!

The Galaxy Team