January 2020 Galaxy Release (v 20.01)
Highlights
- Enhanced workflow functionality
Workflows can now make use of optional datasets and optional parameters, optional parameter default values and dataset inputs can now be restricted to user-defined datatypes. The workflow editor and execution engine have been enhanced to allow defining options to choose from, and these options can be connected to compatible text, integer, float, color and select tool parameters for more flexible and universally re-usable workflows (Pull Request 9086).
- Major update to container and dependency management interface
The manage dependencies interface has been ported to vue.js and displays additional information about installed, missing and available dependencies and conveniently summarizes dependency status for all installed tools. Additionally the user interface now displays whether a Biocontainers image is available at quay.io (Pull Request 8741).
- Extended job metadata collection
Creating Job metadata has been reworked so that the dataset transfer that occurs after a Galaxy job can be initiated by the compute node that runs a Galaxy job. Previously the transfer was often done by Galaxy. Offloading this task to the compute node should allow handling many more jobs in parallel. This work will also make jobs running on remote compute resources much more efficient if the remote resources can write to Galaxy’s object store. To benefit from this new metadata setting Galaxy needs to use object stores that use uuids for storing datasets and set the
metadata_strategy
configuration value toextended
. This feature is in beta testing, but work is ongoing to mix existing object store configuration and choose the most efficient way of transfering datasets automatically (Pull Request 8930 ).
Also check out the 20.01 user release notes
Get Galaxy
The code lives at GitHub and you should have Git to obtain it.
- To get a new Galaxy repository run:
$ git clone -b release_20.01 https://github.com/galaxyproject/galaxy.git
- To update an existing Galaxy repository run:
$ git fetch origin && git checkout release_20.01 && git pull --ff-only origin release_20.01
See the community hub for additional details regarding the source code locations.
Updates to Tools and Converters in the Galaxy distribution
Many tools and Converters have been updated in the 20.01 release. As a consequence deployers need to install dependencies for these in the Admin Menu -> Manage Dependencies.
Deprecation Notice
With the release of Galaxy 20.01 we strongly encourage all deployers to run Galaxy under Python 3. For instructions on how to transition to Python 3 please see here.
Release Notes
Enhancements
CLI tools for installing/testing Docker and Singularity containers (thanks to @simonbray). Pull Request 7936
Markdown-based Galaxy Pages. Pull Request 8512
Make Masthead and workflow page screenreader friendly Pull Request 8700
Add data dialog input element to client form parameter list Pull Request 8774
Add has_size option (thanks to @simonbray). Pull Request 8852
Email notifications on job completion Pull Request 8948
Make tours reloadable Pull Request 9032
Add a config option to allow customizing the brand title Pull Request 9062
IT fixes and 3 new ITs Pull Request 9141
Remove GenomeSpace tools, OpenID, UI and requirements (thanks to @nsoranzo). Pull Request 9363
Hyphy-vision fixes and updates. Pull Request 9384
Datatypes for MassSpec DIA dlib,elib (thanks to @jj-umn). Pull Request 8657
Add bus as binary, remove extra datatype (thanks to @astrovsky01). Pull Request 8705
remove dada2_derep data type (thanks to @bernt-matthias). Pull Request 8902
Add new GROMACS datatypes: ndx, xvg, edr (thanks to @simonbray). Pull Request 8964
Add gff3.gz and gff3.bz2 datatypes (thanks to @FredericBGA). Pull Request 8967
Make mol2 visible (thanks to @galaxyproject). Pull Request 8995
Add a datatype subclass for CAT (Contig Annotation Tool) (thanks to @jj-umn). Pull Request 9087
Add tabular.gz datatype and add embl to uploadable formats Pull Request 9293
Upgrade askomics interactive tool (thanks to @xgaia). Pull Request 8517
Remove collection ID character restriction from ‘Relabel collection from file’ tool (thanks to @brinkmanlab). Pull Request 8571
Allow space, dot and comma in relabel identifier tool Pull Request 8757
Fix script shebangs (thanks to @nsoranzo). Pull Request 8888
Allow admins to import tools/workflows from paths. Pull Request 9003
Move sam_pileup to GALAXY_LIB_TOOLS_VERSIONED Pull Request 9320
Interactive modification of invocation report. Pull Request 8691
User-facing Workflow Invocation Grid Pull Request 8699
PDF Export of Galaxy Markdown Pull Request 8893
Refactor some workflow editor plumbing to VueJS Pull Request 8931
Track non-history content outputs. Pull Request 9009
Default format 2 workflows to be downloaded as YAML. Pull Request 9023
Even Richer Workflow Inputs Pull Request 9086
Separate Markdown Validation into its own module. Pull Request 9099
add option select_type_workflow_threshold (rebase) Pull Request 9105
Upgrade eslint (major version), fix compliance with new standard rules. Pull Request 8708
JobMetrics.vue unit test. Pull Request 8732
Fix the flexible Data Dialog to work independent of the page layout initialization hooks. Pull Request 8744
axios-mock-adapter for cleaner, more expansive testing Pull Request 8749
Added client-side application configs for each target environment Pull Request 8827
Small cleanup for job runners kill (thanks to @nsoranzo). Pull Request 8523
Improvements to Galaxy and Pulsar job execution Pull Request 8625
Allow assigning UUID job ids with Pulsar in MQ mode. Pull Request 8718
Do not re-compute job input_ext/dbkey in job finish. Pull Request 8910
Implement extended job metadata collection Pull Request 8930
Move Jobs updated time display logic to client. Pull Request 8943
Implement extended job metadata collection + fixes Pull Request 8984
Place version output file in a writable directory for containerized jobs. Pull Request 8997
Fix deletion of outputs and working before job Pull Request 9245
Mount tool_data_path (ro) when running containerized tools Pull Request 9314
Allow remote jobs to configure Interactive Tool entry points. Pull Request 9349
Add Tool Manager Component for Admin Panel Pull Request 8601
Add process ID to worker_process table (thanks to @selten). Pull Request 8972
Use official “login with” buttons for OIDC (thanks to @hexylena). Pull Request 8107, Pull Request 9102
Add data types to docs.galaxyproject.org (thanks to @selten). Pull Request 8531
python-ldap python3 update and documentation change (thanks to @selten). Pull Request 8618
GitLab error reporting (thanks to @selten). Pull Request 8620
Extend the list of attributes to be reported. Pull Request 8661
Have CI ensure client PRs are formatted. Pull Request 8662
Add area/configuration label Pull Request 8665
Update BootstrapVue to 2.0.1 Pull Request 8666
add .gitattributes file and normalize line endings to use LF in… Pull Request 8677
Improve CI reliability by annotating external resources. Pull Request 8678
Restrict /api/histories/shared_with_me to logged-in users (thanks to @nsoranzo). Pull Request 8685
Automate path resolution for config options Pull Request 8689
Remove unused Docker Swarm options (thanks to @nsoranzo). Pull Request 8690
LDAP search member of filter (thanks to @BioinformaticsPlatformWIV-ISP). Pull Request 8697
Rebuild and prettify config sample and doc (thanks to @nsoranzo). Pull Request 8713
Update zh/locale.js (thanks to @shenweiyan). Pull Request 8725
Start using path_resolves_to; add integration tests Pull Request 8737
Significant overhaul to container+dependency management Pull Request 8741
Multi subdomains cookie support (thanks to @julozi). Pull Request 8746
One test case to rule them all Pull Request 8793
Allow recording more performance statistics to statsd. Pull Request 8800
Add installation monitor popup to toolshed client Pull Request 8810
General server dependency updates (was/including catalina fix) Pull Request 8843
Add badge to indicate availability of a newer toolshed repository version Pull Request 8860
Add config options to integration test for abs path Pull Request 8862
Make message_box style and visible options reloadable Pull Request 8863
Remove ‘area/cleanup’, edit ‘kind/refactoring’ Pull Request 8874
Unpin bdbag. Pull Request 8883
Add separate knob for masthead colors Pull Request 8904
Refactor galaxy_install and required dependencies into Galaxy’s namespace. Pull Request 8905
Enable interactivetools with other beta tools… Pull Request 8913
Simplify logo replacement, fix height and alignment Pull Request 8914
Simplify Python deps installation in testing-base Dockerfile (thanks to @nsoranzo). Pull Request 8921
Config: drop redundant template_cache attribute (thanks to @nsoranzo). Pull Request 8924
Enable secondary authorization Pull Request 8926
Set up a cache dir for pip when running tests with Docker (thanks to @nsoranzo). Pull Request 8929
Add link for interactive tools (thanks to @anuprulez). Pull Request 8936
Remove deprecated tool dependency scripts. Pull Request 8955
Remove tool shed API examples. Pull Request 8956
Remove script for migrating tools to the tool shed… Pull Request 8957
Move galaxy testing utilities into cleaner package structure. Pull Request 8960
Create packages for Galaxy tests and infrastructure. Pull Request 8962
Remove workflow support from tool shed. Pull Request 8978
Remove capsule feature from tool shed. Pull Request 8979
trim long ldda messages; improve fluid media css Pull Request 9000
Expose python version in api/config (thanks to @hexylena). Pull Request 9010
Hot module reloading using webpack-dev-server for client dev Pull Request 9029
when building ts index log parse errors at debug level Pull Request 9033
Update quickstart.rst (thanks to @galaxyproject). Pull Request 9044
Add Sergey Golitsynskiy to committers group Pull Request 9056
Update Mercurial to Py3-compatible v5.2, and all other deps (thanks to @nsoranzo). Pull Request 9065
Automatically redirect to OIDC provider when only auth option (thanks to @machristie). Pull Request 9115
Automatically use extended-base-image if necessary Pull Request 9116
Add HMR to client readme. Pull Request 9121
Add admin functionality to histories API (thanks to @thomcuddihy). Pull Request 9124
Don’t break generated reStructuredText and YAML on hyphens (thanks to @nsoranzo). Pull Request 9128
mulled: use threading.Event in PrintProgress to avoid unnecessary wai… Pull Request 9163
Set Conda environment variables when running biocontainers Pull Request 9170
Update my username (thanks to @hexylena). Pull Request 9195
Add base_image column to mulled-build-files Pull Request 9199
Allow passing conda_context to any_target_requires_extended_base Pull Request 9202
Backport 9175: Update galaxy.tool_util.cwl for latest changes in CWL branch. Pull Request 9210
Extended base annotation fixes Pull Request 9224
External auth login refinement Pull Request 9233
Fixes for IGV GEDA handling of genomes Pull Request 9413
Factor out yaml read/write from config_manage.py Pull Request 8509
Implementation of the workflow list as vue component Pull Request 8551
Refactor config loading Pull Request 8573
Change reloadable config + misc cleanup Pull Request 8600
Case-insensitive match user-supplied email when logging in (thanks to @nsoranzo). Pull Request 8631
Modify schema defaults and their loading Pull Request 8639
Replace resolve_path() with os.path.join() Pull Request 8646
Simplify config code (minor edits) Pull Request 8648
Refactor/cleanup config loading (misc edits) Pull Request 8651
Remove unused Tags file since it was converted to a Vue component Pull Request 8722
Update slot syntax to v-slot across the board. Pull Request 8730
Remove deprecated Toolshed API endpoints Pull Request 8734
Client build process refinement Pull Request 8742
Add vue component and endpoint to reset repository metadata Pull Request 8754
Cleanup SQL query in
_calculate_or_set_disk_usage()
(thanks to @nsoranzo). Pull Request 8776Drop DetachedInstanceError workaround Pull Request 8785
Simplify code for expand_meta_parameters (thanks to @qiagu). Pull Request 8794
Convert User Preferences page to Vue Component Pull Request 8803
client-side current user store module Pull Request 8813
Refactor database_connection config loading Pull Request 8855
Refactor default_locale config option; test Pull Request 8856
Refactor infrastructure_url/port values; fix tests Pull Request 8857
Refactor pretty-datatime_format config loading Pull Request 8858
Fix misc. integration tests for config values Pull Request 8859
Do not specify None as default in dict.get() Pull Request 8865
Delete ‘resolve path’ integration tests Pull Request 8870
Do not raise Exception to manage control flow Pull Request 8876
Eliminate interactive debugging middleware Pull Request 8877
Normalize myexperiment_target_url / myexperiment_url Pull Request 8882
Do not override schema defaults if they exist Pull Request 8884
Use schema defaults in galaxy masthead Pull Request 8885
Cleanup config loading (minor fixes) Pull Request 8890
Refactor _set_config_base; fix config_dir value Pull Request 8915
Library tags tweaks and tests Pull Request 8947
Refactor Toolshed webapp into tool_shed package. Pull Request 8949
Separate Shed and Galaxy makos, package shed makos. Pull Request 8958
Remove admin tool shed controller code (Part 1) Pull Request 8988
Remove unused helper code from twilltestcase. Pull Request 8989
Fix tool image loading (was broken due to failing repo dir resolution) Pull Request 9006
Convert Custom builds vue Pull Request 9016
Implement datasets grid as vue component Pull Request 9018
remove unused toolshed template code Pull Request 9084
Default the authnz_manager to None Pull Request 9117
Update github username @erasche to @hexylena and correct inconsistencies (thanks to @hexylena). Pull Request 9194
Add requirements, fix legacy tools and move them out of GALAXY_LIB_TOOLS_UNVERSIONED Pull Request 9336
Add requirements to (almost) all converters, profile and minor fixes Pull Request 9345
Fixes
Wrap SQL statement in a text() function as required by sqlalchemy (thanks to @kaukrise). Pull Request 9216
Ensure visible overflow on navbar container (thanks to @kaukrise). Pull Request 9217
Various python 3 fixes from sentry Pull Request 9337
Fix cool sniffing on python3 Pull Request 9153
Fix SearchGuiArchive.set_meta() (thanks to @bernt-matthias). Pull Request 9437
grouping: strip newlines (thanks to @bernt-matthias). Pull Request 8953
Add requirements to interval_to_bed_converter.xml Pull Request 9041
Fix version in which
sam_pileup
was fixed (thanks to @nsoranzo). Pull Request 9350Expunge tool shed repository objects, prevents ParentInstanceDetachedError Pull Request 9354
Fix maf tools Pull Request 9393
Exports reports with workflows (if configured). Pull Request 9055
Rev gxformat2 dependency to fix various export issues. Pull Request 9075
Fix workflow editor save as functionality Pull Request 9315
Various fixes and enhancements to reports Pull Request 9316
Small fixes for k8s. Pull Request 8759
Fix output.file_path writing to read-only path on Docker Pull Request 9048
Add job home to writable paths Pull Request 9051
Fix LocalShell returncode Pull Request 9171
Fix: Added support for specifying k8s job pod priority class (thanks to @nuwang). Pull Request 9219
Fix: Added support for specifying k8s job affinity and tolerations (thanks to @nuwang). Pull Request 9225
Mount job_working_directory/configs rw Pull Request 9234
Limit async hack to async tools Pull Request 9242
Fix moving of data manager output for entries Pull Request 9243
python3 fixes for setups using drmaa and realuser setups (thanks to @bernt-matthias). Pull Request 9246, Pull Request 9253
Email notifications aren’t used from just workflow steps anymore. Pull Request 9276
Use requests for slots instead of limit (thanks to @nuwang). Pull Request 9322
Make set metadata exec after process more robust Pull Request 9395
Make dataset discovery respect final_job_state Pull Request 9406
Strip unicode null characters from traceback and job_messages Pull Request 9430
Encode lines in filter_file_fields (thanks to @selten). Pull Request 8652
Git error reporting - bug fixes and better error handling (thanks to @selten). Pull Request 8900
Assorted admin panel fixes Pull Request 9097
Run integration tests on github, fix various bugs (interactive tools, kubernetes, test framework) Pull Request 9193
Always add shed_tool_config file to tool_configs Pull Request 9364
Documentation update (thanks to @selten). Pull Request 8508
Asynchronous data source: unable to create the file (thanks to @FredericBGA). Pull Request 8515
OIDC config patch (thanks to @almahmoud). Pull Request 8572
set uwsgi umask default (thanks to @bernt-matthias). Pull Request 8581
Fix test_validate_simple_config unit test (thanks to @nsoranzo). Pull Request 8632
Fix a bug that occurs while installing a data manager. (thanks to @selten). Pull Request 8634
Fix bug in build_dependency_manager() Pull Request 8660
Get rid of fix timeout in admin app Selenium test. Pull Request 8667
More robust login/logout with Selenium. Pull Request 8676
Fixes for BootstrapVue Tables Pull Request 8684
More bootstrap vue upgrade fixes. Pull Request 8692
GRT: Ensure correct encoding, no partial lines (thanks to @hexylena). Pull Request 8712
Revise toolshed config dependency check error message in repository install manager Pull Request 8720
Fix data dialog height, modal test case and table slots Pull Request 8724
Do not overwrite existing attributes Pull Request 8728
Replace hardcoded configuration Pull Request 8731
Fix new_secure_hash() when passed an empty string (thanks to @nsoranzo). Pull Request 8739
data_manager_conf.xml.sample is missing (thanks to @lecorguille). Pull Request 8784
Revert “[19.09] Fix detached parent instance error in 8769” (thanks to @galaxyproject). Pull Request 8791
Handles k8s job recently created with no status set yet (thanks to @pcm32). Pull Request 8799
Add postcss-loader and autoprefixer dependencies Pull Request 8821
LDAP fixes (thanks to @nsoranzo). Pull Request 8823
Ensure app.config.config_file is an absolute path Pull Request 8836
Fix rare bug in config loading + add unit test Pull Request 8837
fix plugin name (thanks to @gmauro). Pull Request 8840
Restore portlet section style for repeat blocks Pull Request 8864
Prevent potential bug in config loading; test Pull Request 8866
Fix authnz manager’s oidc config parsing for python3 Pull Request 8871
Update check_py3_compatibility script for new ack versions. Pull Request 8872
If config_file is None, use cwd, not its parent Pull Request 8894
Log LDAP authenticate exceptions at the right level (thanks to @nsoranzo). Pull Request 8907
make pgcleanup also delete HDAs (thanks to @bernt-matthias). Pull Request 8908
Fix unbounded variable error in GET /api/datasets/{encoded_dataset_id} exception handling Pull Request 8935
Set deleted attribute of history datasets when purging histories from grid (thanks to @nsoranzo). Pull Request 8940
Fix default logo path in configuration manager Pull Request 8941
Fix API decorator order (thanks to @nsoranzo). Pull Request 9015
do not use var from inner scope Pull Request 9026
escape double hyphen in IT sample config, making it valid XML Pull Request 9034
Fix input parameter comparisons Pull Request 9047
Restore casting a boolean param to str when comparing with one (thanks to @nsoranzo). Pull Request 9052
Fix comparison if parameter is optional and value is not provided Pull Request 9054
Small refactor of
InputValueWrapper
comparison methods (thanks to @nsoranzo). Pull Request 9059Add create_time to the signature of Repository constructor. Pull Request 9063
Listify value for multiple SelectToolParameterWrapper (thanks to @nsoranzo). Pull Request 9067
Use newer psycopg2 conditional requirement for wheels on 3.7 Pull Request 9068
Fix mulled version comparison Pull Request 9078
linter: accept outputs with auto_format only (thanks to @bernt-matthias). Pull Request 9081
Create temporary files for cgi.FieldsStorage with delete=False Pull Request 9082
Fix importing a directory from user_library_import_dir (thanks to @nsoranzo). Pull Request 9090
Make callbacks for tour and config watchers a control task Pull Request 9092
Don’t watch tool files through watchdog Pull Request 9094
Allow html in registration warning message. Pull Request 9098
Admin job info display fix Pull Request 9120
Client build readme followup Pull Request 9122
Fix mulled cache/data locations, rebuild schema Pull Request 9125
More fixes for publishing decomposed packages. Pull Request 9126
fix url for workflow sharing Pull Request 9132
Add click prevent to workflow list dropdowns Pull Request 9133
Remove legacy admin panel entry for installation monitor, fix typo Pull Request 9134
Parse brand label option in legacy mako Pull Request 9136
Restore returning “None” when casting unselected multiselect param to str (thanks to @nsoranzo). Pull Request 9144
Misc fixes (thanks to @nsoranzo). Pull Request 9145
Add implementation of parse_interactivetool to CwlToolSource class (thanks to @nsoranzo). Pull Request 9146
Fix optional unset integer and float params Pull Request 9149
Assorted OIDC fixes Pull Request 9150
Cast to None for optional params if not comparing strings Pull Request 9156
Ensure that users have a private role before login Pull Request 9168
Add static-safe mapping to config management Pull Request 9179
Fix tool shed tests for #9168. Pull Request 9182
Add require_login bypass for entire authnz controller. Pull Request 9192
Fix runtime error in ToolConfWatcher Pull Request 9212
Fix extended base for singularity and add base_image_for_targets Pull Request 9215
More mulled-build-files fixes Pull Request 9228
Fix mailto hrefs in menus Pull Request 9232
Move meme_fimo out of unversioned lib tools Pull Request 9237
Fallback to binary download if header lies about content encoding Pull Request 9239
Fix lint warning string formatting Pull Request 9240
Fix unselected select parameter comparison to empty string Pull Request 9260
Fix iteritems use in build_from_current_history.mako Pull Request 9266
Fix suffix for single target mulled builds Pull Request 9267
backport 9268: Fix target_str_to_targets Pull Request 9269
Backwards compatible build number parsing Pull Request 9278
Tag display fixes for ui-select. Pull Request 9281
Python3: don’t check for has_key attribute (thanks to @nsoranzo). Pull Request 9290
Prevent caching of bootstrapped client data Pull Request 9294
Add build_sites_config_file option to config_schema Pull Request 9297
Backport reports config/py3 fix Pull Request 9309
Python3: unicodify the output of
subprocess.check_output()
(thanks to @nsoranzo). Pull Request 9319Fix reports disk usage query for python 3 Pull Request 9321
Fix string module use Pull Request 9327
Fix Python 3 error in containers lib Pull Request 9329
Fix use of next() as a meethod instead of a function Pull Request 9330
Automatically resolve and add static-safe mappings for client-based images Pull Request 9331
Various doc fixes (thanks to @nsoranzo). Pull Request 9334
Fix app reference in guess_shed_config Pull Request 9339
Make linters more robust on python 3 Pull Request 9348
Use RenamedTemporaryFile for safer integrated_tool_panel_file writing Pull Request 9359
Only get non-deleted datasets when listing custom dbkeys Pull Request 9360
Prevent duplicate listing of doi link and link (text, with href doi link) Pull Request 9366
Styleguide fixes Pull Request 9367
Prevent tool execution getting stuck in history when a chromInfo data… Pull Request 9385
Move migrated_tools_conf > mutable_config_dir (#9376) Pull Request 9388
Fix dependency for CONVERTER_tar_to_directory Pull Request 9389
Fix for qp method used in GEDAs. Pull Request 9391
Fix hid comparison failing with hdas without hid Pull Request 9398
Don’t raise exception if anon goes to <galaxy_url>/user/resend_verification Pull Request 9404
Update bleach to 3.1.1 (thanks to @nsoranzo). Pull Request 9408
Fix shed image resolution Pull Request 9411
Fixes for get_estimated_display_viewport Pull Request 9415
Fix wrong map use in many places Pull Request 9416
Conda startup fixes (thanks to @nsoranzo). Pull Request 9421
fix two cases of opening a file binary (thanks to @bernt-matthias). Pull Request 9424
Remove ‘.sample’ from default value; rebuild schema Pull Request 9427
Fix login to accounts using different email capitalization Pull Request 9432
Fix TwoBitFile handle not opened in binary mode Pull Request 9433
Fix uploading workflow archives Pull Request 9434
Fix to_dict for upload1 tool Pull Request 9441
Fix data tables vue display Pull Request 9442
Fix UserManager.get_reset_token() after #9432 (thanks to @nsoranzo). Pull Request 9446
To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.
You can always chat with us on Matrix.
Thanks for using Galaxy!