January 2019 Galaxy Release (v 19.01)

Highlights

Revised UI Style

The Galaxy stylesheet has been substantially reworked resulting in the largest visual refresh to Galaxy in years.

Pull Request 6422, Pull Request 6790, Pull Request 6868, Pull Request 6870, Pull Request 6881, Pull Request 6905, Pull Request 6910, Pull Request 6967, Pull Request 6985, Pull Request 7017

More Colorful Tags

Use consistently hashed randomized colors for history dataset name tags to increase visual distinction between tags, making it easier to track data flow in complex histories. Thanks to @hexylena. Pull Request 7072, Pull Request 7081

Extensive Workflow Enhancements (continued)

Galaxy 18.09 featured a bunch of important and diverse fixes and enhancements to workflows on both the frontend and backend. That progress continues in release 19.01.

The workflow editor now features explicit step parameter nodes for non-file data (integers, strings, booleans, etc) and the ability to connect these to all tool inputs - not just files (Pull Request 6925, Pull Request 7150). The YAML-based Galaxy Format 2 workflow language is now included in Galaxy as a beta option and allows import and export of human readable workflows (Pull Request 6776, Pull Request 6807, Pull Request 6811, Pull Request 6746) as well as extensions to allow Galaxy to serve as a file editor for such workflows (Pull Request 7019). The workflow editor features more helpful labels on inputs and outputs (Pull Request 6770) and important fixes including logic for connecting collections (Pull Request 6732) and dealing with missing tools (Pull Request 7227, Pull Request 7258). The workflow run form now deals with default values more correctly (Pull Request 7157).

Enhanced Support for Singularity

Singularity container support in Galaxy has been brought to parity with Docker container support. Specifically, Galaxy can now be configured to fetch Biocontainers automatically for enabled job destinations. Moreover, explicitly annotated Docker containers (either in tools or job destinations) can be used with Singularity. Thanks to @mvdbeek. Pull Request 7125

Get Galaxy

The code lives at GitHub and you should have Git to obtain it.

To get a new Galaxy repository run:

$ git clone -b release_19.01 https://github.com/galaxyproject/galaxy.git

To update an existing Galaxy repository run:

$ git fetch origin && git checkout release_19.01 && git pull --ff-only origin release_19.01

See the community hub for additional details regarding the source code locations.

Security

Permissions Not Checked to Modify Dataset Metadata

Jelle Scholtalbers reported an issue where permissions were not being checked when modifying dataset metadata. This has been resolved with Pull Request 7322 and the fix backported through 18.01.

Release Notes

Enhancements

• Adds Aequatus visualisation plugin (thanks to @anilthanki). Pull Request 6484

• Integrate framework code for interfacing with cloud authorization providers. Pull Request 5903

• Added support for chaining dynamic destinations (thanks to @nuwang). Pull Request 7006

• Upgrade web client building to use webpack 4 Pull Request 6917

• Vue.js rewrite of admin data manager user interface (thanks to @scrathat). Pull Request 6729

• Implement ‘make all datasets private’ functionality. Pull Request 6952

• Add datatype sniff button to edit dataset attributes page (thanks to @qiagu). Pull Request 7018

• Overhaul handler assignment methods. Pull Request 7129, Pull Request 7091

• Add tool for sending dataset to the cloud. Pull Request 6583

• Infrastructure/API Support for composite uploads/downloads. Pull Request 6844

• Improvements to tool testing including allowing remote tests and tests without admin API keys. (thanks to @mvdbeek). Pull Request 7160, Pull Request 7167, Pull Request 7199

• Add ISA-TAB datatype (thanks to @phnmnl). Pull Request 5787

• Adding itp and mdp datatypes (for gromacs) (thanks to @simonbray). Pull Request 6812

• Add pqr datatype (thanks to @jj-umn). Pull Request 6816

• Add nmrML datatype (thanks to @phnmnl). Pull Request 6903

• Add RMA6 datatype (MEGAN6 read-match archive) (thanks to @bernt-matthias). Pull Request 6962

• Add fastg datatype (thanks to @jj-umn). Pull Request 6987

• Add datatype for Mash Sketch files. (thanks to @dfornika). Pull Request 7087

• Add JavaScript client library utility function to create history entries. Pull Request 6693

• Give all model objects with an id a __repr__ that includes the id Pull Request 6757

• Add styled loader vue component (thanks to @inkuzmin). Pull Request 7052

• Restore the ability to track jobs for a WF invocation step. Pull Request 6559

• Improvements to the grouping tool (thanks to @bernt-matthias). Pull Request 6577

• Trim whitespace from data manager conf paths (thanks to @pvanheus). Pull Request 6682

• Track workflow step input definitions in our model. Pull Request 6850, Pull Request 7103

• Deprecate size attribute of <param/> and remove it from tools (thanks to @nsoranzo). Pull Request 6684

• Create container_resolvers_conf.xml.sample (thanks to @pcm32). Pull Request 6700

• Update to latest galaxy-lib. Pull Request 6709

• Updates to OIDC backends default configuration. Pull Request 6727

• Small local testing improvements (thanks to @nsoranzo). Pull Request 6738

• Add backrefs to mapped objects pointing at Dataset. Pull Request 6754

• Fixes for the upload row template. Pull Request 6758

• Disable file name text field after upload. Pull Request 6762

• Remove empty log messages. Pull Request 6763

• Release issue scaffold tweaks. Pull Request 6778

• Warn job runner plugin writers not to emulate condor.py. Pull Request 6783

• Add PB and EB to util.size_to_bytes Pull Request 6786

• Add a pgcleanup.py action to purge HDAs of purged histories. Pull Request 6801

• Various pgcleanup.py enhancements. Pull Request 6736, Pull Request 6877

• Fix typos on “reserve_memory” / “reserve_cpu” (thanks to @pvanheus). Pull Request 6817

• Add more detail about container tags (thanks to @pvanheus). Pull Request 6818

• Add docs about running jobs in containers (thanks to @pvanheus). Pull Request 6820

• Remove trans from OIDC-related types. Pull Request 6826

• Clarify code used in output <filter> tags (thanks to @pvanheus). Pull Request 6827

• Convert user_is_admin and user_can_do_run_as to properties. Pull Request 6835

• Don’t create jobs for inactive users. Pull Request 6837

• Replace deprecated assertEquals() method (thanks to @nsoranzo). Pull Request 6841

• Improvements to GIE docs (thanks to @thobalose). Pull Request 6846

• Indicate language in client docs to enable syntax highlighting.. Pull Request 6848

• Fix panel header margins in history and tool panel Pull Request 6854

• Small fixes (thanks to @nsoranzo). Pull Request 6861

• Datatype upload integration test and fixes (thanks to @mvdbeek). Pull Request 6869

• Use small button for editable text fields in history annotation Pull Request 6878

• Improve Kubernetes job error handling (thanks to @ilveroluca). Pull Request 6886

• Fixes and updates to the Neo4j GIE (thanks to @thobalose). Pull Request 6897

• Allow remove_value filter on data tables (thanks to @nsoranzo). Pull Request 6914

• Update all dependencies (thanks to @nsoranzo). Pull Request 6921

• Be more precise with the job runner worker thread join timeout and improve documentation. Pull Request 6924

• Selenium test enhancements and fixes. Pull Request 6927, Pull Request 6982

• Small testing tweaks from the CWL branch. Pull Request 6929

• Tests need to count data sets and collections (thanks to @bernt-matthias). Pull Request 6931

• Eliminate some mutable defaults in populators.py. Pull Request 6933

• Default shed tool installation to database/shed_tools. Pull Request 6937

• Add option to strip url prefix from pair names to collection builder (thanks to @shiltemann). Pull Request 6943

• Small API workflow test improvements. Pull Request 6954

• Switch from PhantomJS to Chrome Headless for unit tests. Pull Request 6959

• Added a metadata element to the biom1 datatype (thanks to @Slugger70). Pull Request 6972

• Update locale.js (thanks to @istevenshen). Pull Request 6974

• Use virtualenv when conda is on PATH (thanks to @mvdbeek). Pull Request 6975

• Improvements to exit code and regex parsing (thanks to @bernt-matthias). Pull Request 6977

• Optimize images with optipng (thanks to @bgruening). Pull Request 6980

• Fix tool XSD to list tags as attribute of TestCollection (thanks to @mvdbeek). Pull Request 6983

• Use detect_errors="exit_code" in __SET_METADATA__ tool (thanks to @nsoranzo). Pull Request 6989

• Don’t mount external /tmp for tests. Pull Request 6990

• Upgrade nodeenv’s nodejs to 10.13.0 LTS. Pull Request 6991

• Fix broken links on Job Configuration page. Pull Request 6992

• Allow rules module override and refactor submodule imports (thanks to @nuwang). Pull Request 6993

• Fix JSON loading permission problem with run-as-real-user DRMAA job runner (thanks to @bernt-matthias). Pull Request 6995

• CLI plugin for LSF (thanks to @pcm32), Pull Request 7007 Pull Request 6788

• A new UNIVA extension to the DRMAA job runner (thanks to @bernt-matthias). Pull Request 7004, Pull Request 7161

• DRMAA fix to soft catch of external runner fails (thanks to @bernt-matthias). Pull Request 6997

• Replace global Galaxy references with galaxy singleton accessor/setter. Pull Request 7005

• Workflow test case - test marking inputs as outputs. Pull Request 7016

• Expose published workflows through API (thanks to @frederikcoppens). Pull Request 7020

• Pass a JobWrapper instead of a Job to stop_job() methods (thanks to @nsoranzo). Pull Request 7023

• Fix element_identifier access (thanks to @mvdbeek). Pull Request 7031

• Avoid replacing data-referenced input values with default values Pull Request 7043

• Update error parsing docs (thanks to @bernt-matthias). Pull Request 7049

• Add resubmit handlers on Pulsar connection failures (thanks to @nuwang). Pull Request 7053

• Add new “Help” item to the help menu. Pull Request 7054

• Eliminate dependency on make for automatic client build. Pull Request 7060

• Remove legacy job handling hacks. Pull Request 7082

• De-couple objectstore abstraction from XML. Pull Request 7085

• Localize version strings to the working directory w/predictable path. Pull Request 7086

• Replace display applications mako with vue component Pull Request 7097

• Run singularity in silent mode (thanks to @mvdbeek). Pull Request 7104

• Eliminate unused code block in output_collect. Pull Request 7109

• Add to_dict() to ObjectStores. Pull Request 7110

• Enable tagging and annotation of jobs regardless of state. Pull Request 7115

• Remove unused __collected_datasets__ from tool hook dict. Pull Request 7123

• Add a Vue.js component to display tool errors in the admin panel. Pull Request 7124

• Update help text in filter_from_file tool. (thanks to @pvanheus). Pull Request 7127

• Remove unneeded session flushes and adds when discovering outputs. Pull Request 7134

• Drop tools without installable dependencies (thanks to @mvdbeek). Pull Request 7136

• Small de-duplication cleanups to galaxy.jobs Pull Request 7155

• Improve tool form client to add error listener for input elements. Pull Request 7183, Pull Request 7229

• Stop tool migrations checking by default (was broken in some cases). Pull Request 7275

• Replace trans with session and request in PSA Storage. Pull Request 6838

• Share ssh key generation across tests (thanks to @mvdbeek). Pull Request 6865

• Specify POST and GET actions for cloudauthz API actions. Pull Request 6876

• Prevent sanitized history names being saved in database. Pull Request 6898

• Rework language in cloud data fetch tool from “Upload” to “Get” Pull Request 6899

• Refactor collection mapping workflows toward independence from tools. Pull Request 6911

• Normalized jQuery bundle. Pull Request 6970

• Replaced jStorage with store. Pull Request 6994

• Remove sudo: false from .travis.yml, it will stop being supported soon Pull Request 7035

• Fixes state-change animations causing ‘block’ display. Pull Request 7122

Fixes

• Fixes to allow config schema to validate (thanks to @hexylena). Pull Request 5919

• Patch update.sh to address dependency update issues on a macOS. Pull Request 6694

• Fix wrong version being returned if requested version doesn’t exist (thanks to @mvdbeek). Pull Request 6702

• Small mixed fixes (thanks to @nsoranzo). Pull Request 6722

• Prevent errors when a library dataset has been purged from disk (thanks to @dpryan79). Pull Request 6784

• Fix apply rules bug in filter first/last N rows. (thanks to @shiltemann). Pull Request 6814

• Updated volume kwds. (thanks to @thobalose). Pull Request 6840

• Remove some model objects added in #5013 but unused before the PR was merged, and fix a model object typo Pull Request 6843

• Revert selective metadata flushing - breaks multiple uploads/outputs. Pull Request 6858

• prevent double sanitizing history name when renaming (thanks to @almahmoud). Pull Request 6871

• Fix rule editor element alignment. Pull Request 6874

• Fix styleguide; change permissions on jstree.scss. Pull Request 6879

• Fix username filtering for published workflows (thanks to @almahmoud). Pull Request 6882

• Align toolshed makos used in the admin panel, fix slider height. Pull Request 6883

• Improve resilience on handling sharing attributes. Pull Request 6884

• Update Bug Reports docs to use YAML syntax (thanks to @nsoranzo). Pull Request 6909

• Fix expr: non-integer argument error when running make (thanks to @nsoranzo). Pull Request 6913

• Do not log exception info twice (thanks to @nsoranzo). Pull Request 6916

• Fix typos in scripts/bootstrap_history.py (thanks to @nsoranzo). Pull Request 6941

• Make BAM index only available for coordinate sorted BAM files (thanks to @mvdbeek). Pull Request 6946

• Use recently introduced overflow utility instead of overwriting card defaults Pull Request 6948

• Fix sniffing of readname sorted BAM files (thanks to @mvdbeek). Pull Request 6950

• Fix for workflow parameter collection in workflows view. Pull Request 6955

• Topic/empty data table (thanks to @bernt-matthias). Pull Request 6957

• Allow deferring ‘to_ext’ extension to the server Pull Request 6960

• Allow upload for rdata.sce format (thanks to @nsoranzo). Pull Request 6963

• Fix traceback when starting under Python3 with galaxy.ini (thanks to @nsoranzo). Pull Request 6978

• Fix charts entrypoint for compatibility with newer chunk changes. Pull Request 6984

• reclaim ownership for failed jobs (thanks to @bernt-matthias). Pull Request 6996

• Change join two dataset output format to tabular (thanks to @qiagu). Pull Request 6998

• Fixes for Python 3.7 (thanks to @nsoranzo). Pull Request 7008

• Fix PID file handling for gunicorn (thanks to @nsoranzo). Pull Request 7013

• Fix display_in_tool_panel status for shared workflows (thanks to @mvdbeek). Pull Request 7014

• Use solid background for text field autocomplete options Pull Request 7027

• Follow-up on #6850 (thanks to @nsoranzo). Pull Request 7029

• Fix error when building images (thanks to @scholtalbers). Pull Request 7036

• Fix involucro_auto_init default value (thanks to @nsoranzo). Pull Request 7038

• Fix resubmission on tool error detection. Pull Request 7040

• GRT: Rip out all parameters, round file sizes (thanks to @hexylena). Pull Request 7041

• Pass through --daemon option to gunicorn (thanks to @nsoranzo). Pull Request 7042

• Escape dataset names in job submission success message Pull Request 7062

• Update galaxy.yml.sample and galaxy_options.rst (thanks to @nsoranzo). Pull Request 7064

• Fix link to requirements.txt. Pull Request 7065

• Improved parsing of sizes (bytes) (thanks to @bernt-matthias). Pull Request 7067

• Downgrade bdbag - 1.5.0 requires python:>=2.7.9 Pull Request 7070, Pull Request 7079

• Fix process key (thanks to @mvdbeek). Pull Request 7071

• Fix welcome on redirect from facebook auth Pull Request 7074

• Fix set_peek method for copied datasets (thanks to @mvdbeek). Pull Request 7078

• Fix toolshed repository error display (thanks to @jvanbraekel). Pull Request 7080

• Fix via refactor set_metadata so more output metadata pruned properly. Pull Request 7084

• Replace credentials with authz_id Pull Request 7088

• TravisCI configuration workaround for problems with boto configuration. Pull Request 7092

• Fix size_to_bytes for some float values (thanks to @nsoranzo). Pull Request 7096

• Client readme updates. Pull Request 7102

• Don’t write outputs to object store twice. Pull Request 7111

• Fix make client-watch extra error message. Pull Request 7113

• Properly escape history names in scratchbook titles. Pull Request 7114

• Fix activation of _galaxy_ conda env (thanks to @nsoranzo). Pull Request 7116

• Increase timeout on flakey unit test. Pull Request 7120

• Fixes and tests for collecting extra files for discovered datasets. Pull Request 7131

• Address CORS issue in toolshed iframe handlers. Pull Request 7135

• Use regular js-event handling instead of jquery for uploader Pull Request 7145

• Fixes toolshed and reports styling that was previously applied at the… Pull Request 7146

• Correct code error in ‘build job runner’ tutorial. Pull Request 7148

• Added store library to exported libs in bundleEntries. Pull Request 7149

• Fixes for library support of YAML/dictified tools. Pull Request 7152, Pull Request 7153

• Fix for authnz managers (thanks to @machristie). Pull Request 7166

• Misc fixes (thanks to @nsoranzo). Pull Request 7168

• Fix process_key test and deal with unordered repeat elements (thanks to @mvdbeek). Pull Request 7169

• Fix test_test_data_download (thanks to @mvdbeek). Pull Request 7171

• Document message_box_class option (thanks to @nsoranzo). Pull Request 7174

• Prevent XSS on toolshed when browsing repo changeset Pull Request 7180

• Fix HDA accessibility check in Page’s renderForSave name fetching Pull Request 7181

• Fix for workflow parameter test case. Pull Request 7188

• Fixes for library dataset permission API. Pull Request 7190

• Properly document handler assignment method configuration and fix logging configuration docs. Pull Request 7191

• Re-enable script staging. Pull Request 7196

• Fix error in view_tool_metadata (thanks to @mvdbeek). Pull Request 7201

• Data library: fix tag_using_filenames not properly converted to bool (thanks to @abretaud). Pull Request 7204

• Fixes to event handling re-enabling multi-history panel drag and drop functionality. Pull Request 7218

• Fix displaying tool help images on python 3 (thanks to @mvdbeek). Pull Request 7228

• Fix data_source tools on python 3 (thanks to @mvdbeek). Pull Request 7236

• Interval datatype fixes (thanks to @mvdbeek). Pull Request 7237

• strip() text when parsing xml (thanks to @mvdbeek). Pull Request 7252

• Fix install operation in installed_repository_grid (thanks to @mvdbeek). Pull Request 7253

• Update sqlalchemy-migrate requirement to 0.12.0 (thanks to @nsoranzo). Pull Request 7265

• Fix _.escape typo in toolshed group detail view. Pull Request 7267

• Fix installing tools via tool_id (thanks to @mvdbeek). Pull Request 7277

• Remove errant space in intermine_simple_display.loc.sample. Pull Request 7298

To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.

You can always chat with us on Matrix.

Thanks for using Galaxy!

The Galaxy Team