May 2018 Galaxy Release (v 18.05)

Highlights

New Interface Style Library

The design of many components of the Galaxy user interface has been subtly improved as the result of upgrading to the Bootstrap 4 style library. The upgrade process was a large community effort with commits from six contributors and testing and bug reports from many others. A huge thanks goes out to everyone who helped - it is a great step forward toward a more standardized Galaxy client infrastructure built on reactive VueJS components. Read more about whats new and changed in the underlying Bootstrap 4 JavaScript library here.

Pull Request 5807, Galaxy Project Board for Bootstrap 4.

Unlimited Browser Upload Size

Support for chunked and resumable file uploads out-of-the-box. This should effectively eliminate browser-based limitations on the size of files that can be uploaded to Galaxy.

Pull Request 5516

New Powerful Rule-Based Data Uploader

Galaxy now features a powerful new interface for manipulating lists of rules for processing metadata that can be used to describe the upload of arbitrarily complex collections as well as individual datasets. These rules can be used to build collections from lists of URLs, FTP directory contents, data library folders, and tabular sample sheets or exports from external data sources. The same technique can be applied to the metadata of an existing collection, enabling very flexible filtering, sorting, relabelling, grouping, flattening, and general re-organization of collections. This client widget is backed by a powerful new hierarchical upload API and a new collection operation tool.

Check out this tutorial describing new techniques for loading large numbers of datasets and complex collections into Galaxy.

Pull Request 5365, Pull Request 5220, Pull Request 5819, Pull Request 5823, Pull Request 5929, Pull Request 5940

GDPR Beta Mode

We introduce beta GDPR privacy mode. Admins can enable it to help them comply with the European Union law known as General Data Protection Regulation, which goes into effect on May 25, 2018. Thanks to @hexylena.

Pull Request 6069

New HIV-TRACE Plugin

Integration of a new visualization plugin - HIV-TRACE. Thanks to @veg!

Pull Request 5869

Visualization Frameworks Unified

The charts and generic visualization frameworks have been merged together into a single coherent one. You can use a new menu option to list and search all available visualizations.

Pull Request 5633, Pull Request 5505

Get Galaxy

The code lives at Github and you should have Git to obtain it.

To get a new Galaxy repository run:

$ git clone -b release_18.05 https://github.com/galaxyproject/galaxy.git

To update an existing Galaxy repository run:

$ git fetch origin && git checkout release_18.05 && git pull --ff-only origin release_18.05

See the community hub for additional details regarding the source code locations.

Security

vcffilter by devteam allowed RCE

Tracked as GX-2018-0004.

Older versions of the VCF filter tool in the vcffilter repository by devteam allowed remote code execution due to missing character escaping. If you have this repository installed in your Galaxy, please update to the revision 4:6b935ab36d7b or newer and disable/uninstall all older versions.

Breaking Changes

• The charts repository that powers Galaxy built-in visualizations that need backend computation has been moved to the IUC account. In order for these visualization options to keep working, admins need to install the following repository: iuc/charts.

• Removed support for the Graphite statistics middleware. We continue to further support the statsd middleware.

• We reworked how compressed archives (zip, gz, bz etc.) are being defined and handled in Galaxy. As a result, if you have a custom datatypes_conf.xml configuration file, you should remove the compressed sniffers that used to be in the sample file but now are not. Otherwise you will encounter non-breaking errors in the logs.

• We have removed the deprecated symlink static/scripts/packed -> . If you used it please change your (proxy) configuration. An example can be seen here.

Release Notes

Enhancements

• Allow user authentication using OpenID Connect protocol (thanks to @VJalili). Pull Request 4474, Pull Request 5908, Pull Request 5906

• Add basic cgroup metric collection easing the collection of job memory data (thanks to @scholtalbers). Pull Request 5610

• Enhance Galaxy’s API and test framework to allow running tool tests on running Galaxy instances. Check out the new Ephemeris functionality it is powering here. Pull Request 5628, Pull Request 5667

• Global and profile-bound language configuration (thanks to @abretaud). Pull Request 5282

• Tool that adds/sets tags for collection elements from a file (thanks to @mvdbeek). Pull Request 5462

• Implement an extension point during workflow running to describe site specific job scheduling parameters (thanks to @JeffreyThiessen). Pull Request 4830

• Update Dynamic Tool Destinations with configurable priorities and better validation (thanks to @tizzir). Pull Request 5645

• More great Kubernetes job runner enhancements to improve robustness and configurability (thanks to @pcm32 and the PhenoMeNal project). Pull Request 5760, Pull Request 5761

• Documentation version selector on docs.galaxyproject.org. Pull Request 5334

• Update sort tool to allow ignoring headers (thanks to @lecorguille). Pull Request 4235

• Add GALAXY_MEMORY_MB_PER_SLOT environment variable to job environment (thanks to @bernt-matthias). Pull Request 5625

• Allow sending slow request timing to Sentry. Pull Request 5479

• Allow formatting StatsD data for InfluxDB (thanks to @hexylena). Pull Request 5742

• Tool versions should follow PEP 440 (thanks to @nsoranzo). Pull Request 5746

• Expand pipenv dependency management to include all Python dependencies (thanks to @nsoranzo). Pull Request 5786

• Sniffing framework with constrained memory and I/O. Pull Request 5793

• Infrastructure for dynamic compressed datatypes. Pull Request 5794

• Capture more HDA state transitions in the history panel. Pull Request 5749

• Add script for propagating hashtags in a Galaxy history (thanks to @anuprulez). Pull Request 5669

• Add proteomics proBAM and proBED datatypes (thanks to @jj-umn). Pull Request 5360

• Add fasta.gz datatype. Pull Request 5553

• Add genbank.gz datatype (thanks to @Delphine-L). Pull Request 5772

• Add RDS object and SingleCellExperiment datatypes (thanks to @mtekman). Pull Request 5716

• Allow registration with empty email (thanks to @bernt-matthias). Pull Request 5320

• Avoid silent tool upgrades during workflow imports. Pull Request 5821

• Many awesome Python 3 related fixes (thanks to @nsoranzo and @mvdbeek). Pull Request 5359, Pull Request 5422, Pull Request 5453, Pull Request 5461, Pull Request 5492, Pull Request 5494, Pull Request 5458, Pull Request 5636,`Pull Request 5642`_, Pull Request 5427, Pull Request 5451

• Install Node.js and Yarn on startup if running a non-release branch. Pull Request 5375

• Add --skip_flakey_fails flag to CI script for API tests (thanks to @nsoranzo). Pull Request 5388

• Remove unused mapping code. Pull Request 5450

• Show parameter name in DataToolParameter exception (thanks to @peterjc). Pull Request 5465

• Unify config-rebuild Makefile targets (thanks to @nsoranzo). Pull Request 5488

• Rev SQLAlchemy from 1.0.15 to 1.2.2. Pull Request 5538

• Add middleware and instrumentation to allow per-request SQL debugging. Pull Request 5539

• Overhaul UCSC tools with more specific, working Conda compatible requirements (with help from @nsoranzo). Pull Request 5544, Pull Request 5973

• More and more robust Selenium tests. Pull Request 5559, Pull Request 5882, Pull Request 5886, Pull Request 5876, Pull Request 5836, Pull Request 5701, Pull Request 5771, Pull Request 5754, Pull Request 5714, Pull Request 5705, Pull Request 5708, Pull Request 5710, Pull Request 5706, Pull Request 5707

• Implement basic cache on toolbox to_dict objects. Pull Request 5570

• Log errors from external chown script. (thanks to @Petraea). Pull Request 5581

• Updates against galaxy-lib changes. Pull Request 5602, Pull Request 5621, Pull Request 5936, Pull Request 5653

• Performance optimization to avoid producing deep-copies of tool input dictionaries. Pull Request 5646

• Disable tool input datatype validation for dropped history items. Pull Request 5657

• Add a script for dumping library file paths for backup. Pull Request 5436

• Rebuild client only if there are changes (thanks to @nsoranzo). Pull Request 5679

• Populate column selector for datasets which are not ready yet. Pull Request 5680

• Fix role handling library manager. Pull Request 5681

• GFF updates to handle ##FASTA directive. Pull Request 5686

• Memory and performance fixes for upload tool. Pull Request 5689

• Fixes for testing Dockerfile recipe. Pull Request 5693

• Various improvements for API tests. Pull Request 5697, Pull Request 5699

• Small log level improvement (thanks to @hexylena). Pull Request 5715

• Swap password checking for the constant time comparison algorithm (thanks to @hexylena). Pull Request 5719

• Upgrade clone failure logging to error (thanks to @pvanheus). Pull Request 5724

• Improve galaxy update commands, update some bootstrap checkpoints. Pull Request 5731

• Remove extract_genomic_dna tool from tool sample. Pull Request 5741

• Revise uWSGI defaults and display. Pull Request 5743

• Documentation typo fixes and small refactors (thanks to @nsoranzo). Pull Request 5747

• More backend speed ups for user’s workflow index query. Pull Request 5755

• Fix all bundled scripts for the switch to YAML configuration. Pull Request 5769

• Rev and re-lock Galaxy dependencies (thanks to @nsoranzo). Pull Request 5776

• Lock pysam to 0.14.1 in Pipfile. Pull Request 5798

• Client related readme and Makefile clarifications. Pull Request 5800

• Use VueJS and API to handle workflow imports. Pull Request 5804

• Add link that explains quota management to quota indicator. Pull Request 5811

• Improve startup interoperability with Conda. Pull Request 5854

• Implement the API version of the Docker containers interface. Pull Request 5861

• Use extras to specify additional social_auth_core dependencies (thanks to @nsoranzo). Pull Request 5877

• Update cluster docs for a Virtualenv issue in DRMAA real user jobs (thanks to @moskalenko). Pull Request 5881

• Add Mocha/Chai client test framework. Pull Request 5888

• Small fix to run more integration tests (thanks to @nsoranzo). Pull Request 5892

• Use random password generation function of user class (thanks to @VJalili). Pull Request 5904

• Remove unused client route. Pull Request 5907

• Fix to provide ToolDataTableManager class for backwards compatibility (thanks to @mvdbeek). Pull Request 5924

• Document Github bug reporter (thanks to @hexylena). Pull Request 5928

• Rewrite history structure display in VueJS (thanks to @anuprulez). Pull Request 5317

• Remove static/scripts/* from the dev branch, add to .gitignore. Pull Request 5434

• History contents client code refactoring. Pull Request 5448

• Drop short-lived make client-production warning from build scripts. Pull Request 5508

• Use with statement in lib/galaxy/datatypes/ (thanks to @nsoranzo). Pull Request 5623

• Update tool conf samples for recent changes. Pull Request 5637

• Extend grid framework with formal support for delayed columns. Pull Request 5789

• Fixes for plugin directory staging. Pull Request 5810

• Remove defunct CasperJS tests. Pull Request 5567

• Eliminate unused pre-created datasets concept in upload API. Pull Request 5604

Fixes

• Fix workflow step cloning. (thanks to @mvdbeek). Pull Request 6085

• Fix tool state performance for large collections. Pull Request 5997

• Fix overly strict sanitizing in the filter tool (thanks to @abretaud). Pull Request 5872

• Allow whitelisted symlinks to be imported (thanks to @bwlang). Pull Request 5596

• Fix no dataset found issue (thanks to @osallou). Pull Request 5829

• Fix active tab highlighting (thanks to @qiagu). Pull Request 5675

• Fix initial binding for slow query logging. Pull Request 5401

• Default slow request tracking to disabled. Pull Request 5513

• Fixed stale link (thanks to @bernt-matthias). Pull Request 5548

• Fix active tab highlighting issue #4846 (thanks to @qiagu). Pull Request 5593

• Small misc fixes and cleanups (thanks to @nsoranzo). Pull Request 5638

• Fix for running Yarn in Docker for tests that do not need it. Pull Request 5690

• Various test fixes (thanks to @nsoranzo). Pull Request 5691

• Hack to fix bug in uploader where it sometimes posts with empty “file_type”. Pull Request 5692

• Bump psycopg2 to 2.7.4. Pull Request 5698, Pull Request 5825

• Do not show composite datatypes for regular dataset upload (thanks to @nsoranzo). Pull Request 5700

• Relax validation condition for dynamic parameters. Pull Request 5702

• Fix issue where some elements of collections wouldn’t render in the UI. Pull Request 5703

• Fix default role type on auto created roles (thanks to @scholtalbers). Pull Request 5709

• Fix to properly order @flakey and @selenium_test Selenium test decorators (thanks to @nsoranzo). Pull Request 5711

• Clarify that nginx-extras needs to be install for the upload_store documentation (thanks to @mvdbeek). Pull Request 5717

• Remove subclass="true" from Phylip datatype definition (thanks to @nsoranzo). Pull Request 5718

• Sentry bug reports working for anonymous users (thanks to @hexylena). Pull Request 5721

• Add Galaxy lib directory to Python path for 2 scripts (thanks to @nsoranzo). Pull Request 5725

• Fix LDAP when role_search_attribute is not defined (thanks to @nsoranzo). Pull Request 5732

• Fix mulled Docker caching. Pull Request 5735

• Drop old charts building from makefile/package scripts Pull Request 5737

• Configuration fixes (thanks to @nsoranzo). Pull Request 5745

• Add additional validation to numeric input element. Pull Request 5750

• Various scatterplot fixes. Pull Request 5752

• Fix linking option for library dataset uploads (thanks to @nsoranzo). Pull Request 5758

• Fix copying tags for collections when copying histories. Pull Request 5766

• Fix /api/whoami API endpoint for master key. Pull Request 5773

• Log the pid of the first uWSGI process (thanks to @nsoranzo). Pull Request 5774

• Fix SQL query for “workflows shared with me” as part of workflow index. Pull Request 5782

• Fix model operation tools that produce standalone datasets. Pull Request 5784

• Fix missing “from Import Directory” button in data libraries (thanks to @nsoranzo). Pull Request 5797

• Fix metadata setting for CRAM files (thanks to @mvdbeek). Pull Request 5805

• Use app.datatypes_registry in ValidationContext.from_app() (thanks to @nsoranzo). Pull Request 5806

• Fix upload_common.py ensure hda is added to a session. Pull Request 5816

• Exclude the JS GIE proxy’s Node.js modules from linting. Pull Request 5826

• Do not append ‘/’ to ‘js’ in GIE static path, this breaks uWSGI static routing. Pull Request 5859

• Fix conditional parameter handling. Pull Request 5864

• Increase Yarn’s network timeout. Pull Request 5870

• Move the memory statement below _galaxy_setup_environment (thanks to @moskalenko). Pull Request 5880

• Remove deprecated controller endpoint and link to preferred implementation. Pull Request 5887

• Fix update_repository when hg repo has local modifications (thanks to @nsoranzo). Pull Request 5890

• Fix admin data library dataset permissions. Pull Request 5899

• Fix display of the tour index names. Pull Request 5900

• Fix Trackster links. Pull Request 5916

• Fix uploads of link-only datasets to data library (thanks to @mvdbeek). Pull Request 5917

• Correct signal number in admin documentation (thanks to @hexylena). Pull Request 5921

• Fix error with data managers when trying to move_merge nonexistent extra_files dir (thanks to @abretaud). Pull Request 5922

• Fix bug with removing workflow outputs from a node. Pull Request 5935

• Fix for running in containers when job_tmp is set in job_conf.xml. Pull Request 5937

• Fix for running jobs in containers. Pull Request 5938

• Fix dropdown click and highlighting behavior. Pull Request 5945

• Resort to os.path.exists() also for paths not ending in .loc (thanks to @nsoranzo). Pull Request 5947

• Remove ini config parameter from cleanup scripts (thanks to @bernt-matthias). Pull Request 5956

• Fix shared history list bug. Pull Request 5962

• Pin the version of Node.js used in CI. Pull Request 5972

• Fix 2 typos in the docs (thanks to @nsoranzo). Pull Request 5986

To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.

You can always chat with us on Matrix.

Thanks for using Galaxy!

The Galaxy Team