18.01

Enhancements

• Performance and User Experience Improvements Pull Request 5514, Pull Request 5515, Pull Request 5542, Pull Request 5574, Pull Request 5576, Pull Request 5582, Pull Request 5533, Pull Request 5615, Pull Request 5616

• Client Architecture Pull Request 4435, Pull Request 4978, Pull Request 4436, Pull Request 5173 Pull Request 5293, Pull Request 5191, Pull Request 5190, Pull Request 4827, Pull Request 4964, Pull Request 5371, Pull Request 5128, Pull Request 5262, Pull Request 5081, Pull Request 5054 Pull Request 5093, Pull Request 5001, Pull Request 5261, Pull Request 5278, Pull Request 5294

• Add option to clone tool steps in the workflow editor (thanks to @anatskiy). Pull Request 5087, Pull Request 5124, Pull Request 5185

• Implement a generic Cloud ObjectStore using multi-cloud support from cloudbridge (thanks to @VJalili). Pull Request 4487

• Reload tools immediately if a macro changes (thanks to @mvdbeek). Pull Request 4635

• Add an interactive environment for AskOmics a visual SPARQL query builder for RDF databases (thanks to @xgaia). Pull Request 4737

• Implement a Github error report backend for (thanks to @hexylena). Pull Request 4765

• Add maintenance script to simplify setting up a cleanup cron job (thanks to @scholtalbers). Pull Request 4916

• Create the $GALAXY_MEMORY_MB job environment variable for tools. Pull Request 4958

• Improved home and temp directory handling for tools and jobs. Pull Request 5193

• Tool-based annotation of out of memory errors. Pull Request 5196

• Added support for recursively discovering output datasets (thanks to @nuwang). Pull Request 5240

• Add an API endpoint that allows admins to decode galaxy object IDs. Pull Request 5018

• Add sorting options for data library folders. Pull Request 4877

• Add option to auto create user_library_import_dir directories upon user login. (thanks to @scholtalbers). Pull Request 5138

• Add option to check file permissions for the user library imports (thanks to @scholtalbers). Pull Request 5154

• Add an admin form to pre-populate an LDAP user (thanks to @scholtalbers). Pull Request 5238

• Add a CRAM to BAM datatype converter (thanks to @mvdbeek). Pull Request 4939

• Add infernal-cm datatype (thanks to @mmiladi). Pull Request 4977

• Add ICM datatype for glimmer (thanks to @abretaud). Pull Request 5042

• Add Analyze75 datatype (thanks to @bgruening). Pull Request 4841

• Add Classical linkage and haplotype datatypes (thanks to @mtekman). Pull Request 4894

• Add ideaspre datatype (thanks to @gregvonkuster). Pull Request 5445

• Add compressed fasta support. Pull Request 5561

• Add cool datatype (thanks to @bgruening). Pull Request 5023

• Add neostore zip datatype (thanks to @zipho). Pull Request 4178

• Add fast5_archive datatype (thanks to @jvolkening). Pull Request 4569

• Add postgresql datatype (thanks to @bgruening). Pull Request 4730

• Add IQtree datatypes (thanks to @mtekman). Pull Request 4849

• Add Phylip datatype (thanks to @khillion). Pull Request 5301

• Option to group Galaxy Tours for viewing clarity (thanks to @selten). Pull Request 4829

• Allow users to upload multiple FTP files within a single request Pull Request 4513

• Replaces the admin form builder grid and forms Pull Request 4480

• Update pysam dependency to 0.14 and eliminate the need to have samtools on Galaxy’s PATH (with huge help from @mvdbeek) Pull Request 4497, Pull Request 5512, Pull Request 5037

• Expand and improve Selenium testing of the web UI (with help from @anatskiy). Pull Request 4543, Pull Request 4655, Pull Request 4676, Pull Request 4753, Pull Request 4732, Pull Request 4909, Pull Request 4935, Pull Request 4925, Pull Request 4687, Pull Request 4660, Pull Request 5333, Pull Request 5068, Pull Request 4757, Pull Request 4971, Pull Request 4653

• Allow multiple simultaneous uploads via single POST. Pull Request 4563

• Various bits of proactive security-related hardening (thanks to @hexylena). Pull Request 4604

• Move test helper wait_for_dataset_collection for reuse (thanks to @mvdbeek). Pull Request 4658

• Extend job search for hdcas (thanks to @mvdbeek). Pull Request 4665

• Add a custom TRACE log level for even more verbosity. Pull Request 4672

• Migrate various mako templates to modern JavaScript driven views. Pull Request 4681, Pull Request 4682, Pull Request 4711, Pull Request 4755, Pull Request 4661, Pull Request 4993, Pull Request 4664, Pull Request 4996, Pull Request 5014, Pull Request 4833, Pull Request 5212, Pull Request 5114

• More HDCA copying tests, add option to copy HDAs into new history. Pull Request 5004

• Use only latest publication in CITATION (thanks to @nsoranzo). Pull Request 4688

• Nicer logging message for sentry (thanks to @hexylena). Pull Request 4693

• Use enumeration to provide history input index names. Pull Request 4700

• Cleanup delayed action in history-view.js. Pull Request 4704

• Add logging during for script integrity exceptions (thanks to @hexylena). Pull Request 4720

• Fix to respect galaxy.json supplied metadata for empty outputs. Pull Request 4722

• Update reports.md with a link to the community hub (thanks to @scholtalbers). Pull Request 4723

• Typo fixes and small cleanups (thanks to @nsoranzo). Pull Request 4725

• Fix that implements stricter checks for SDF datatype sniffing (thanks to @nsoranzo). Pull Request 4729

• When reloading the toolbox, use the existing dependency manager if possible (thanks to @mvdbeek). Pull Request 4739

• Many more upload tests - big improvements to uploads will be coming to Galaxy 18.05. Pull Request 4746, Pull Request 4762, Pull Request 4790, Pull Request 4863, Pull Request 5266, Pull Request 5209

• Kubernetes resources requests/limits support (thanks to @pcm32 and the @phnmnl project). Pull Request 4751

• Send all influxdb stats to a single bucket (thanks to @hexylena). Pull Request 4758

• PEP-8 fixes in random doctests. Pull Request 4764

• Avoid circular import when determining max metadata value size. Pull Request 4770

• Remove unused concept of dataset children. Pull Request 4771

• Allow dataset attribute changes without refreshing the page. Pull Request 4779

• Include check for updates to dependencies in tox, Makefile, Travis. Pull Request 4788

• Add labels to selection options of the collection flattening tool. Pull Request 4791

• Expose state, stdout, stderr, uuid in the library API Pull Request 4839

• Significant improvements to client QUnit tests including new Karma based test driver. Pull Request 4855, Pull Request 5066, Pull Request 5063, Pull Request 5074

• Remove tool reloading buttons and packaging endpoint, revise admin labels Pull Request 4866

• Add various elements to the Tool XML doc page (thanks to @nsoranzo). Pull Request 4871

• Absolute pins on linting dependencies. Pull Request 4876

• Update various Galaxy requirements. Pull Request 4884

• Updates and enhancements for the testing Dockerfile. Pull Request 4885, Pull Request 4879, Pull Request 4900

• Add nicer composite dataset names by default (thanks to @bgruening). Pull Request 4886

• Improved initialization and isolation of Postgres databases used in integration testing. Pull Request 4887, Pull Request 4914

• Rework dev dependencies to be Pipfile generated, locked, and hashed. Pull Request 4891

• Install dev and regular dependencies at the same time. Pull Request 4903

• More structured app shutdown. Pull Request 4910

• Touch up issue labels in repository procedures document. Pull Request 4912

• Improve error handling for incorrect datatype upload. Pull Request 4922

• Activate mulled dependencies once per job (thanks to @nsoranzo). Pull Request 4945

• Remove invalid user preferences beta (no longer a beta, always enabled) Pull Request 4957

• Add DELETE /api/jobs/<job_id> as a job cancellation API endpoint. Pull Request 4961

• Kill off processes when stopping the local job runner. Pull Request 4962

• Test both endpoints for collection creation in history contents API. Pull Request 4965

• Add @flakey test annotation and “skip” them if the fail on Jenkins. Pull Request 4972

• Include path in job script integrity errors (thanks to @nsoranzo). Pull Request 4991

• Fix tox envlist to contain all envs that should pass (thanks to @nsoranzo). Pull Request 4992

• Changes made while working on uWSGI mules that don’t belong in the mules PR Pull Request 4994

• Add warning for sort-of-defunct workflow options in galaxy.ini. Pull Request 5002

• Fix Python import order everywhere (thanks to @nsoranzo). Pull Request 4697, Pull Request 5005

• dataset info: present decoded IDs for admins and add peek Pull Request 5006

• Removed unneeded skip_install tags in tox.ini. Pull Request 5011

• Add metric to the core plugin to collect $GALAXY_MEMORY_MB Pull Request 5016

• Test renaming multiple different outputs in workflows. Pull Request 5017

• Render arbitrarily nested lists in the UI Pull Request 5038

• Expose proper element_identifier in tool action template expansions Pull Request 5049

• Document fastest way to iterate on client & client tests together. Pull Request 5051

• Support separate DSN for job sentry error reports (thanks to @hexylena). Pull Request 5056

• Build endpoint separation Pull Request 5058

• Remove wchartype dependency and is_multi_byte attribute (thanks to @nsoranzo). Pull Request 5062, Pull Request 5088

• Implement multilingualization support on workflow page (thanks to @manabuishii). Pull Request 5077

• Update Jupyter IE default image (thanks to @galaxyproject). Pull Request 5095

• Determine dynamically if an extension corresponds to an unsniffable binary datatype (thanks to @nsoranzo). Pull Request 5109

• Update to flake8-import-order 0.16 (thanks to @nsoranzo). Pull Request 4869, Pull Request 5110

• Change the default to not add tags based on filename Pull Request 5115

• Improve tool validation and metadata generation (thanks to @mvdbeek). Pull Request 5119

• Download button fixes. Pull Request 5125

• Datatype conversion check optimization (thanks to @nsoranzo). Pull Request 5137

• Test resubmit for dynamic JobDestination structures. Pull Request 5139

• Various dataset hashtag display improvements. Pull Request 5155

• i18n improvements for French localization (thanks to @ValentinChCloud). Pull Request 5089, Pull Request 5157

• Change hicexplorer image name (thanks to @bgruening). Pull Request 5177

• Fix #3002 with update to newer version of virtualenv. (thanks to @selten). Pull Request 5202

• Huge steps toward Python 3 compatiblity (thanks to @nsoranzo). Pull Request 5215, Pull Request 5221, Pull Request 5225, Pull Request 5228, Pull Request 5200, Pull Request 5199, Pull Request 4799, Pull Request 4811

• Add chinese support (thanks to @istevenshen). Pull Request 5232

• Make pam a conditional dependency (thanks to @nsoranzo). Pull Request 5251

• Update api quickstart doc to use default port, with a little bit of extra guidance. Pull Request 5260

• Allow sorting collection elements based on file (thanks to @mvdbeek). Pull Request 5273

• Add Biostar configuration options to the configuration sample file. Pull Request 5276

• Update Japanese translation (thanks to @istevenshen). Pull Request 5288

• Improve the migration message when having to downgrade the database (thanks to @mvdbeek). Pull Request 5299

• Allow message attribute to be modified on a library dataset. Pull Request 5312

• Enforce a size constraint on the tag editor. Pull Request 5337

• Various webhooks improvements. Pull Request 5342

• Make TabularToolDataTable._deduplicate_data() faster (thanks to @nsoranzo). Pull Request 5383

• Add --skip_flakey_fails flag to CI script for API tests Pull Request 5455

• Fix for toolbox filters (thanks to @bgruening). Pull Request 5526

• Fix missing job stdout/stderr on Pulsar transfer failures. Pull Request 5535

• Optimization to fix repeated fetching dataset permissions in dynamic output collection. Pull Request 5578

• Backport toolbox cache Pull Request 5583

• Replace distutils.version with packaging.version (fixing lineage handling for freebayes among other things). Pull Request 5588

• Move the cache from toolbox level to the tool level. Pull Request 5600

• Removes Sample request and External services makos/controllers Pull Request 4526, Pull Request 4872, Pull Request 5103, Pull Request 5204

• Remove unused abstraction layers from Visualizations Registry. Pull Request 4620

• Visible keys standardization in model classes. Pull Request 4744

• Remove legacy library interface Pull Request 4908

• ES6 client/galaxy/scripts/utils/ajax-queue.js Pull Request 4933

• Remove unused galaxy-side set_tool_version code (thanks to @mvdbeek). Pull Request 4986

• Fix allow-register option in auth module (thanks to @nsoranzo). Pull Request 4989

• Tried to fix issue with Conda environment building failing jobs that had to be rolled back. (thanks to @mvdbeek). Pull Request 5036, Pull Request 5470

• Add packed symlink back pending a deprecation cycle. Pull Request 5094

• Onload webhook toggles. Pull Request 5116

• Re-organize edge case upload options for greater readability. Pull Request 5206

• Refactor upload.py toward reuse. Pull Request 5229

• JavaScript router cleanup. Pull Request 5235

• Comment PlantTribes datatypes (thanks to @gregvonkuster). Pull Request 5254

• Dataset error interface cleanup. Pull Request 5279

• Mention how to specify the config file location (thanks to @mvdbeek). Pull Request 5297

• Uninstall a tool by default (vs. deactivating it). Pull Request 5300

Fixes

• Allow mapping over collections when discovering dataset collections in tools (thanks to @mvdbeek). Pull Request 5413

• Make liftover tool use data tables (thanks to @mvdbeek). Pull Request 4645

• Pass job output file unqualified names to Pulsar so that it can create them before running the job. Pull Request 4662

• Fix transiently failing saved histories due to success message disappearing. Pull Request 4669

• Update CITATION file with 2016 paper (thanks to @mvdbeek). Pull Request 4686

• Stop running common_startup.sh twice when starting from run.sh Pull Request 4759

• Fix docstring linting. Pull Request 4766

• Mothur.freq sniffer more stringent (thanks to @yhoogstrate). Pull Request 4781

• MacOS X fix for a test tool. Pull Request 4806

• Minor architecture slides fixes for typos/grammar. Pull Request 4822

• Fix two invalid targets in dataset list item. Pull Request 4823

• Fix double return in form-parameters. Pull Request 4826

• Fix the case where ‘Labels’ in a docker node inspect exists but is null. Pull Request 4838

• Fix all E722 errors and ignore E741 (thanks to @nsoranzo). Pull Request 4847

• Fix jsutils time - fixes charts visualize. Pull Request 4852

• Bugfix during error handling with linked files during upload (thanks to @ieguinoa). Pull Request 4858

• Add a gulp plumber. Pull Request 4867

• Minor admin panel fixes. Pull Request 4895

• Fix list collection creator dragdrop event bindings. Pull Request 4901

• Fix status handling for grids. Pull Request 4913

• Fix revision selection during (beta) repository installation (thanks to @mvdbeek). Pull Request 4944

• Fix Trackster styles bleeding into the app (the slightly off-center upload .icon buttons). Pull Request 4948

• Fix multiple selections for grid operations redirecting to center panel. Pull Request 4951

• Fix visualization link in import success message. Pull Request 4952

• Fix up reports JavaScript. Pull Request 4967

• Run hg clone in a subprocess instead of using the Mercurial API (thanks to @nsoranzo). Pull Request 4979

• Tolerate IOError in tool and data table watcher (thanks to @mvdbeek). Pull Request 4981

• Install numpy before bx-python in case you are installing from sdists Pull Request 4982

• Specify TagAssociation class when copying a tag (thanks to @mvdbeek). Pull Request 4984

• Fix initialization of named ajax queue. Pull Request 4985

• Encode file content with utf-8 (thanks to @mvdbeek). Pull Request 4987

• Revert export to file. Pull Request 4988

• Remove extraneous subclass for DMND datatype (thanks to @nsoranzo). Pull Request 4990

• Fix to allow get_history calls with create=False when evaluating workflows. Pull Request 4997

• Add missing ipaddress requirement (thanks to @nsoranzo). Pull Request 5000

• Use github instead of NCBI for problematic data manager test FASTA. Pull Request 5007

• Fix for history contents API to remove the default flag override. Pull Request 5008

• Fix to avoid visiting child inputs of invalid conditionals. Pull Request 5010

• Fix for parse_interpreter - log a warning only if interpreter is set (thanks to @nsoranzo). Pull Request 5015

• Fix a few stray wiki links. Pull Request 5020

• Workflow editor outputs fixes. Pull Request 5021

• Fix workflow editor output attributes. Pull Request 5022

• Metadata parameters rely on their own optional setting not on the field attribute. Pull Request 5027

• A Python 3 fix for the gsummary tool (thanks to @bernt-matthias). Pull Request 5043

• Cleanup SA objects between workflow invocation scheduling attempts. Pull Request 5045

• Safely handle possible None value encountered in processing and execution of post-job action arguments (thanks to @hexylena). Pull Request 5050

• Restore admin form routes. Pull Request 5065

• Fixes galaxy startup when LC_TYPE=UTF-8 on OS X (thanks to @mvdbeek). Pull Request 5070

• Ignore OSError when chmod’ing integrated_tool_panel_conf.xml (thanks to @mvdbeek). Pull Request 5071

• Fix metadata setting for otu datatypes (thanks to @mvdbeek). Pull Request 5072

• Fix exception in admin panel (thanks to @mvdbeek). Pull Request 5079

• Always fill message to avoid KeyError for sentry ERROR_TEMPLATE (thanks to @mvdbeek). Pull Request 5086

• Attempt to make data manager integration test more robust. Pull Request 5098

• Toolshed install stability improvements (thanks to @mvdbeek). Pull Request 5099

• Fix Trackster link to custom builds view. Pull Request 5104

• Improve resilience of filter detection, add error messages, fix identifiers. Pull Request 5106

• Fix ICM datatype sniffer config (thanks to @mvdbeek). Pull Request 5121

• Fix attribute error that was missed during refactor of sentry (thanks to @hexylena). Pull Request 5122

• Fix connecting non-input modules to subworkflow inputs. Pull Request 5140

• Fix for history copy. Pull Request 5144

• Fixes for handling delayed steps within subworkflows. Pull Request 5145

• Fix localization and build client. Pull Request 5156

• Resolve broken link for “all workflows” (thanks to @mmiladi). Pull Request 5161

• Fix workflow collection attachments. Pull Request 5166

• Provide UI collection output information for subworkflows. Pull Request 5172

• Remove legacy request controller from reports app. Pull Request 5182

• Fixes genome decode handling. Pull Request 5198

• Remove broken Yahoo OpenID support (thanks to @VJalili). Pull Request 5208

• Force window jQuery access for charts. Pull Request 5210

• Mark transiently failing test_run_data as flakey. Pull Request 5211

• Fail job if tools that use galaxy.json write to stderr (thanks to @mvdbeek). Pull Request 5217

• Allow shell plugins with different parameters to co-exist (thanks to @mvdbeek). Pull Request 5223

• paster.pid/log to galaxy.pid/log reference update (thanks to @nsoranzo). Pull Request 5226

• Refine Conda version numbers allowed for using --offline flag again (thanks to @NCEichner). Pull Request 5233

• Workflow list show-in-tool-panel callback bugfix. Pull Request 5234

• Fix image proxy prefix in tool form. Pull Request 5237

• Fix wrong tool id after switching versions (thanks to @mvdbeek). Pull Request 5249

• Catch and display error when displaying broken BAM file (thanks to @mvdbeek). Pull Request 5253

• Fix for getmicrobedata tool (thanks to @bernt-matthias). Pull Request 5258

• Adjust note about nginx enabled sites configuration (thanks to @scholtalbers). Pull Request 5259

• Mark the test rerun_with_use_cached_job as flakey (thanks to @mvdbeek). Pull Request 5286

• Fix dataset edit link. Pull Request 5289

• Fix link to getgalaxy.org (thanks to @selten). Pull Request 5295

• Fix tool report submission with sentry (thanks to @mvdbeek). Pull Request 5303

• Fix common_startup.sh bash-isms (thanks to @nsoranzo). Pull Request 5304

• Fix Galaxy instance startup error (thanks to @bgruening). Pull Request 5306

• Fix link to library from history structure view. Pull Request 5327

• Prevent reloading when data table is being modified (thanks to @mvdbeek). Pull Request 5331

• Work around (temporarily) wrong getsize() output (thanks to @mvdbeek). Pull Request 5335

• Disable galaxy tour ‘path’ navigation. Pull Request 5340

• Fix tool-shed-config-validate Makefile target. (thanks to @ValentinChCloud). Pull Request 5343

• History import fixes (thanks to @mvdbeek). Pull Request 5344

• Fix dataset purging, dataset cleanup script and db_shell script (thanks to @mvdbeek). Pull Request 5346

• Fix BAM conversion test tool (thanks to @mvdbeek). Pull Request 5351

• Remove charlimit for fetching URLs. Pull Request 5353

• Use print() function in common_startup.sh (thanks to @mvdbeek). Pull Request 5357

• Handle None in job owner in API (thanks to @hexylena). Pull Request 5358

• Fix form logic in deactivate or uninstall. Pull Request 5363

• History multipanel fixes. Pull Request 5364

• Remove additional checkbox workarounds. Pull Request 5367

• Workflow tool menu fix for deleted workflows. Pull Request 5368

• Bump conditional Pygments version to 2.2.0 so it matches the dev. Pull Request 5376

• Fix make docs when the virtualenv is not .venv . Pull Request 5377

• Force a fixed node image version when building client for Jenkins. Pull Request 5382

• Install latest Conda and don’t use shell=True (thanks to @nsoranzo). Pull Request 5397

• Fix for subworkflow creation (thanks to @mvdbeek). Pull Request 5398

• Follow IUC and drop r channel from the default Conda channels (thanks to @mvdbeek). Pull Request 5406

• Implement workaround for extra metadata revision creation. Pull Request 5433

• Don’t silently fail rest of multi-action queue when attempting to purge a collection. Pull Request 5443

• Ensure jobStateSummariesCollection exists. Pull Request 5444

• Fix server_name when config is a dict. Pull Request 5447

• Fixes for workflow sharing links. Pull Request 5467

• Fix workflow execution post action - rename (thanks to @scholtalbers). Pull Request 5485

• Fix common_startup.sh for galaxy tarballs (thanks to @nsoranzo). Pull Request 5491

• Close tempfile handles. Pull Request 5506, Pull Request 5552

• Do not allow codefiles during tool shed tool loading. Pull Request 5510

• Fix for GenomeSpace importer/exporter (thanks to @nuwang). Pull Request 5528

• Fix search overlay JS errors (thanks to @bgruening). Pull Request 5531

• Allow LDAP options not starting with OPT_ (thanks to @nsoranzo). Pull Request 5556

• Fix broken admin create new user when registration =='challenge' (thanks to @scholtalbers). Pull Request 5569

• Do not remove external path files during library uploads (thanks to @nsoranzo). Pull Request 5573

• Bump sqlite3 dependency for web proxy. Pull Request 5575

• Fix small bam (header-only) infinite fetching. Pull Request 5579

• Fix migration 0137’s downgrade. Pull Request 5605

• Conda fix for commands using stdout redirection. Pull Request 5620

• Move Extract genomic DNA 1 to list of versioned galaxy tools requiring galaxy to be importable (thanks to @mvdbeek). Pull Request 5626

• Changed GenomeSpace token handling to use manual OpenID association only. Pull Request 5631

• Use sample file path when adding missing indexes (thanks to @mvdbeek). Pull Request 5650

• Fix exception if user preference value undefined (thanks to @mvdbeek). Pull Request 5662

• Fix exception if email is not specified or email is wrong (thanks to @mvdbeek). Pull Request 5663

• Show error message if user tries purging datasets (thanks to @mvdbeek). Pull Request 5664

• Show error message if history structure can’t be shown (thanks to @mvdbeek). Pull Request 5665

• Consistent sniffing regardless of in_place. Pull Request 5643