16.07

Highlighted Enhancements

• Implement an Azure backend for the Galaxy objectstore (thanks to @zfrenchee.) Pull Request 2621

• HTCondor Docker Integration (thanks to @bgruening.) Pull Request 2278

• Implement a Kubernetes job runner (thanks to @pcm32.) Pull Request 2314, Pull Request 2498, Pull Request 2559, Pull Request 2578, Pull Request 2616, Pull Request 2528

• Implement collection operations - tool-like entities allowing users to zip, unzip, flatten, and filter failed datasets out of collections. (You may need to sync your tool_conf.xml against sample.) Pull Request 2434

• Allow history panel to display list of lists dataset collections. Pull Request 2613

• Add a Tool Shed API endpoint method to check the current user. Pull Request 2243

• Add a Galaxy API endpoint to update a user object. Pull Request 2595

• Improvements to toolbox search. Pull Request 2273

• New pages and general improvements to the Reports application (thanks to @BalthazarPavot.) Pull Request 2664

• Nicer bug report emails (thanks to @hexylena.) Pull Request 2143

• Tools with auto_format=’true’ now connectable in workflow editor (thanks to @yhoogstrate.) Pull Request 2144

• Drop “Galaxy pip” in favor of manylinux1 wheels (which are supported in pip >= 8.1). Pull Request 2383

• Allow tool form to listen to history (thanks to @nturaga.) Pull Request 2476

• Add EDAM operations to tools and expose in API (thanks to @mvdbeek, @nsoranzo and @odoppelt.) Pull Request 2379

• Add support to PDBQT molecule datatype (thanks to @leobiscassi.) Pull Request 2452

• Add the trackhub composite datatype (thanks to @remimarenco.) Pull Request 2348, Pull Request 2646, Pull Request 2701

• Add the Mothur datatypes (thanks to @shiltemann.) Pull Request 2038, Pull Request 2429

• Add the dynamic tool destination tool to Galaxy for defining finer grained tool destination rules. (thanks to @ericenns.) Pull Request 2579

• Galaxy can now install Conda dependencies in addition to tool shed dependencies during repository installations (thanks to @mvdbeek and @bwlang.) Pull Request 2554

• Implement the Galaxy Communication Server (a simple in-Galaxy chat server) (thanks to @anuprulez and @bgruening.) Pull Request 2515

Enhancements

• Install Conda at startup; enable conda_auto_init in sample conf Pull Request 2759

• Add filter_failed_datasets_from_collection.py script to scripts/ (thanks to @chambm.) Pull Request 2358

• Enhanced workflow run form using newer client forms architecture. Pull Request 2077

• Tweaks to various library functions for galaxy-lib. Pull Request 2110, Pull Request 2494, Pull Request 2354, Pull Request 2414, Pull Request 2436

• Automate and clarify more of the release process. Pull Request 2117, Pull Request 2366, Pull Request 2372, Pull Request 2388

• Update template to include the release of the Docker container (thanks to @bgruening.) Pull Request 2378

• Add server_starttime to configuration API. Pull Request 2124

• Show line breaks in commit log in the Tool Shed (thanks to @nturaga.) Pull Request 2142

• Replacing the _getIndex method by UnderscoreJS method _.findIndex() (thanks to @anuprulez.) Pull Request 2157

• Small enhancements to user activation process. Pull Request 2158

• Implement single_user = <email> config option. Pull Request 2165

• More documentation and flexibility for FTP imports. Pull Request 2168

• Make stats prefix configurable and add a conditional wheel requirement. Pull Request 2171

• Fix running tool tests if remote user middleware is enabled. Pull Request 2173

• Improve handling of a missing R environment during Tool Shed dependency installations. Pull Request 2215

• Quote some parameters in the trim tool command (thanks to @nsoranzo.) Pull Request 2248

• Remove overly chatty tool dependency resolution logging statement. Pull Request 2254

• Update the list of contributors. Pull Request 2256, Pull Request 2257, Pull Request 2258

• Add grunt watcher on .less files (thanks to @remimarenco.) Pull Request 2274

• Remove dependency on custom Whoosh (requirements update). Pull Request 2276

• Various enhancements for library functions related to tools. Pull Request 2279

• Shift asbool handling in datatypes API so that an appropriate error message is now returned. Pull Request 2286

• More small fixes from downstream projects. Pull Request 2292

• For published workflow display, provide a better error when the workflow or user cannot be found Pull Request 2302

• Bump sqlite3 version (thanks to @matthdsm.) Pull Request 2307

• Show collection counts in collection and history collection views. Pull Request 2326

• Various client library updates. Pull Request 2328

• Revise form data visitor. Pull Request 2330

• Update using xhr date header. Pull Request 2335

• Implement an API for managing history/sharable model user sharing. Pull Request 2340

• Revise portlet view, integrate collapsible option. Pull Request 2360

• Fix issue with lowercasing in datatypes/registry.py (thanks to @remimarenco.) Pull Request 2361

• Improve composite output tool test. Pull Request 2363

• Upgrade shared ui libraries, add test cases Pull Request 2368

• Throughout backend use open() instead of file() for Python 3 (thanks to @nsoranzo.) Pull Request 2370

• Remove backported importlib and ordereddict. Pull Request 2380

• Note on updating tool testing (thanks to @hexylena.) Pull Request 2381

• Docker swarm support for Galaxy Interactive Environments (GIEs) Pull Request 2386, Pull Request 2392

• Add support for EDAM data annotations to datatypes (thanks to @nsoranzo, @bgruening, @hmenager and @mvdbeek.) Pull Request 2387

• Add a new workflow building mode variation which does access the history Pull Request 2390

• Add conda search function to conda_util library function (thanks to @bgruening.) Pull Request 2395

• Add history navigation options for dataset frames to scratch book Pull Request 2396

• Add EDAM topics to tools and expose in API (thanks to @nsoranzo.) Pull Request 2397

• Update SLURM runner so when scontrol fails, Galaxy attempts to get job state with sacct (thanks to @nsoranzo.) Pull Request 2403

• Enhancements to tool test output verification. Pull Request 2409

• Show history dataset id (hid) in dataset ‘view details’ page. Pull Request 2411

• Updates and fixes for the run workflow UI elements. Pull Request 2416

• Raise Exception if watch_tools has an unrecognized value (thanks to @nsoranzo.) Pull Request 2418

• Use log.warning() instead of deprecated log.warn() (for Python 3) (thanks to @nsoranzo.) Pull Request 2420

• Minor updates to documentation. Pull Request 2427

• Correct various EDAM data/format annotations in datatypes (thanks to @hmenager.) Pull Request 2428

• Fix $defaults according to #2430 (thanks to @ThomasWollmann.) Pull Request 2438

• Properly list subworkflows in run workflow form. Pull Request 2440

• Make some files compatible with Python3 (thanks to @nsoranzo.) Pull Request 2453

• Add parameter expansion caller and test cases as preparation for workflow API extension. Pull Request 2464

• Set a default for tool_dependencies_dir. Pull Request 2480

• Add a password expiration config option (resolves #2485) (thanks to @benfulton.) Pull Request 2486

• Display TIFF format in upload (thanks to @bgruening.) Pull Request 2516

• Use self.decode_id instead of trans.security.decode_id (thanks to @nsoranzo.) Pull Request 2529

• Charts revision. Pull Request 2531

• Add Makefile target for fetching updated bower dependencies. Pull Request 2534

• Make conda autoinstall of packages safer: (thanks to @pvanheus.) Pull Request 2538

• Updates to Interactive Environment docs (thanks to @pvanheus.) Pull Request 2543

• Allow displaying bigWig files in IGV (thanks to @dpryan79.) Pull Request 2545

• Allow users to purge datasets by default. Pull Request 2553

• Add Jupyter IE to the allowed visualizations for BAM datasets (thanks to @hexylena.) Pull Request 2572

• Python3: port 215 files (thanks to @nsoranzo, @tshtatland, @xiaoqian1984, @alanryanwang and @ericenns.) Pull Request 2574

• Add python-ldap module to conditional requirements (thanks to @abretaud.) Pull Request 2576

• Update gsummary.xml (thanks to @lecorguille.) Pull Request 2590

• Add Warning when Conda is installed (thanks to @remimarenco.) Pull Request 2596

• Added the neostore (neo4j db) for noe4jdb dataset (thanks to @zipho.) Pull Request 2605

• Fix smart_str for Python 3. Add doctests (thanks to @nsoranzo.) Pull Request 2608

• Added Neo4j IE (thanks to @thobalose.) Pull Request 2610

• Committing format changes to the user impersonation dropdown. (thanks to @sszakony.) Pull Request 2612

• Use pagination and filtering for select2 fields Pull Request 2615

• Update workflow share dropdown width (thanks to @sszakony.) Pull Request 2617

• Graph visualization: add parsers, change datatypes Pull Request 2620

• Fix certain combos of dynamic deeply nested collections in workflows. Pull Request 2634

• add tar to datatypes_conf.xml (thanks to @mariabernard.) Pull Request 2636

• Optimize get_file_peek() (thanks to @nsoranzo.) Pull Request 2637

• Move workflow run and history options to top, add loading indicator Pull Request 2643

• Enhance workflow run button to indicate workflow loading progress, keep header fixed Pull Request 2647

• Allow uploading strict BED datatypes. Pull Request 2648

• Reduce server sided refresh requests to changes in data associated input fields Pull Request 2652

• Add import order linting using tox and flake8-import-order. Pull Request 2661

• Add hg38 to main for ucsc_build_sites.txt.sample Pull Request 2670

• Extend the list of files tested with flake8-import-order to all files already passing (thanks to @nsoranzo.) Pull Request 2682

• List resolver Tool Shed packages (thanks to @mvdbeek.) Pull Request 2750

• Remove buildbot remnants Pull Request 2125

• Try pushing again when we get unexpected exception (test fix) Pull Request 2130

• Refactor checking tool files toward reuse with workflows in Planemo. Pull Request 2183

• Rename/refactor Utils.validate() to Utils.isEmpty() Pull Request 2239

• Cleanup of the install and test framework. Pull Request 2249

• cleanup TS and Reports configs Pull Request 2267

• Move specific methods from shed_util_common.py to specific_util.py Pull Request 2269

• Revise selection, options and drilldown wrappers, add tests Pull Request 2305

• Visualizations: remove graphview Pull Request 2329

• Update qunit and sinon Pull Request 2338

• Build job success messages within tool form view Pull Request 2355

• Cleanup of E203 primarily, and some other misc formatting Pull Request 2364

• Isolate the data libraries css to affect only the libraries app Pull Request 2444

• Re-add get_updated_changeset_revisions_from_tool_shed() method (thanks to @nsoranzo.) Pull Request 2472

• Workaround for circular import. Refactor. Pull Request 2477

• run_tests.sh: Clarify when/where the report file is created. (thanks to @NickSto.) Pull Request 2564

• Reroute consistent workflow endpoint to internal API Pull Request 2622

• Add tool test for what happens when output filter throws exception. Pull Request 2631

• Ignore .tox and eggs directories for flake8 Pull Request 2650

• Reorder base templates meta tags Pull Request 2651

• Import pyparsing in testing so twill’s older version doesn’t get loaded. Pull Request 2656

• Change default refresh mode for form views Pull Request 2667

• Fix unit tests (thanks to @nsoranzo.) Pull Request 2691

• Create a uWSGI postfork function registry and start the tool conf watcher thread post-fork Pull Request 2774

Fixes

• Do not encode dict items that start with metadata\_ in return value of api/tools/create. Pull Request 2459

• Generate metadata in separate conda environment (thanks to @mvdbeek.) Pull Request 2253

• Fix to use “$NCPUS” if defined to set GALAXY_SLOTS (thanks to @mvdbeek.) Pull Request 2145

• fix blocking issue check Pull Request 2113

• Correctly check set_environment declarations when updating repository. Pull Request 2116

• Hide hidden data parameters in workflow editor Pull Request 2120

• Place $headers before integrity_check (thanks to @mvdbeek.) Pull Request 2146

• Fix bug when determining locale (thanks to @xiaojay.) Pull Request 2213

• Add pyparsing dependency of svgwrite. Unpin mercurial. (thanks to @nsoranzo.) Pull Request 2223

• Sanitize poorly escaped input value in run workflow form (thanks to @hexylena.) Pull Request 2231

• Multi-data collection reductions in repeat blocks are broken. Pull Request 2233

• Encode collection reduce in serializable fashion Pull Request 2241

• Cast everything to a string (thanks to @hexylena.) Pull Request 2252

• Use standard model.orm.now for toolshed’s password reset token datetime math Pull Request 2261

• fixed ui bugs in tours (thanks to @ThomasWollmann.) Pull Request 2262

• Preliminary fix for displaying raw text/html Pull Request 2263

• Correctly use defaults specified in galaxy.ini.sample for statsd. Pull Request 2268

• Fix encoding issues in test/unit/test_objectstore.py. Pull Request 2280

• Workflow sharing slug edit fix. Pull Request 2293

• Always remove the working directory after creating the capsule archive. Pull Request 2294

• update check for Conda env. (thanks to @matthdsm.) Pull Request 2295

• Make grunt watch (for scripts) work again, settings were being overwritten by the new watch-style command Pull Request 2296

• Monkeypatch Whoosh to use mkdtemp rather than a fixed temporary directory Pull Request 2310

• Build: remove redundant webpack step in makefile Pull Request 2327

• Unicodify has different semantics to str, causing bug reporter bug (thanks to @hexylena.) Pull Request 2336

• Wrap statsd timing middleware more closely around the core application Pull Request 2342

• UI & API: fix display of inaccessible datasets Pull Request 2351

• Update README to remove 2.6 (thanks to @remimarenco.) Pull Request 2367

• Revise asserts triggering false exceptions for dynamic parameters Pull Request 2374

• Fix the installed conda version to 3.19.3. (thanks to @bgruening.) Pull Request 2394

• Check variable’s presence before calling str methods (thanks to @hexylena.) Pull Request 2408

• UI, history: correctly handle index API defaults Pull Request 2413

• Fix to 4a61a4c: add hids for dataset inputs Pull Request 2415

• wrap bare str raises with generic Exception() Pull Request 2421

• Fix raven.js import order in galaxy.panels.mako Pull Request 2445

• Fix usage of super() in the mothur datatypes. (thanks to @bgruening.) Pull Request 2457

• Make pykube an optional dependency since we don’t have wheels for it. Pull Request 2468

• fix markup for button links on sharing template Pull Request 2481

• fix image generation for workflows with missing tools Pull Request 2488

• mothur datatypes: don’t generate error for pairwise distance matrices (thanks to @shiltemann.) Pull Request 2489

• Toolbox filter fixes Pull Request 2499

• Fix Post Job Action problem Pull Request 2505

• Copy PJAs when importing a workflow. Pull Request 2508

• Fix two errors in sff_extract.py detected by flake8 Pull Request 2509

• Fix HTML errors on 2 templates (thanks to @nsoranzo.) Pull Request 2524

• Fix tabular display chunking Pull Request 2527

• Fix dataset matcher when no tool available. (thanks to @zipho.) Pull Request 2540

• Initialize objectstore for fix_set_dataset_sizes script Pull Request 2544

• Fix old bug in non-allowed_images.yml GIEs (thanks to @hexylena.) Pull Request 2548

• More defensive GIE Proxying (thanks to @hexylena.) Pull Request 2552

• Fixes for showing min/max labels in Trackster. Pull Request 2555

• Fix tag_autocomplete issue in workflow editor. Pull Request 2569

• Fix element_identifier unavailable with data_collection input param (thanks to @abretaud.) Pull Request 2570

• Ignore W503 (line break before binary operator) as PEP 8 has reversed its position on this. Pull Request 2594

• Fix hex vs float parsing in request body Pull Request 2597

• Fixes for workflow extraction of mapping collection jobs. Pull Request 2625

• Workflow SVG generation fix Pull Request 2628

• Properly initialize datasets when workflow is loaded for execution Pull Request 2632

• typo in summary statistics tool (thanks to @nturaga.) Pull Request 2639

• Set allow_duplicate_entries=”False” for built-in Tool Data Tables. Pull Request 2644

• Ensure Galaxy’s python binary is used for metadata generation. Pull Request 2660

• Fix an error with getting the size of an object from Azure Pull Request 2678

• Limit creation of multitudes of default, empty histories. Pull Request 2684

• Fix the sanitization of None values in utils Pull Request 2693

• Fix metadata collection on workdir outputs for Pulsar. Pull Request 2721

• Allow link buttons to trigger regular click events Pull Request 2743

• Fix unused href parameter when adding scratchbook frames Pull Request 2744

• Update common_startup.sh to require python2.7 in .venv (thanks to @BenjaminHCCarr.) Pull Request 2756

• Separate styles of new and old libraries fixing various ui bugs Pull Request 2780

• Make conda install and resolver status more robust (thanks to @mvdbeek.) Pull Request 2789

• Smart string handling on the wsgi response – this fixes uWSGI blanking out when handling endpoints that return Unicode Pull Request 2630

• Do not instantiate the raven (sentry) client or tool conf watchdog threads until uWSGI postfork Pull Request 2792

• API, history contents: allow source in payload Pull Request 2809