July 2016 Galaxy Release (v 16.07)

Highlights

Shift of Galaxy tool dependencies to Conda

Galaxy admins obtained the ability to install Galaxy tool’s dependencies using the Conda package manager. This is a Beta feature and we encourage interested deployers to opt-in by modifying configuration. Documentation that explains this switch and answers FAQ is available.

Dynamic tool destinations

Our friends from Canada National Microbiology Laboratory enhanced Galaxy with feature that allows dynamic mapping of tools to destinations based on finer grained admin-specified rules. Please see the wiki. Implemented in PR #2579

Galaxy chat

Admins can now plug in the included communication server to enable users of their instance to use real-time chat within the Galaxy interface. Implemented in PR #2515

Github

New Galaxy repository

$ git clone -b release_16.07 https://github.com/galaxyproject/galaxy.git

Update of existing Galaxy repository

$ git checkout release_16.07 && git pull --ff-only origin release_16.07

See our wiki for additional details regarding the source code locations.

Deprecation Notices

The following URLs will be removed and should be considered deprecated:

• <galaxy>\history\as_xml

• <galaxy>\history\list_as_xml

Release Notes

Highlighted Enhancements

• Implement an Azure backend for the Galaxy objectstore (thanks to @zfrenchee.) Pull Request 2621

• HTCondor Docker Integration (thanks to @bgruening.) Pull Request 2278

• Implement a Kubernetes job runner (thanks to @pcm32.) Pull Request 2314, Pull Request 2498, Pull Request 2559, Pull Request 2578, Pull Request 2616, Pull Request 2528

• Implement collection operations - tool-like entities allowing users to zip, unzip, flatten, and filter failed datasets out of collections. (You may need to sync your tool_conf.xml against sample.) Pull Request 2434

• Allow history panel to display list of lists dataset collections. Pull Request 2613

• Add a Tool Shed API endpoint method to check the current user. Pull Request 2243

• Add a Galaxy API endpoint to update a user object. Pull Request 2595

• Improvements to toolbox search. Pull Request 2273

• New pages and general improvements to the Reports application (thanks to @BalthazarPavot.) Pull Request 2664

• Nicer bug report emails (thanks to @hexylena.) Pull Request 2143

• Tools with auto_format=’true’ now connectable in workflow editor (thanks to @yhoogstrate.) Pull Request 2144

• Drop “Galaxy pip” in favor of manylinux1 wheels (which are supported in pip >= 8.1). Pull Request 2383

• Allow tool form to listen to history (thanks to @nturaga.) Pull Request 2476

• Add EDAM operations to tools and expose in API (thanks to @mvdbeek, @nsoranzo and @odoppelt.) Pull Request 2379

• Add support to PDBQT molecule datatype (thanks to @leobiscassi.) Pull Request 2452

• Add the trackhub composite datatype (thanks to @remimarenco.) Pull Request 2348, Pull Request 2646, Pull Request 2701

• Add the Mothur datatypes (thanks to @shiltemann.) Pull Request 2038, Pull Request 2429

• Add the dynamic tool destination tool to Galaxy for defining finer grained tool destination rules. (thanks to @ericenns.) Pull Request 2579

• Galaxy can now install Conda dependencies in addition to tool shed dependencies during repository installations (thanks to @mvdbeek and @bwlang.) Pull Request 2554

• Implement the Galaxy Communication Server (a simple in-Galaxy chat server) (thanks to @anuprulez and @bgruening.) Pull Request 2515

Enhancements

• Install Conda at startup; enable conda_auto_init in sample conf Pull Request 2759

• Add filter_failed_datasets_from_collection.py script to scripts/ (thanks to @chambm.) Pull Request 2358

• Enhanced workflow run form using newer client forms architecture. Pull Request 2077

• Tweaks to various library functions for galaxy-lib. Pull Request 2110, Pull Request 2494, Pull Request 2354, Pull Request 2414, Pull Request 2436

• Automate and clarify more of the release process. Pull Request 2117, Pull Request 2366, Pull Request 2372, Pull Request 2388

• Update template to include the release of the Docker container (thanks to @bgruening.) Pull Request 2378

• Add server_starttime to configuration API. Pull Request 2124

• Show line breaks in commit log in the Tool Shed (thanks to @nturaga.) Pull Request 2142

• Replacing the _getIndex method by UnderscoreJS method _.findIndex() (thanks to @anuprulez.) Pull Request 2157

• Small enhancements to user activation process. Pull Request 2158

• Implement single_user = <email> config option. Pull Request 2165

• More documentation and flexibility for FTP imports. Pull Request 2168

• Make stats prefix configurable and add a conditional wheel requirement. Pull Request 2171

• Fix running tool tests if remote user middleware is enabled. Pull Request 2173

• Improve handling of a missing R environment during Tool Shed dependency installations. Pull Request 2215

• Quote some parameters in the trim tool command (thanks to @nsoranzo.) Pull Request 2248

• Remove overly chatty tool dependency resolution logging statement. Pull Request 2254

• Update the list of contributors. Pull Request 2256, Pull Request 2257, Pull Request 2258

• Add grunt watcher on .less files (thanks to @remimarenco.) Pull Request 2274

• Remove dependency on custom Whoosh (requirements update). Pull Request 2276

• Various enhancements for library functions related to tools. Pull Request 2279

• Shift asbool handling in datatypes API so that an appropriate error message is now returned. Pull Request 2286

• More small fixes from downstream projects. Pull Request 2292

• For published workflow display, provide a better error when the workflow or user cannot be found Pull Request 2302

• Bump sqlite3 version (thanks to @matthdsm.) Pull Request 2307

• Show collection counts in collection and history collection views. Pull Request 2326

• Various client library updates. Pull Request 2328

• Revise form data visitor. Pull Request 2330

• Update using xhr date header. Pull Request 2335

• Implement an API for managing history/sharable model user sharing. Pull Request 2340

• Revise portlet view, integrate collapsible option. Pull Request 2360

• Fix issue with lowercasing in datatypes/registry.py (thanks to @remimarenco.) Pull Request 2361

• Improve composite output tool test. Pull Request 2363

• Upgrade shared ui libraries, add test cases Pull Request 2368

• Throughout backend use open() instead of file() for Python 3 (thanks to @nsoranzo.) Pull Request 2370

• Remove backported importlib and ordereddict. Pull Request 2380

• Note on updating tool testing (thanks to @hexylena.) Pull Request 2381

• Docker swarm support for Galaxy Interactive Environments (GIEs) Pull Request 2386, Pull Request 2392

• Add support for EDAM data annotations to datatypes (thanks to @nsoranzo, @bgruening, @hmenager and @mvdbeek.) Pull Request 2387

• Add a new workflow building mode variation which does access the history Pull Request 2390

• Add conda search function to conda_util library function (thanks to @bgruening.) Pull Request 2395

• Add history navigation options for dataset frames to scratch book Pull Request 2396

• Add EDAM topics to tools and expose in API (thanks to @nsoranzo.) Pull Request 2397

• Update SLURM runner so when scontrol fails, Galaxy attempts to get job state with sacct (thanks to @nsoranzo.) Pull Request 2403

• Enhancements to tool test output verification. Pull Request 2409

• Show history dataset id (hid) in dataset ‘view details’ page. Pull Request 2411

• Updates and fixes for the run workflow UI elements. Pull Request 2416

• Raise Exception if watch_tools has an unrecognized value (thanks to @nsoranzo.) Pull Request 2418

• Use log.warning() instead of deprecated log.warn() (for Python 3) (thanks to @nsoranzo.) Pull Request 2420

• Minor updates to documentation. Pull Request 2427

• Correct various EDAM data/format annotations in datatypes (thanks to @hmenager.) Pull Request 2428

• Fix $defaults according to #2430 (thanks to @ThomasWollmann.) Pull Request 2438

• Properly list subworkflows in run workflow form. Pull Request 2440

• Make some files compatible with Python3 (thanks to @nsoranzo.) Pull Request 2453

• Add parameter expansion caller and test cases as preparation for workflow API extension. Pull Request 2464

• Set a default for tool_dependencies_dir. Pull Request 2480

• Add a password expiration config option (resolves #2485) (thanks to @benfulton.) Pull Request 2486

• Display TIFF format in upload (thanks to @bgruening.) Pull Request 2516

• Use self.decode_id instead of trans.security.decode_id (thanks to @nsoranzo.) Pull Request 2529

• Charts revision. Pull Request 2531

• Add Makefile target for fetching updated bower dependencies. Pull Request 2534

• Make conda autoinstall of packages safer: (thanks to @pvanheus.) Pull Request 2538

• Updates to Interactive Environment docs (thanks to @pvanheus.) Pull Request 2543

• Allow displaying bigWig files in IGV (thanks to @dpryan79.) Pull Request 2545

• Allow users to purge datasets by default. Pull Request 2553

• Add Jupyter IE to the allowed visualizations for BAM datasets (thanks to @hexylena.) Pull Request 2572

• Python3: port 215 files (thanks to @nsoranzo, @tshtatland, @xiaoqian1984, @alanryanwang and @ericenns.) Pull Request 2574

• Add python-ldap module to conditional requirements (thanks to @abretaud.) Pull Request 2576

• Update gsummary.xml (thanks to @lecorguille.) Pull Request 2590

• Add Warning when Conda is installed (thanks to @remimarenco.) Pull Request 2596

• Added the neostore (neo4j db) for noe4jdb dataset (thanks to @zipho.) Pull Request 2605

• Fix smart_str for Python 3. Add doctests (thanks to @nsoranzo.) Pull Request 2608

• Added Neo4j IE (thanks to @thobalose.) Pull Request 2610

• Committing format changes to the user impersonation dropdown. (thanks to @sszakony.) Pull Request 2612

• Use pagination and filtering for select2 fields Pull Request 2615

• Update workflow share dropdown width (thanks to @sszakony.) Pull Request 2617

• Graph visualization: add parsers, change datatypes Pull Request 2620

• Fix certain combos of dynamic deeply nested collections in workflows. Pull Request 2634

• add tar to datatypes_conf.xml (thanks to @mariabernard.) Pull Request 2636

• Optimize get_file_peek() (thanks to @nsoranzo.) Pull Request 2637

• Move workflow run and history options to top, add loading indicator Pull Request 2643

• Enhance workflow run button to indicate workflow loading progress, keep header fixed Pull Request 2647

• Allow uploading strict BED datatypes. Pull Request 2648

• Reduce server sided refresh requests to changes in data associated input fields Pull Request 2652

• Add import order linting using tox and flake8-import-order. Pull Request 2661

• Add hg38 to main for ucsc_build_sites.txt.sample Pull Request 2670

• Extend the list of files tested with flake8-import-order to all files already passing (thanks to @nsoranzo.) Pull Request 2682

• List resolver Tool Shed packages (thanks to @mvdbeek.) Pull Request 2750

• Remove buildbot remnants Pull Request 2125

• Try pushing again when we get unexpected exception (test fix) Pull Request 2130

• Refactor checking tool files toward reuse with workflows in Planemo. Pull Request 2183

• Rename/refactor Utils.validate() to Utils.isEmpty() Pull Request 2239

• Cleanup of the install and test framework. Pull Request 2249

• cleanup TS and Reports configs Pull Request 2267

• Move specific methods from shed_util_common.py to specific_util.py Pull Request 2269

• Revise selection, options and drilldown wrappers, add tests Pull Request 2305

• Visualizations: remove graphview Pull Request 2329

• Update qunit and sinon Pull Request 2338

• Build job success messages within tool form view Pull Request 2355

• Cleanup of E203 primarily, and some other misc formatting Pull Request 2364

• Isolate the data libraries css to affect only the libraries app Pull Request 2444

• Re-add get_updated_changeset_revisions_from_tool_shed() method (thanks to @nsoranzo.) Pull Request 2472

• Workaround for circular import. Refactor. Pull Request 2477

• run_tests.sh: Clarify when/where the report file is created. (thanks to @NickSto.) Pull Request 2564

• Reroute consistent workflow endpoint to internal API Pull Request 2622

• Add tool test for what happens when output filter throws exception. Pull Request 2631

• Ignore .tox and eggs directories for flake8 Pull Request 2650

• Reorder base templates meta tags Pull Request 2651

• Import pyparsing in testing so twill’s older version doesn’t get loaded. Pull Request 2656

• Change default refresh mode for form views Pull Request 2667

• Fix unit tests (thanks to @nsoranzo.) Pull Request 2691

• Create a uWSGI postfork function registry and start the tool conf watcher thread post-fork Pull Request 2774

Fixes

• Do not encode dict items that start with metadata\_ in return value of api/tools/create. Pull Request 2459

• Generate metadata in separate conda environment (thanks to @mvdbeek.) Pull Request 2253

• Fix to use “$NCPUS” if defined to set GALAXY_SLOTS (thanks to @mvdbeek.) Pull Request 2145

• fix blocking issue check Pull Request 2113

• Correctly check set_environment declarations when updating repository. Pull Request 2116

• Hide hidden data parameters in workflow editor Pull Request 2120

• Place $headers before integrity_check (thanks to @mvdbeek.) Pull Request 2146

• Fix bug when determining locale (thanks to @xiaojay.) Pull Request 2213

• Add pyparsing dependency of svgwrite. Unpin mercurial. (thanks to @nsoranzo.) Pull Request 2223

• Sanitize poorly escaped input value in run workflow form (thanks to @hexylena.) Pull Request 2231

• Multi-data collection reductions in repeat blocks are broken. Pull Request 2233

• Encode collection reduce in serializable fashion Pull Request 2241

• Cast everything to a string (thanks to @hexylena.) Pull Request 2252

• Use standard model.orm.now for toolshed’s password reset token datetime math Pull Request 2261

• fixed ui bugs in tours (thanks to @ThomasWollmann.) Pull Request 2262

• Preliminary fix for displaying raw text/html Pull Request 2263

• Correctly use defaults specified in galaxy.ini.sample for statsd. Pull Request 2268

• Fix encoding issues in test/unit/test_objectstore.py. Pull Request 2280

• Workflow sharing slug edit fix. Pull Request 2293

• Always remove the working directory after creating the capsule archive. Pull Request 2294

• update check for Conda env. (thanks to @matthdsm.) Pull Request 2295

• Make grunt watch (for scripts) work again, settings were being overwritten by the new watch-style command Pull Request 2296

• Monkeypatch Whoosh to use mkdtemp rather than a fixed temporary directory Pull Request 2310

• Build: remove redundant webpack step in makefile Pull Request 2327

• Unicodify has different semantics to str, causing bug reporter bug (thanks to @hexylena.) Pull Request 2336

• Wrap statsd timing middleware more closely around the core application Pull Request 2342

• UI & API: fix display of inaccessible datasets Pull Request 2351

• Update README to remove 2.6 (thanks to @remimarenco.) Pull Request 2367

• Revise asserts triggering false exceptions for dynamic parameters Pull Request 2374

• Fix the installed conda version to 3.19.3. (thanks to @bgruening.) Pull Request 2394

• Check variable’s presence before calling str methods (thanks to @hexylena.) Pull Request 2408

• UI, history: correctly handle index API defaults Pull Request 2413

• Fix to 4a61a4c: add hids for dataset inputs Pull Request 2415

• wrap bare str raises with generic Exception() Pull Request 2421

• Fix raven.js import order in galaxy.panels.mako Pull Request 2445

• Fix usage of super() in the mothur datatypes. (thanks to @bgruening.) Pull Request 2457

• Make pykube an optional dependency since we don’t have wheels for it. Pull Request 2468

• fix markup for button links on sharing template Pull Request 2481

• fix image generation for workflows with missing tools Pull Request 2488

• mothur datatypes: don’t generate error for pairwise distance matrices (thanks to @shiltemann.) Pull Request 2489

• Toolbox filter fixes Pull Request 2499

• Fix Post Job Action problem Pull Request 2505

• Copy PJAs when importing a workflow. Pull Request 2508

• Fix two errors in sff_extract.py detected by flake8 Pull Request 2509

• Fix HTML errors on 2 templates (thanks to @nsoranzo.) Pull Request 2524

• Fix tabular display chunking Pull Request 2527

• Fix dataset matcher when no tool available. (thanks to @zipho.) Pull Request 2540

• Initialize objectstore for fix_set_dataset_sizes script Pull Request 2544

• Fix old bug in non-allowed_images.yml GIEs (thanks to @hexylena.) Pull Request 2548

• More defensive GIE Proxying (thanks to @hexylena.) Pull Request 2552

• Fixes for showing min/max labels in Trackster. Pull Request 2555

• Fix tag_autocomplete issue in workflow editor. Pull Request 2569

• Fix element_identifier unavailable with data_collection input param (thanks to @abretaud.) Pull Request 2570

• Ignore W503 (line break before binary operator) as PEP 8 has reversed its position on this. Pull Request 2594

• Fix hex vs float parsing in request body Pull Request 2597

• Fixes for workflow extraction of mapping collection jobs. Pull Request 2625

• Workflow SVG generation fix Pull Request 2628

• Properly initialize datasets when workflow is loaded for execution Pull Request 2632

• typo in summary statistics tool (thanks to @nturaga.) Pull Request 2639

• Set allow_duplicate_entries=”False” for built-in Tool Data Tables. Pull Request 2644

• Ensure Galaxy’s python binary is used for metadata generation. Pull Request 2660

• Fix an error with getting the size of an object from Azure Pull Request 2678

• Limit creation of multitudes of default, empty histories. Pull Request 2684

• Fix the sanitization of None values in utils Pull Request 2693

• Fix metadata collection on workdir outputs for Pulsar. Pull Request 2721

• Allow link buttons to trigger regular click events Pull Request 2743

• Fix unused href parameter when adding scratchbook frames Pull Request 2744

• Update common_startup.sh to require python2.7 in .venv (thanks to @BenjaminHCCarr.) Pull Request 2756

• Separate styles of new and old libraries fixing various ui bugs Pull Request 2780

• Make conda install and resolver status more robust (thanks to @mvdbeek.) Pull Request 2789

• Smart string handling on the wsgi response – this fixes uWSGI blanking out when handling endpoints that return Unicode Pull Request 2630

• Do not instantiate the raven (sentry) client or tool conf watchdog threads until uWSGI postfork Pull Request 2792

• API, history contents: allow source in payload Pull Request 2809

To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.

You can always chat with us on Matrix.

Thanks for using Galaxy!

The Galaxy Team