July 2016 Galaxy Release (v 16.07)

Highlights

Shift of Galaxy tool dependencies to Conda: Galaxy admins obtained the ability to install Galaxy tool’s dependencies using the Conda package manager. This is a Beta feature and we encourage interested deployers to opt-in by modifying configuration. Documentation that explains this switch and answers FAQ is available.
Dynamic tool destinations: Our friends from Canada National Microbiology Laboratory enhanced Galaxy with feature that allows dynamic mapping of tools to destinations based on finer grained admin-specified rules. Please see the wiki. Implemented in PR #2579
Galaxy chat: Admins can now plug in the included communication server to enable users of their instance to use real-time chat within the Galaxy interface. Implemented in PR #2515

Github 

New Galaxy repository

$ git clone -b release_16.07 https://github.com/galaxyproject/galaxy.git

Update of existing Galaxy repository

$ git checkout release_16.07 && git pull --ff-only origin release_16.07

See our wiki for additional details regarding the source code locations.

Deprecation Notices

The following URLs will be removed and should be considered deprecated:

<galaxy>\history\as_xml
<galaxy>\history\list_as_xml

Release Notes

Highlighted Enhancements

Implement an Azure backend for the Galaxy objectstore (thanks to @zfrenchee.) Pull Request 2621
HTCondor Docker Integration (thanks to @bgruening.) Pull Request 2278
Implement a Kubernetes job runner (thanks to @pcm32.) Pull Request 2314, Pull Request 2498, Pull Request 2559, Pull Request 2578, Pull Request 2616, Pull Request 2528
Implement collection operations - tool-like entities allowing users to zip, unzip, flatten, and filter failed datasets out of collections. (You may need to sync your tool_conf.xml against sample.) Pull Request 2434
Allow history panel to display list of lists dataset collections. Pull Request 2613
Add a Tool Shed API endpoint method to check the current user. Pull Request 2243
Add a Galaxy API endpoint to update a user object. Pull Request 2595
Improvements to toolbox search. Pull Request 2273
New pages and general improvements to the Reports application (thanks to @BalthazarPavot.) Pull Request 2664
Nicer bug report emails (thanks to @hexylena.) Pull Request 2143
Tools with auto_format=’true’ now connectable in workflow editor (thanks to @yhoogstrate.) Pull Request 2144
Drop “Galaxy pip” in favor of manylinux1 wheels (which are supported in pip >= 8.1). Pull Request 2383
Allow tool form to listen to history (thanks to @nturaga.) Pull Request 2476
Add EDAM operations to tools and expose in API (thanks to @mvdbeek, @nsoranzo and @odoppelt.) Pull Request 2379
Add support to PDBQT molecule datatype (thanks to @leobiscassi.) Pull Request 2452
Add the trackhub composite datatype (thanks to @remimarenco.) Pull Request 2348, Pull Request 2646, Pull Request 2701
Add the Mothur datatypes (thanks to @shiltemann.) Pull Request 2038, Pull Request 2429
Add the dynamic tool destination tool to Galaxy for defining finer grained tool destination rules. (thanks to @ericenns.) Pull Request 2579
Galaxy can now install Conda dependencies in addition to tool shed dependencies during repository installations (thanks to @mvdbeek and @bwlang.) Pull Request 2554
Implement the Galaxy Communication Server (a simple in-Galaxy chat server) (thanks to @anuprulez and @bgruening.) Pull Request 2515

Enhancements

Install Conda at startup; enable conda_auto_init in sample conf Pull Request 2759
Add filter_failed_datasets_from_collection.py script to scripts/ (thanks to @chambm.) Pull Request 2358
Enhanced workflow run form using newer client forms architecture. Pull Request 2077
Tweaks to various library functions for galaxy-lib. Pull Request 2110, Pull Request 2494, Pull Request 2354, Pull Request 2414, Pull Request 2436
Automate and clarify more of the release process. Pull Request 2117, Pull Request 2366, Pull Request 2372, Pull Request 2388
Update template to include the release of the Docker container (thanks to @bgruening.) Pull Request 2378
Add server_starttime to configuration API. Pull Request 2124
Show line breaks in commit log in the Tool Shed (thanks to @nturaga.) Pull Request 2142
Replacing the _getIndex method by UnderscoreJS method _.findIndex() (thanks to @anuprulez.) Pull Request 2157
Small enhancements to user activation process. Pull Request 2158
Implement single_user = <email> config option. Pull Request 2165
More documentation and flexibility for FTP imports. Pull Request 2168
Make stats prefix configurable and add a conditional wheel requirement. Pull Request 2171
Fix running tool tests if remote user middleware is enabled. Pull Request 2173
Improve handling of a missing R environment during Tool Shed dependency installations. Pull Request 2215
Quote some parameters in the trim tool command (thanks to @nsoranzo.) Pull Request 2248
Remove overly chatty tool dependency resolution logging statement. Pull Request 2254
Update the list of contributors. Pull Request 2256, Pull Request 2257, Pull Request 2258
Add grunt watcher on .less files (thanks to @remimarenco.) Pull Request 2274
Remove dependency on custom Whoosh (requirements update). Pull Request 2276
Various enhancements for library functions related to tools. Pull Request 2279
Shift asbool handling in datatypes API so that an appropriate error message is now returned. Pull Request 2286
More small fixes from downstream projects. Pull Request 2292
For published workflow display, provide a better error when the workflow or user cannot be found Pull Request 2302
Bump sqlite3 version (thanks to @matthdsm.) Pull Request 2307
Show collection counts in collection and history collection views. Pull Request 2326
Various client library updates. Pull Request 2328
Revise form data visitor. Pull Request 2330
Update using xhr date header. Pull Request 2335
Implement an API for managing history/sharable model user sharing. Pull Request 2340
Revise portlet view, integrate collapsible option. Pull Request 2360
Fix issue with lowercasing in datatypes/registry.py (thanks to @remimarenco.) Pull Request 2361
Improve composite output tool test. Pull Request 2363
Upgrade shared ui libraries, add test cases Pull Request 2368
Throughout backend use open() instead of file() for Python 3 (thanks to @nsoranzo.) Pull Request 2370
Remove backported importlib and ordereddict. Pull Request 2380
Note on updating tool testing (thanks to @hexylena.) Pull Request 2381
Docker swarm support for Galaxy Interactive Environments (GIEs) Pull Request 2386, Pull Request 2392
Add support for EDAM data annotations to datatypes (thanks to @nsoranzo, @bgruening, @hmenager and @mvdbeek.) Pull Request 2387
Add a new workflow building mode variation which does access the history Pull Request 2390
Add conda search function to conda_util library function (thanks to @bgruening.) Pull Request 2395
Add history navigation options for dataset frames to scratch book Pull Request 2396
Add EDAM topics to tools and expose in API (thanks to @nsoranzo.) Pull Request 2397
Update SLURM runner so when scontrol fails, Galaxy attempts to get job state with sacct (thanks to @nsoranzo.) Pull Request 2403
Enhancements to tool test output verification. Pull Request 2409
Show history dataset id (hid) in dataset ‘view details’ page. Pull Request 2411
Updates and fixes for the run workflow UI elements. Pull Request 2416
Raise Exception if watch_tools has an unrecognized value (thanks to @nsoranzo.) Pull Request 2418
Use log.warning() instead of deprecated log.warn() (for Python 3) (thanks to @nsoranzo.) Pull Request 2420
Minor updates to documentation. Pull Request 2427
Correct various EDAM data/format annotations in datatypes (thanks to @hmenager.) Pull Request 2428
Fix $defaults according to #2430 (thanks to @ThomasWollmann.) Pull Request 2438
Properly list subworkflows in run workflow form. Pull Request 2440
Make some files compatible with Python3 (thanks to @nsoranzo.) Pull Request 2453
Add parameter expansion caller and test cases as preparation for workflow API extension. Pull Request 2464
Set a default for tool_dependencies_dir. Pull Request 2480
Add a password expiration config option (resolves #2485) (thanks to @benfulton.) Pull Request 2486
Display TIFF format in upload (thanks to @bgruening.) Pull Request 2516
Use self.decode_id instead of trans.security.decode_id (thanks to @nsoranzo.) Pull Request 2529
Charts revision. Pull Request 2531
Add Makefile target for fetching updated bower dependencies. Pull Request 2534
Make conda autoinstall of packages safer: (thanks to @pvanheus.) Pull Request 2538
Updates to Interactive Environment docs (thanks to @pvanheus.) Pull Request 2543
Allow displaying bigWig files in IGV (thanks to @dpryan79.) Pull Request 2545
Allow users to purge datasets by default. Pull Request 2553
Add Jupyter IE to the allowed visualizations for BAM datasets (thanks to @hexylena.) Pull Request 2572
Python3: port 215 files (thanks to @nsoranzo, @tshtatland, @xiaoqian1984, @alanryanwang and @ericenns.) Pull Request 2574
Add python-ldap module to conditional requirements (thanks to @abretaud.) Pull Request 2576
Update gsummary.xml (thanks to @lecorguille.) Pull Request 2590
Add Warning when Conda is installed (thanks to @remimarenco.) Pull Request 2596
Added the neostore (neo4j db) for noe4jdb dataset (thanks to @zipho.) Pull Request 2605
Fix smart_str for Python 3. Add doctests (thanks to @nsoranzo.) Pull Request 2608
Added Neo4j IE (thanks to @thobalose.) Pull Request 2610
Committing format changes to the user impersonation dropdown. (thanks to @sszakony.) Pull Request 2612
Use pagination and filtering for select2 fields Pull Request 2615
Update workflow share dropdown width (thanks to @sszakony.) Pull Request 2617
Graph visualization: add parsers, change datatypes Pull Request 2620
Fix certain combos of dynamic deeply nested collections in workflows. Pull Request 2634
add tar to datatypes_conf.xml (thanks to @mariabernard.) Pull Request 2636
Optimize get_file_peek() (thanks to @nsoranzo.) Pull Request 2637
Move workflow run and history options to top, add loading indicator Pull Request 2643
Enhance workflow run button to indicate workflow loading progress, keep header fixed Pull Request 2647
Allow uploading strict BED datatypes. Pull Request 2648
Reduce server sided refresh requests to changes in data associated input fields Pull Request 2652
Add import order linting using tox and flake8-import-order. Pull Request 2661
Add hg38 to main for ucsc_build_sites.txt.sample Pull Request 2670
Extend the list of files tested with flake8-import-order to all files already passing (thanks to @nsoranzo.) Pull Request 2682
List resolver Tool Shed packages (thanks to @mvdbeek.) Pull Request 2750

Remove buildbot remnants Pull Request 2125
Try pushing again when we get unexpected exception (test fix) Pull Request 2130
Refactor checking tool files toward reuse with workflows in Planemo. Pull Request 2183
Rename/refactor Utils.validate() to Utils.isEmpty() Pull Request 2239
Cleanup of the install and test framework. Pull Request 2249
cleanup TS and Reports configs Pull Request 2267
Move specific methods from shed_util_common.py to specific_util.py Pull Request 2269
Revise selection, options and drilldown wrappers, add tests Pull Request 2305
Visualizations: remove graphview Pull Request 2329
Update qunit and sinon Pull Request 2338
Build job success messages within tool form view Pull Request 2355
Cleanup of E203 primarily, and some other misc formatting Pull Request 2364
Isolate the data libraries css to affect only the libraries app Pull Request 2444
Re-add get_updated_changeset_revisions_from_tool_shed() method (thanks to @nsoranzo.) Pull Request 2472
Workaround for circular import. Refactor. Pull Request 2477
run_tests.sh: Clarify when/where the report file is created. (thanks to @NickSto.) Pull Request 2564
Reroute consistent workflow endpoint to internal API Pull Request 2622
Add tool test for what happens when output filter throws exception. Pull Request 2631
Ignore .tox and eggs directories for flake8 Pull Request 2650
Reorder base templates meta tags Pull Request 2651
Import pyparsing in testing so twill’s older version doesn’t get loaded. Pull Request 2656
Change default refresh mode for form views Pull Request 2667
Fix unit tests (thanks to @nsoranzo.) Pull Request 2691
Create a uWSGI postfork function registry and start the tool conf watcher thread post-fork Pull Request 2774

Fixes

Do not encode dict items that start with metadata\_ in return value of api/tools/create. Pull Request 2459
Generate metadata in separate conda environment (thanks to @mvdbeek.) Pull Request 2253
Fix to use “$NCPUS” if defined to set GALAXY_SLOTS (thanks to @mvdbeek.) Pull Request 2145

fix blocking issue check Pull Request 2113
Correctly check set_environment declarations when updating repository. Pull Request 2116
Hide hidden data parameters in workflow editor Pull Request 2120
Place $headers before integrity_check (thanks to @mvdbeek.) Pull Request 2146
Fix bug when determining locale (thanks to @xiaojay.) Pull Request 2213
Add pyparsing dependency of svgwrite. Unpin mercurial. (thanks to @nsoranzo.) Pull Request 2223
Sanitize poorly escaped input value in run workflow form (thanks to @hexylena.) Pull Request 2231
Multi-data collection reductions in repeat blocks are broken. Pull Request 2233
Encode collection reduce in serializable fashion Pull Request 2241
Cast everything to a string (thanks to @hexylena.) Pull Request 2252
Use standard model.orm.now for toolshed’s password reset token datetime math Pull Request 2261
fixed ui bugs in tours (thanks to @ThomasWollmann.) Pull Request 2262
Preliminary fix for displaying raw text/html Pull Request 2263
Correctly use defaults specified in galaxy.ini.sample for statsd. Pull Request 2268
Fix encoding issues in test/unit/test_objectstore.py. Pull Request 2280
Workflow sharing slug edit fix. Pull Request 2293
Always remove the working directory after creating the capsule archive. Pull Request 2294
update check for Conda env. (thanks to @matthdsm.) Pull Request 2295
Make grunt watch (for scripts) work again, settings were being overwritten by the new watch-style command Pull Request 2296
Monkeypatch Whoosh to use mkdtemp rather than a fixed temporary directory Pull Request 2310
Build: remove redundant webpack step in makefile Pull Request 2327
Unicodify has different semantics to str, causing bug reporter bug (thanks to @hexylena.) Pull Request 2336
Wrap statsd timing middleware more closely around the core application Pull Request 2342
UI & API: fix display of inaccessible datasets Pull Request 2351
Update README to remove 2.6 (thanks to @remimarenco.) Pull Request 2367
Revise asserts triggering false exceptions for dynamic parameters Pull Request 2374
Fix the installed conda version to 3.19.3. (thanks to @bgruening.) Pull Request 2394
Check variable’s presence before calling str methods (thanks to @hexylena.) Pull Request 2408
UI, history: correctly handle index API defaults Pull Request 2413
Fix to 4a61a4c: add hids for dataset inputs Pull Request 2415
wrap bare str raises with generic Exception() Pull Request 2421
Fix raven.js import order in galaxy.panels.mako Pull Request 2445
Fix usage of super() in the mothur datatypes. (thanks to @bgruening.) Pull Request 2457
Make pykube an optional dependency since we don’t have wheels for it. Pull Request 2468
fix markup for button links on sharing template Pull Request 2481
fix image generation for workflows with missing tools Pull Request 2488
mothur datatypes: don’t generate error for pairwise distance matrices (thanks to @shiltemann.) Pull Request 2489
Toolbox filter fixes Pull Request 2499
Fix Post Job Action problem Pull Request 2505
Copy PJAs when importing a workflow. Pull Request 2508
Fix two errors in sff_extract.py detected by flake8 Pull Request 2509
Fix HTML errors on 2 templates (thanks to @nsoranzo.) Pull Request 2524
Fix tabular display chunking Pull Request 2527
Fix dataset matcher when no tool available. (thanks to @zipho.) Pull Request 2540
Initialize objectstore for fix_set_dataset_sizes script Pull Request 2544
Fix old bug in non-allowed_images.yml GIEs (thanks to @hexylena.) Pull Request 2548
More defensive GIE Proxying (thanks to @hexylena.) Pull Request 2552
Fixes for showing min/max labels in Trackster. Pull Request 2555
Fix tag_autocomplete issue in workflow editor. Pull Request 2569
Fix element_identifier unavailable with data_collection input param (thanks to @abretaud.) Pull Request 2570
Ignore W503 (line break before binary operator) as PEP 8 has reversed its position on this. Pull Request 2594
Fix hex vs float parsing in request body Pull Request 2597
Fixes for workflow extraction of mapping collection jobs. Pull Request 2625
Workflow SVG generation fix Pull Request 2628
Properly initialize datasets when workflow is loaded for execution Pull Request 2632
typo in summary statistics tool (thanks to @nturaga.) Pull Request 2639
Set allow_duplicate_entries=”False” for built-in Tool Data Tables. Pull Request 2644
Ensure Galaxy’s python binary is used for metadata generation. Pull Request 2660
Fix an error with getting the size of an object from Azure Pull Request 2678
Limit creation of multitudes of default, empty histories. Pull Request 2684
Fix the sanitization of None values in utils Pull Request 2693
Fix metadata collection on workdir outputs for Pulsar. Pull Request 2721
Allow link buttons to trigger regular click events Pull Request 2743
Fix unused href parameter when adding scratchbook frames Pull Request 2744
Update common_startup.sh to require python2.7 in .venv (thanks to @BenjaminHCCarr.) Pull Request 2756
Separate styles of new and old libraries fixing various ui bugs Pull Request 2780
Make conda install and resolver status more robust (thanks to @mvdbeek.) Pull Request 2789
Smart string handling on the wsgi response – this fixes uWSGI blanking out when handling endpoints that return Unicode Pull Request 2630
Do not instantiate the raven (sentry) client or tool conf watchdog threads until uWSGI postfork Pull Request 2792
API, history contents: allow source in payload Pull Request 2809

To stay up to date with Galaxy’s progress, watch our screencasts; visit our community Hub; and follow us on Bluesky, Mastodon, and LinkedIn.

You can always chat with us on Matrix.

Thanks for using Galaxy!

The Galaxy Team