3.6.0

• New Toil landing page at http://bd2kgenomics.github.io/toil/
• Tags all Toil-provisioned instances with an 'Owner' tag to improve cost tracking
• Allows users to arbitrarily tag their Toil-provisioned instances
• Adds a new Torque batch system class
• Improved documentation
• Support for piping commands within DockerCall
• Improved cluster usage statistics
• Various bug fixes improving support for large file exporting, the Slurm batch system, and general Toil robustness

3.5.1

Hotfix release for 3.5.x

• Fixes import error raised when 'AWS' extra not installed

3.5.0

• Toil now includes a native EC2 provisioner for launching and controlling Toil clusters.
• Toil now has the ability to autoscale - if enabled, Toil will increase or decrease the number of worker nodes in its cluster depending on the current demands of the workflow
• Log output now refers to each Job by name.

3.3.4

https://pypi.python.org/pypi?:action=display&name=toil&version=3.3.4

• Fix: Toil fails completely when not installed with [aws] extra (#1115)
• Prevent virtualenv from silently upgrading pip and setuptools, potentially breaking build (#1270)

Core production is advised to skip this release.

3.3.3

• Fix --restart not working at all on distributed clusters (resolves #1127)
• Work around apparent eventual consistency with SDB (resolves #760)

This release is recommended to replace 3.3.1 and 3.3.0 for production use.

CGCloud 1.5.5 was released to deploy this release of Toil on toil-box images/instances.

The 3.3.2 release was invalidated due a error late in the release process.

3.4.0a1.dev228

This pre-release of Toil contains bug fixes and the addition of experimental, CGCloud-based auto-scaling.

https://github.com/BD2KGenomics/toil/milestone/17?closed=1

Due to the experimental nature of the auto-scaling feature, we decided to skip publishing a stable release for it. To install this release with pip, use pip install toil==3.4.0a1.dev228.

The next stable release of Toil will contain docker-based autoscaling for EC2.

3.3.1

This hotfix release adds disk tracking to the single machine batch system (#770). Previously, that batch system ignored any disk requirements associated with jobs. The fix was contributed by @arkal.

3.3.0

This major release adds bug fixes and a minor feature, specifically:

• toil clean is more robust and can clean partially corrupt job stores (#869 et al)
• job.fileStore.writeGlobalFIle(…) returns a magic string with a .size attribute reflecting the size of the file just having been written. In conjunction with promised requirements, this small change greatly simplifies dynamic computation of job requirements as a function of the size of another job's output (#1014)
• Two race conditions in the caching layer are fixed (#1046 and #1033)
• A show-stopping race condition with promised requirements was fixed (#1039)
• Several other bug fixes and code quality improvements
• The undocumented cache= job requirement has been removed

3.2.1

This major release adds many new features, improvements and bug fixes. It is the new stable release and users are encouraged to upgrade to it. Support for the previous stable release, now considered legacy, will continue but be limited to security fixes.

The important new features are:

• Completely rewritten caching allows for sharing of files between worker processes running on the same worker node. Previously, only a chain of jobs running inside a single worker process would benefit from caching.
• Improved ability to run long-running jobs that provide services to other, regular jobs in the workflow.
• Building on top of the improved service jobs feature, we added support for ephemeral Spark sub-clusters that are formed on-demand during a Toil workflow and can be used by jobs to perform ADAM or other Spark workloads.
• Support for hosting job stores in Google Cloud Storage was added. Install Toil with the google extra to enable this support. The Google job store should be considered experimental. The Toil team is working on gaining more experience on the consistency and atomicity guarantees provided by Google Cloud Storage to ready that job store implementation for production use.
• Support for hosting job stores in Azure Storage was demoted to experimental status due to atomicity and consistency issues. The Toil team will be working closely with Microsoft engineers to resolve these.
• Added ability to import and export files into a job store on both the leader prior/after to invoking a workflow and on the workers as part of a workflow. Importing files from http://, https:// and file:// URLs is supported out of the box. So is exporting to file:// URLs. Installing Toil with any or all of the azure, aws or google extras extends the import/export facility to files hosted on Azure Blob Storage, S3 and Google Cloud Storage, respectively.
• Toil scripts can now have dependencies and those dependencies are deployed automatically to each worker node. The script and the dependencies must be installed in a virtualenv and must be pure-Python. Native dependencies must be installed by the user on each worker prior to running a workflow. The Toil team uses the CGCloud project to handle those native dependencies.
• CWL support has matured.

3.2.0

This release contains critical bugs (#1005 and #1007) and should not be used.