'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "TOIL" "1" "Apr 29, 2024" "6.1.0" "Toil" .SH NAME toil \- Toil Documentation .sp Toil is an open\-source pure\-Python workflow engine that lets people write better pipelines. .sp Check out our \fI\%website\fP for a comprehensive list of Toil\(aqs features and read our \fI\%paper\fP to learn what Toil can do in the real world. Please subscribe to our low\-volume \fI\%announce\fP mailing list and feel free to also join us on \fI\%GitHub\fP and \fI\%Gitter\fP\&. .sp If using Toil for your research, please cite .INDENT 0.0 .INDENT 3.5 Vivian, J., Rao, A. A., Nothaft, F. A., Ketchum, C., Armstrong, J., Novak, A., … Paten, B. (2017). Toil enables reproducible, open source, big biomedical data analyses. Nature Biotechnology, 35(4), 314–316. \fI\%http://doi.org/10.1038/nbt.3772\fP .UNINDENT .UNINDENT .SH QUICKSTART EXAMPLES .SS Running a basic CWL workflow .sp The \fI\%Common Workflow Language\fP (CWL) is an emerging standard for writing workflows that are portable across multiple workflow engines and platforms. Running CWL workflows using Toil is easy. .INDENT 0.0 .IP 1. 3 Copy and paste the following code block into \fBexample.cwl\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX cwlVersion: v1.0 class: CommandLineTool baseCommand: echo stdout: output.txt inputs: message: type: string inputBinding: position: 1 outputs: output: type: stdout .EE .UNINDENT .UNINDENT .sp and this code into \fBexample\-job.yaml\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX message: Hello world! .EE .UNINDENT .UNINDENT .IP 2. 3 To run the workflow simply enter .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil\-cwl\-runner example.cwl example\-job.yaml .EE .UNINDENT .UNINDENT .sp Your output will be in \fBoutput.txt\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX $ cat output.txt Hello world! .EE .UNINDENT .UNINDENT .UNINDENT .sp Congratulations! You\(aqve run your first Toil workflow using the default \fI\%Batch System\fP, \fBsingle_machine\fP, and the default \fBfile\fP job store (which was placed in a temporary directory for you by \fBtoil\-cwl\-runner\fP). .sp Toil uses batch systems to manage the jobs it creates. .sp The \fBsingle_machine\fP batch system is primarily used to prepare and debug workflows on a local machine. Once validated, try running them on a full\-fledged batch system (see \fI\%Batch System API\fP). Toil supports many different batch systems such as \fI\%Kubernetes\fP and Grid Engine; its versatility makes it easy to run your workflow in all kinds of places. .sp Toil\(aqs CWL runner is totally customizable! Run \fBtoil\-cwl\-runner \-\-help\fP to see a complete list of available options. .sp To learn more about CWL, see the \fI\%CWL User Guide\fP (from where this example was shamelessly borrowed). For information on using CWL with Toil see the section \fI\%CWL in Toil\fP\&. And for an example of CWL on an AWS cluster, have a look at \fI\%Running a CWL Workflow on AWS\fP\&. .SS Running a basic WDL workflow .sp The \fI\%Workflow Description Language\fP (WDL) is another emerging language for writing workflows that are portable across multiple workflow engines and platforms. Running WDL workflows using Toil is still in alpha, and currently experimental. Toil currently supports basic workflow syntax (see \fI\%WDL in Toil\fP for more details and examples). Here we go over running a basic WDL helloworld workflow. .INDENT 0.0 .IP 1. 3 Copy and paste the following code block into \fBwdl\-helloworld.wdl\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX workflow write_simple_file { call write_file } task write_file { String message command { echo ${message} > wdl\-helloworld\-output.txt } output { File test = \(dqwdl\-helloworld\-output.txt\(dq } } .EE .UNINDENT .UNINDENT .sp and this code into \fBwdl\-helloworld.json\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX { \(dqwrite_simple_file.write_file.message\(dq: \(dqHello world!\(dq } .EE .UNINDENT .UNINDENT .IP 2. 3 To run the workflow simply enter .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil\-wdl\-runner wdl\-helloworld.wdl wdl\-helloworld.json .EE .UNINDENT .UNINDENT .sp Your output will be in \fBwdl\-helloworld\-output.txt\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX $ cat wdl\-helloworld\-output.txt Hello world! .EE .UNINDENT .UNINDENT .UNINDENT .sp This will, like the CWL example above, use the \fBsingle_machine\fP batch system and an automatically\-located \fBfile\fP job store by default. You can customize Toil\(aqs execution of the workflow with command\-line options; run \fBtoil\-wdl\-runner \-\-help\fP to learn about them. .sp To learn more about WDL in general, see the \fI\%Terra WDL documentation\fP . For more on using WDL in Toil, see \fI\%WDL in Toil\fP\&. .SS Running a basic Python workflow .sp In addition to workflow languages like CWL and WDL, Toil supports running workflows written against its Python API. .sp An example Toil Python workflow can be run with just three steps: .INDENT 0.0 .IP 1. 3 Install Toil (see \fI\%Installation\fP) .IP 2. 3 Copy and paste the following code block into a new file called \fBhelloWorld.py\fP: .UNINDENT .INDENT 0.0 .INDENT 3.5 .sp .EX from toil.common import Toil from toil.job import Job def helloWorld(message, memory=\(dq1G\(dq, cores=1, disk=\(dq1G\(dq): return f\(dqHello, world!, here\(aqs a message: {message}\(dq if __name__ == \(dq__main__\(dq: parser = Job.Runner.getDefaultArgumentParser() options = parser.parse_args() options.clean = \(dqalways\(dq with Toil(options) as toil: output = toil.start(Job.wrapFn(helloWorld, \(dqYou did it!\(dq)) print(output) .EE .UNINDENT .UNINDENT .INDENT 0.0 .IP 3. 3 Specify the name of the \fI\%job store\fP and run the workflow: .INDENT 3.0 .INDENT 3.5 .sp .EX $ python3 helloWorld.py file:my\-job\-store .EE .UNINDENT .UNINDENT .UNINDENT .sp For something beyond a \(dqHello, world!\(dq example, refer to \fI\%A (more) real\-world example\fP\&. .sp Toil\(aqs customization options are available in Python workflows. Run \fBpython3 helloWorld.py \-\-help\fP to see a complete list of available options. .SS A (more) real\-world example .sp For a more detailed example and explanation, we\(aqve developed a sample pipeline that merge\-sorts a temporary file. This is not supposed to be an efficient sorting program, rather a more fully worked example of what Toil is capable of. .SS Running the example .INDENT 0.0 .IP 1. 3 Download \fBthe example code\fP .IP 2. 3 Run it with the default settings: .INDENT 3.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore .EE .UNINDENT .UNINDENT .sp The workflow created a file called \fBsortedFile.txt\fP in your current directory. Have a look at it and notice that it contains a whole lot of sorted lines! .sp This workflow does a smart merge sort on a file it generates, \fBfileToSort.txt\fP\&. The sort is \fIsmart\fP because each step of the process\-\-\-splitting the file into separate chunks, sorting these chunks, and merging them back together\-\-\-is compartmentalized into a \fBjob\fP\&. Each job can specify its own resource requirements and will only be run after the jobs it depends upon have run. Jobs without dependencies will be run in parallel. .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Delete \fBfileToSort.txt\fP before moving on to #3. This example introduces options that specify dimensions for \fBfileToSort.txt\fP, if it does not already exist. If it exists, this workflow will use the existing file and the results will be the same as #2. .UNINDENT .UNINDENT .INDENT 0.0 .IP 3. 3 Run with custom options: .INDENT 3.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore \e \-\-numLines=5000 \e \-\-lineLength=10 \e \-\-overwriteOutput=True \e \-\-workDir=/tmp/ .EE .UNINDENT .UNINDENT .sp Here we see that we can add our own options to a Toil Python workflow. As noted above, the first two options, \fB\-\-numLines\fP and \fB\-\-lineLength\fP, determine the number of lines and how many characters are in each line. \fB\-\-overwriteOutput\fP causes the current contents of \fBsortedFile.txt\fP to be overwritten, if it already exists. The last option, \fB\-\-workDir\fP, is an option built into Toil to specify where temporary files unique to a job are kept. .UNINDENT .SS Describing the source code .sp To understand the details of what\(aqs going on inside. Let\(aqs start with the \fBmain()\fP function. It looks like a lot of code, but don\(aqt worry\-\-\-we\(aqll break it down piece by piece. .INDENT 0.0 .INDENT 3.5 .sp .EX def main(options=None): if not options: # deal with command line arguments parser = ArgumentParser() Job.Runner.addToilOptions(parser) parser.add_argument(\(aq\-\-numLines\(aq, default=defaultLines, help=\(aqNumber of lines in file to sort.\(aq, type=int) parser.add_argument(\(aq\-\-lineLength\(aq, default=defaultLineLen, help=\(aqLength of lines in file to sort.\(aq, type=int) parser.add_argument(\(dq\-\-fileToSort\(dq, help=\(dqThe file you wish to sort\(dq) parser.add_argument(\(dq\-\-outputFile\(dq, help=\(dqWhere the sorted output will go\(dq) parser.add_argument(\(dq\-\-overwriteOutput\(dq, help=\(dqWrite over the output file if it already exists.\(dq, default=True) parser.add_argument(\(dq\-\-N\(dq, dest=\(dqN\(dq, help=\(dqThe threshold below which a serial sort function is used to sort file. \(dq \(dqAll lines must of length less than or equal to N or program will fail\(dq, default=10000) parser.add_argument(\(aq\-\-downCheckpoints\(aq, action=\(aqstore_true\(aq, help=\(aqIf this option is set, the workflow will make checkpoints on its way through\(aq \(aqthe recursive \(dqdown\(dq part of the sort\(aq) parser.add_argument(\(dq\-\-sortMemory\(dq, dest=\(dqsortMemory\(dq, help=\(dqMemory for jobs that sort chunks of the file.\(dq, default=None) parser.add_argument(\(dq\-\-mergeMemory\(dq, dest=\(dqmergeMemory\(dq, help=\(dqMemory for jobs that collate results.\(dq, default=None) options = parser.parse_args() if not hasattr(options, \(dqsortMemory\(dq) or not options.sortMemory: options.sortMemory = sortMemory if not hasattr(options, \(dqmergeMemory\(dq) or not options.mergeMemory: options.mergeMemory = sortMemory # do some input verification sortedFileName = options.outputFile or \(dqsortedFile.txt\(dq if not options.overwriteOutput and os.path.exists(sortedFileName): print(f\(aqOutput file {sortedFileName} already exists. \(aq f\(aqDelete it to run the sort example again or use \-\-overwriteOutput=True\(aq) exit() fileName = options.fileToSort if options.fileToSort is None: # make the file ourselves fileName = \(aqfileToSort.txt\(aq if os.path.exists(fileName): print(f\(aqSorting existing file: {fileName}\(aq) else: print(f\(aqNo sort file specified. Generating one automatically called: {fileName}.\(aq) makeFileToSort(fileName=fileName, lines=options.numLines, lineLen=options.lineLength) else: if not os.path.exists(options.fileToSort): raise RuntimeError(\(dqFile to sort does not exist: %s\(dq % options.fileToSort) if int(options.N) <= 0: raise RuntimeError(\(dqInvalid value of N: %s\(dq % options.N) # Now we are ready to run with Toil(options) as workflow: sortedFileURL = \(aqfile://\(aq + os.path.abspath(sortedFileName) if not workflow.options.restart: sortFileURL = \(aqfile://\(aq + os.path.abspath(fileName) sortFileID = workflow.importFile(sortFileURL) sortedFileID = workflow.start(Job.wrapJobFn(setup, sortFileID, int(options.N), options.downCheckpoints, options=options, memory=sortMemory)) else: sortedFileID = workflow.restart() workflow.exportFile(sortedFileID, sortedFileURL) .EE .UNINDENT .UNINDENT .sp First we make a parser to process command line arguments using the \fI\%argparse\fP module. It\(aqs important that we add the call to \fBJob.Runner.addToilOptions()\fP to initialize our parser with all of Toil\(aqs default options. Then we add the command line arguments unique to this workflow, and parse the input. The help message listed with the arguments should give you a pretty good idea of what they can do. .sp Next we do a little bit of verification of the input arguments. The option \fB\-\-fileToSort\fP allows you to specify a file that needs to be sorted. If this option isn\(aqt given, it\(aqs here that we make our own file with the call to \fBmakeFileToSort()\fP\&. .sp Finally we come to the context manager that initializes the workflow. We create a path to the input file prepended with \fB\(aqfile://\(aq\fP as per the documentation for \fI\%toil.common.Toil()\fP when staging a file that is stored locally. Notice that we have to check whether or not the workflow is restarting so that we don\(aqt import the file more than once. Finally we can kick off the workflow by calling \fI\%toil.common.Toil.start()\fP on the job \fBsetup\fP\&. When the workflow ends we capture its output (the sorted file\(aqs fileID) and use that in \fI\%toil.common.Toil.exportFile()\fP to move the sorted file from the job store back into \(dquserland\(dq. .sp Next let\(aqs look at the job that begins the actual workflow, \fBsetup\fP\&. .INDENT 0.0 .INDENT 3.5 .sp .EX def setup(job, inputFile, N, downCheckpoints, options): \(dq\(dq\(dq Sets up the sort. Returns the FileID of the sorted file \(dq\(dq\(dq RealtimeLogger.info(\(dqStarting the merge sort\(dq) return job.addChildJobFn(down, inputFile, N, \(aqroot\(aq, downCheckpoints, options = options, preemptible=True, memory=sortMemory).rv() .EE .UNINDENT .UNINDENT .sp \fBsetup\fP really only does two things. First it writes to the logs using \fBJob.log()\fP and then calls \fBaddChildJobFn()\fP\&. Child jobs run directly after the current job. This function turns the \(aqjob function\(aq \fBdown\fP into an actual job and passes in the inputs including an optional resource requirement, \fBmemory\fP\&. The job doesn\(aqt actually get run until the call to \fBJob.rv()\fP\&. Once the job \fBdown\fP finishes, its output is returned here. .sp Now we can look at what \fBdown\fP does. .INDENT 0.0 .INDENT 3.5 .sp .EX def down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory): \(dq\(dq\(dq Input is a file, a subdivision size N, and a path in the hierarchy of jobs. If the range is larger than a threshold N the range is divided recursively and a follow on job is then created which merges back the results else the file is sorted and placed in the output. \(dq\(dq\(dq RealtimeLogger.info(\(dqDown job starting: %s\(dq % path) # Read the file inputFile = job.fileStore.readGlobalFile(inputFileStoreID, cache=False) length = os.path.getsize(inputFile) if length > N: # We will subdivide the file RealtimeLogger.critical(\(dqSplitting file: %s of size: %s\(dq % (inputFileStoreID, length)) # Split the file into two copies midPoint = getMidPoint(inputFile, 0, length) t1 = job.fileStore.getLocalTempFile() with open(t1, \(aqw\(aq) as fH: fH.write(copySubRangeOfFile(inputFile, 0, midPoint+1)) t2 = job.fileStore.getLocalTempFile() with open(t2, \(aqw\(aq) as fH: fH.write(copySubRangeOfFile(inputFile, midPoint+1, length)) # Call down recursively. By giving the rv() of the two jobs as inputs to the follow\-on job, up, # we communicate the dependency without hindering concurrency. result = job.addFollowOnJobFn(up, job.addChildJobFn(down, job.fileStore.writeGlobalFile(t1), N, path + \(aq/0\(aq, downCheckpoints, checkpoint=downCheckpoints, options=options, preemptible=True, memory=options.sortMemory).rv(), job.addChildJobFn(down, job.fileStore.writeGlobalFile(t2), N, path + \(aq/1\(aq, downCheckpoints, checkpoint=downCheckpoints, options=options, preemptible=True, memory=options.mergeMemory).rv(), path + \(aq/up\(aq, preemptible=True, options=options, memory=options.sortMemory).rv() else: # We can sort this bit of the file RealtimeLogger.critical(\(dqSorting file: %s of size: %s\(dq % (inputFileStoreID, length)) # Sort the copy and write back to the fileStore shutil.copyfile(inputFile, inputFile + \(aq.sort\(aq) sort(inputFile + \(aq.sort\(aq) result = job.fileStore.writeGlobalFile(inputFile + \(aq.sort\(aq) RealtimeLogger.info(\(dqDown job finished: %s\(dq % path) return result .EE .UNINDENT .UNINDENT .sp Down is the recursive part of the workflow. First we read the file into the local filestore by calling \fBjob.fileStore.readGlobalFile()\fP\&. This puts a copy of the file in the temp directory for this particular job. This storage will disappear once this job ends. For a detailed explanation of the filestore, job store, and their interfaces have a look at \fI\%Managing files within a workflow\fP\&. .sp Next \fBdown\fP checks the base case of the recursion: is the length of the input file less than \fBN\fP (remember \fBN\fP was an option we added to the workflow in \fBmain\fP)? In the base case, we just sort the file, and return the file ID of this new sorted file. .sp If the base case fails, then the file is split into two new tempFiles using \fBjob.fileStore.getLocalTempFile()\fP and the helper function \fBcopySubRangeOfFile\fP\&. Finally we add a follow on Job \fBup\fP with \fBjob.addFollowOnJobFn()\fP\&. We\(aqve already seen child jobs. A follow\-on Job is a job that runs after the current job and \fIall\fP of its children (and their children and follow\-ons) have completed. Using a follow\-on makes sense because \fBup\fP is responsible for merging the files together and we don\(aqt want to merge the files together until we \fIknow\fP they are sorted. Again, the return value of the follow\-on job is requested using \fBJob.rv()\fP\&. .sp Looking at \fBup\fP .INDENT 0.0 .INDENT 3.5 .sp .EX def up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory): \(dq\(dq\(dq Merges the two files and places them in the output. \(dq\(dq\(dq RealtimeLogger.info(\(dqUp job starting: %s\(dq % path) with job.fileStore.writeGlobalFileStream() as (fileHandle, outputFileStoreID): fileHandle = codecs.getwriter(\(aqutf\-8\(aq)(fileHandle) with job.fileStore.readGlobalFileStream(inputFileID1) as inputFileHandle1: inputFileHandle1 = codecs.getreader(\(aqutf\-8\(aq)(inputFileHandle1) with job.fileStore.readGlobalFileStream(inputFileID2) as inputFileHandle2: inputFileHandle2 = codecs.getreader(\(aqutf\-8\(aq)(inputFileHandle2) RealtimeLogger.info(\(dqMerging %s and %s to %s\(dq % (inputFileID1, inputFileID2, outputFileStoreID)) merge(inputFileHandle1, inputFileHandle2, fileHandle) # Cleanup up the input files \- these deletes will occur after the completion is successful. job.fileStore.deleteGlobalFile(inputFileID1) job.fileStore.deleteGlobalFile(inputFileID2) RealtimeLogger.info(\(dqUp job finished: %s\(dq % path) return outputFileStoreID .EE .UNINDENT .UNINDENT .sp we see that the two input files are merged together and the output is written to a new file using \fBjob.fileStore.writeGlobalFileStream()\fP\&. After a little cleanup, the output file is returned. .sp Once the final \fBup\fP finishes and all of the \fBrv()\fP promises are fulfilled, \fBmain\fP receives the sorted file\(aqs ID which it uses in \fBexportFile\fP to send it to the user. .sp There are other things in this example that we didn\(aqt go over such as \fI\%Checkpoints\fP and the details of much of the \fI\%Toil Class API\fP\&. .sp At the end of the script the lines .INDENT 0.0 .INDENT 3.5 .sp .EX if __name__ == \(aq__main__\(aq main() .EE .UNINDENT .UNINDENT .sp are included to ensure that the main function is only run once in the \(aq__main__\(aq process invoked by you, the user. In Toil terms, by invoking the script you created the \fIleader process\fP in which the \fBmain()\fP function is run. A \fIworker process\fP is a separate process whose sole purpose is to host the execution of one or more jobs defined in that script. In any Toil workflow there is always one leader process, and potentially many worker processes. .sp When using the single\-machine batch system (the default), the worker processes will be running on the same machine as the leader process. With full\-fledged batch systems like Kubernetes the worker processes will typically be started on separate machines. The boilerplate ensures that the pipeline is only started once\-\-\-on the leader\-\-\-but not when its job functions are imported and executed on the individual workers. .sp Typing \fBpython3 sort.py \-\-help\fP will show the complete list of arguments for the workflow which includes both Toil\(aqs and ones defined inside \fBsort.py\fP\&. A complete explanation of Toil\(aqs arguments can be found in \fI\%Commandline Options\fP\&. .SS Logging .sp By default, Toil logs a lot of information related to the current environment in addition to messages from the batch system and jobs. This can be configured with the \fB\-\-logLevel\fP flag. For example, to only log \fBCRITICAL\fP level messages to the screen: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore \e \-\-logLevel=critical \e \-\-overwriteOutput=True .EE .UNINDENT .UNINDENT .sp This hides most of the information we get from the Toil run. For more detail, we can run the pipeline with \fB\-\-logLevel=debug\fP to see a comprehensive output. For more information, see \fI\%Commandline Options\fP\&. .SS Error Handling and Resuming Pipelines .sp With Toil, you can recover gracefully from a bug in your pipeline without losing any progress from successfully completed jobs. To demonstrate this, let\(aqs add a bug to our example code to see how Toil handles a failure and how we can resume a pipeline after that happens. Add a bad assertion at line 52 of the example (the first line of \fBdown()\fP): .INDENT 0.0 .INDENT 3.5 .sp .EX def down(job, inputFileStoreID, N, downCheckpoints, memory=sortMemory): ... assert 1 == 2, \(dqTest error!\(dq .EE .UNINDENT .UNINDENT .sp When we run the pipeline, Toil will show a detailed failure log with a traceback: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore \&... \-\-\-TOIL WORKER OUTPUT LOG\-\-\- \&... m/j/jobonrSMP Traceback (most recent call last): m/j/jobonrSMP File \(dqtoil/src/toil/worker.py\(dq, line 340, in main m/j/jobonrSMP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) m/j/jobonrSMP File \(dqtoil/src/toil/job.py\(dq, line 1270, in _runner m/j/jobonrSMP returnValues = self._run(jobGraph, fileStore) m/j/jobonrSMP File \(dqtoil/src/toil/job.py\(dq, line 1217, in _run m/j/jobonrSMP return self.run(fileStore) m/j/jobonrSMP File \(dqtoil/src/toil/job.py\(dq, line 1383, in run m/j/jobonrSMP rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) m/j/jobonrSMP File \(dqtoil/example.py\(dq, line 30, in down m/j/jobonrSMP assert 1 == 2, \(dqTest error!\(dq m/j/jobonrSMP AssertionError: Test error! .EE .UNINDENT .UNINDENT .sp If we try and run the pipeline again, Toil will give us an error message saying that a job store of the same name already exists. By default, in the event of a failure, the job store is preserved so that the workflow can be restarted, starting from the previously failed jobs. We can restart the pipeline by running .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore \e \-\-restart \e \-\-overwriteOutput=True .EE .UNINDENT .UNINDENT .sp We can also change the number of times Toil will attempt to retry a failed job: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore \e \-\-retryCount 2 \e \-\-restart \e \-\-overwriteOutput=True .EE .UNINDENT .UNINDENT .sp You\(aqll now see Toil attempt to rerun the failed job until it runs out of tries. \fB\-\-retryCount\fP is useful for non\-systemic errors, like downloading a file that may experience a sporadic interruption, or some other non\-deterministic failure. .sp To successfully restart our pipeline, we can edit our script to comment out line 30, or remove it, and then run .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py file:jobStore \e \-\-restart \e \-\-overwriteOutput=True .EE .UNINDENT .UNINDENT .sp The pipeline will run successfully, and the job store will be removed on the pipeline\(aqs completion. .SS Collecting Statistics .sp Please see the \fI\%Status Command\fP section for more on gathering runtime and resource info on jobs. .SS Launching a Toil Workflow in AWS .sp After having installed the \fBaws\fP extra for Toil during the \fI\%Installation\fP and set up AWS (see \fI\%Preparing your AWS environment\fP), the user can run the basic \fBhelloWorld.py\fP script (\fI\%Running a basic Python workflow\fP) on a VM in AWS just by modifying the run command. .sp Note that when running in AWS, users can either run the workflow on a single instance or run it on a cluster (which is running across multiple containers on multiple AWS instances). For more information on running Toil workflows on a cluster, see \fI\%Running in AWS\fP\&. .sp Also! Remember to use the \fI\%Destroy\-Cluster Command\fP command when finished to destroy the cluster! Otherwise things may not be cleaned up properly. .INDENT 0.0 .IP 1. 3 Launch a cluster in AWS using the \fI\%Launch\-Cluster Command\fP command: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil launch\-cluster \e \-\-clusterType kubernetes \e \-\-keyPairName \e \-\-leaderNodeType t2.medium \e \-\-nodeTypes t2.medium \-w 1 \e \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .sp The arguments \fBkeyPairName\fP, \fBleaderNodeType\fP, and \fBzone\fP are required to launch a cluster. .IP 2. 3 Copy \fBhelloWorld.py\fP to the \fB/tmp\fP directory on the leader node using the \fI\%Rsync\-Cluster Command\fP command: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil rsync\-cluster \-\-zone us\-west\-2a helloWorld.py :/tmp .EE .UNINDENT .UNINDENT .sp Note that the command requires defining the file to copy as well as the target location on the cluster leader node. .IP 3. 3 Login to the cluster leader node using the \fI\%Ssh\-Cluster Command\fP command: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil ssh\-cluster \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .sp Note that this command will log you in as the \fBroot\fP user. .IP 4. 3 Run the workflow on the cluster: .INDENT 3.0 .INDENT 3.5 .sp .EX $ python3 /tmp/helloWorld.py aws:us\-west\-2:my\-S3\-bucket .EE .UNINDENT .UNINDENT .sp In this particular case, we create an S3 bucket called \fBmy\-S3\-bucket\fP in the \fBus\-west\-2\fP availability zone to store intermediate job results. .sp Along with some other \fBINFO\fP log messages, you should get the following output in your terminal window: \fBHello, world!, here\(aqs a message: You did it!\fP\&. .IP 5. 3 Exit from the SSH connection. .INDENT 3.0 .INDENT 3.5 .sp .EX $ exit .EE .UNINDENT .UNINDENT .IP 6. 3 Use the \fI\%Destroy\-Cluster Command\fP command to destroy the cluster: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil destroy\-cluster \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .sp Note that this command will destroy the cluster leader node and any resources created to run the job, including the S3 bucket. .UNINDENT .SS Running a CWL Workflow on AWS .sp After having installed the \fBaws\fP and \fBcwl\fP extras for Toil during the \fI\%Installation\fP and set up AWS (see \fI\%Preparing your AWS environment\fP), the user can run a CWL workflow with Toil on AWS. .sp Also! Remember to use the \fI\%Destroy\-Cluster Command\fP command when finished to destroy the cluster! Otherwise things may not be cleaned up properly. .INDENT 0.0 .IP 1. 3 First launch a node in AWS using the \fI\%Launch\-Cluster Command\fP command: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil launch\-cluster \e \-\-clusterType kubernetes \e \-\-keyPairName \e \-\-leaderNodeType t2.medium \e \-\-nodeTypes t2.medium \-w 1 \e \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .IP 2. 3 Copy \fBexample.cwl\fP and \fBexample\-job.yaml\fP from the \fI\%CWL example\fP to the node using the \fI\%Rsync\-Cluster Command\fP command: .INDENT 3.0 .INDENT 3.5 .sp .EX toil rsync\-cluster \-\-zone us\-west\-2a example.cwl :/tmp toil rsync\-cluster \-\-zone us\-west\-2a example\-job.yaml :/tmp .EE .UNINDENT .UNINDENT .IP 3. 3 SSH into the cluster\(aqs leader node using the \fI\%Ssh\-Cluster Command\fP utility: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil ssh\-cluster \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .IP 4. 3 Once on the leader node, command line tools such as \fBkubectl\fP will be available to you. It\(aqs also a good idea to update and install the following: .INDENT 3.0 .INDENT 3.5 .sp .EX sudo apt\-get update sudo apt\-get \-y upgrade sudo apt\-get \-y dist\-upgrade sudo apt\-get \-y install git .EE .UNINDENT .UNINDENT .IP 5. 3 Now create a new \fBvirtualenv\fP with the \fB\-\-system\-site\-packages\fP option and activate: .INDENT 3.0 .INDENT 3.5 .sp .EX virtualenv \-\-system\-site\-packages venv source venv/bin/activate .EE .UNINDENT .UNINDENT .IP 6. 3 Now run the CWL workflow with the Kubernetes batch system: .INDENT 3.0 .INDENT 3.5 .sp .EX (venv) $ toil\-cwl\-runner \e \-\-provisioner aws \e \-\-batchSystem kubernetes \e \-\-jobStore aws:us\-west\-2:any\-name \e /tmp/example.cwl /tmp/example\-job.yaml .EE .UNINDENT .UNINDENT .sp \fBTIP:\fP .INDENT 3.0 .INDENT 3.5 When running a CWL workflow on AWS, input files can be provided either on the local file system or in S3 buckets using \fBs3://\fP URI references. Final output files will be copied to the local file system of the leader node. .UNINDENT .UNINDENT .IP 7. 3 Finally, log out of the leader node and from your local computer, destroy the cluster: .INDENT 3.0 .INDENT 3.5 .sp .EX $ toil destroy\-cluster \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .UNINDENT .SS Running a Workflow with Autoscaling \- Cactus .sp \fI\%Cactus\fP is a reference\-free, whole\-genome multiple alignment program that can be run on any of the cloud platforms Toil supports. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 \fBCloud Independence\fP: .sp This example provides a \(dqcloud agnostic\(dq view of running Cactus with Toil. Most options will not change between cloud providers. However, each provisioner has unique inputs for \fB\-\-leaderNodeType\fP, \fB\-\-nodeType\fP and \fB\-\-zone\fP\&. We recommend the following: .INDENT 0.0 .INDENT 3.5 .TS center; |l|l|l|l|. _ T{ Option T} T{ Used in T} T{ AWS T} T{ Google T} _ T{ \fB\-\-leaderNodeType\fP T} T{ launch\-cluster T} T{ t2.medium T} T{ n1\-standard\-1 T} _ T{ \fB\-\-zone\fP T} T{ launch\-cluster T} T{ us\-west\-2a T} T{ us\-west1\-a T} _ T{ \fB\-\-zone\fP T} T{ cactus T} T{ us\-west\-2 T} _ T{ \fB\-\-nodeType\fP T} T{ cactus T} T{ c3.4xlarge T} T{ n1\-standard\-8 T} _ .TE .UNINDENT .UNINDENT .sp When executing \fBtoil launch\-cluster\fP with \fBgce\fP specified for \fB\-\-provisioner\fP, the option \fB\-\-boto\fP must be specified and given a path to your .boto file. See \fI\%Running in Google Compute Engine (GCE)\fP for more information about the \fB\-\-boto\fP option. .UNINDENT .UNINDENT .sp Also! Remember to use the \fI\%Destroy\-Cluster Command\fP command when finished to destroy the cluster! Otherwise things may not be cleaned up properly. .INDENT 0.0 .IP 1. 4 Download \fBpestis.tar.gz\fP .IP 2. 4 Launch a cluster using the \fI\%Launch\-Cluster Command\fP command: .INDENT 4.0 .INDENT 3.5 .sp .EX $ toil launch\-cluster \e \-\-provisioner \e \-\-keyPairName \e \-\-leaderNodeType \e \-\-nodeType \e \-w 1\-2 \e \-\-zone .EE .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 4.0 .INDENT 3.5 \fBA Helpful Tip\fP .sp When using AWS, setting the environment variable eliminates having to specify the \fB\-\-zone\fP option for each command. This will be supported for GCE in the future. .INDENT 0.0 .INDENT 3.5 .sp .EX $ export TOIL_AWS_ZONE=us\-west\-2c .EE .UNINDENT .UNINDENT .UNINDENT .UNINDENT .IP 3. 4 Create appropriate directory for uploading files: .INDENT 4.0 .INDENT 3.5 .sp .EX $ toil ssh\-cluster \-\-provisioner $ mkdir /root/cact_ex $ exit .EE .UNINDENT .UNINDENT .IP 4. 4 Copy the required files, i.e., seqFile.txt (a text file containing the locations of the input sequences as well as their phylogenetic tree, see \fI\%here\fP), organisms\(aq genome sequence files in FASTA format, and configuration files (e.g. blockTrim1.xml, if desired), up to the leader node: .INDENT 4.0 .INDENT 3.5 .sp .EX $ toil rsync\-cluster \-\-provisioner pestis\-short\-aws\-seqFile.txt :/root/cact_ex $ toil rsync\-cluster \-\-provisioner GCF_000169655.1_ASM16965v1_genomic.fna :/root/cact_ex $ toil rsync\-cluster \-\-provisioner GCF_000006645.1_ASM664v1_genomic.fna :/root/cact_ex $ toil rsync\-cluster \-\-provisioner GCF_000182485.1_ASM18248v1_genomic.fna :/root/cact_ex $ toil rsync\-cluster \-\-provisioner GCF_000013805.1_ASM1380v1_genomic.fna :/root/cact_ex $ toil rsync\-cluster \-\-provisioner setup_leaderNode.sh :/root/cact_ex $ toil rsync\-cluster \-\-provisioner blockTrim1.xml :/root/cact_ex $ toil rsync\-cluster \-\-provisioner blockTrim3.xml :/root/cact_ex .EE .UNINDENT .UNINDENT .IP 5. 4 Log in to the leader node: .INDENT 4.0 .INDENT 3.5 .sp .EX $ toil ssh\-cluster \-\-provisioner .EE .UNINDENT .UNINDENT .IP 6. 4 Set up the environment of the leader node to run Cactus: .INDENT 4.0 .INDENT 3.5 .sp .EX $ bash /root/cact_ex/setup_leaderNode.sh $ source cact_venv/bin/activate (cact_venv) $ cd cactus (cact_venv) $ pip install \-\-upgrade . .EE .UNINDENT .UNINDENT .IP 7. 4 Run \fI\%Cactus\fP as an autoscaling workflow: .INDENT 4.0 .INDENT 3.5 .sp .EX (cact_venv) $ cactus \e \-\-retry 10 \e \-\-batchSystem kubernetes \e \-\-logDebug \e \-\-logFile /logFile_pestis3 \e \-\-configFile \e /root/cact_ex/blockTrim3.xml ::cactus\-pestis \e /root/cact_ex/pestis\-short\-aws\-seqFile.txt \e /root/cact_ex/pestis_output3.hal .EE .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 4.0 .INDENT 3.5 \fBPieces of the Puzzle\fP: .sp \fB\-\-logDebug\fP \-\-\- equivalent to \fB\-\-logLevel DEBUG\fP\&. .sp \fB\-\-logFile /logFile_pestis3\fP \-\-\- writes logs in a file named \fIlogFile_pestis3\fP under \fB/\fP folder. .sp \fB\-\-configFile\fP \-\-\- this is not required depending on whether a specific configuration file is intended to run the alignment. .sp \fB::cactus\-pestis\fP \-\-\- creates a bucket, named \fBcactus\-pestis\fP, with the specified cloud provider to store intermediate job files and metadata. \fBNOTE\fP: If you want to use a GCE\-based jobstore, specify \fBgoogle\fP here, not \fBgce\fP\&. .sp The result file, named \fBpestis_output3.hal\fP, is stored under \fB/root/cact_ex\fP folder of the leader node. .sp Use \fBcactus \-\-help\fP to see all the Cactus and Toil flags available. .UNINDENT .UNINDENT .IP 8. 4 Log out of the leader node: .INDENT 4.0 .INDENT 3.5 .sp .EX (cact_venv) $ exit .EE .UNINDENT .UNINDENT .IP 9. 4 Download the resulted output to local machine: .INDENT 4.0 .INDENT 3.5 .sp .EX (venv) $ toil rsync\-cluster \e \-\-provisioner \e :/root/cact_ex/pestis_output3.hal \e .EE .UNINDENT .UNINDENT .IP 10. 4 Destroy the cluster: .INDENT 4.0 .INDENT 3.5 .sp .EX (venv) $ toil destroy\-cluster \-\-provisioner .EE .UNINDENT .UNINDENT .UNINDENT .SH CWL IN TOIL .sp The Common Workflow Language (CWL) is an emerging standard for writing workflows that are portable across multiple workflow engines and platforms. Toil has full support for the CWL v1.0, v1.1, and v1.2 standards. .sp You can use Toil to \fI\%run CWL workflows\fP or develop and test new ones. .SH RUNNING CWL WORKFLOWS .sp The \fItoil\-cwl\-runner\fP command provides CWL parsing functionality using cwltool, and leverages the job\-scheduling and batch system support of Toil. You can use it to run CWL workflows locally or in the cloud. .SS Running CWL Locally .sp To run in local batch mode, provide the CWL file and the input object file: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil\-cwl\-runner example.cwl example\-job.yml .EE .UNINDENT .UNINDENT .sp For a simple example of CWL with Toil see \fI\%Running a basic CWL workflow\fP\&. .SS Note for macOS + Docker + Toil .sp When invoking CWL documents that make use of Docker containers if you see errors that look like .INDENT 0.0 .INDENT 3.5 .sp .EX docker: Error response from daemon: Mounts denied: The paths /var/...tmp are not shared from OS X and are not known to Docker. .EE .UNINDENT .UNINDENT .sp you may need to add .INDENT 0.0 .INDENT 3.5 .sp .EX export TMPDIR=/tmp/docker_tmp .EE .UNINDENT .UNINDENT .sp either in your startup file (\fB\&.bashrc\fP) or add it manually in your shell before invoking toil. .SS Detailed Usage Instructions .sp Help information can be found by using this toil command: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil\-cwl\-runner \-h .EE .UNINDENT .UNINDENT .sp A more detailed example shows how we can specify both Toil and cwltool arguments for our workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil\-cwl\-runner \e \-\-singularity \e \-\-jobStore my_jobStore \e \-\-batchSystem lsf \e \-\-workDir \(gapwd\(ga \e \-\-outdir \(gapwd\(ga \e \-\-logFile cwltoil.log \e \-\-writeLogs \(gapwd\(ga \e \-\-logLevel DEBUG \e \-\-retryCount 2 \e \-\-maxLogFileSize 20000000000 \e \-\-stats \e standard_bam_processing.cwl \e inputs.yaml .EE .UNINDENT .UNINDENT .sp In this example, we set the following options, which are all passed to Toil: .sp \fB\-\-singularity\fP: Specifies that all jobs with Docker format containers specified should be run using the Singularity container engine instead of the Docker container engine. .sp \fB\-\-jobStore\fP: Path to a folder which doesn\(aqt exist yet, which will contain the Toil jobstore and all related job\-tracking information. .sp \fB\-\-batchSystem\fP: Use the specified HPC or Cloud\-based cluster platform. .sp \fB\-\-workDir\fP: The directory where all temporary files will be created for the workflow. A subdirectory of this will be set as the \fB$TMPDIR\fP environment variable and this subdirectory can be referenced using the CWL parameter reference \fB$(runtime.tmpdir)\fP in CWL tools and workflows. .sp \fB\-\-outdir\fP: Directory where final \fBFile\fP and \fBDirectory\fP outputs will be written. References to these and other output types will be in the JSON object printed to the stdout stream after workflow execution. .sp \fB\-\-logFile\fP: Path to the main logfile. .sp \fB\-\-writeLogs\fP: Directory where job logs will be stored. At \fBDEBUG\fP log level, this will contain logs for each Toil job run, as well as \fBstdout\fP/\fBstderr\fP logs for each CWL \fBCommandLineTool\fP that didn\(aqt use the \fBstdout\fP/\fBstderr\fP directives to redirect output. .sp \fB\-\-retryCount\fP: How many times to retry each Toil job. .sp \fB\-\-maxLogFileSize\fP: Logs that get larger than this value will be truncated. .sp \fB\-\-stats\fP: Save resources usages in json files that can be collected with the \fBtoil stats\fP command after the workflow is done. .sp \fB\-\-disable\-streaming\fP: Does not allow streaming of input files. This is enabled by default for files marked with \fBstreamable\fP flag True and only for remote files when the jobStore is not on local machine. .SS Running CWL in the Cloud .sp To run in cloud and HPC configurations, you may need to provide additional command line parameters to select and configure the batch system to use. .sp To run a CWL workflow in AWS with toil see \fI\%Running a CWL Workflow on AWS\fP\&. .SS Running CWL within Toil Scripts .sp A CWL workflow can be run from a Toil Python workflow. However, this is not the \fI\%standard\fP way to run CWL workflows with Toil and doing so comes at the cost of job efficiency. For some use cases, such as running one process on multiple files, it may be useful. For example, if you want to run a CWL workflow with 3 different input files specifying different samples inputs, it could look something like: .INDENT 0.0 .INDENT 3.5 .sp .EX import os import subprocess from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def initialize_jobs(job): job.fileStore.log_to_leader(\(aqinitialize_jobs\(aq) def runQC(job, cwl_file, cwl_filename, yml_file, yml_filename, outputs_dir, output_num): job.fileStore.log_to_leader(\(dqrunQC\(dq) tempDir = job.fileStore.getLocalTempDir() cwl = job.fileStore.readGlobalFile(cwl_file, userPath=os.path.join(tempDir, cwl_filename)) yml = job.fileStore.readGlobalFile(yml_file, userPath=os.path.join(tempDir, yml_filename)) subprocess.check_call([\(dqtoil\-cwl\-runner\(dq, cwl, yml]) output_filename = \(dqoutput.txt\(dq output_file = job.fileStore.writeGlobalFile(output_filename) job.fileStore.readGlobalFile(output_file, userPath=os.path.join(outputs_dir, \(dqsample_\(dq + output_num + \(dq_\(dq + output_filename)) return output_file if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_cwlexample\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: # specify the folder where the cwl and yml files live inputs_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), \(dqcwlExampleFiles\(dq) # specify where you wish the outputs to be written outputs_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), \(dqcwlExampleFiles\(dq) job0 = Job.wrapJobFn(initialize_jobs) cwl_filename = \(dqhello.cwl\(dq cwl_file = toil.importFile(\(dqfile://\(dq + os.path.abspath(os.path.join(inputs_dir, cwl_filename))) # add list of yml config inputs here or import and construct from file yml_files = [\(dqhello1.yml\(dq, \(dqhello2.yml\(dq, \(dqhello3.yml\(dq] i = 0 for yml in yml_files: i = i + 1 yml_file = toil.importFile(\(dqfile://\(dq + os.path.abspath(os.path.join(inputs_dir, yml))) yml_filename = yml job = Job.wrapJobFn(runQC, cwl_file, cwl_filename, yml_file, yml_filename, outputs_dir, output_num=str(i)) job0.addChild(job) toil.start(job0) .EE .UNINDENT .UNINDENT .SS Running CWL workflows with InplaceUpdateRequirement .sp Some CWL workflows use the \fBInplaceUpdateRequirement\fP feature, which requires that operations on files have visible side effects that Toil\(aqs file store cannot support. If you need to run a workflow like this, you can make sure that all of your worker nodes have a shared filesystem, and use the \fB\-\-bypass\-file\-store\fP option to \fBtoil\-cwl\-runner\fP\&. This will make it leave all CWL intermediate files on disk and share them between jobs using file paths, instead of storing them in the file store and downloading them when jobs need them. .SS Toil & CWL Tips .sp \fBSee logs for just one job by using the full log file\fP .sp This requires knowing the job\(aqs toil\-generated ID, which can be found in the log files. .INDENT 0.0 .INDENT 3.5 .sp .EX cat cwltoil.log | grep jobVM1fIs .EE .UNINDENT .UNINDENT .sp \fBGrep for full tool commands from toil logs\fP .sp This gives you a more concise view of the commands being run (note that this information is only available from Toil when running with \fI\-\-logDebug\fP). .INDENT 0.0 .INDENT 3.5 .sp .EX pcregrep \-M \(dq\e[job .*\e.cwl.*$\en(.* .*$\en)*\(dq cwltoil.log # ^allows for multiline matching .EE .UNINDENT .UNINDENT .sp \fBFind Bams that have been generated for specific step while pipeline is running:\fP .INDENT 0.0 .INDENT 3.5 .sp .EX find . | grep \-P \(aq^./out_tmpdir.*_MD\e.bam$\(aq .EE .UNINDENT .UNINDENT .sp \fBSee what jobs have been run\fP .INDENT 0.0 .INDENT 3.5 .sp .EX cat log/cwltoil.log | grep \-oP \(dq\e[job .*.cwl\e]\(dq | sort | uniq .EE .UNINDENT .UNINDENT .sp or: .INDENT 0.0 .INDENT 3.5 .sp .EX cat log/cwltoil.log | grep \-i \(dqissued job\(dq .EE .UNINDENT .UNINDENT .sp \fBGet status of a workflow\fP .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil status /home/johnsoni/TEST_RUNS_3/TEST_run/tmp/jobstore\-09ae0acc\-c800\-11e8\-9d09\-70106fb1697e 2018\-10\-04 15:01:44,184 MainThread INFO toil.lib.bioio: Root logger is at level \(aqINFO\(aq, \(aqtoil\(aq logger at level \(aqINFO\(aq. 2018\-10\-04 15:01:44,185 MainThread INFO toil.utils.toilStatus: Parsed arguments 2018\-10\-04 15:01:47,081 MainThread INFO toil.utils.toilStatus: Traversing the job graph gathering jobs. This may take a couple of minutes. Of the 286 jobs considered, there are 179 jobs with children, 107 jobs ready to run, 0 zombie jobs, 0 jobs with services, 0 services, and 0 jobs with log files currently in file:/home/user/jobstore\-09ae0acc\-c800\-11e8\-9d09\-70106fb1697e. .EE .UNINDENT .UNINDENT .sp \fBToil Stats\fP .sp You can get run statistics broken down by CWL file. This only works once the workflow is finished: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil stats /path/to/jobstore .EE .UNINDENT .UNINDENT .sp This will report resource usage information for all the CWL jobs executed by the workflow. .sp See \fI\%Stats Command\fP for an explanation of what the different fields mean. .sp \fBUnderstanding toil log files\fP .sp There is a \fIworker_log.txt\fP file for each Toil job. This file is written to while the job is running, and uploaded at the end if the job finishes or if running at debug log level. If uploaded, the contents are printed to the main log file and transferred to a log file in the \fI\-\-logDir\fP folder. .sp The new log file will be named something like: .INDENT 0.0 .INDENT 3.5 .sp .EX CWLJob__.log .EE .UNINDENT .UNINDENT .sp Standard output/error files will be named like: .INDENT 0.0 .INDENT 3.5 .sp .EX .stdout_.log .EE .UNINDENT .UNINDENT .sp If you have a workflow \fBrevsort.cwl\fP which has a step \fBrev\fP which calls the tool \fBrevtool.cwl\fP, the CWL job name ends up being all those parts strung together with \fB\&.\fP: \fBrevsort.cwl.rev.revtool.cwl\fP\&. .SH WDL IN TOIL .sp The \fI\%Workflow Description Language\fP (WDL) is a programming language designed for writing workflows that execute a set of tasks in a pipeline distributed across multiple computers. Workflows enable scientific analyses to be reproducible, by wrapping up a whole sequence of commands, whose outputs feed into other commands, into a workflow that can be executed the same way every time. .sp Toil can be used to \fI\%run\fP and to \fI\%develop\fP WDL workflows. The Toil team also maintains a set of \fI\%WDL conformance tests\fP for evaluating Toil and other WDL runners. .SH RUNNING WDL WITH TOIL .sp Toil has beta support for running WDL workflows, using the \fBtoil\-wdl\-runner\fP command. This command comes with the \fB[wdl]\fP extra; see \fI\%Installing Toil with Extra Features\fP for how to install it if you do not have it. .sp You can run WDL workflows with \fBtoil\-wdl\-runner\fP\&. Currently, \fBtoil\-wdl\-runner\fP works by using \fI\%MiniWDL\fP to parse and interpret the WDL workflow, and has support for workflows in WDL 1.0 or later (which are required to declare a \fBversion\fP, and which use \fBinputs\fP and \fBoutputs\fP sections). .sp \fBTIP:\fP .INDENT 0.0 .INDENT 3.5 The last release of Toil that supported unversioned, \fBdraft\-2\fP WDL workflows was \fI\%5.12.0\fP\&. .UNINDENT .UNINDENT .sp Toil is, for compatible workflows, a drop\-in replacement for the \fI\%Cromwell\fP WDL runner. Instead of running a workflow with Cromwell: .INDENT 0.0 .INDENT 3.5 .sp .EX java \-jar Cromwell.jar run myWorkflow.wdl \-\-inputs myWorkflow_inputs.json .EE .UNINDENT .UNINDENT .sp You can run the workflow with \fBtoil\-wdl\-runner\fP: .INDENT 0.0 .INDENT 3.5 .sp .EX toil\-wdl\-runner myWorkflow.wdl \-\-inputs myWorkflow_inputs.json .EE .UNINDENT .UNINDENT .sp This will default to executing on the current machine, with a job store in an automatically determined temporary location, but you can add a few Toil options to use other Toil\-supported batch systems, such as Kubernetes: .INDENT 0.0 .INDENT 3.5 .sp .EX toil\-wdl\-runner \-\-jobStore aws:us\-west\-2:wdl\-job\-store \-\-batchSystem kubernetes myWorkflow.wdl \-\-inputs myWorkflow_inputs.json .EE .UNINDENT .UNINDENT .sp For Toil, the \fB\-\-inputs\fP is optional, and inputs can be passed as a positional argument: .INDENT 0.0 .INDENT 3.5 .sp .EX toil\-wdl\-runner myWorkflow.wdl myWorkflow_inputs.json .EE .UNINDENT .UNINDENT .sp You can also run workflows from URLs. For example, to run the MiniWDL self test workflow, you can do: .INDENT 0.0 .INDENT 3.5 .sp .EX toil\-wdl\-runner https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/self_test.wdl https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/inputs.json .EE .UNINDENT .UNINDENT .SS Toil WDL Runner Options .sp \fB\-\-jobStore\fP: Specifies where to keep the Toil state information while running the workflow. Must be accessible from all machines. .sp \fB\-o\fP or \fB\-\-outputDirectory\fP: Specifies the output folder or URI prefix to save workflow output files in. Defaults to a new directory in the current directory. .sp \fB\-m\fP or \fB\-\-outputFile\fP: Specifies a JSON file name or URI to save workflow output values at. Defaults to standard output. .sp \fB\-i\fP or \fB\-\-input\fP: Alternative to the positional argument for the input JSON file, for compatibility with other WDL runners. .sp \fB\-\-outputDialect\fP: Specifies an output format dialect. Can be \fBcromwell\fP to just return the workflow\(aqs output values as JSON or \fBminiwdl\fP to nest that under an \fBoutputs\fP key and includes a \fBdir\fP key. .sp \fB\-\-container\fP: Specifies the container engine to use to run tasks. By default this is \fBauto\fP, which tries Singularity if it is installed and Docker if it isn\(aqt. Can also be set to \fBdocker\fP or \fBsingularity\fP explicitly. .sp Any number of other Toil options may also be specified. For defined Toil options, see \fI\%Commandline Options\fP\&. .SS Managing Workflow Logs .sp At the default settings, if a WDL task succeeds, the standard output and standard error will be printed in the \fBtoil\-wdl\-runner\fP output, unless they are captured by the workflow (with the \fBstdout()\fP and \fBstderr()\fP WDL built\-in functions). If a WDL task fails, they will be printed whether they were meant to be captured or not. Complete logs from Toil for failed jobs will also be printed. .sp If you would like to save the logs organized by WDL task, you can use the \fB\-\-writeLogs\fP or \fB\-\-writeLogsGzip\fP options to specify a directory where the log files should be saved. Log files will be named after the same dotted, hierarchical workflow and task names used to set values from the input JSON, except that scatters will add an additional numerical component. In addition to the logs for WDL tasks, Toil job logs for failed jobs will also appear here when running at the default log level. .sp For example, if you run: .INDENT 0.0 .INDENT 3.5 .sp .EX toil\-wdl\-runner \-\-writeLogs logs https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/self_test.wdl https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/inputs.json .EE .UNINDENT .UNINDENT .sp You will end up with a \fBlogs/\fP directory containing: .INDENT 0.0 .INDENT 3.5 .sp .EX hello_caller.0.hello.stderr_000.log hello_caller.1.hello.stderr_000.log hello_caller.2.hello.stderr_000.log .EE .UNINDENT .UNINDENT .sp The final number is a sequential counter: if a step has to be retried, or if you run the workflow multiple times without clearing out the logs directory, it will increment. .SH DEVELOPING A WDL WORKFLOW .sp Toil can be used as a development tool for writing and locally testing WDL workflows. These workflows can then be run on Toil against a cloud or cluster backend, or used with other WDL implementations such as \fI\%Terra\fP, \fI\%Cromwell\fP, or \fI\%MiniWDL\fP\&. .sp The easiest way to get started with writing WDL workflows is by following a tutorial. .SS Using the UCSC Genomics Institute Tutorial .sp The UCSC Genomics Institute (home of the Toil project) has \fI\%a tutorial on writing WDL workflows with Toil\fP\&. You can follow this tutorial to be walked through writing your own WDL workflow with Toil. They also have \fI\%tips on debugging WDL workflows with Toil\fP\&. .sp These tutorials and tips are aimed at users looking to run WDL workflows with Toil in a Slurm environment, but they can also apply in other situations. .SS Using the Official WDL tutorials .sp You can also learn to write WDL workflows for Toil by following the \fI\%official WDL tutorials\fP\&. .sp When you reach the point of \fI\%executing your workflow\fP, instead of running with Cromwell: .INDENT 0.0 .INDENT 3.5 .sp .EX java \-jar Cromwell.jar run myWorkflow.wdl \-\-inputs myWorkflow_inputs.json .EE .UNINDENT .UNINDENT .sp you can instead run with \fBtoil\-wdl\-runner\fP: .INDENT 0.0 .INDENT 3.5 .sp .EX toil\-wdl\-runner myWorkflow.wdl \-\-inputs myWorkflow_inputs.json .EE .UNINDENT .UNINDENT .SS Using the Learn WDL Video Tutorials .sp For people who prefer video tutorials, Lynn Langit has a \fI\%Learn WDL video course\fP that will teach you how to write and run WDL workflows. The course is taught using Cromwell, but Toil should also be compatible with the course\(aqs workflows. .SS WDL Specifications .sp WDL language specifications can be found here: \fI\%https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md\fP .sp Toil is not yet fully conformant with the WDL specification, but it inherits most of the functionality of \fI\%MiniWDL\fP\&. .SH WDL CONFORMANCE TESTING .sp The Toil team maintains a set of \fI\%WDL Conformance Tests\fP\&. Much like the \fI\%CWL Conformance Tests\fP for CWL, the WDL Conformance Tests are useful for determining if a WDL implementation actually follows the \fI\%WDL specification\fP\&. .sp The WDL Conformance Tests include a runner harness that is able to test \fBtoil\-wdl\-runner\fP, as well as Cromwell and MiniWDL, and supports testing conformance with the 1.1, 1.0, and \fBdraft\-2\fP versions of WDL. .sp If you would like to evaluate Toil\(aqs WDL conformance for yourself, first make sure that you have \fBtoil\-wdl\-runner\fP installed. It comes with the \fB[wdl]\fP extra; see \fI\%Installing Toil with Extra Features\fP\&. .sp Then, you can check out the test repository: .INDENT 0.0 .INDENT 3.5 .sp .EX $ git clone https://github.com/DataBiosphere/wdl\-conformance\-tests $ cd wdl\-conformance\-tests .EE .UNINDENT .UNINDENT .sp Most tests will need a Docker daemon available, so make sure yours is working properly: .INDENT 0.0 .INDENT 3.5 .sp .EX $ docker info $ docker run \-\-rm docker/whalesay cowsay \(dqDocker is working\(dq .EE .UNINDENT .UNINDENT .sp Then, you can test \fBtoil\-wdl\-runner\fP against a particular WDL spec version, say 1.1: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 run.py \-\-runner toil\-wdl\-runner \-\-versions 1.1 .EE .UNINDENT .UNINDENT .sp For any failed tests, the test number and the log of the failing test will be reported. .sp After the tests run, you can clean up intermediate files with: .INDENT 0.0 .INDENT 3.5 .sp .EX $ make clean .EE .UNINDENT .UNINDENT .sp For more options, see: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 run.py \-\-help .EE .UNINDENT .UNINDENT .sp Or, consult the \fI\%conformance test documentation\fP\&. .SH INTRODUCTION .sp Toil runs in various environments, including \fI\%locally\fP and \fI\%in the cloud\fP (Amazon Web Services and Google Compute Engine). Toil also supports workflows written in two DSLs: \fI\%CWL\fP and \fI\%WDL\fP, as well as workflows written in Python (see \fI\%Developing a Python Workflow\fP). .sp Toil is built in a modular way so that it can be used on lots of different systems, and with different configurations. The three configurable pieces are the .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 \fI\%Job Store\fP: A filepath or url that can host and centralize all files for a workflow (e.g. a local folder, or an AWS s3 bucket url). .IP \(bu 2 \fI\%Batch System\fP: Specifies either a local single\-machine or a currently supported HPC environment (lsf, mesos, slurm, torque, htcondor, kubernetes, or grid_engine). .IP \(bu 2 \fI\%Provisioner\fP: For running in the cloud only. This specifies which cloud provider provides instances to do the \(dqwork\(dq of your workflow. .UNINDENT .UNINDENT .UNINDENT .SS Job Store .sp The job store is a storage abstraction which contains all of the information used in a Toil run. This centralizes all of the files used by jobs in the workflow and also the details of the progress of the run. If a workflow crashes or fails, the job store contains all of the information necessary to resume with minimal repetition of work. .sp Several different job stores are supported, including the file job store and cloud job stores. For information on developing job stores, see \fI\%Job Store API\fP\&. .SS File Job Store .sp The file job store is for use locally, and keeps the workflow information in a directory on the machine where the workflow is launched. This is the simplest and most convenient job store for testing or for small runs. .sp For an example that uses the file job store, see \fI\%Running a basic CWL workflow\fP\&. .SS Cloud Job Stores .sp Toil currently supports the following cloud storage systems as job stores: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 \fI\%AWS Job Store\fP: An AWS S3 bucket formatted as \(dqaws::\(dq where only numbers, letters, and dashes are allowed in the bucket name. Example: \fIaws:us\-west\-2:my\-aws\-jobstore\-name\fP\&. .IP \(bu 2 \fI\%Google Job Store\fP: A Google Cloud Storage bucket formatted as \(dqgce::\(dq where only numbers, letters, and dashes are allowed in the bucket name. Example: \fIgce:us\-west2\-a:my\-google\-jobstore\-name\fP\&. .UNINDENT .UNINDENT .UNINDENT .sp These use cloud buckets to house all of the files. This is useful if there are several different worker machines all running jobs that need to access the job store. .SS Batch System .sp A Toil batch system is either a local single\-machine (one computer) or a currently supported cluster of computers (lsf, mesos, slurm, torque, htcondor, or grid_engine) These environments manage individual worker nodes under a leader node to process the work required in a workflow. The leader and its workers all coordinate their tasks and files through a centralized job store location. .sp See \fI\%Batch System API\fP for a more detailed description of different batch systems, or information on developing batch systems. .SS Provisioner .sp The Toil provisioner provides a tool set for running a Toil workflow on a particular cloud platform. .sp The \fI\%Toil Cluster Utilities\fP are command line tools used to provision nodes in your desired cloud platform. They allows you to launch nodes, ssh to the leader, and rsync files back and forth. .sp For detailed instructions for using the provisioner see \fI\%Running in AWS\fP or \fI\%Running in Google Compute Engine (GCE)\fP\&. .SH COMMANDLINE OPTIONS .sp A quick way to see all of Toil\(aqs commandline options is by executing the following on a workflow language front\-end: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil\-wdl\-runner \-\-help .EE .UNINDENT .UNINDENT .sp Or a Toil Python workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 example.py \-\-help .EE .UNINDENT .UNINDENT .sp For a basic toil workflow, Toil has one mandatory argument, the job store. All other arguments are optional. .SS The Config File .sp Instead of changing the arguments on the command line, Toil offers support for using a configuration file. .sp Options will be applied with priority: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP 1. 3 Command line options .IP 2. 3 Environmental Variables .IP 3. 3 Config file values .INDENT 3.0 .IP a. 3 Provided config file through \fB\-\-config\fP .IP b. 3 Default config value in \fB$HOME/.toil/default.yaml\fP .UNINDENT .IP 4. 3 Defaults .UNINDENT .UNINDENT .UNINDENT .sp You can manually generate an example configuration file to a path you select. To generate a configuration file, run: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil config [filename].yaml .EE .UNINDENT .UNINDENT .sp Then uncomment options as necessary and change/provide new values. .sp After editing the config file, you can run Toil with its settings by passing it on the command line: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 example.py \-\-config=[filename].yaml .EE .UNINDENT .UNINDENT .sp Alternatively, you can edit the default config file, which is located at \fB$HOME/.toil/default.yaml\fP .sp If CLI options are used in addition to the configuration file, the CLI options will overwrite the configuration file options. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 example.py \-\-config=[filename].yaml \-\-defaultMemory 80Gi .EE .UNINDENT .UNINDENT .sp This will result in a default memory per job of 80GiB no matter what is in the configuration file provided. .SS The Job Store .sp Running Toil workflows requires a file path or URL to a central location for all of the intermediate files for the workflow: the job store. For \fBtoil\-cwl\-runner\fP and \fBtoil\-wdl\-runner\fP a job store can often be selected automatically or can be specified with the \fB\-\-jobStore\fP option; Toil Python workflows generally require the job store as a positional command line argument. To use the \fI\%Python quickstart\fP example, if you\(aqre on a node that has a large \fB/scratch\fP volume, you can specify that the jobstore be created there by executing: \fBpython3 HelloWorld.py /scratch/my\-job\-store\fP, or more explicitly, \fBpython3 HelloWorld.py file:/scratch/my\-job\-store\fP\&. .sp Syntax for specifying different job stores: .INDENT 0.0 .INDENT 3.5 Local: \fBfile:job\-store\-name\fP .sp AWS: \fBaws:region\-here:job\-store\-name\fP .sp Google: \fBgoogle:projectID\-here:job\-store\-name\fP .UNINDENT .UNINDENT .sp Different types of job store options can be found below. .SS Commandline Options .sp \fBCore Toil Options\fP Options to specify the location of the Toil workflow and turn on stats collation about the performance of jobs. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-workDir \ WORKDIR Absolute path to directory where temporary files generated during the Toil run should be placed. Standard output and error from batch system jobs (unless \fB\-\-noStdOutErr\fP is set) will be placed in this directory. A cache directory may be placed in this directory. Temp files and folders will be placed in a directory \fBtoil\-\fP within workDir. The workflowID is generated by Toil and will be reported in the workflow logs. Default is determined by the variables (TMPDIR, TEMP, TMP) via mkdtemp. This directory needs to exist on all machines running jobs; if capturing standard output and error from batch system jobs is desired, it will generally need to be on a shared file system. When sharing a cache between containers on a host, this directory must be shared between the containers. .TP .BI \-\-coordinationDir \ COORDINATION_DIR Absolute path to directory where Toil will keep state and lock files. When sharing a cache between containers on a host, this directory must be shared between the containers. .TP .B \-\-noStdOutErr Do not capture standard output and error from batch system jobs. .TP .B \-\-stats Records statistics about the toil workflow to be used by \(aqtoil stats\(aq. .TP .BI \-\-clean\fB= STATE Determines the deletion of the jobStore upon completion of the program. Choices: \(aqalways\(aq, \(aqonError\(aq,\(aqnever\(aq, or \(aqonSuccess\(aq. The \fB\-\-stats\fP option requires information from the jobStore upon completion so the jobStore will never be deleted with that flag. If you wish to be able to restart the run, choose \(aqnever\(aq or \(aqonSuccess\(aq. Default is \(aqnever\(aq if stats is enabled, and \(aqonSuccess\(aq otherwise .TP .BI \-\-cleanWorkDir \ STATE Determines deletion of temporary worker directory upon completion of a job. Choices: \(aqalways\(aq, \(aqonError\(aq, \(aqnever\(aq, or \(aqonSuccess\(aq. Default = always. WARNING: This option should be changed for debugging only. Running a full pipeline with this option could fill your disk with intermediate data. .TP .BI \-\-clusterStats \ FILEPATH If enabled, writes out JSON resource usage statistics to a file. The default location for this file is the current working directory, but an absolute path can also be passed to specify where this file should be written. This option only applies when using scalable batch systems. .TP .B \-\-restart If \fB\-\-restart\fP is specified then will attempt to restart existing workflow at the location pointed to by the \fB\-\-jobStore\fP option. Will raise an exception if the workflow does not exist. .UNINDENT .UNINDENT .UNINDENT .sp \fBLogging Options\fP Toil hides stdout and stderr by default except in case of job failure. Log levels in toil are based on priority from the logging module: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \-\-logOff Only CRITICAL log levels are shown. Equivalent to \fB\-\-logLevel=OFF\fP or \fB\-\-logLevel=CRITICAL\fP\&. .TP .B \-\-logCritical Only CRITICAL log levels are shown. Equivalent to \fB\-\-logLevel=OFF\fP or \fB\-\-logLevel=CRITICAL\fP\&. .TP .B \-\-logError Only ERROR, and CRITICAL log levels are shown. Equivalent to \fB\-\-logLevel=ERROR\fP\&. .TP .B \-\-logWarning Only WARN, ERROR, and CRITICAL log levels are shown. Equivalent to \fB\-\-logLevel=WARNING\fP\&. .TP .B \-\-logInfo All log statements are shown, except DEBUG. Equivalent to \fB\-\-logLevel=INFO\fP\&. .TP .B \-\-logDebug All log statements are shown. Equivalent to \fB\-\-logLevel=DEBUG\fP\&. .TP .BI \-\-logLevel\fB= LOGLEVEL May be set to: \fBOFF\fP (or \fBCRITICAL\fP), \fBERROR\fP, \fBWARN\fP (or \fBWARNING\fP), \fBINFO\fP, or \fBDEBUG\fP\&. .TP .BI \-\-logFile \ FILEPATH Specifies a file path to write the logging output to. .TP .B \-\-rotatingLogging Turn on rotating logging, which prevents log files from getting too big (set using \fB\-\-maxLogFileSize BYTESIZE\fP). .TP .BI \-\-maxLogFileSize \ BYTESIZE The maximum size of a job log file to keep (in bytes), log files larger than this will be truncated to the last X bytes. Setting this option to zero will prevent any truncation. Setting this option to a negative value will truncate from the beginning. Default=100MiB Sets the maximum log file size in bytes (\fB\-\-rotatingLogging\fP must be active). .TP .BI \-\-log\-dir \ DIRPATH For CWL and local file system only. Log stdout and stderr (if tool requests stdout/stderr) to the DIRPATH. .UNINDENT .UNINDENT .UNINDENT .sp \fBBatch System Options\fP .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-batchSystem \ BATCHSYSTEM The type of batch system to run the job(s) with. Default = single_machine. .TP .B \-\-disableAutoDeployment Should auto\-deployment of Toil Python workflows be deactivated? If True, the workflow\(aqs Python code should be present at the same location on all workers. Default = False. .TP .BI \-\-maxJobs \ MAXJOBS Specifies the maximum number of jobs to submit to the backing scheduler at once. Not supported on Mesos or AWS Batch. Use 0 for unlimited. Defaults to unlimited. .TP .BI \-\-maxLocalJobs \ MAXLOCALJOBS Specifies the maximum number of housekeeping jobs to run simultaneously on the local system. Use 0 for unlimited. Defaults to the number of local cores. .TP .B \-\-manualMemArgs Do not add the default arguments: \(aqhv=MEMORY\(aq & \(aqh_vmem=MEMORY\(aq to the qsub call, and instead rely on TOIL_GRIDGENGINE_ARGS to supply alternative arguments. Requires that TOIL_GRIDGENGINE_ARGS be set. .TP .B \-\-runCwlInternalJobsOnWorkers Whether to run CWL internal jobs (e.g. CWLScatter) on the worker nodes instead of the primary node. If false (default), then all such jobs are run on the primary node. Setting this to true can speed up the pipeline for very large workflows with many sub\-workflows and/or scatters, provided that the worker pool is large enough. .TP .BI \-\-statePollingWait \ STATEPOLLINGWAIT Time, in seconds, to wait before doing a scheduler query for job state. Return cached results if within the waiting period. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf. .TP .BI \-\-batchLogsDir \ BATCHLOGSDIR Directory to tell the backing batch system to log into. Should be available on both the leader and the workers, if the backing batch system writes logs to the worker machines\(aq filesystems, as many HPC schedulers do. If unset, the Toil work directory will be used. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf. .TP .BI \-\-mesosEndpoint \ MESOSENDPOINT The host and port of the Mesos server separated by a colon. (default: :5050) .TP .BI \-\-mesosFrameworkId \ MESOSFRAMEWORKID Use a specific Mesos framework ID. .TP .BI \-\-mesosRole \ MESOSROLE Use a Mesos role. .TP .BI \-\-mesosName \ MESOSNAME The Mesos name to use. (default: toil) .TP .BI \-\-kubernetesHostPath \ KUBERNETES_HOST_PATH Path on Kubernetes hosts to use as shared inter\-pod temp directory. .TP .BI \-\-kubernetesOwner \ KUBERNETES_OWNER Username to mark Kubernetes jobs with. .TP .BI \-\-kubernetesServiceAccount \ KUBERNETES_SERVICE_ACCOUNT Service account to run jobs as. .TP .BI \-\-kubernetesPodTimeout \ KUBERNETES_POD_TIMEOUT Seconds to wait for a scheduled Kubernetes pod to start running. (default: 120s) .TP .BI \-\-awsBatchRegion \ AWS_BATCH_REGION The AWS region containing the AWS Batch queue to submit to. .TP .BI \-\-awsBatchQueue \ AWS_BATCH_QUEUE The name or ARN of the AWS Batch queue to submit to. .TP .BI \-\-awsBatchJobRoleArn \ AWS_BATCH_JOB_ROLE_ARN The ARN of an IAM role to run AWS Batch jobs as, so they can e.g. access a job store. Must be assumable by ecs\-tasks.amazonaws.com .TP .BI \-\-scale \ SCALE A scaling factor to change the value of all submitted tasks\(aq submitted cores. Used in single_machine batch system. Useful for running workflows on smaller machines than they were designed for, by setting a value less than 1. (default: 1) .UNINDENT .UNINDENT .UNINDENT .sp \fBData Storage Options\fP Allows configuring Toil\(aqs data storage. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-symlinkImports \ BOOL When using a filesystem based job store, CWL input files are by default symlinked in. Setting this option to True instead copies the files into the job store, which may protect them from being modified externally. When set to False and as long as caching is enabled, Toil will protect the file automatically by changing the permissions to read\-only. (Default=True) .TP .BI \-\-moveOutputs \ BOOL When using a filesystem based job store, output files are by default moved to the output directory, and a symlink to the moved exported file is created at the initial location. Setting this option to True instead copies the files into the output directory. Applies to filesystem\-based job stores only. (Default=False) .TP .BI \-\-caching \ BOOL Set caching options. This must be set to \(dqfalse\(dq to use a batch system that does not support cleanup. Set to \(dqtrue\(dq if caching is desired. .UNINDENT .UNINDENT .UNINDENT .sp \fBAutoscaling Options\fP Allows the specification of the minimum and maximum number of nodes in an autoscaled cluster, as well as parameters to control the level of provisioning. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-provisioner \ CLOUDPROVIDER The provisioner for cluster auto\-scaling. This is the main Toil \fB\-\-provisioner\fP option, and defaults to None for running on single_machine and non\-auto\-scaling batch systems. The currently supported choices are \(aqaws\(aq or \(aqgce\(aq. .TP .BI \-\-nodeTypes \ NODETYPES Specifies a list of comma\-separated node types, each of which is composed of slash\-separated instance types, and an optional spot bid set off by a colon, making the node type preemptible. Instance types may appear in multiple node types, and the same node type may appear as both preemptible and non\-preemptible. .INDENT 7.0 .TP .B Valid argument specifying two node types: c5.4xlarge/c5a.4xlarge:0.42,t2.large .TP .B Node types: c5.4xlarge/c5a.4xlarge:0.42 and t2.large .TP .B Instance types: c5.4xlarge, c5a.4xlarge, and t2.large .TP .B Semantics: Bid $0.42/hour for either c5.4xlarge or c5a.4xlarge instances, treated interchangeably, while they are available at that price, and buy t2.large instances at full price .UNINDENT .TP .BI \-\-minNodes \ MINNODES Minimum number of nodes of each type in the cluster, if using auto\-scaling. This should be provided as a comma\-separated list of the same length as the list of node types. default=0 .TP .BI \-\-maxNodes \ MAXNODES Maximum number of nodes of each type in the cluster, if using autoscaling, provided as a comma\-separated list. The first value is used as a default if the list length is less than the number of nodeTypes. default=10 .TP .BI \-\-targetTime \ TARGETTIME Sets how rapidly you aim to complete jobs in seconds. Shorter times mean more aggressive parallelization. The autoscaler attempts to scale up/down so that it expects all queued jobs will complete within targetTime seconds. (Default: 1800) .TP .BI \-\-betaInertia \ BETAINERTIA A smoothing parameter to prevent unnecessary oscillations in the number of provisioned nodes. This controls an exponentially weighted moving average of the estimated number of nodes. A value of 0.0 disables any smoothing, and a value of 0.9 will smooth so much that few changes will ever be made. Must be between 0.0 and 0.9. (Default: 0.1) .TP .BI \-\-scaleInterval \ SCALEINTERVAL The interval (seconds) between assessing if the scale of the cluster needs to change. (Default: 60) .TP .BI \-\-preemptibleCompensation \ PREEMPTIBLECOMPENSATION The preference of the autoscaler to replace preemptible nodes with non\-preemptible nodes, when preemptible nodes cannot be started for some reason. Defaults to 0.0. This value must be between 0.0 and 1.0, inclusive. A value of 0.0 disables such compensation, a value of 0.5 compensates two missing preemptible nodes with a non\-preemptible one. A value of 1.0 replaces every missing pre\-emptable node with a non\-preemptible one. .TP .BI \-\-nodeStorage \ NODESTORAGE Specify the size of the root volume of worker nodes when they are launched in gigabytes. You may want to set this if your jobs require a lot of disk space. The default value is 50. .TP .BI \-\-nodeStorageOverrides \ NODESTORAGEOVERRIDES Comma\-separated list of nodeType:nodeStorage that are used to override the default value from \fB\-\-nodeStorage\fP for the specified nodeType(s). This is useful for heterogeneous jobs where some tasks require much more disk than others. .TP .B \-\-metrics Enable the prometheus/grafana dashboard for monitoring CPU/RAM usage, queue size, and issued jobs. .TP .B \-\-assumeZeroOverhead Ignore scheduler and OS overhead and assume jobs can use every last byte of memory and disk on a node when autoscaling. .UNINDENT .UNINDENT .UNINDENT .sp \fBService Options\fP Allows the specification of the maximum number of service jobs in a cluster. By keeping this limited we can avoid nodes occupied with services causing deadlocks. (Not for CWL). .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-maxServiceJobs \ MAXSERVICEJOBS The maximum number of service jobs that can be run concurrently, excluding service jobs running on preemptible nodes. default=9223372036854775807 .TP .BI \-\-maxPreemptibleServiceJobs \ MAXPREEMPTIBLESERVICEJOBS The maximum number of service jobs that can run concurrently on preemptible nodes. default=9223372036854775807 .TP .BI \-\-deadlockWait \ DEADLOCKWAIT Time, in seconds, to tolerate the workflow running only the same service jobs, with no jobs to use them, before declaring the workflow to be deadlocked and stopping. default=60 .TP .BI \-\-deadlockCheckInterval \ DEADLOCKCHECKINTERVAL Time, in seconds, to wait between checks to see if the workflow is stuck running only service jobs, with no jobs to use them. Should be shorter than \fB\-\-deadlockWait\fP\&. May need to be increased if the batch system cannot enumerate running jobs quickly enough, or if polling for running jobs is placing an unacceptable load on a shared cluster. default=30 .UNINDENT .UNINDENT .UNINDENT .sp \fBResource Options\fP The options to specify default cores/memory requirements (if not specified by the jobs themselves), and to limit the total amount of memory/cores requested from the batch system. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-defaultMemory \ INT The default amount of memory to request for a job. Only applicable to jobs that do not specify an explicit value for this requirement. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. Default is 2.0Gi .TP .BI \-\-defaultCores \ FLOAT The default number of CPU cores to dedicate a job. Only applicable to jobs that do not specify an explicit value for this requirement. Fractions of a core (for example 0.1) are supported on some batch systems, namely Mesos and single_machine. Default is 1.0 .TP .BI \-\-defaultDisk \ INT The default amount of disk space to dedicate a job. Only applicable to jobs that do not specify an explicit value for this requirement. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. Default is 2.0Gi .TP .BI \-\-defaultAccelerators \ ACCELERATOR The default amount of accelerators to request for a job. Only applicable to jobs that do not specify an explicit value for this requirement. Each accelerator specification can have a type (gpu [default], nvidia, amd, cuda, rocm, opencl, or a specific model like nvidia\-tesla\-k80), and a count [default: 1]. If both a type and a count are used, they must be separated by a colon. If multiple types of accelerators are used, the specifications are separated by commas. Default is []. .TP .BI \-\-defaultPreemptible \ BOOL Make all jobs able to run on preemptible (spot) nodes by default. .TP .BI \-\-maxCores \ INT The maximum number of CPU cores to request from the batch system at any one time. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. .TP .BI \-\-maxMemory \ INT The maximum amount of memory to request from the batch system at any one time. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. .TP .BI \-\-maxDisk \ INT The maximum amount of disk space to request from the batch system at any one time. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. .UNINDENT .UNINDENT .UNINDENT .sp \fBOptions for rescuing/killing/restarting jobs.\fP The options for jobs that either run too long/fail or get lost (some batch systems have issues!). .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-retryCount \ RETRYCOUNT Number of times to retry a failing job before giving up and labeling job failed. default=1 .TP .B \-\-enableUnlimitedPreemptibleRetries If set, preemptible failures (or any failure due to an instance getting unexpectedly terminated) will not count towards job failures and \fB\-\-retryCount\fP\&. .TP .B \-\-doubleMem If set, batch jobs which die due to reaching memory limit on batch schedulers will have their memory doubled and they will be retried. The remaining retry count will be reduced by 1. Currently only supported by LSF. default=False. .TP .BI \-\-maxJobDuration \ MAXJOBDURATION Maximum runtime of a job (in seconds) before we kill it (this is a lower bound, and the actual time before killing the job may be longer). .TP .BI \-\-rescueJobsFrequency \ RESCUEJOBSFREQUENCY Period of time to wait (in seconds) between checking for missing/overlong jobs, that is jobs which get lost by the batch system. Expert parameter. .UNINDENT .UNINDENT .UNINDENT .sp \fBLog Management Options\fP .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .BI \-\-maxLogFileSize \ MAXLOGFILESIZE The maximum size of a job log file to keep (in bytes), log files larger than this will be truncated to the last X bytes. Setting this option to zero will prevent any truncation. Setting this option to a negative value will truncate from the beginning. Default=62.5 K .TP .BI \-\-writeLogs \ FILEPATH Write worker logs received by the leader into their own files at the specified path. Any non\-empty standard output and error from failed batch system jobs will also be written into files at this path. The current working directory will be used if a path is not specified explicitly. Note: By default only the logs of failed jobs are returned to leader. Set log level to \(aqdebug\(aq or enable \fB\-\-writeLogsFromAllJobs\fP to get logs back from successful jobs, and adjust \fB\-\-maxLogFileSize\fP to control the truncation limit for worker logs. .TP .BI \-\-writeLogsGzip \ FILEPATH Identical to \fB\-\-writeLogs\fP except the logs files are gzipped on the leader. .TP .BI \-\-writeMessages \ FILEPATH File to send messages from the leader\(aqs message bus to. .TP .B \-\-realTimeLogging Enable real\-time logging from workers to leader. .UNINDENT .UNINDENT .UNINDENT .sp \fBMiscellaneous Options\fP .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \-\-disableChaining Disables chaining of jobs (chaining uses one job\(aqs resource allocation for its successor job if possible). .TP .B \-\-disableJobStoreChecksumVerification Disables checksum verification for files transferred to/from the job store. Checksum verification is a safety check to ensure the data is not corrupted during transfer. Currently only supported for non\-streaming AWS files .TP .BI \-\-sseKey \ SSEKEY Path to file containing 32 character key to be used for server\-side encryption on awsJobStore or googleJobStore. SSE will not be used if this flag is not passed. .TP .BI \-\-setEnv \ NAME\fR,\fB \ \-e \ NAME NAME=VALUE or NAME, \-e NAME=VALUE or NAME are also valid. Set an environment variable early on in the worker. If VALUE is omitted, it will be looked up in the current environment. Independently of this option, the worker will try to emulate the leader\(aqs environment before running a job, except for some variables known to vary across systems. Using this option, a variable can be injected into the worker process itself before it is started. .TP .BI \-\-servicePollingInterval \ SERVICEPOLLINGINTERVAL Interval of time service jobs wait between polling for the existence of the keep\-alive flag (default=60) .TP .B \-\-forceDockerAppliance Disables sanity checking the existence of the docker image specified by TOIL_APPLIANCE_SELF, which Toil uses to provision mesos for autoscaling. .TP .BI \-\-statusWait \ INT Seconds to wait between reports of running jobs. (default=3600) .TP .B \-\-disableProgress Disables the progress bar shown when standard error is a terminal. .UNINDENT .UNINDENT .UNINDENT .sp \fBDebug Options\fP Debug options for finding problems or helping with testing. .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \-\-debugWorker Experimental no forking mode for local debugging. Specifically, workers are not forked and stderr/stdout are not redirected to the log. (default=False) .TP .B \-\-disableWorkerOutputCapture Let worker output go to worker\(aqs standard out/error instead of per\-job logs. .TP .BI \-\-badWorker \ BADWORKER For testing purposes randomly kill \fB\-\-badWorker\fP proportion of jobs using SIGKILL. (Default: 0.0) .TP .BI \-\-badWorkerFailInterval \ BADWORKERFAILINTERVAL When killing the job pick uniformly within the interval from 0.0 to \fB\-\-badWorkerFailInterval\fP seconds after the worker starts. (Default: 0.01) .TP .BI \-\-kill_polling_interval \ KILL_POLLING_INTERVAL Interval of time (in seconds) the leader waits between polling for the kill flag inside the job store set by the \(dqtoil kill\(dq command. (default=5) .UNINDENT .UNINDENT .UNINDENT .SS Restart Option .sp In the event of failure, Toil can resume the pipeline by adding the argument \fB\-\-restart\fP and rerunning the workflow. Toil Python workflows (but not CWL or WDL workflows) can even be edited and resumed, which is useful for development or troubleshooting. .SS Running Workflows with Services .sp Toil supports jobs, or clusters of jobs, that run as \fIservices\fP to other \fIaccessor\fP jobs. Example services include server databases or Apache Spark Clusters. As service jobs exist to provide services to accessor jobs their runtime is dependent on the concurrent running of their accessor jobs. The dependencies between services and their accessor jobs can create potential deadlock scenarios, where the running of the workflow hangs because only service jobs are being run and their accessor jobs can not be scheduled because of too limited resources to run both simultaneously. To cope with this situation Toil attempts to schedule services and accessors intelligently, however to avoid a deadlock with workflows running service jobs it is advisable to use the following parameters: .INDENT 0.0 .IP \(bu 2 \fB\-\-maxServiceJobs\fP: The maximum number of service jobs that can be run concurrently, excluding service jobs running on preemptible nodes. .IP \(bu 2 \fB\-\-maxPreemptibleServiceJobs\fP: The maximum number of service jobs that can run concurrently on preemptible nodes. .UNINDENT .sp Specifying these parameters so that at a maximum cluster size there will be sufficient resources to run accessors in addition to services will ensure that such a deadlock can not occur. .sp If too low a limit is specified then a deadlock can occur in which toil can not schedule sufficient service jobs concurrently to complete the workflow. Toil will detect this situation if it occurs and throw a \fBtoil.DeadlockException\fP exception. Increasing the cluster size and these limits will resolve the issue. .SS Setting Options directly in a Python Workflow .sp It\(aqs good to remember that commandline options can be overridden in the code of a Python workflow. For example, \fI\%toil.job.Job.Runner.getDefaultOptions()\fP can be used to get the default Toil options, ignoring what was passed on the command line. In this example, this is used to ignore command\-line options and always run with the \(dq./toilWorkflow\(dq directory as the jobstore: .INDENT 0.0 .INDENT 3.5 .sp .EX options = Job.Runner.getDefaultOptions(\(dq./toilWorkflow\(dq) # Get the options object with Toil(options) as toil: toil.start(Job()) # Run the root job .EE .UNINDENT .UNINDENT .sp However, each option can be explicitly set within the workflow by modifying the options object. In this example, we are setting \fBlogLevel = \(dqDEBUG\(dq\fP (all log statements are shown) and \fBclean=\(dqALWAYS\(dq\fP (always delete the jobstore) like so: .INDENT 0.0 .INDENT 3.5 .sp .EX options = Job.Runner.getDefaultOptions(\(dq./toilWorkflow\(dq) # Get the options object options.logLevel = \(dqDEBUG\(dq # Set the log level to the debug level. options.clean = \(dqALWAYS\(dq # Always delete the jobStore after a run with Toil(options) as toil: toil.start(Job()) # Run the root job .EE .UNINDENT .UNINDENT .sp However, the usual incantation is to accept commandline args from the user with the following: .INDENT 0.0 .INDENT 3.5 .sp .EX parser = Job.Runner.getDefaultArgumentParser() # Get the parser options = parser.parse_args() # Parse user args to create the options object with Toil(options) as toil: toil.start(Job()) # Run the root job .EE .UNINDENT .UNINDENT .sp We can also have code in the workflow to overwrite user supplied arguments: .INDENT 0.0 .INDENT 3.5 .sp .EX parser = Job.Runner.getDefaultArgumentParser() # Get the parser options = parser.parse_args() # Parse user args to create the options object options.logLevel = \(dqDEBUG\(dq # Set the log level to the debug level. options.clean = \(dqALWAYS\(dq # Always delete the jobStore after a run with Toil(options) as toil: toil.start(Job()) # Run the root job .EE .UNINDENT .UNINDENT .SH TOIL UTILITIES .sp Toil includes some utilities for inspecting or manipulating workflows during and after their execution. (There are additional \fI\%Toil Cluster Utilities\fP available for working with Toil\-managed clusters in the cloud.) .sp The generic \fBtoil\fP subcommand utilities are: .INDENT 0.0 .INDENT 3.5 \fBstats\fP \-\-\- Reports runtime and resource usage for all jobs in a specified jobstore (workflow must have originally been run using the \fB\-\-stats\fP option). .sp \fBstatus\fP \-\-\- Inspects a job store to see which jobs have failed, run successfully, etc. .sp \fBclean\fP \-\-\- Delete the job store used by a previous Toil workflow invocation. .sp \fBkill\fP \-\-\- Kills any running jobs in a rogue toil. .UNINDENT .UNINDENT .sp For information on a specific utility, run it with the \fB\-\-help\fP option: .INDENT 0.0 .INDENT 3.5 .sp .EX toil stats \-\-help .EE .UNINDENT .UNINDENT .SS Stats Command .sp To use the stats command, a workflow must first be run using the \fB\-\-stats\fP option. Using this command makes certain that toil does not delete the job store, no matter what other options are specified (i.e. normally the option \fB\-\-clean=always\fP would delete the job store, but \fB\-\-stats\fP will override this). .SS Running an Example .sp We can run an example workflow and record stats: .INDENT 0.0 .INDENT 3.5 .sp .EX python3 discoverfiles.py file:my\-jobstore \-\-stats .EE .UNINDENT .UNINDENT .sp Where \fBdiscoverfiles.py\fP is the following: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job import math import time from multiprocessing import Process def think(seconds): start = time.time() while time.time() \- start < seconds: # Use CPU math.sqrt(123456) class TimeWaster(Job): def __init__(self, time_to_think, time_to_waste, space_to_waste, *args, **kwargs): self.time_to_think = time_to_think self.time_to_waste = time_to_waste self.space_to_waste = space_to_waste super().__init__(*args, **kwargs) def run(self, fileStore): # Waste some space file_path = fileStore.getLocalTempFile() with open(file_path, \(aqw\(aq) as stream: for i in range(self.space_to_waste): stream.write(\(dqX\(dq) # Do some \(dquseful\(dq compute processes = [] for core_number in range(max(1, self.cores)): # Use all the assigned cores to think p = Process(target=think, args=(self.time_to_think,)) p.start() processes.append(p) for p in processes: p.join() # Also waste some time time.sleep(self.time_to_waste) def main(): options = Job.Runner.getDefaultArgumentParser().parse_args() job1 = TimeWaster(0, 0, 0, displayName=\(aqdoNothing\(aq) job2 = TimeWaster(10, 0, 4096, displayName=\(aqefficientJob\(aq) job3 = TimeWaster(10, 0, 1024, cores=4, displayName=\(aqmultithreadedJob\(aq) job4 = TimeWaster(1, 9, 65536, displayName=\(aqinefficientJob\(aq) job1.addChild(job2) job1.addChild(job3) job3.addChild(job4) with Toil(options) as toil: if not toil.options.restart: toil.start(job1) else: toil.restart() if __name__ == \(aq__main__\(aq: main() .EE .UNINDENT .UNINDENT .sp Notice the \fBdisplayName\fP key, which can rename a job, giving it an alias when it is finally displayed in stats. .SS Displaying Stats .sp To see the runtime and resources used for each job when it was run, type .INDENT 0.0 .INDENT 3.5 .sp .EX toil stats file:my\-jobstore .EE .UNINDENT .UNINDENT .sp This should output something like the following: .INDENT 0.0 .INDENT 3.5 .sp .EX Batch System: single_machine Default Cores: 1 Default Memory: 2097152KiB Max Cores: unlimited Local CPU Time: 55.54 core·s Overall Runtime: 26.23 s Worker Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 3 | 0.34 10.83 10.80 21.23 32.40 | 0.33 10.43 17.94 43.07 53.83 | 0.01 0.40 14.08 41.85 42.25 | 177168Ki 179312Ki 178730Ki 179712Ki 536192Ki | 0Ki 4Ki 22Ki 64Ki 68Ki Job Worker Jobs | min med ave max | 1 1 1.3333 2 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 4 | 0.33 10.83 8.10 10.85 32.38 | 0.33 10.43 13.46 41.70 53.82 | 0.01 1.68 2.78 9.02 11.10 | 177168Ki 179488Ki 178916Ki 179696Ki 715664Ki | 0Ki 4Ki 18Ki 64Ki 72Ki multithreadedJob Total Cores: 4.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 10.85 10.85 10.85 10.85 10.85 | 41.70 41.70 41.70 41.70 41.70 | 1.68 1.68 1.68 1.68 1.68 | 179488Ki 179488Ki 179488Ki 179488Ki 179488Ki | 4Ki 4Ki 4Ki 4Ki 4Ki efficientJob Total Cores: 1.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 10.83 10.83 10.83 10.83 10.83 | 10.43 10.43 10.43 10.43 10.43 | 0.40 0.40 0.40 0.40 0.40 | 179312Ki 179312Ki 179312Ki 179312Ki 179312Ki | 4Ki 4Ki 4Ki 4Ki 4Ki inefficientJob Total Cores: 1.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 10.38 10.38 10.38 10.38 10.38 | 1.36 1.36 1.36 1.36 1.36 | 9.02 9.02 9.02 9.02 9.02 | 179696Ki 179696Ki 179696Ki 179696Ki 179696Ki | 64Ki 64Ki 64Ki 64Ki 64Ki doNothing Total Cores: 1.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 0.33 0.33 0.33 0.33 0.33 | 0.33 0.33 0.33 0.33 0.33 | 0.01 0.01 0.01 0.01 0.01 | 177168Ki 177168Ki 177168Ki 177168Ki 177168Ki | 0Ki 0Ki 0Ki 0Ki 0Ki .EE .UNINDENT .UNINDENT .sp This report gives information on the resources used by your workflow. \fBNote that right now it does NOT track CPU and memory used inside Docker containers\fP, only Singularity containers. .sp There are three parts to this report. .SS Overall Summary .sp At the top is a section with overall summary statistics for the run: .INDENT 0.0 .INDENT 3.5 .sp .EX Batch System: single_machine Default Cores: 1 Default Memory: 2097152KiB Max Cores: unlimited Local CPU Time: 55.54 core·s Overall Runtime: 26.23 s .EE .UNINDENT .UNINDENT .sp This lists some important the settings for the Toil batch system that actually executed jobs. It also lists: .INDENT 0.0 .IP \(bu 2 The CPU time used on the local machine, in core seconds. This includes time used by the Toil leader itself (excluding some startup time), and time used by jobs that run under the leader (which, for the \fBsingle_machine\fP batch system, is all jobs). It does \fBnot\fP include CPU used by jobs that ran on other machines. .IP \(bu 2 The overall wall\-clock runtime of the workflow in seconds, as measured by the leader. .UNINDENT .sp These latter two numbers don\(aqt count some startup/shutdown time spent loading and saving files, so you still may want to use the \fBtime\fP shell built\-in to time your Toil runs overall. .SS Worker Summary .sp After the overall summary, there is a section with statistics about the Toil worker processes, which Toil used to execute your workflow\(aqs jobs: .INDENT 0.0 .INDENT 3.5 .sp .EX Worker Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 3 | 0.34 10.83 10.80 21.23 32.40 | 0.33 10.43 17.94 43.07 53.83 | 0.01 0.40 14.08 41.85 42.25 | 177168Ki 179312Ki 178730Ki 179712Ki 536192Ki | 0Ki 4Ki 22Ki 64Ki 68Ki .EE .UNINDENT .UNINDENT .INDENT 0.0 .IP \(bu 2 The \fBCount\fP column shows that, to run this workflow, Toil had to submit 3 Toil worker processes to the backing scheduler. (In this case, it ran them all on the local machine.) .IP \(bu 2 The \fBReal Time\fP column shows satistics about the wall clock times that all the worker process took. All the sub\-column values are in seconds. .IP \(bu 2 The \fBCPU Time\fP column shows statistics about the CPU usage amounts of all the worker processes. All the sub\-column values are in core seconds. .IP \(bu 2 The \fBCPU Wait\fP column shows statistics about CPU time reserved for but \fBnot\fP consumed by worker processes. In this example, the \fBmax\fP and \fBtotal\fP are relatively high compared to both real time and CPU time, indicating that a lot of reserved CPU time went unused. This can indicate that the workflow is overestimating its required cores, that small jobs are running in the same resource reservations as large jobs via chaining, or that the workflow is having to wait around for slow disk I/O. .IP \(bu 2 The \fBMemory\fP column shows the peak memory usage of each worker process and its child processes. .IP \(bu 2 The \fBDisk\fP column shows the disk usage in each worker. This is polled at the \fBend\fP of each job that is run by the worker, so it may not always reflect the actual peak disk usage. .UNINDENT .SS Job Breakdown .sp Finally, there is the breakdown of resource usage by jobs. This starts with a table summarizing the counts of jobs that ran on each worker: .INDENT 0.0 .INDENT 3.5 .sp .EX Job Worker Jobs | min med ave max | 1 1 1.3333 2 .EE .UNINDENT .UNINDENT .sp In this example, most of the workers ran one job each, but one worker managed to run two jobs, via chaining. (Jobs will chain when a job has only one dependant job, which in turn depends on only that first job, and the second job needs no more resources than the first job did.) .sp Next, we have statistics for resource usage over all jobs together: .INDENT 0.0 .INDENT 3.5 .sp .EX Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 4 | 0.33 10.83 8.10 10.85 32.38 | 0.33 10.43 13.46 41.70 53.82 | 0.01 1.68 2.78 9.02 11.10 | 177168Ki 179488Ki 178916Ki 179696Ki 715664Ki | 0Ki 4Ki 18Ki 64Ki 72Ki .EE .UNINDENT .UNINDENT .sp And finally, for each kind of job (as determined by the job\(aqs \fBdisplayName\fP), we have statistics summarizing the resources used by the instances of that kind of job: .INDENT 0.0 .INDENT 3.5 .sp .EX multithreadedJob Total Cores: 4.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 10.85 10.85 10.85 10.85 10.85 | 41.70 41.70 41.70 41.70 41.70 | 1.68 1.68 1.68 1.68 1.68 | 179488Ki 179488Ki 179488Ki 179488Ki 179488Ki | 4Ki 4Ki 4Ki 4Ki 4Ki efficientJob Total Cores: 1.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 10.83 10.83 10.83 10.83 10.83 | 10.43 10.43 10.43 10.43 10.43 | 0.40 0.40 0.40 0.40 0.40 | 179312Ki 179312Ki 179312Ki 179312Ki 179312Ki | 4Ki 4Ki 4Ki 4Ki 4Ki inefficientJob Total Cores: 1.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 10.38 10.38 10.38 10.38 10.38 | 1.36 1.36 1.36 1.36 1.36 | 9.02 9.02 9.02 9.02 9.02 | 179696Ki 179696Ki 179696Ki 179696Ki 179696Ki | 64Ki 64Ki 64Ki 64Ki 64Ki doNothing Total Cores: 1.0 Count | Real Time (s)* | CPU Time (core·s) | CPU Wait (core·s) | Memory (B) | Disk (B) n | min med* ave max total | min med ave max total | min med ave max total | min med ave max total | min med ave max total 1 | 0.33 0.33 0.33 0.33 0.33 | 0.33 0.33 0.33 0.33 0.33 | 0.01 0.01 0.01 0.01 0.01 | 177168Ki 177168Ki 177168Ki 177168Ki 177168Ki | 0Ki 0Ki 0Ki 0Ki 0Ki .EE .UNINDENT .UNINDENT .sp For each job, we first list its name, and then the total cores that it asked for, summed across all instances of it. Then we show a table of statistics. .sp Here the \fB*\fP marker in the table headers becomes relevant; it shows that jobs are being sorted by the median of the real time used. You can control this with the \fB\-\-sortCategory\fP option. .sp The columns meanings are the same as for the workers: .INDENT 0.0 .IP \(bu 2 The \fBCount\fP column shows the number of jobs of each type that ran. .IP \(bu 2 The \fBReal Time\fP column shows satistics about the wall clock times that instances of the job type took. All the sub\-column values are in seconds. .IP \(bu 2 The \fBCPU Time\fP column shows statistics about the CPU usage amounts of each job. Note that \fBmultithreadedJob\fP managed to use CPU time at faster than one core second per second, because it reserved multiple cores and ran multiple threads. .IP \(bu 2 The \fBCPU Wait\fP column shows statistics about CPU time reserved for but \fBnot\fP consumed by jobs. Note that \fBinefficientJob\fP used hardly any of the cores it requested for most of its real time. .IP \(bu 2 The \fBMemory\fP column shows the peak memory usage of each job. .IP \(bu 2 The \fBDisk\fP column shows the disk usage at the \fBend\fP of each job. It may not always reflect the actual peak disk usage. .UNINDENT .SS Example Cleanup .sp Once we\(aqre done looking at the stats, we can clean up the job store by running: .INDENT 0.0 .INDENT 3.5 .sp .EX toil clean file:my\-jobstore .EE .UNINDENT .UNINDENT .SS Status Command .sp Continuing the example from the stats section above, if we ran our workflow with the command .INDENT 0.0 .INDENT 3.5 .sp .EX python3 discoverfiles.py file:my\-jobstore \-\-stats .EE .UNINDENT .UNINDENT .sp We could interrogate our jobstore with the status command, for example: .INDENT 0.0 .INDENT 3.5 .sp .EX toil status file:my\-jobstore .EE .UNINDENT .UNINDENT .sp If the run was successful, this would not return much valuable information, something like .INDENT 0.0 .INDENT 3.5 .sp .EX 2018\-01\-11 19:31:29,739 \- toil.lib.bioio \- INFO \- Root logger is at level \(aqINFO\(aq, \(aqtoil\(aq logger at level \(aqINFO\(aq. 2018\-01\-11 19:31:29,740 \- toil.utils.toilStatus \- INFO \- Parsed arguments 2018\-01\-11 19:31:29,740 \- toil.utils.toilStatus \- INFO \- Checking if we have files for Toil The root job of the job store is absent, the workflow completed successfully. .EE .UNINDENT .UNINDENT .sp Otherwise, the \fBstatus\fP command should return the following: .INDENT 0.0 .INDENT 3.5 There are \fBx\fP unfinished jobs, \fBy\fP parent jobs with children, \fBz\fP jobs with services, \fBa\fP services, and \fBb\fP totally failed jobs currently in \fBc\fP\&. .UNINDENT .UNINDENT .SS Clean Command .sp If a Toil pipeline didn\(aqt finish successfully, or was run using \fB\-\-clean=always\fP or \fB\-\-stats\fP, the job store will exist until it is deleted. \fBtoil clean \fP ensures that all artifacts associated with a job store are removed. This is particularly useful for deleting AWS job stores, which reserves an SDB domain as well as an S3 bucket. .sp The deletion of the job store can be modified by the \fB\-\-clean\fP argument, and may be set to \fBalways\fP, \fBonError\fP, \fBnever\fP, or \fBonSuccess\fP (default). .sp Temporary directories where jobs are running can also be saved from deletion using the \fB\-\-cleanWorkDir\fP, which has the same options as \fB\-\-clean\fP\&. This option should only be run when debugging, as intermediate jobs will fill up disk space. .SS Kill Command .sp To kill all currently running jobs for a given jobstore, use the command .INDENT 0.0 .INDENT 3.5 .sp .EX toil kill file:my\-jobstore .EE .UNINDENT .UNINDENT .SH TOIL DEBUGGING .sp Toil has a number of tools to assist in debugging. Here we provide help in working through potential problems that a user might encounter in attempting to run a workflow. .SS Introspecting the Jobstore .sp Note: Currently these features are only implemented for use locally (single machine) with the fileJobStore. .sp To view what files currently reside in the jobstore, run the following command: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil debug\-file file:path\-to\-jobstore\-directory \e \-\-listFilesInJobStore .EE .UNINDENT .UNINDENT .sp When run from the commandline, this should generate a file containing the contents of the job store (in addition to displaying a series of log messages to the terminal). This file is named \(dqjobstore_files.txt\(dq by default and will be generated in the current working directory. .sp If one wishes to copy any of these files to a local directory, one can run for example: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil debug\-file file:path\-to\-jobstore \e \-\-fetch overview.txt *.bam *.fastq \e \-\-localFilePath=/home/user/localpath .EE .UNINDENT .UNINDENT .sp To fetch \fBoverview.txt\fP, and all \fB\&.bam\fP and \fB\&.fastq\fP files. This can be used to recover previously used input and output files for debugging or reuse in other workflows, or use in general debugging to ensure that certain outputs were imported into the jobStore. .SS Stats and Status .sp See \fI\%Status Command\fP for more about gathering statistics about job success, runtime, and resource usage from workflows. .SS Using a Python debugger .sp If you execute a workflow using the \fB\-\-debugWorker\fP flag, Toil will not fork in order to run jobs, which means you can either use \fI\%pdb\fP, or an \fI\%IDE that supports debugging Python\fP as you would normally. Note that the \fB\-\-debugWorker\fP flag will only work with the \fBsingle_machine\fP batch system (the default), and not any of the custom job schedulers. .SH RUNNING IN THE CLOUD .sp Toil supports Amazon Web Services (AWS) and Google Compute Engine (GCE) in the cloud and has autoscaling capabilities that can adapt to the size of your workflow, whether your workflow requires 10 instances or 20,000. .sp Toil does this by creating a virtual cluster running \fI\%Kubernetes\fP\&. Kubernetes requires a leader node to coordinate the workflow, and worker nodes to execute the various tasks within the workflow. As the workflow runs, Kubernetes will \(dqautoscale\(dq, creating and terminating workers as needed to meet the demands of the workflow. Historically, Toil has spun up clusters with \fI\%Apache Mesos\fP, but it is no longer recommended. .sp Once a user is familiar with the basics of running Toil locally (specifying a \fI\%jobStore\fP, and how to write a workflow), they can move on to the guides below to learn how to translate these workflows into cloud ready workflows. .SS Managing a Cluster of Virtual Machines (Provisioning) .sp Toil can launch and manage a cluster of virtual machines to run using the \fIprovisioner\fP to run a workflow distributed over several nodes. The provisioner also has the ability to automatically scale up or down the size of the cluster to handle dynamic changes in computational demand (autoscaling). Currently we have working provisioners with AWS and GCE (Azure support has been deprecated). .sp Toil uses \fI\%Kubernetes\fP as the \fI\%Batch System\fP\&. .sp See here for instructions for \fI\%Running in AWS\fP\&. .sp See here for instructions for \fI\%Running in Google Compute Engine (GCE)\fP\&. .sp Toil offers a suite of commands for using the provisioners to manage clusters. .SS Toil Cluster Utilities .sp In addition to the generic \fI\%Toil Utilities\fP, there are several utilities used for starting and managing a Toil cluster using the AWS or GCE provisioners. They are installed via the \fB[aws]\fP or \fB[google]\fP extra. For installation details see \fI\%Toil Provisioner\fP\&. .sp The \fBtoil\fP cluster subcommands are: .INDENT 0.0 .INDENT 3.5 \fBdestroy\-cluster\fP \-\-\- For autoscaling. Terminates the specified cluster and associated resources. .sp \fBlaunch\-cluster\fP \-\-\- For autoscaling. This is used to launch a toil leader instance with the specified provisioner. .sp \fBrsync\-cluster\fP \-\-\- For autoscaling. Used to transfer files to a cluster launched with \fBtoil launch\-cluster\fP\&. .sp \fBssh\-cluster\fP \-\-\- SSHs into the toil appliance container running on the leader of the cluster. .UNINDENT .UNINDENT .sp For information on a specific utility, run it with the \fB\-\-help\fP option: .INDENT 0.0 .INDENT 3.5 .sp .EX toil launch\-cluster \-\-help .EE .UNINDENT .UNINDENT .sp The cluster utilities can be used for \fI\%Running in Google Compute Engine (GCE)\fP and \fI\%Running in AWS\fP\&. .sp \fBTIP:\fP .INDENT 0.0 .INDENT 3.5 By default, all of the cluster utilities expect to be running on AWS. To run with Google you will need to specify the \fB\-\-provisioner gce\fP option for each utility. .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Boto must be \fI\%configured\fP with AWS credentials before using cluster utilities. .sp \fI\%Running in Google Compute Engine (GCE)\fP contains instructions for .UNINDENT .UNINDENT .SS Launch\-Cluster Command .sp Running \fBtoil launch\-cluster\fP starts up a leader for a cluster. Workers can be added to the initial cluster by specifying the \fB\-w\fP option. An example would be .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil launch\-cluster my\-cluster \e \-\-leaderNodeType t2.small \-z us\-west\-2a \e \-\-keyPairName your\-AWS\-key\-pair\-name \e \-\-nodeTypes m3.large,t2.micro \-w 1,4 .EE .UNINDENT .UNINDENT .sp Options are listed below. These can also be displayed by running .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil launch\-cluster \-\-help .EE .UNINDENT .UNINDENT .sp launch\-cluster\(aqs main positional argument is the clusterName. This is simply the name of your cluster. If it does not exist yet, Toil will create it for you. .sp \fBLaunch\-Cluster Options\fP .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \-\-help \-h also accepted. Displays this help menu. .TP .BI \-\-tempDirRoot \ TEMPDIRROOT Path to the temporary directory where all temp files are created, by default uses the current working directory as the base. .TP .B \-\-version Display version. .TP .BI \-\-provisioner \ CLOUDPROVIDER \-p CLOUDPROVIDER also accepted. The provisioner for cluster auto\-scaling. Both AWS and GCE are currently supported. .TP .BI \-\-zone \ ZONE \-z ZONE also accepted. The availability zone of the leader. This parameter can also be set via the TOIL_AWS_ZONE or TOIL_GCE_ZONE environment variables, or by the ec2_region_name parameter in your .boto file if using AWS, or derived from the instance metadata if using this utility on an existing EC2 instance. .TP .BI \-\-leaderNodeType \ LEADERNODETYPE Non\-preemptable node type to use for the cluster leader. .TP .BI \-\-keyPairName \ KEYPAIRNAME The name of the AWS or ssh key pair to include on the instance. .TP .BI \-\-owner \ OWNER The owner tag for all instances. If not given, the value in TOIL_OWNER_TAG will be used, or else the value of \fB\-\-keyPairName\fP\&. .TP .BI \-\-boto \ BOTOPATH The path to the boto credentials directory. This is transferred to all nodes in order to access the AWS jobStore from non\-AWS instances. .TP .BI \-\-tag \ KEYVALUE KEYVALUE is specified as KEY=VALUE. \-t KEY=VALUE also accepted. Tags are added to the AWS cluster for this node and all of its children. Tags are of the form: \fB\-t key1=value1\fP \fB\-\-tag key2=value2\fP\&. Multiple tags are allowed and each tag needs its own flag. By default the cluster is tagged with: { \(dqName\(dq: clusterName, \(dqOwner\(dq: IAM username }. .TP .BI \-\-vpcSubnet \ VPCSUBNET VPC subnet ID to launch cluster leader in. Uses default subnet if not specified. This subnet needs to have auto assign IPs turned on. .TP .BI \-\-nodeTypes \ NODETYPES Comma\-separated list of node types to create while launching the leader. The syntax for each node type depends on the provisioner used. For the AWS provisioner this is the name of an EC2 instance type followed by a colon and the price in dollars to bid for a spot instance, for example \(aqc3.8xlarge:0.42\(aq. Must also provide the \fB\-\-workers\fP argument to specify how many workers of each node type to create. .TP .BI \-\-workers \ WORKERS \-w WORKERS also accepted. Comma\-separated list of the number of workers of each node type to launch alongside the leader when the cluster is created. This can be useful if running toil without auto\-scaling but with need of more hardware support. .TP .BI \-\-leaderStorage \ LEADERSTORAGE Specify the size (in gigabytes) of the root volume for the leader instance. This is an EBS volume. .TP .BI \-\-nodeStorage \ NODESTORAGE Specify the size (in gigabytes) of the root volume for any worker instances created when using the \-w flag. This is an EBS volume. .TP .BI \-\-nodeStorageOverrides \ NODESTORAGEOVERRIDES Comma\-separated list of nodeType:nodeStorage that are used to override the default value from \fB\-\-nodeStorage\fP for the specified nodeType(s). This is useful for heterogeneous jobs where some tasks require much more disk than others. .UNINDENT .UNINDENT .UNINDENT .sp \fBLogging Options\fP .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \-\-logOff Same as \fB\-\-logCritical\fP\&. .TP .B \-\-logCritical Turn on logging at level CRITICAL and above. (default is INFO) .TP .B \-\-logError Turn on logging at level ERROR and above. (default is INFO) .TP .B \-\-logWarning Turn on logging at level WARNING and above. (default is INFO) .TP .B \-\-logInfo Turn on logging at level INFO and above. (default is INFO) .TP .B \-\-logDebug Turn on logging at level DEBUG and above. (default is INFO) .TP .BI \-\-logLevel \ LOGLEVEL Log at given level (may be either OFF (or CRITICAL), ERROR, WARN (or WARNING), INFO or DEBUG). (default is INFO) .TP .BI \-\-logFile \ LOGFILE File to log in. .TP .B \-\-rotatingLogging Turn on rotating logging, which prevents log files getting too big. .UNINDENT .UNINDENT .UNINDENT .SS Ssh\-Cluster Command .sp Toil provides the ability to ssh into the leader of the cluster. This can be done as follows: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil ssh\-cluster CLUSTER\-NAME\-HERE .EE .UNINDENT .UNINDENT .sp This will open a shell on the Toil leader and is used to start an \fI\%Running a Workflow with Autoscaling\fP run. Issues with docker prevent using \fBscreen\fP and \fBtmux\fP when sshing the cluster (The shell doesn\(aqt know that it is a TTY which prevents it from allocating a new screen session). This can be worked around via .INDENT 0.0 .INDENT 3.5 .sp .EX $ script $ screen .EE .UNINDENT .UNINDENT .sp Simply running \fBscreen\fP within \fBscript\fP will get things working properly again. .sp Finally, you can execute remote commands with the following syntax: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil ssh\-cluster CLUSTER\-NAME\-HERE remoteCommand .EE .UNINDENT .UNINDENT .sp It is not advised that you run your Toil workflow using remote execution like this unless a tool like \fI\%nohup\fP is used to ensure the process does not die if the SSH connection is interrupted. .sp For an example usage, see \fI\%Running a Workflow with Autoscaling\fP\&. .SS Rsync\-Cluster Command .sp The most frequent use case for the \fBrsync\-cluster\fP utility is deploying your workflow code to the Toil leader. Note that the syntax is the same as traditional \fI\%rsync\fP with the exception of the hostname before the colon. This is not needed in \fBtoil rsync\-cluster\fP since the hostname is automatically determined by Toil. .sp Here is an example of its usage: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil rsync\-cluster CLUSTER\-NAME\-HERE \e ~/localFile :/remoteDestination .EE .UNINDENT .UNINDENT .SS Destroy\-Cluster Command .sp The \fBdestroy\-cluster\fP command is the advised way to get rid of any Toil cluster launched using the \fI\%Launch\-Cluster Command\fP command. It ensures that all attached nodes, volumes, security groups, etc. are deleted. If a node or cluster is shut down using Amazon\(aqs online portal residual resources may still be in use in the background. To delete a cluster run .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil destroy\-cluster CLUSTER\-NAME\-HERE .EE .UNINDENT .UNINDENT .SS Storage (Toil jobStore) .sp Toil can make use of cloud storage such as AWS or Google buckets to take care of storage needs. .sp This is useful when running Toil in single machine mode on any cloud platform since it allows you to make use of their integrated storage systems. .sp For an overview of the job store see \fI\%Job Store\fP\&. .sp For instructions configuring a particular job store see: .INDENT 0.0 .IP \(bu 2 \fI\%AWS Job Store\fP .IP \(bu 2 \fI\%Google Job Store\fP .UNINDENT .SH CLOUD PLATFORMS .SS Running on Kubernetes .sp \fI\%Kubernetes\fP is a very popular container orchestration tool that has become a \fIde facto\fP cross\-cloud\-provider API for accessing cloud resources. Major cloud providers like \fI\%Amazon\fP, \fI\%Microsoft\fP, Kubernetes owner \fI\%Google\fP, and \fI\%DigitalOcean\fP have invested heavily in making Kubernetes work well on their platforms, by writing their own deployment documentation and developing provider\-managed Kubernetes\-based products. Using \fI\%minikube\fP, Kubernetes can even be run on a single machine. .sp Toil supports running Toil workflows against a Kubernetes cluster, either in the cloud or deployed on user\-owned hardware. .SS Preparing your Kubernetes environment .INDENT 0.0 .IP 1. 3 \fBGet a Kubernetes cluster\fP .sp To run Toil workflows on Kubernetes, you need to have a Kubernetes cluster set up. This will not be covered here, but there are many options available, and which one you choose will depend on which cloud ecosystem if any you use already, and on pricing. If you are just following along with the documentation, use \fBminikube\fP on your local machine. .sp Alternatively, Toil can set up a Kubernetes cluster for you with the \fI\%Toil provisioner\fP\&. Follow \fI\%this\fP guide to get started with a Toil\-managed Kubernetes cluster on AWS. .sp \fBNote that currently the only way to run a Toil workflow on Kubernetes is to use the AWS Job Store, so your Kubernetes workflow will currently have to store its data in Amazon\(aqs cloud regardless of where you run it. This can result in significant egress charges from Amazon if you run it outside of Amazon.\fP .sp Kubernetes Cluster Providers: .INDENT 3.0 .IP \(bu 2 Your own institution .IP \(bu 2 \fI\%Amazon EKS\fP .IP \(bu 2 \fI\%Microsoft Azure AKS\fP .IP \(bu 2 \fI\%Google GKE\fP .IP \(bu 2 \fI\%DigitalOcean Kubernetes\fP .IP \(bu 2 \fI\%minikube\fP .UNINDENT .IP 2. 3 \fBGet a Kubernetes context on your local machine\fP .sp There are two main ways to run Toil workflows on Kubernetes. You can either run the Toil leader on a machine outside the cluster, with jobs submitted to and run on the cluster, or you can submit the Toil leader itself as a job and have it run inside the cluster. Either way, you will need to configure your own machine to be able to submit jobs to the Kubernetes cluster. Generally, this involves creating and populating a file named \fB\&.kube/config\fP in your user\(aqs home directory, and specifying the cluster to connect to, the certificate and token information needed for mutual authentication, and the Kubernetes namespace within which to work. However, Kubernetes configuration can also be picked up from other files in the \fB\&.kube\fP directory, environment variables, and the enclosing host when running inside a Kubernetes\-managed container. .sp You will have to do different things here depending on where you got your Kubernetes cluster: .INDENT 3.0 .IP \(bu 2 \fI\%Configuring for Amazon EKS\fP .IP \(bu 2 \fI\%Configuring for Microsoft Azure AKS\fP .IP \(bu 2 \fI\%Configuring for Google GKE\fP .IP \(bu 2 \fI\%Configuring for DigitalOcean Kubernetes Clusters\fP .IP \(bu 2 \fI\%Configuring for minikube\fP .UNINDENT .sp Toil\(aqs internal Kubernetes configuration logic mirrors that of the \fBkubectl\fP command. Toil workflows will use the current \fBkubectl\fP context to launch their Kubernetes jobs. .IP 3. 3 \fBIf running the Toil leader in the cluster, get a service account\fP .sp If you are going to run your workflow\(aqs leader within the Kubernetes cluster (see \fI\%Option 1: Running the Leader Inside Kubernetes\fP), you will need a service account in your chosen Kubernetes namespace. Most namespaces should have a service account named \fBdefault\fP which should work fine. If your cluster requires you to use a different service account, you will need to obtain its name and use it when launching the Kubernetes job containing the Toil leader. .IP 4. 3 \fBSet up appropriate permissions\fP .sp Your local Kubernetes context and/or the service account you are using to run the leader in the cluster will need to have certain permissions in order to run the workflow. Toil needs to be able to interact with jobs and pods in the cluster, and to retrieve pod logs. You as a user may need permission to set up an AWS credentials secret, if one is not already available. Additionally, it is very useful for you as a user to have permission to interact with nodes, and to shell into pods. .sp The appropriate permissions may already be available to you and your service account by default, especially in managed or ease\-of\-use\-optimized setups such as EKS or minikube. .sp However, if the appropriate permissions are not already available, you or your cluster administrator will have to grant them manually. The following \fBRole\fP (\fBtoil\-user\fP) and \fBClusterRole\fP (\fBnode\-reader\fP), to be applied with \fBkubectl apply \-f filename.yaml\fP, should grant sufficient permissions to run Toil workflows when bound to your account and the service account used by Toil workflows. Be sure to replace \fBYOUR_NAMESPACE_HERE\fP with the namespace you are running your workflows in .INDENT 3.0 .INDENT 3.5 .sp .EX apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: YOUR_NAMESPACE_HERE name: toil\-user rules: \- apiGroups: [\(dq*\(dq] resources: [\(dq*\(dq] verbs: [\(dqexplain\(dq, \(dqget\(dq, \(dqwatch\(dq, \(dqlist\(dq, \(dqdescribe\(dq, \(dqlogs\(dq, \(dqattach\(dq, \(dqexec\(dq, \(dqport\-forward\(dq, \(dqproxy\(dq, \(dqcp\(dq, \(dqauth\(dq] \- apiGroups: [\(dqbatch\(dq] resources: [\(dq*\(dq] verbs: [\(dqget\(dq, \(dqwatch\(dq, \(dqlist\(dq, \(dqcreate\(dq, \(dqrun\(dq, \(dqset\(dq, \(dqdelete\(dq] \- apiGroups: [\(dq\(dq] resources: [\(dqsecrets\(dq, \(dqpods\(dq, \(dqpods/attach\(dq, \(dqpodtemplates\(dq, \(dqconfigmaps\(dq, \(dqevents\(dq, \(dqservices\(dq] verbs: [\(dqpatch\(dq, \(dqget\(dq, \(dqupdate\(dq, \(dqwatch\(dq, \(dqlist\(dq, \(dqcreate\(dq, \(dqrun\(dq, \(dqset\(dq, \(dqdelete\(dq, \(dqexec\(dq] \- apiGroups: [\(dq\(dq] resources: [\(dqpods\(dq, \(dqpods/log\(dq] verbs: [\(dqget\(dq, \(dqlist\(dq] \- apiGroups: [\(dq\(dq] resources: [\(dqpods/exec\(dq] verbs: [\(dqcreate\(dq] .EE .UNINDENT .UNINDENT .INDENT 3.0 .INDENT 3.5 .sp .EX apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node\-reader rules: \- apiGroups: [\(dq\(dq] resources: [\(dqnodes\(dq] verbs: [\(dqget\(dq, \(dqlist\(dq, \(dqdescribe\(dq] \- apiGroups: [\(dq\(dq] resources: [\(dqnamespaces\(dq] verbs: [\(dqget\(dq, \(dqlist\(dq, \(dqdescribe\(dq] \- apiGroups: [\(dqmetrics.k8s.io\(dq] resources: [\(dq*\(dq] verbs: [\(dq*\(dq] .EE .UNINDENT .UNINDENT .sp To bind a user or service account to the \fBRole\fP or \fBClusterRole\fP and actually grant the permissions, you will need a \fBRoleBinding\fP and a \fBClusterRoleBinding\fP, respectively. Make sure to fill in the namespace, username, and service account name, and add more user stanzas if your cluster is to support multiple Toil users. .INDENT 3.0 .INDENT 3.5 .sp .EX apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: toil\-developer\-member namespace: toil subjects: \- kind: User name: YOUR_KUBERNETES_USERNAME_HERE apiGroup: rbac.authorization.k8s.io \- kind: ServiceAccount name: YOUR_SERVICE_ACCOUNT_NAME_HERE namespace: YOUR_NAMESPACE_HERE roleRef: kind: Role name: toil\-user apiGroup: rbac.authorization.k8s.io .EE .UNINDENT .UNINDENT .INDENT 3.0 .INDENT 3.5 .sp .EX apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: read\-nodes subjects: \- kind: User name: YOUR_KUBERNETES_USERNAME_HERE apiGroup: rbac.authorization.k8s.io \- kind: ServiceAccount name: YOUR_SERVICE_ACCOUNT_NAME_HERE namespace: YOUR_NAMESPACE_HERE roleRef: kind: ClusterRole name: node\-reader apiGroup: rbac.authorization.k8s.io .EE .UNINDENT .UNINDENT .UNINDENT .SS AWS Job Store for Kubernetes .sp Currently, the only job store, which is what Toil uses to exchange data between jobs, that works with jobs running on Kubernetes is the AWS Job Store. This requires that the Toil leader and Kubernetes jobs be able to connect to and use Amazon S3 and Amazon SimpleDB. It also requires that you have an Amazon Web Services account. .INDENT 0.0 .IP 1. 3 \fBGet access to AWS S3 and SimpleDB\fP .sp In your AWS account, you need to create an AWS access key. First go to the IAM dashboard; for \(dqus\-west1\(dq, the link would be: .INDENT 3.0 .INDENT 3.5 .sp .EX https://console.aws.amazon.com/iam/home?region=us\-west\-1#/home .EE .UNINDENT .UNINDENT .sp Then create an access key, and save the Access Key ID and the Secret Key. As documented in \fI\%the AWS documentation\fP: .INDENT 3.0 .IP 1. 3 On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials. .IP 2. 3 Expand the Access keys (access key ID and secret access key) section. .IP 3. 3 Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can\(aqt retrieve this secret access key again. .UNINDENT .sp Make sure that, if your AWS infrastructure requires your user to authenticate with a multi\-factor authentication (MFA) token, you obtain a second secret key and access key that don\(aqt have this requirement. The secret key and access key used to populate the Kubernetes secret that allows the jobs to contact the job store need to be usable without human intervention. .IP 2. 3 \fBConfigure AWS access from the local machine\fP .sp This only really needs to happen if you run the leader on the local machine. But we need the files in place to fill in the secret in the next step. Run: .INDENT 3.0 .INDENT 3.5 .sp .EX $ aws configure .EE .UNINDENT .UNINDENT .sp Then when prompted, enter your secret key and access key. This should create a file \fB~/.aws/credentials\fP that looks like this: .INDENT 3.0 .INDENT 3.5 .sp .EX [default] aws_access_key_id = BLAH aws_secret_access_key = blahblahblah .EE .UNINDENT .UNINDENT .IP 3. 3 \fBCreate a Kubernetes secret to give jobs access to AWS\fP .UNINDENT .INDENT 0.0 .INDENT 3.5 Go into the directory where the \fBcredentials\fP file is: .INDENT 0.0 .INDENT 3.5 .sp .EX $ cd ~/.aws .EE .UNINDENT .UNINDENT .sp Then, create a Kubernetes secret that contains it. We\(aqll call it \fBaws\-credentials\fP: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl create secret generic aws\-credentials \-\-from\-file credentials .EE .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SS Configuring Toil for your Kubernetes environment .sp To configure your workflow to run on Kubernetes, you will have to configure several environment variables, in addition to passing the \fB\-\-batchSystem kubernetes\fP option. Doing the research to figure out what values to give these variables may require talking to your cluster provider. .INDENT 0.0 .IP 1. 3 \fBTOIL_AWS_SECRET_NAME\fP is the most important, and \fBmust\fP be set to the secret that contains your AWS \fBcredentials\fP file, \fBif\fP your cluster nodes don\(aqt otherwise have access to S3 and SimpleDB (such as through IAM roles). This is required for the AWS job store to work, which is currently the only job store that can be used on Kubernetes. In this example we are using \fBaws\-credentials\fP\&. .IP 2. 3 \fBTOIL_KUBERNETES_HOST_PATH\fP \fBcan\fP be set to allow Toil jobs on the same physical host to share a cache. It should be set to a path on the host where the shared cache should be stored. It will be mounted as \fB/var/lib/toil\fP, or at \fBTOIL_WORKDIR\fP if specified, inside the container. This path must already exist on the host, and must have as much free space as your Kubernetes node offers to jobs. In this example, we are using \fB/data/scratch\fP\&. To actually make use of caching, make sure not to use \fB\-\-disableCaching\fP\&. .IP 3. 3 \fBTOIL_KUBERNETES_OWNER\fP \fBshould\fP be set to the username of the user running the Toil workflow. The jobs that Toil creates will include this username, so they can be more easily recognized, and cleaned up by the user if anything happens to the Toil leader. In this example we are using \fBdemo\-user\fP\&. .UNINDENT .sp Note that Docker containers cannot be run inside of unprivileged Kubernetes pods (which are themselves containers). The Docker daemon does not (yet) support this. Other tools, such as Singularity in its user\-namespace mode, are able to run containers from within containers. If using Singularity to run containerized tools, and you want downloaded container images to persist between Toil jobs, some setup may be required: .sp \fBOn non\-Toil managed clusters:\fP You will also want to set \fBTOIL_KUBERNETES_HOST_PATH\fP, and make sure that Singularity is downloading its containers under the Toil work directory (\fB/var/lib/toil\fP by default) by setting \fBSINGULARITY_CACHEDIR\fP\&. .sp \fBOn Toil\-managed clusters:\fP On clusters created with the \fBlaunch\-cluster\fP command, no setup is required. \fBTOIL_KUBERNETES_HOST_PATH\fP is already set to \fB/var/lib/toil\fP\&. \fBSINGULARITY_CACHEDIR\fP is set to \fB/var/lib/toil/singularity\fP which is a shared location; however, you may need to implement Singularity locking as shown below or change the Singularity cache location to somewhere else. .sp If using \fBtoil\-wdl\-runner\fP, all the necessary locking for Singularity is already in place and no work should be necessary. Else, for both Toil managed and non\-Toil managed clusters, you will need to make sure that no two jobs try to download the same container at the same time; Singularity has no synchronization or locking around its cache, but the cache is also not safe for simultaneous access by multiple Singularity invocations. Some Toil workflows use their own custom workaround logic for this problem; for example, \fI\%see this section in toil\-wdl\-runner\fP\&. .SS Running workflows .sp To run the workflow, you will need to run the Toil leader process somewhere. It can either be run inside Kubernetes as a Kubernetes job, or outside Kubernetes as a normal command. .SS Option 1: Running the Leader Inside Kubernetes .sp Once you have determined a set of environment variable values for your workflow run, write a YAML file that defines a Kubernetes job to run your workflow with that configuration. Some configuration items (such as your username, and the name of your AWS credentials secret) need to be written into the YAML so that they can be used from the leader as well. .sp Note that the leader pod will need your workflow, its other dependencies, and Toil all installed. An easy way to get Toil installed is to start with the Toil appliance image for the version of Toil you want to use. In this example, we use \fBquay.io/ucsc_cgl/toil:5.5.0\fP\&. .sp Here\(aqs an example YAML file to run a test workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX apiVersion: batch/v1 kind: Job metadata: # It is good practice to include your username in your job name. # Also specify it in TOIL_KUBERNETES_OWNER name: demo\-user\-toil\-test # Do not try and rerun the leader job if it fails spec: backoffLimit: 0 template: spec: # Do not restart the pod when the job fails, but keep it around so the # log can be retrieved restartPolicy: Never volumes: \- name: aws\-credentials\-vol secret: # Make sure the AWS credentials are available as a volume. # This should match TOIL_AWS_SECRET_NAME secretName: aws\-credentials # You may need to replace this with a different service account name as # appropriate for your cluster. serviceAccountName: default containers: \- name: main image: quay.io/ucsc_cgl/toil:5.5.0 env: # Specify your username for inclusion in job names \- name: TOIL_KUBERNETES_OWNER value: demo\-user # Specify where to find the AWS credentials to access the job store with \- name: TOIL_AWS_SECRET_NAME value: aws\-credentials # Specify where per\-host caches should be stored, on the Kubernetes hosts. # Needs to be set for Toil\(aqs caching to be efficient. \- name: TOIL_KUBERNETES_HOST_PATH value: /data/scratch volumeMounts: # Mount the AWS credentials volume \- mountPath: /root/.aws name: aws\-credentials\-vol resources: # Make sure to set these resource limits to values large enough # to accommodate the work your workflow does in the leader # process, but small enough to fit on your cluster. # # Since no request values are specified, the limits are also used # for the requests. limits: cpu: 2 memory: \(dq4Gi\(dq ephemeral\-storage: \(dq10Gi\(dq command: \- /bin/bash \- \-c \- | # This Bash script will set up Toil and the workflow to run, and run them. set \-e # We make sure to create a work directory; Toil can\(aqt hot\-deploy a # Python file from the root of the filesystem, which is where we start. mkdir /tmp/work cd /tmp/work # We make a virtual environment to allow workflow dependencies to be # hot\-deployed. # # We don\(aqt really make use of it in this example, but for workflows # that depend on PyPI packages we will need this. # # We use \-\-system\-site\-packages so that the Toil installed in the # appliance image is still available. virtualenv \-\-python python3 \-\-system\-site\-packages venv . venv/bin/activate # Now we install the workflow. Here we\(aqre using a demo workflow # from Toil itself. wget https://raw.githubusercontent.com/DataBiosphere/toil/releases/4.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py # Now we run the workflow. We make sure to use the Kubernetes batch # system and an AWS job store, and we set some generally useful # logging options. We also make sure to enable caching. python3 tutorial_helloworld.py \e aws:us\-west\-2:demouser\-toil\-test\-jobstore \e \-\-batchSystem kubernetes \e \-\-realTimeLogging \e \-\-logInfo .EE .UNINDENT .UNINDENT .sp You can save this YAML as \fBleader.yaml\fP, and then run it on your Kubernetes installation with: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl apply \-f leader.yaml .EE .UNINDENT .UNINDENT .sp To monitor the progress of the leader job, you will want to read its logs. If you are using a Kubernetes dashboard such as \fI\%k9s\fP, you can simply find the pod created for the job in the dashboard, and view its logs there. If not, you will need to locate the pod by hand. .SS Monitoring and Debugging Kubernetes Jobs and Pods .sp The following techniques are most useful for looking at the pod which holds the Toil leader, but they can also be applied to individual Toil jobs on Kubernetes, even when the leader is outside the cluster. .sp Kubernetes names pods for jobs by appending a short random string to the name of the job. You can find the name of the pod for your job by doing: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl get pods | grep demo\-user\-toil\-test demo\-user\-toil\-test\-g5496 1/1 Running 0 2m .EE .UNINDENT .UNINDENT .sp Assuming you have set \fBTOIL_KUBERNETES_OWNER\fP correctly, you should be able to find all of your workflow\(aqs pods by searching for your username: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl get pods | grep demo\-user .EE .UNINDENT .UNINDENT .sp If the status of a pod is anything other than \fBPending\fP, you will be able to view its logs with: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl logs demo\-user\-toil\-test\-g5496 .EE .UNINDENT .UNINDENT .sp This will dump the pod\(aqs logs from the beginning to now and terminate. To follow along with the logs from a running pod, add the \fB\-f\fP option: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl logs \-f demo\-user\-toil\-test\-g5496 .EE .UNINDENT .UNINDENT .sp A status of \fBImagePullBackoff\fP suggests that you have requested to use an image that is not available. Check the \fBimage\fP section of your YAML if you are looking at a leader, or the value of \fBTOIL_APPLIANCE_SELF\fP if you are delaying with a worker job. You also might want to check your Kubernetes node\(aqs Internet connectivity and DNS function; in Kubernetes, DNS depends on system\-level pods which can be terminated or evicted in cases of resource oversubscription, just like user workloads. .sp If your pod seems to be stuck \fBPending\fP, \fBContainerCreating\fP, you can get information on what is wrong with it by using \fBkubectl describe pod\fP: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl describe pod demo\-user\-toil\-test\-g5496 .EE .UNINDENT .UNINDENT .sp Pay particular attention to the \fBEvents:\fP section at the end of the output. An indication that a job is too big for the available nodes on your cluster, or that your cluster is too busy for your jobs, is \fBFailedScheduling\fP events: .INDENT 0.0 .INDENT 3.5 .sp .EX Type Reason Age From Message \-\-\-\- \-\-\-\-\-\- \-\-\-\- \-\-\-\- \-\-\-\-\-\-\- Warning FailedScheduling 13s (x79 over 100m) default\-scheduler 0/4 nodes are available: 1 Insufficient cpu, 1 Insufficient ephemeral\-storage, 4 Insufficient memory. .EE .UNINDENT .UNINDENT .sp If a pod is running but seems to be behaving erratically, or seems stuck, you can shell into it and look around: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl exec \-ti demo\-user\-toil\-test\-g5496 /bin/bash .EE .UNINDENT .UNINDENT .sp One common cause of stuck pods is attempting to use more memory than allowed by Kubernetes (or by the Toil job\(aqs memory resource requirement), but in a way that does not trigger the Linux OOM killer to terminate the pod\(aqs processes. In these cases, the pod can remain stuck at nearly 100% memory usage more or less indefinitely, and attempting to shell into the pod (which needs to start a process within the pod, using some of its memory) will fail. In these cases, the recommended solution is to kill the offending pod and increase its (or its Toil job\(aqs) memory requirement, or reduce its memory needs by adapting user code. .SS When Things Go Wrong .sp The Toil Kubernetes batch system includes cleanup code to terminate worker jobs when the leader shuts down. However, if the leader pod is removed by Kubernetes, is forcibly killed or otherwise suffers a sudden existence failure, it can go away while its worker jobs live on. It is not recommended to restart a workflow in this state, as jobs from the previous invocation will remain running and will be trying to modify the job store concurrently with jobs from the new invocation. .sp To clean up dangling jobs, you can use the following snippet: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl get jobs | grep demo\-user | cut \-f1 \-d\(aq \(aq | xargs \-n10 kubectl delete job .EE .UNINDENT .UNINDENT .sp This will delete all jobs with \fBdemo\-user\fP\(aqs username in their names, in batches of 10. You can also use the UUID that Toil assigns to a particular workflow invocation in the filter, to clean up only the jobs pertaining to that workflow invocation. .SS Option 2: Running the Leader Outside Kubernetes .sp If you don\(aqt want to run your Toil leader inside Kubernetes, you can run it locally instead. This can be useful when developing a workflow; files can be hot\-deployed from your local machine directly to Kubernetes. However, your local machine will have to have (ideally role\-assumption\- and MFA\-free) access to AWS, and access to Kubernetes. Real time logging will not work unless your local machine is able to listen for incoming UDP packets on arbitrary ports on the address it uses to contact the IPv4 Internet; Toil does no NAT traversal or detection. .sp Note that if you set \fBTOIL_WORKDIR\fP when running your workflow like this, it will need to be a directory that exists both on the host and in the Toil appliance. .sp Here is an example of running our test workflow leader locally, outside of Kubernetes: .INDENT 0.0 .INDENT 3.5 .sp .EX $ export TOIL_KUBERNETES_OWNER=demo\-user # This defaults to your local username if not set $ export TOIL_AWS_SECRET_NAME=aws\-credentials $ export TOIL_KUBERNETES_HOST_PATH=/data/scratch $ virtualenv \-\-python python3 \-\-system\-site\-packages venv $ . venv/bin/activate $ wget https://raw.githubusercontent.com/DataBiosphere/toil/releases/4.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py $ python3 tutorial_helloworld.py \e aws:us\-west\-2:demouser\-toil\-test\-jobstore \e \-\-batchSystem kubernetes \e \-\-realTimeLogging \e \-\-logInfo .EE .UNINDENT .UNINDENT .SS Running CWL Workflows .sp Running CWL workflows on Kubernetes can be challenging, because executing CWL can require \fBtoil\-cwl\-runner\fP to orchestrate containers of its own, within a Kubernetes job running in the Toil appliance container. .sp Normally, running a CWL workflow should Just Work, as long as the workflow\(aqs Docker containers are able to be executed with Singularity, your Kubernetes cluster does not impose extra capability\-based confinement (i.e. SELinux, AppArmor) that interferes with Singularity\(aqs use of user\-mode namespaces, and you make sure to configure Toil so that its workers know where to store their data within the Kubernetes pods (which would be done for you if using a Toil\-managed cluster). For example, you should be able to run a CWL workflow like this: .INDENT 0.0 .INDENT 3.5 .sp .EX $ export TOIL_KUBERNETES_OWNER=demo\-user # This defaults to your local username if not set $ export TOIL_AWS_SECRET_NAME=aws\-credentials $ export TOIL_KUBERNETES_HOST_PATH=/data/scratch $ virtualenv \-\-python python3 \-\-system\-site\-packages venv $ . venv/bin/activate $ pip install toil[kubernetes,cwl]==5.8.0 $ toil\-cwl\-runner \e \-\-jobStore aws:us\-west\-2:demouser\-toil\-test\-jobstore \e \-\-batchSystem kubernetes \e \-\-realTimeLogging \e \-\-logInfo \e \-\-disableCaching \e path/to/cwl/workflow \e path/to/cwl/input/object .EE .UNINDENT .UNINDENT .sp Additional \fBcwltool\fP options that your workflow might require, such as \fB\-\-no\-match\-user\fP, can be passed to \fBtoil\-cwl\-runner\fP, which inherits most \fBcwltool\fP options. .SS AppArmor and Singularity .sp Kubernetes clusters based on Ubuntu hosts often will have AppArmor enabled on the host. AppArmor is a capability\-based security enhancement system that integrates with the Linux kernel to enforce lists of things which programs may or may not do, called \fBprofiles\fP\&. For example, an AppArmor profile could be applied to a web server process to stop it from using the \fBmount()\fP system call to manipulate the filesystem, because it has no business doing that under normal circumstances but might attempt to do it if compromised by hackers. .sp Kubernetes clusters also often use Docker as the backing container runtime, to run pod containers. When AppArmor is enabled, Docker will load an AppArmor profile and apply it to all of its containers by default, with the ability for the profile to be overridden on a per\-container basis. This profile unfortunately prevents some of the \fImount()\fP system calls that Singularity uses to set up user\-mode containers from working inside the pod, even though these calls would be allowed for an unprivileged user under normal circumstances. .sp On the UCSC Kubernetes cluster, \fI\%we configure our Ubuntu hosts with an alternative default AppArmor profile for Docker containers\fP which allows these calls. Other solutions include turning off AppArmor on the host, configuring Kubernetes with a container runtime other than Docker, or \fI\%using Kubernetes\(aqs AppArmor integration\fP to apply a more permissive profile or the \fBunconfined\fP profile to pods that Toil launches. .sp Toil does not yet have a way to apply a \fBcontainer.apparmor.security.beta.kubernetes.io/runner\-container: unconfined\fP annotation to its pods, \fI\%as described in the Kubernetes AppArmor documentation\fP\&. This feature is tracked in \fI\%issue #4331\fP\&. .SS Running in AWS .sp Toil jobs can be run on a variety of cloud platforms. Of these, Amazon Web Services (AWS) is currently the best\-supported solution. Toil provides the \fI\%Toil Cluster Utilities\fP to conveniently create AWS clusters, connect to the leader of the cluster, and then launch a workflow. The leader handles distributing the jobs over the worker nodes and autoscaling to optimize costs. .sp The \fI\%Running a Workflow with Autoscaling\fP section details how to create a cluster and run a workflow that will dynamically scale depending on the workflow\(aqs needs. .sp The \fI\%Static Provisioning\fP section explains how a static cluster (one that won\(aqt automatically change in size) can be created and provisioned (grown, shrunk, destroyed, etc.). .SS Preparing your AWS environment .sp To use Amazon Web Services (AWS) to run Toil or to just use S3 to host the files during the computation of a workflow, first set up and configure an account with AWS: .INDENT 0.0 .IP 1. 4 If necessary, create and activate an \fI\%AWS account\fP .IP 2. 4 Next, generate a key pair for AWS with the command (do NOT generate your key pair with the Amazon browser): .INDENT 4.0 .INDENT 3.5 .sp .EX $ ssh\-keygen \-t rsa .EE .UNINDENT .UNINDENT .IP 3. 4 This should prompt you to save your key. Please save it in .INDENT 4.0 .INDENT 3.5 .sp .EX ~/.ssh/id_rsa .EE .UNINDENT .UNINDENT .IP 4. 4 Now move this to where your OS can see it as an authorized key: .INDENT 4.0 .INDENT 3.5 .sp .EX $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys .EE .UNINDENT .UNINDENT .IP 5. 4 Next, you\(aqll need to add your key to the \fIssh\-agent\fP: .INDENT 4.0 .INDENT 3.5 .sp .EX $ eval \(gassh\-agent \-s\(ga $ ssh\-add .EE .UNINDENT .UNINDENT .sp If your key has a passphrase, you will be prompted to enter it here once. .IP 6. 4 You\(aqll also need to chmod your private key (good practice but also enforced by AWS): .INDENT 4.0 .INDENT 3.5 .sp .EX $ chmod 400 id_rsa .EE .UNINDENT .UNINDENT .IP 7. 4 Now you\(aqll need to add the key to AWS via the browser. For example, on us\-west1, this address would accessible at: .INDENT 4.0 .INDENT 3.5 .sp .EX https://us\-west\-1.console.aws.amazon.com/ec2/v2/home?region=us\-west\-1#KeyPairs:sort=keyName .EE .UNINDENT .UNINDENT .IP 8. 4 Now click on the \(dqImport Key Pair\(dq button to add your key: .INDENT 4.0 .INDENT 3.5 \fI\%Adding an Amazon Key Pair\fP.UNINDENT .UNINDENT .IP 9. 4 Next, you need to create an AWS access key. First go to the IAM dashboard, again; for \(dqus\-west1\(dq, the example link would be here: .INDENT 4.0 .INDENT 3.5 .sp .EX https://console.aws.amazon.com/iam/home?region=us\-west\-1#/home .EE .UNINDENT .UNINDENT .IP 10. 4 The directions (transcribed from: \fI\%https://docs.aws.amazon.com/general/latest/gr/managing\-aws\-access\-keys.html\fP ) are now: .INDENT 4.0 .INDENT 3.5 .INDENT 0.0 .IP 1. 3 On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials. .IP 2. 3 Expand the Access keys (access key ID and secret access key) section. .IP 3. 3 Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can\(aqt retrieve this secret access key again. .UNINDENT .UNINDENT .UNINDENT .IP 11. 4 Now you should have a newly generated \(dqAWS Access Key ID\(dq and \(dqAWS Secret Access Key\(dq. We can now install the AWS CLI and make sure that it has the proper credentials: .INDENT 4.0 .INDENT 3.5 .sp .EX $ pip install awscli \-\-upgrade \-\-user .EE .UNINDENT .UNINDENT .IP 12. 4 Now configure your AWS credentials with: .INDENT 4.0 .INDENT 3.5 .sp .EX $ aws configure .EE .UNINDENT .UNINDENT .IP 13. 4 Add your \(dqAWS Access Key ID\(dq and \(dqAWS Secret Access Key\(dq from earlier and your region and output format: .INDENT 4.0 .INDENT 3.5 .sp .EX \(dq AWS Access Key ID [****************Q65Q]: \(dq \(dq AWS Secret Access Key [****************G0ys]: \(dq \(dq Default region name [us\-west\-1]: \(dq \(dq Default output format [json]: \(dq .EE .UNINDENT .UNINDENT .sp This will create the files \fI~/.aws/config\fP and \fI~/.aws/credentials\fP\&. .IP 14. 4 If not done already, install toil (example uses version 5.12.0, but we recommend the latest release): .INDENT 4.0 .INDENT 3.5 .sp .EX $ virtualenv venv $ source venv/bin/activate $ pip install toil[all]==5.12.0 .EE .UNINDENT .UNINDENT .IP 15. 4 Now that toil is installed and you are running a virtualenv, an example of launching a toil leader node would be the following (again, note that we set TOIL_APPLIANCE_SELF to toil version 5.3.0 in this example, but please set the version to the installed version that you are using if you\(aqre using a different version): .INDENT 4.0 .INDENT 3.5 .sp .EX $ toil launch\-cluster \e \-\-clusterType kubernetes \e \-\-leaderNodeType t2.medium \e \-\-nodeTypes t2.medium \-w 1 \e \-\-zone us\-west\-1a \e \-\-keyPairName id_rsa .EE .UNINDENT .UNINDENT .UNINDENT .sp To further break down each of these commands: .INDENT 0.0 .INDENT 3.5 \fBtoil launch\-cluster\fP \-\-\- Base command in toil to launch a cluster. .sp \fB\fP \-\-\- Just choose a name for your cluster. .sp \fB\-\-clusterType kubernetes\fP \-\-\- Specify the type of cluster to coordinate and execute your workflow. Kubernetes is the recommended option. .sp \fB\-\-leaderNodeType t2.medium\fP \-\-\- Specify the leader node type. Make a t2.medium (2CPU; 4Gb RAM; $0.0464/Hour). List of available AWS instances: \fI\%https://aws.amazon.com/ec2/pricing/on\-demand/\fP .sp \fB\-\-nodeTypes t2.medium \-w 1\fP \-\-\- Specify the worker node type and the number of worker nodes to launch. The Kubernetes cluster requires at least 1 worker node. .sp \fB\-\-zone us\-west\-1a\fP \-\-\- Specify the AWS zone you want to launch the instance in. Must have the same prefix as the zone in your awscli credentials (which, in the example of this tutorial is: \(dqus\-west\-1\(dq). .sp \fB\-\-keyPairName id_rsa\fP \-\-\- The name of your key pair, which should be \(dqid_rsa\(dq if you\(aqve followed this tutorial. .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 You can set the \fBTOIL_AWS_TAGS\fP environment variable to a JSON object to specify arbitrary tags for AWS resources. For example, if you \fBexport TOIL_AWS_TAGS=\(aq{\(dqproject\-name\(dq: \(dqvariant\-calling\(dq}\(aq\fP in your shell before using Toil, AWS resources created by Toil will be tagged with a \fBproject\-name\fP tag with the value \fBvariant\-calling\fP\&. .sp You can also set the \fBTOIL_APPLIANCE_SELF\fP environment variable to one of the \fI\%Toil project\(aqs Docker images\fP, if you would like to launch a cluster using a different version of Toil than the one you have installed. .UNINDENT .UNINDENT .SS AWS Job Store .sp Using the AWS job store is straightforward after you\(aqve finished \fI\%Preparing your AWS environment\fP; all you need to do is specify the prefix for the job store name. .sp To run the sort example \fI\%sort example\fP with the AWS job store you would type .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py aws:us\-west\-2:my\-aws\-sort\-jobstore .EE .UNINDENT .UNINDENT .SS Toil Provisioner .sp The Toil provisioner is the component responsible for creating resources in Amazon\(aqs cloud. It is included in Toil alongside the \fB[aws]\fP extra and allows us to spin up a cluster. .sp Getting started with the provisioner is simple: .INDENT 0.0 .IP 1. 3 Make sure you have Toil installed with the AWS extras. For detailed instructions see \fI\%Installing Toil with Extra Features\fP\&. .IP 2. 3 You will need an AWS account and you will need to save your AWS credentials on your local machine. For help setting up an AWS account see \fI\%here\fP\&. For setting up your AWS credentials follow instructions \fI\%here\fP\&. .UNINDENT .sp The Toil provisioner makes heavy use of the Toil Appliance, a Docker image that bundles Toil and all its requirements (e.g. Kubernetes). This makes deployment simple across platforms, and you can even simulate a cluster locally (see \fI\%Developing with Docker\fP for details). .INDENT 0.0 .INDENT 3.5 .IP "Choosing Toil Appliance Image" .sp When using the Toil provisioner, the appliance image will be automatically chosen based on the pip\-installed version of Toil on your system. That choice can be overridden by setting the environment variables \fBTOIL_DOCKER_REGISTRY\fP and \fBTOIL_DOCKER_NAME\fP or \fBTOIL_APPLIANCE_SELF\fP\&. See \fI\%Environment Variables\fP for more information on these variables. If you are developing with autoscaling and want to test and build your own appliance have a look at \fI\%Developing with Docker\fP\&. .UNINDENT .UNINDENT .sp For information on using the Toil Provisioner have a look at \fI\%Running a Workflow with Autoscaling\fP\&. .SS Details about Launching a Cluster in AWS .sp Using the provisioner to launch a Toil leader instance is simple using the \fBlaunch\-cluster\fP command. For example, to launch a Kubernetes cluster named \(dqmy\-cluster\(dq with a t2.medium leader in the us\-west\-2a zone, run .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster my\-cluster \e \-\-clusterType kubernetes \e \-\-leaderNodeType t2.medium \e \-\-nodeTypes t2.medium \-w 1 \e \-\-zone us\-west\-2a \e \-\-keyPairName .EE .UNINDENT .UNINDENT .sp The cluster name is used to uniquely identify your cluster and will be used to populate the instance\(aqs \fBName\fP tag. Also, the Toil provisioner will automatically tag your cluster with an \fBOwner\fP tag that corresponds to your keypair name to facilitate cost tracking. In addition, the \fBToilNodeType\fP tag can be used to filter \(dqleader\(dq vs. \(dqworker\(dq nodes in your cluster. .sp The leaderNodeType is an \fI\%EC2 instance type\fP\&. This only affects the leader node. .sp The \fB\-\-zone\fP parameter specifies which EC2 availability zone to launch the cluster in. Alternatively, you can specify this option via the \fBTOIL_AWS_ZONE\fP environment variable. Note: the zone is different from an EC2 region. A region corresponds to a geographical area like \fBus\-west\-2 (Oregon)\fP, and availability zones are partitions of this area like \fBus\-west\-2a\fP\&. .sp By default, Toil creates an IAM role for each cluster with sufficient permissions to perform cluster operations (e.g. full S3, EC2, and SDB access). If the default permissions are not sufficient for your use case (e.g. if you need access to ECR), you may create a custom IAM role with all necessary permissions and set the \fB\-\-awsEc2ProfileArn\fP parameter when launching the cluster. Note that your custom role must at least have \fI\%these permissions\fP in order for the Toil cluster to function properly. .sp In addition, Toil creates a new security group with the same name as the cluster name with default rules (e.g. opens port 22 for SSH access). If you require additional security groups, you may use the \fB\-\-awsEc2ExtraSecurityGroupId\fP parameter when launching the cluster. \fBNote:\fP Do not use the same name as the cluster name for the extra security groups as any security group matching the cluster name will be deleted once the cluster is destroyed. .sp For more information on options try: .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster \-\-help .EE .UNINDENT .UNINDENT .SS Static Provisioning .sp Toil can be used to manage a cluster in the cloud by using the \fI\%Toil Cluster Utilities\fP\&. The cluster utilities also make it easy to run a toil workflow directly on this cluster. We call this static provisioning because the size of the cluster does not change. This is in contrast with \fI\%Running a Workflow with Autoscaling\fP\&. .sp To launch worker nodes alongside the leader we use the \fB\-w\fP option: .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster my\-cluster \e \-\-clusterType kubernetes \e \-\-leaderNodeType t2.small \-z us\-west\-2a \e \-\-keyPairName \e \-\-nodeTypes m3.large,t2.micro \-w 1,4 \e \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .sp This will spin up a leader node of type t2.small with five additional workers \-\-\- one m3.large instance and four t2.micro. .sp Currently static provisioning is only possible during the cluster\(aqs creation. The ability to add new nodes and remove existing nodes via the native provisioner is in development. Of course the cluster can always be deleted with the \fI\%Destroy\-Cluster Command\fP utility. .SS Uploading Workflows .sp Now that our cluster is launched, we use the \fI\%Rsync\-Cluster Command\fP utility to copy the workflow to the leader. For a simple workflow in a single file this might look like .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil rsync\-cluster \-z us\-west\-2a my\-cluster toil\-workflow.py :/ .EE .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 If your toil workflow has dependencies have a look at the \fI\%Auto\-Deployment\fP section for a detailed explanation on how to include them. .UNINDENT .UNINDENT .SS Running a Workflow with Autoscaling .sp Toil can create an autoscaling Kubernetes cluster for you using the AWS provisioner. Autoscaling is a feature of running Toil in a cloud whereby additional cloud instances are launched as needed to run the workflow. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Make sure you\(aqve done the AWS setup in \fI\%Preparing your AWS environment\fP\&. .UNINDENT .UNINDENT .sp To set up a Kubernetes cluster, simply use the \fB\-\-clusterType=kubernetes\fP command line option to \fBtoil launch\-cluster\fP\&. To make it autoscale, specify a range of possible node counts for a node type (such as \fB\-w 1\-4\fP). The cluster will automatically add and remove nodes, within that range, depending on how many seem to be needed to run the jobs submitted to the cluster. .sp For example, to launch a Toil cluster with a Kubernetes scheduler, run: .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster \e \-\-provisioner=aws \e \-\-clusterType kubernetes \e \-\-zone us\-west\-2a \e \-\-keyPairName \e \-\-leaderNodeType t2.medium \e \-\-leaderStorage 50 \e \-\-nodeTypes t2.medium \-w 1\-4 \e \-\-nodeStorage 20 \e \-\-logDebug .EE .UNINDENT .UNINDENT .sp Behind the scenes, Toil installs kubeadm and configures the kubelet on the Toil leader and all worker nodes. This Toil cluster can then schedule jobs using Kubernetes. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 You should set at least one worker node, otherwise Kubernetes would not be able to schedule any jobs. It is also normal for this step to take a while. .UNINDENT .UNINDENT .sp As a demonstration, we will use \fBsort.py\fP again, but run it on a Toil cluster with Kubernetes. First, download this file and put it to the current working directory. .sp We then need to copy over the workflow file and SSH into the cluster: .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil rsync\-cluster \-z us\-west\-2a sort.py :/root (venv) $ toil ssh\-cluster \-z us\-west\-2a .EE .UNINDENT .UNINDENT .sp Remember to replace \fB\fP with your actual cluster name, and feel free to use your own cluster configuration and/or workflow files. For more information on this step, see the corresponding section of the \fI\%Static Provisioning\fP tutorial. .sp \fBIMPORTANT:\fP .INDENT 0.0 .INDENT 3.5 Some important caveats about starting a toil run through an ssh session are explained in the \fI\%Ssh\-Cluster Command\fP section. .UNINDENT .UNINDENT .sp Now that we are inside the cluster, a Kubernetes environment should already be configured and running. To verify this, simply run: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl get nodes .EE .UNINDENT .UNINDENT .sp You should see a leader node with the \fBReady\fP status. Depending on the number of worker nodes you set to create upfront, you should also see them displayed here. .sp Additionally, you can also verify that the metrics server is running: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl get \-\-raw \(dq/apis/metrics.k8s.io/v1beta1/nodes\(dq .EE .UNINDENT .UNINDENT .sp If there is a JSON response (similar to the output below), and you are not seeing any errors, that means the metrics server is set up and running, and you are good to start running workflows. .INDENT 0.0 .INDENT 3.5 .sp .EX {\(dqkind\(dq:\(dqNodeMetricsList\(dq,\(dqapiVersion\(dq:\(dqmetrics.k8s.io/v1beta1\(dq, ...} .EE .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 It\(aqll take a while for all nodes to get set up and running, so you might not be able to see all nodes running at first. You can start running workflows already, but Toil might complain until the necessary resources are set up and running. .UNINDENT .UNINDENT .sp Now we can run the workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py \e \-\-batchSystem kubernetes \e aws:: .EE .UNINDENT .UNINDENT .sp Make sure to replace \fB\fP and \fB\fP\&. It is \fBrequired\fP to use a cloud\-accessible job store like AWS or Google when using the Kubernetes batch system. .sp The sort workflow should start running on the Kubernetes cluster set up by Toil. This workflow would take a while to execute, so you could put the job in the background and monitor the Kubernetes cluster using \fBkubectl\fP\&. For example, you can check out the pods that are running: .INDENT 0.0 .INDENT 3.5 .sp .EX $ kubectl get pods .EE .UNINDENT .UNINDENT .sp You should see an output like: .INDENT 0.0 .INDENT 3.5 .sp .EX NAME READY STATUS RESTARTS AGE root\-toil\-a864e1b0\-2e1f\-48db\-953c\-038e5ad293c7\-11\-4cwdl 0/1 ContainerCreating 0 85s root\-toil\-a864e1b0\-2e1f\-48db\-953c\-038e5ad293c7\-14\-5dqtk 0/1 Completed 0 18s root\-toil\-a864e1b0\-2e1f\-48db\-953c\-038e5ad293c7\-7\-gkwc9 0/1 ContainerCreating 0 107s root\-toil\-a864e1b0\-2e1f\-48db\-953c\-038e5ad293c7\-9\-t7vsb 1/1 Running 0 96s .EE .UNINDENT .UNINDENT .sp If a pod failed for whatever reason or if you want to make sure a pod isn\(aqt stuck, you can use \fBkubectl describe pod \fP or \fBkubectl logs \fP to inspect the pod. .sp If everything is successful, you should be able to see an output file from the sort workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX $ head sortedFile.txt .EE .UNINDENT .UNINDENT .sp You can now run your own workflows! .SS Preemptibility .sp Toil can run on a heterogeneous cluster of both preemptible and non\-preemptible nodes. Being a preemptible node simply means that the node may be shut down at any time, while jobs are running. These jobs can then be restarted later somewhere else. .sp A node type can be specified as preemptible by adding a \fI\%spot bid\fP in dollars, after a colon, to its entry in the list of node types provided with the \fB\-\-nodeTypes\fP flag. If spot instance prices rise above your bid, the preemptible nodes will be shut down. .sp For example, this cluster will have both preemptible and non\-preemptible nodes: .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster \e \-\-provisioner=aws \e \-\-clusterType kubernetes \e \-\-zone us\-west\-2a \e \-\-keyPairName \e \-\-leaderNodeType t2.medium \e \-\-leaderStorage 50 \e \-\-nodeTypes t2.medium \-w 1\-4 \e \-\-nodeTypes t2.large:0.20 \-w 1\-4 \e \-\-nodeStorage 20 \e \-\-logDebug .EE .UNINDENT .UNINDENT .sp Individual jobs can explicitly specify whether they should be run on preemptible nodes via the boolean \fBpreemptible\fP resource requirement in Toil\(aqs Python API. In CWL, this is \fI\%exposed as a hint\fP \fBUsePreemptible\fP in the \fBhttp://arvados.org/cwl#\fP namespace (usually imported as \fBarv\fP). In WDL, this is \fI\%exposed as a runtime attribute\fP \fBpreemptible\fP as recognized by Cromwell. Toil\(aqs Kubernetes batch system will prefer to schedule preemptible jobs on preemptible nodes. .sp If a job is not specified to be preemptible, the job will not run on preemptible nodes even if preemptible nodes are available, unless the workflow is run with the \fB\-\-defaultPreemptible\fP flag. The \fB\-\-defaultPreemptible\fP flag will allow jobs without an explicit \fBpreemptible\fP requirement to run on preemptible machines. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 /root/sort.py aws:us\-west\-2: \e \-\-batchSystem kubernetes \e \-\-defaultPreemptible .EE .UNINDENT .UNINDENT .INDENT 0.0 .INDENT 3.5 .IP "Specify Preemptibility Carefully" .sp Ensure that your choices for \fB\-\-nodeTypes\fP and \fB\-\-maxNodes <>\fP make sense for your workflow and won\(aqt cause it to hang. You should make sure the provisioner is able to create nodes large enough to run the largest job in the workflow, and that non\-preemptible node types are allowed if there are non\-preemptible jobs in the workflow. .UNINDENT .UNINDENT .SS Using MinIO and S3\-Compatible object stores .sp Toil can be configured to access files stored in an \fI\%S3\-compatible object store\fP such as \fI\%MinIO\fP\&. The following environment variables can be used to configure the S3 connection used: .INDENT 0.0 .IP \(bu 2 \fBTOIL_S3_HOST\fP: the IP address or hostname to use for connecting to S3 .IP \(bu 2 \fBTOIL_S3_PORT\fP: the port number to use for connecting to S3, if needed .IP \(bu 2 \fBTOIL_S3_USE_SSL\fP: enable or disable the usage of SSL for connecting to S3 (\fBTrue\fP by default) .UNINDENT .sp Examples: .INDENT 0.0 .INDENT 3.5 .sp .EX TOIL_S3_HOST=127.0.0.1 TOIL_S3_PORT=9010 TOIL_S3_USE_SSL=False .EE .UNINDENT .UNINDENT .SS In\-Workflow Autoscaling with Mesos .sp Instead of the normal Kubernetes\-based autoscaling, you can also use Toil\(aqs old Mesos\-based autoscaling method, where the scaling logic runs inside the Toil workflow. With this approach, a Toil cluster can only run one workflow at a time. This method also does not work on the ARM architecture. .sp In this mode, the \fB\-\-preemptibleCompensation\fP flag can be used to handle cases where preemptible nodes may not be available but are required for your workflow. With this flag enabled, the autoscaler will attempt to compensate for a shortage of preemptible nodes of a certain type by creating non\-preemptible nodes of that type, if non\-preemptible nodes of that type were specified in \fB\-\-nodeTypes\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This approach is deprecated, because the Mesos project is no longer publishing up\-to\-date builds. .UNINDENT .UNINDENT .INDENT 0.0 .IP 1. 3 Download \fBsort.py\fP .IP 2. 3 Launch a Mesos leader node in AWS using the \fI\%Launch\-Cluster Command\fP command, without using any ranges of node counts: .INDENT 3.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster \e \-\-clusterType mesos \e \-\-keyPairName \e \-\-leaderNodeType t2.medium \e \-\-zone us\-west\-2a .EE .UNINDENT .UNINDENT .IP 3. 3 Copy the \fBsort.py\fP workflow up to the leader node: .INDENT 3.0 .INDENT 3.5 .sp .EX (venv) $ toil rsync\-cluster \-z us\-west\-2a sort.py :/root .EE .UNINDENT .UNINDENT .IP 4. 3 Login to the leader node: .INDENT 3.0 .INDENT 3.5 .sp .EX (venv) $ toil ssh\-cluster \-z us\-west\-2a .EE .UNINDENT .UNINDENT .IP 5. 3 Run the workflow with in\-workflow autoscaling, specifying a provisioner and node types and counts as workflow arguments: .INDENT 3.0 .INDENT 3.5 .sp .EX $ python3 /root/sort.py aws:us\-west\-2: \e \-\-provisioner aws \e \-\-nodeTypes c3.large \e \-\-maxNodes 2 \e \-\-batchSystem mesos .EE .UNINDENT .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 In this example, the autoscaling Toil code creates up to two instances of type \fBc3.large\fP and launches Mesos agent containers inside them. The containers are then available to run jobs defined by the \fBsort.py\fP workflow. Toil also creates a bucket in S3 called \fBaws:us\-west\-2:autoscaling\-sort\-jobstore\fP to store intermediate job results. The Toil autoscaler can also provision multiple different node types, which is useful for workflows that have jobs with varying resource requirements. For example, one could execute the workflow with \fB\-\-nodeTypes c3.large,r3.xlarge \-\-maxNodes 5,1\fP, which would allow the provisioner to create up to five c3.large nodes and one r3.xlarge node for memory\-intensive jobs. In this situation, the autoscaler would avoid creating the more expensive r3.xlarge node until needed, running most jobs on the c3.large nodes. .UNINDENT .UNINDENT .INDENT 0.0 .IP 1. 3 View the generated file to sort: .INDENT 3.0 .INDENT 3.5 .sp .EX $ head fileToSort.txt .EE .UNINDENT .UNINDENT .IP 2. 3 View the sorted file: .INDENT 3.0 .INDENT 3.5 .sp .EX $ head sortedFile.txt .EE .UNINDENT .UNINDENT .UNINDENT .SS Dashboard .sp Toil provides a dashboard for viewing the RAM and CPU usage of each node, the number of issued jobs of each type, the number of failed jobs, and the size of the jobs queue. To launch this dashboard for a Toil workflow, pass the \fB\-\-metrics\fP flag on the workflow\(aqs command line. The dashboard can then be viewed in your browser at localhost:3000 while connected to the leader node through \fBtoil ssh\-cluster\fP: .sp To change the default port number, you can use the \fB\-\-grafana_port\fP argument: .INDENT 0.0 .INDENT 3.5 .sp .EX (venv) $ toil ssh\-cluster \-z us\-west\-2a \-\-grafana_port 8000 .EE .UNINDENT .UNINDENT .sp On AWS, the dashboard keeps track of every node in the cluster to monitor CPU and RAM usage, but it can also be used while running a workflow on a single machine. The dashboard uses Grafana as the front end for displaying real\-time plots, and Prometheus for tracking metrics exported by toil: [image] .sp In order to use the dashboard for a non\-released toil version, you will have to build the containers locally with \fBmake docker\fP, since the prometheus, grafana, and mtail containers used in the dashboard are tied to a specific toil version. .SS Running in Google Compute Engine (GCE) .sp Toil supports a provisioner with Google, and a \fI\%Google Job Store\fP\&. To get started, follow instructions for \fI\%Preparing your Google environment\fP\&. .SS Preparing your Google environment .sp Toil supports using the \fI\%Google Cloud Platform\fP\&. Setting this up is easy! .INDENT 0.0 .IP 1. 3 Make sure that the \fBgoogle\fP extra (\fI\%Installing Toil with Extra Features\fP) is installed .IP 2. 3 Follow \fI\%Google\(aqs Instructions\fP to download credentials and set the \fBGOOGLE_APPLICATION_CREDENTIALS\fP environment variable .IP 3. 3 Create a new ssh key with the proper format. To create a new ssh key run the command .INDENT 3.0 .INDENT 3.5 .sp .EX $ ssh\-keygen \-t rsa \-f ~/.ssh/id_rsa \-C [USERNAME] .EE .UNINDENT .UNINDENT .sp where \fB[USERNAME]\fP is something like \fBjane@example.com\fP\&. Make sure to leave your password blank. .sp \fBWARNING:\fP .INDENT 3.0 .INDENT 3.5 This command could overwrite an old ssh key you may be using. If you have an existing ssh key you would like to use, it will need to be called id_rsa and it needs to have no password set. .UNINDENT .UNINDENT .sp Make sure only you can read the SSH keys: .INDENT 3.0 .INDENT 3.5 .sp .EX $ chmod 400 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub .EE .UNINDENT .UNINDENT .IP 4. 3 Add your newly formatted public key to Google. To do this, log into your Google Cloud account and go to \fI\%metadata\fP section under the Compute tab. [image] .sp Near the top of the screen click on \(aqSSH Keys\(aq, then edit, add item, and paste the key. Then save: [image] .UNINDENT .sp For more details look at Google\(aqs instructions for \fI\%adding SSH keys\fP\&. .SS Google Job Store .sp To use the Google Job Store you will need to set the \fBGOOGLE_APPLICATION_CREDENTIALS\fP environment variable by following \fI\%Google\(aqs instructions\fP\&. .sp Then to run the sort example with the Google job store you would type .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 sort.py google:my\-project\-id:my\-google\-sort\-jobstore .EE .UNINDENT .UNINDENT .SS Running a Workflow with Autoscaling .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 Google Autoscaling is in beta! .UNINDENT .UNINDENT .sp The steps to run a GCE workflow are similar to those of AWS (\fI\%Running a Workflow with Autoscaling\fP), except you will need to explicitly specify the \fB\-\-provisioner gce\fP option which otherwise defaults to \fBaws\fP\&. .INDENT 0.0 .IP 1. 3 Download \fBsort.py\fP .IP 2. 3 Launch the leader node in GCE using the \fI\%Launch\-Cluster Command\fP command: .INDENT 3.0 .INDENT 3.5 .sp .EX (venv) $ toil launch\-cluster \e \-\-provisioner gce \e \-\-leaderNodeType n1\-standard\-1 \e \-\-keyPairName \e \-\-zone us\-west1\-a .EE .UNINDENT .UNINDENT .sp Where \fB\fP is the first part of \fB[USERNAME]\fP used when setting up your ssh key. For example if \fB[USERNAME]\fP was \fI\%jane@example.com\fP, \fB\fP should be \fBjane\fP\&. .sp The \fB\-\-keyPairName\fP option is for an SSH key that was added to the Google account. If your ssh key \fB[USERNAME]\fP was \fBjane@example.com\fP, then your key pair name will be just \fBjane\fP\&. .IP 3. 3 Upload the sort example and ssh into the leader: .INDENT 3.0 .INDENT 3.5 .sp .EX (venv) $ toil rsync\-cluster \-\-provisioner gce sort.py :/root (venv) $ toil ssh\-cluster \-\-provisioner gce .EE .UNINDENT .UNINDENT .IP 4. 3 Run the workflow: .INDENT 3.0 .INDENT 3.5 .sp .EX $ python3 /root/sort.py google:: \e \-\-provisioner gce \e \-\-batchSystem mesos \e \-\-nodeTypes n1\-standard\-2 \e \-\-maxNodes 2 .EE .UNINDENT .UNINDENT .IP 5. 3 Clean up: .INDENT 3.0 .INDENT 3.5 .sp .EX $ exit # this exits the ssh from the leader node (venv) $ toil destroy\-cluster \-\-provisioner gce .EE .UNINDENT .UNINDENT .UNINDENT .SH HPC ENVIRONMENTS .sp Toil is a flexible framework that can be leveraged in a variety of environments, including high\-performance computing (HPC) environments. Toil provides support for a number of batch systems, including \fI\%Grid Engine\fP, \fI\%Slurm\fP, \fI\%Torque\fP and \fI\%LSF\fP, which are popular schedulers used in these environments. Toil also supports \fI\%HTCondor\fP, which is a popular scheduler for high\-throughput computing (HTC). To use one of these batch systems specify the \fB\-\-batchSystem\fP argument to the workflow. .sp Due to the cost and complexity of maintaining support for these schedulers we currently consider all but Slurm to be \(dqcommunity supported\(dq, that is the core development team does not regularly test or develop support for these systems. However, there are members of the Toil community currently deploying Toil in a wide variety of HPC environments and we welcome external contributions. .sp Developing the support of a new or existing batch system involves extending the abstract batch system class \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP\&. .SS Running on Slurm .sp When running Toil workflows on Slurm, you usually want to run the workflow itself from the head node. Toil will take care of running all the required \fBsbatch\fP commands for you. You probably do not want to submit the Toil workflow as a Slurm job with \fBsbatch\fP (although you can if you have a large number of workflows to run). You also probably do not want to manually allocate resources with \fBsallocate\fP\&. .sp To run a Toil workflow on Slurm, include \fB\-\-batchSystem slurm\fP in your command line arguments. Generally Slurm clusters have shared filesystems, meaning the file job store would be appropriate. You want to make sure to use a job store location that is shared across your Slurm cluster. Additionally, you will likely want to provide \fIanother\fP shared directory with the \fB\-\-batchLogsDir\fP option, to allow the Slurm job logs to be retrieved by Toil in case something goes wrong with a job. .sp For example, to run the sort example \fI\%sort example\fP on Slurm, assuming you are currently in a shared directory, you would type, on the cluster head node: .INDENT 0.0 .INDENT 3.5 .sp .EX $ mkdir \-p logs $ python3 sort.py ./store \-\-batchSystem slurm \-\-batchLogsDir ./logs .EE .UNINDENT .UNINDENT .SS Slurm Tips .INDENT 0.0 .IP 1. 3 If using Toil workflows that run containers with Singularity on Slurm (such as WDL workflows), you will want to make sure that Singularity caching, and Toil\(aqs MiniWDL caching, use a shared directory across your cluster nodes. By default, Toil will configure Singularity to cache per\-workflow and per\-node, but in Slurm a shared filesystem is almost always available. Assuming your home directory is shared, to set this up, you can: .INDENT 3.0 .INDENT 3.5 .sp .EX $ echo \(aqexport SINGULARITY_CACHEDIR=\(dq${HOME}/.singularity/cache\(dq\(aq >>~/.bashrc $ echo \(aqexport MINIWDL__SINGULARITY__IMAGE_CACHE=\(dq${HOME}/.cache/miniwdl\(dq\(aq >>~/.bashrc .EE .UNINDENT .UNINDENT .sp Then make sure to log out and back in again for the setting to take effect. .IP 2. 3 If your home directory is \fInot\fP shared across the cluster nodes, make sure that you have installed Toil in such a way that it is in your \fBPATH\fP on the cluster nodes. .IP 3. 3 Slurm sandboxing and resource limitation does \fInot\fP apply to Docker containers, because there is no relationship between the sandbox cgroup that your Toil job runs in and the sandbox cgroup that the Docker daemon creates to run the Docker container your job requested to run. If you want your Toil jobs\(aq containers to actually be \fIinside\fP their Slurm job resource allocations, you should make sure to run containers with Singularity or another user\-mode or daemon\-less containerization system. .IP 4. 3 Slurm can sometimes report that a job has finished before that job\(aqs changes to the cluster\(aqs shared filesystem are visible to other nodes or to the head node. Toil \fItries\fP to anticipate and compensate for this situation, but there is no amount of waiting or retrying that Toil could do to guarantee correct behavior \fIin theory\fP in these situations; the shared filesystem could in theory be days or months behind. In practice, the delay is usually no more than a few seconds, and Toil can handle it. But if you are seeing odd behavior from Toil related to files not existing when they should or still existing when they shouldn\(aqt, your problem could be that your cluster\(aqs filesystem is unusually slow to reach consistency across nodes. .IP 5. 3 If you see warnings about \fBXDG_RUNTIME_DIR\fP, your Slurm cluster might not be managing XDG login sessions correctly for Slurm jobs. Toil can work around this, but as a result of the workaround it might have trouble finding an appropriate \(dqcoordination directory\(dq where it can store state files local to each Slurm node. If you are seeing unusual behavior like Toil jobs on one node waiting for operations on a different node, you can try giving Toil a path to a per\-node, writable directory with the \fB\-\-coordinationDir\fP option, to tell it where to put those files explicitly. .IP 6. 3 With a shared filesystem, Toil\(aqs caching system is not necessarily going to help your workflow. Try running and timing test workflows with \fB\-\-caching true\fP and with \fB\-\-caching false\fP, to determine whether it is worth it for your workload to copy files from the shared filesystem to local storage on each node. .IP 7. 3 If running CWL workflows on Slurm, with a shared filesystem, you can try the \fB\-\-bypass\-file\-store\fP option to \fBtoil\-cwl\-runner\fP\&. It may speed up your workflow, but you may also need to make sure to change Toil\(aqs work directory to a shared directory provided with the \fB\-\-workDir\fP option in order for it to work properly across machines. .UNINDENT .SS Standard Output/Error from Batch System Jobs .sp Standard output and error from batch system jobs (except for the Mesos batch system) are redirected to files in the \fBtoil\-\fP directory created within the temporary directory specified by the \fB\-\-workDir\fP option; see \fI\%Commandline Options\fP\&. Each file is named as follows: \fBtoil_job__batch___.log\fP, where \fB\fP is \fBstd_output\fP for standard output, and \fBstd_error\fP for standard error. HTCondor will also write job event log files with \fB = job_events\fP\&. .sp If capturing standard output and error is desired, \fB\-\-workDir\fP will generally need to be on a shared file system; otherwise if these are written to local temporary directories on each node (e.g. \fB/tmp\fP) Toil will not be able to retrieve them. Alternatively, the \fB\-\-noStdOutErr\fP option forces Toil to discard all standard output and error from batch system jobs. .SH WORKFLOW EXECUTION SERVICE (WES) .sp The GA4GH Workflow Execution Service (WES) is a standardized API for submitting and monitoring workflows. Toil has experimental support for setting up a WES server and executing CWL, WDL, and Toil workflows using the WES API. More information about the WES API specification can be found \fI\%here\fP\&. .sp To get started with the Toil WES server, make sure that the \fBserver\fP extra (\fI\%Installing Toil with Extra Features\fP) is installed. .SS Preparing your WES environment .sp The WES server requires \fI\%Celery\fP to distribute and execute workflows. To set up Celery: .INDENT 0.0 .IP 1. 3 Start RabbitMQ, which is the broker between the WES server and Celery workers: .INDENT 3.0 .INDENT 3.5 .sp .EX docker run \-d \-\-name wes\-rabbitmq \-p 5672:5672 rabbitmq:3.9.5 .EE .UNINDENT .UNINDENT .IP 2. 3 Start Celery workers: .INDENT 3.0 .INDENT 3.5 .sp .EX celery \-A toil.server.celery_app worker \-\-loglevel=INFO .EE .UNINDENT .UNINDENT .UNINDENT .SS Starting a WES server .sp To start a WES server on the default port 8080, run the Toil command: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil server .EE .UNINDENT .UNINDENT .sp The WES API will be hosted on the following URL: .INDENT 0.0 .INDENT 3.5 .sp .EX http://localhost:8080/ga4gh/wes/v1 .EE .UNINDENT .UNINDENT .sp To use another port, e.g.: 3000, you can specify the \fB\-\-port\fP argument: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil server \-\-port 3000 .EE .UNINDENT .UNINDENT .sp There are many other command line options. Help information can be found by using this command: .INDENT 0.0 .INDENT 3.5 .sp .EX $ toil server \-\-help .EE .UNINDENT .UNINDENT .sp Below is a detailed summary of all server\-specific options: .INDENT 0.0 .TP .B \-\-debug Enable debug mode. .TP .B \-\-bypass_celery Skip sending workflows to Celery and just run them under the server. For testing. .TP .BI \-\-host \ HOST The host interface that the Toil server binds on. (default: \(dq127.0.0.1\(dq). .TP .BI \-\-port \ PORT The port that the Toil server listens on. (default: 8080). .TP .B \-\-swagger_ui If True, the swagger UI will be enabled and hosted on the \fB{api_base_path}/ui\fP endpoint. (default: False) .TP .B \-\-cors Enable Cross Origin Resource Sharing (CORS). This should only be turned on if the server is intended to be used by a website or domain. (default: False). .TP .BI \-\-cors_origins \ CORS_ORIGIN Ignored if \fB\-\-cors\fP is False. This sets the allowed origins for CORS. For details about CORS and its security risks, see the \fI\%GA4GH docs on CORS\fP\&. (default: \(dq*\(dq). .TP .BI \-\-workers \ WORKERS\fR,\fB \ \-w \ WORKERS Ignored if \fB\-\-debug\fP is True. The number of worker processes launched by the WSGI server. (default: 2). .TP .BI \-\-work_dir \ WORK_DIR The directory where workflows should be stored. This directory should be empty or only contain previous workflows. (default: \(aq./workflows\(aq). .TP .BI \-\-state_store \ STATE_STORE The local path or S3 URL where workflow state metadata should be stored. (default: in \fB\-\-work_dir\fP) .TP .BI \-\-opt \ OPT\fR,\fB \ \-o \ OPT Specify the default parameters to be sent to the workflow engine for each run. Options taking arguments must use = syntax. Accepts multiple values. Example: \fB\-\-opt=\-\-logLevel=CRITICAL\fP \fB\-\-opt=\-\-workDir=/tmp\fP\&. .TP .BI \-\-dest_bucket_base \ DEST_BUCKET_BASE Direct CWL workflows to save output files to dynamically generated unique paths under the given URL. Supports AWS S3. .TP .BI \-\-wes_dialect \ DIALECT Restrict WES responses to a dialect compatible with clients that do not fully implement the WES standard. (default: \(aqstandard\(aq) .UNINDENT .SS Running the Server with \fIdocker\-compose\fP .sp Instead of manually setting up the server components (\fBtoil server\fP, RabbitMQ, and Celery), you can use the following \fBdocker\-compose.yml\fP file to orchestrate and link them together. .sp Make sure to change the credentials for basic authentication by updating the \fBtraefik.http.middlewares.auth.basicauth.users\fP label. The passwords can be generated with tools like \fBhtpasswd\fP \fI\%like this\fP\&. (Note that single \fB$\fP signs need to be replaced with \fB$$\fP in the yaml file). .sp When running on a different host other than \fBlocalhost\fP, make sure to change the \fBHost\fP to your tartget host in the \fBtraefik.http.routers.wes.rule\fP and \fBtraefik.http.routers.wespublic.rule\fP labels. .sp You can also change \fB/tmp/toil\-workflows\fP if you want Toil workflows to live somewhere else, and create the directory before starting the server. .sp In order to run workflows that require Docker, the \fBdocker.sock\fP socket must be mounted as volume for Celery. Additionally, the \fBTOIL_WORKDIR\fP directory (defaults to: \fB/var/lib/toil\fP) and \fB/var/lib/cwl\fP (if running CWL workflows with \fBDockerRequirement\fP) should exist on the host and also be mounted as volumes. .sp Also make sure to run it behind a firewall; it opens up the Toil server on port 8080 to anyone who connects. .INDENT 0.0 .INDENT 3.5 .sp .EX # docker\-compose.yml version: \(dq3.8\(dq services: rabbitmq: image: rabbitmq:3.9.5 hostname: rabbitmq celery: image: ${TOIL_APPLIANCE_SELF} volumes: \- /var/run/docker.sock:/var/run/docker.sock \- /var/lib/docker:/var/lib/docker \- /var/lib/toil:/var/lib/toil \- /var/lib/cwl:/var/lib/cwl \- /tmp/toil\-workflows:/tmp/toil\-workflows command: celery \-\-broker=amqp://guest:guest@rabbitmq:5672// \-A toil.server.celery_app worker \-\-loglevel=INFO depends_on: \- rabbitmq wes\-server: image: ${TOIL_APPLIANCE_SELF} volumes: \- /tmp/toil\-workflows:/tmp/toil\-workflows environment: \- TOIL_WES_BROKER_URL=amqp://guest:guest@rabbitmq:5672// command: toil server \-\-host 0.0.0.0 \-\-port 8000 \-\-work_dir /tmp/toil\-workflows expose: \- 8000 labels: \- \(dqtraefik.enable=true\(dq \- \(dqtraefik.http.routers.wes.rule=Host(\(galocalhost\(ga)\(dq \- \(dqtraefik.http.routers.wes.entrypoints=web\(dq \- \(dqtraefik.http.routers.wes.middlewares=auth\(dq \- \(dqtraefik.http.middlewares.auth.basicauth.users=test:$$2y$$12$$ci.4U63YX83CwkyUrjqxAucnmi2xXOIlEF6T/KdP9824f1Rf1iyNG\(dq \- \(dqtraefik.http.routers.wespublic.rule=Host(\(galocalhost\(ga) && Path(\(ga/ga4gh/wes/v1/service\-info\(ga)\(dq depends_on: \- rabbitmq \- celery traefik: image: traefik:v2.2 command: \- \(dq\-\-providers.docker\(dq \- \(dq\-\-providers.docker.exposedbydefault=false\(dq \- \(dq\-\-entrypoints.web.address=:8080\(dq ports: \- \(dq8080:8080\(dq volumes: \- /var/run/docker.sock:/var/run/docker.sock .EE .UNINDENT .UNINDENT .sp Further customization can also be made as needed. For example, if you have a domain, you can \fI\%set up HTTPS with Let\(aqs Encrypt\fP\&. .sp Once everything is configured, simply run \fBdocker\-compose up\fP to start the containers. Run \fBdocker\-compose down\fP to stop and remove all containers. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 \fBdocker\-compose\fP is not installed on the Toil appliance by default. See the following section to set up the WES server on a Toil cluster. .UNINDENT .UNINDENT .SS Running on a Toil cluster .sp To run the server on a Toil leader instance on EC2: .INDENT 0.0 .IP 1. 3 Launch a Toil cluster with the \fBtoil launch\-cluster\fP command with the AWS provisioner .IP 2. 3 SSH into your cluster with the \fB\-\-sshOption=\-L8080:localhost:8080\fP option to forward port \fB8080\fP .IP 3. 3 Install Docker Compose by running the following commands from the \fI\%Docker docs\fP: .INDENT 3.0 .INDENT 3.5 .sp .EX curl \-L \(dqhttps://github.com/docker/compose/releases/download/1.29.2/docker\-compose\-$(uname \-s)\-$(uname \-m)\(dq \-o /usr/local/bin/docker\-compose chmod +x /usr/local/bin/docker\-compose # check installation docker\-compose \-\-version .EE .UNINDENT .UNINDENT .sp or, install a different version of Docker Compose by changing \fB\(dq1.29.2\(dq\fP to another version. .IP 4. 3 Copy the \fBdocker\-compose.yml\fP file from (\fI\%Running the Server with docker\-compose\fP) to an empty directory, and modify the configuration as needed. .IP 5. 3 Now, run \fBdocker\-compose up \-d\fP to start the WES server in detach mode on the Toil appliance. .IP 6. 3 To stop the server, run \fBdocker\-compose down\fP\&. .UNINDENT .SS WES API Endpoints .sp As defined by the GA4GH WES API specification, the following endpoints with base path \fBga4gh/wes/v1/\fP are supported by Toil: .TS center; |l|l|. _ T{ GET /service\-info T} T{ Get information about the Workflow Execution Service. T} _ T{ GET /runs T} T{ List the workflow runs. T} _ T{ POST /runs T} T{ Run a workflow. This endpoint creates a new workflow run and returns a \fBrun_id\fP to monitor its progress. T} _ T{ GET /runs/{run_id} T} T{ Get detailed info about a workflow run. T} _ T{ POST /runs/{run_id}/cancel T} T{ Cancel a running workflow. T} _ T{ GET /runs/{run_id}/status T} T{ Get the status (overall state) of a workflow run. T} _ .TE .sp When running the WES server with the \fBdocker\-compose\fP setup above, most endpoints (except \fBGET /service\-info\fP) will be protected with basic authentication. Make sure to set the \fBAuthorization\fP header with the correct credentials when submitting or retrieving a workflow. .SS Submitting a Workflow .sp Now that the WES API is up and running, we can submit and monitor workflows remotely using the WES API endpoints. A workflow can be submitted for execution using the \fBPOST /runs\fP endpoint. .sp As a quick example, we can submit the example CWL workflow from \fI\%Running a basic CWL workflow\fP to our WES API: .INDENT 0.0 .INDENT 3.5 .sp .EX # example.cwl cwlVersion: v1.0 class: CommandLineTool baseCommand: echo stdout: output.txt inputs: message: type: string inputBinding: position: 1 outputs: output: type: stdout .EE .UNINDENT .UNINDENT .sp using cURL: .INDENT 0.0 .INDENT 3.5 .sp .EX $ curl \-\-location \-\-request POST \(aqhttp://localhost:8080/ga4gh/wes/v1/runs\(aq \e \-\-user test:test \e \-\-form \(aqworkflow_url=\(dqexample.cwl\(dq\(aq \e \-\-form \(aqworkflow_type=\(dqcwl\(dq\(aq \e \-\-form \(aqworkflow_type_version=\(dqv1.0\(dq\(aq \e \-\-form \(aqworkflow_params=\(dq{\e\(dqmessage\e\(dq: \e\(dqHello world!\e\(dq}\(dq\(aq \e \-\-form \(aqworkflow_attachment=@\(dq./toil_test_files/example.cwl\(dq\(aq { \(dqrun_id\(dq: \(dq4deb8beb24894e9eb7c74b0f010305d1\(dq } .EE .UNINDENT .UNINDENT .sp Note that the \fB\-\-user\fP argument is used to attach the basic authentication credentials along with the request. Make sure to change \fBtest:test\fP to the username and password you configured for your WES server. Alternatively, you can also set the \fBAuthorization\fP header manually as \fB\(dqAuthorization: Basic base64_encoded_auth\(dq\fP\&. .sp If the workflow is submitted successfully, a JSON object containing a \fBrun_id\fP will be returned. The \fBrun_id\fP is a unique identifier of your requested workflow, which can be used to monitor or cancel the run. .sp There are a few required parameters that have to be set for all workflow submissions, which are the following: .TS center; |l|l|. _ T{ workflow_url T} T{ The URL of the workflow to run. This can refer to a file from \fBworkflow_attachment\fP\&. T} _ T{ workflow_type T} T{ The type of workflow language. Toil currently supports one of the following: \fB\(dqCWL\(dq\fP, \fB\(dqWDL\(dq\fP, or \fB\(dqpy\(dq\fP\&. To run a Toil Python workflow, set this to \fB\(dqpy\(dq\fP\&. T} _ T{ workflow_type_version T} T{ The version of the workflow language. Supported versions can be found by accessing the \fBGET /service\-info\fP endpoint of your WES server. T} _ T{ workflow_params T} T{ A JSON object that specifies the inputs of the workflow. T} _ .TE .sp Additionally, the following optional parameters are also available: .TS center; |l|l|. _ T{ workflow_attachment T} T{ A list of files associated with the workflow run. T} _ T{ workflow_engine_parameters T} T{ A JSON key\-value map of workflow engine parameters to send to the runner. .sp Example: \fB{\(dq\-\-logLevel\(dq: \(dqINFO\(dq, \(dq\-\-workDir\(dq: \(dq/tmp/\(dq}\fP T} _ T{ tags T} T{ A JSON key\-value map of metadata associated with the workflow. T} _ .TE .sp For more details about these parameters, refer to the \fI\%Run Workflow section\fP in the WES API spec. .SS Upload multiple files .sp Looking at the body of the request of the previous example, note that the \fBworkflow_url\fP is a relative URL that refers to the \fBexample.cwl\fP file uploaded from the local path \fB\&./toil_test_files/example.cwl\fP\&. .sp To specify the file name (or subdirectory) of the remote destination file, set the \fBfilename\fP field in the \fBContent\-Disposition\fP header. You could also upload more than one file by providing the \fBworkflow_attachment\fP parameter multiple times with different files. .sp This can be shown by the following example: .INDENT 0.0 .INDENT 3.5 .sp .EX $ curl \-\-location \-\-request POST \(aqhttp://localhost:8080/ga4gh/wes/v1/runs\(aq \e \-\-user test:test \e \-\-form \(aqworkflow_url=\(dqexample.cwl\(dq\(aq \e \-\-form \(aqworkflow_type=\(dqcwl\(dq\(aq \e \-\-form \(aqworkflow_type_version=\(dqv1.0\(dq\(aq \e \-\-form \(aqworkflow_params=\(dq{\e\(dqmessage\e\(dq: \e\(dqHello world!\e\(dq}\(dq\(aq \e \-\-form \(aqworkflow_attachment=@\(dq./toil_test_files/example.cwl\(dq\(aq \e \-\-form \(aqworkflow_attachment=@\(dq./toil_test_files/2.fasta\(dq;filename=inputs/test.fasta\(aq \e \-\-form \(aqworkflow_attachment=@\(dq./toil_test_files/2.fastq\(dq;filename=inputs/test.fastq\(aq .EE .UNINDENT .UNINDENT .sp On the server, the execution directory would have the following structure from the above request: .INDENT 0.0 .INDENT 3.5 .sp .EX execution/ ├── example.cwl ├── inputs │ ├── test.fasta | └── test.fastq └── wes_inputs.json .EE .UNINDENT .UNINDENT .SS Specify Toil options .sp To pass Toil\-specific parameters to the workflow, you can include the \fBworkflow_engine_parameters\fP parameter along with your request. .sp For example, to set the logging level to \fBINFO\fP, and change the working directory of the workflow, simply include the following as \fBworkflow_engine_parameters\fP: .INDENT 0.0 .INDENT 3.5 .sp .EX {\(dq\-\-logLevel\(dq: \(dqINFO\(dq, \(dq\-\-workDir\(dq: \(dq/tmp/\(dq} .EE .UNINDENT .UNINDENT .sp These options would be appended at the end of existing parameters during command construction, which would override the default parameters if provided. (Default parameters that can be passed multiple times would not be overridden). .SS Monitoring a Workflow .sp With the \fBrun_id\fP returned when submitting the workflow, we can check the status or get the full logs of the workflow run. .SS Checking the state .sp The \fBGET /runs/{run_id}/status\fP endpoint can be used to get a simple result with the overall state of your run: .INDENT 0.0 .INDENT 3.5 .sp .EX $ curl \-\-user test:test http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1/status { \(dqrun_id\(dq: \(dq4deb8beb24894e9eb7c74b0f010305d1\(dq, \(dqstate\(dq: \(dqRUNNING\(dq } .EE .UNINDENT .UNINDENT .sp The possible states here are: \fBQUEUED\fP, \fBINITIALIZING\fP, \fBRUNNING\fP, \fBCOMPLETE\fP, \fBEXECUTOR_ERROR\fP, \fBSYSTEM_ERROR\fP, \fBCANCELING\fP, and \fBCANCELED\fP\&. .SS Getting the full logs .sp To get the detailed information about a workflow run, use the \fBGET /runs/{run_id}\fP endpoint: .INDENT 0.0 .INDENT 3.5 .sp .EX $ curl \-\-user test:test http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1 { \(dqrun_id\(dq: \(dq4deb8beb24894e9eb7c74b0f010305d1\(dq, \(dqrequest\(dq: { \(dqworkflow_attachment\(dq: [ \(dqexample.cwl\(dq ], \(dqworkflow_url\(dq: \(dqexample.cwl\(dq, \(dqworkflow_type\(dq: \(dqcwl\(dq, \(dqworkflow_type_version\(dq: \(dqv1.0\(dq, \(dqworkflow_params\(dq: { \(dqmessage\(dq: \(dqHello world!\(dq } }, \(dqstate\(dq: \(dqRUNNING\(dq, \(dqrun_log\(dq: { \(dqcmd\(dq: [ \(dqtoil\-cwl\-runner \-\-outdir=/home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/outputs \-\-jobStore=file:/home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/toil_job_store /home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/execution/example.cwl /home/workflows/4deb8beb24894e9eb7c74b0f010305d1/execution/wes_inputs.json\(dq ], \(dqstart_time\(dq: \(dq2021\-08\-30T17:35:50Z\(dq, \(dqend_time\(dq: null, \(dqstdout\(dq: null, \(dqstderr\(dq: null, \(dqexit_code\(dq: null }, \(dqtask_logs\(dq: [], \(dqoutputs\(dq: {} } .EE .UNINDENT .UNINDENT .SS Canceling a run .sp To cancel a workflow run, use the \fBPOST /runs/{run_id}/cancel\fP endpoint: .INDENT 0.0 .INDENT 3.5 .sp .EX $ curl \-\-location \-\-request POST \(aqhttp://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1/cancel\(aq \e \-\-user test:test { \(dqrun_id\(dq: \(dq4deb8beb24894e9eb7c74b0f010305d1\(dq } .EE .UNINDENT .UNINDENT .SH DEVELOPING A PYTHON WORKFLOW .sp This tutorial walks through the features of Toil necessary for developing a workflow using the Toil Python API. .SS Scripting Quick Start .sp To begin, consider this short Toil Python workflow which illustrates defining a workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def helloWorld(message): return f\(dqHello, world!, here\(aqs a message: {message}\(dq if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_quickstart\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqOFF\(dq options.clean = \(dqalways\(dq hello_job = Job.wrapFn(helloWorld, \(dqWoot\(dq) with Toil(options) as toil: print(toil.start(hello_job)) # prints \(dqHello, world!, ...\(dq .EE .UNINDENT .UNINDENT .sp The workflow consists of a single job. The resource requirements for that job are (optionally) specified by keyword arguments (memory, cores, disk). The workflow is run using \fI\%toil.job.Job.Runner.getDefaultOptions()\fP\&. Below we explain the components of this code in detail. .SS Job Basics .sp The atomic unit of work in a Toil workflow is a \fI\%Job\fP\&. User code extends this base class, or uses helper methods like \fI\%toil.job.Job.addChildJobFn()\fP, to define units of work. For example, here is a more long\-winded class\-based version of the job in the quick start example: .INDENT 0.0 .INDENT 3.5 .sp .EX from toil.job import Job class HelloWorld(Job): def __init__(self, message): Job.__init__(self, memory=\(dq2G\(dq, cores=2, disk=\(dq3G\(dq) self.message = message def run(self, fileStore): return f\(dqHello, world! Here\(aqs a message: {self.message}\(dq .EE .UNINDENT .UNINDENT .sp In the example a class, HelloWorld, is defined. The constructor requests 2 gigabytes of memory, 2 cores and 3 gigabytes of local disk to complete the work. .sp The \fI\%toil.job.Job.run()\fP method is the function the user overrides to get work done. Here it just returns a message. .sp It is also possible to log a message using \fI\%toil.job.Job.log()\fP, which will be registered in the log output of the leader process of the workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX \&... def run(self, fileStore): self.log(f\(dqHello, world! Here\(aqs a message: {self.message}\(dq) .EE .UNINDENT .UNINDENT .SS Invoking a Workflow .sp We can add to the previous example to turn it into a complete workflow by adding the necessary function calls to create an instance of HelloWorld and to run this as a workflow containing a single job. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp class HelloWorld(Job): def __init__(self, message): Job.__init__(self) self.message = message def run(self, fileStore): return f\(dqHello, world!, here\(aqs a message: {self.message}\(dq if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_invokeworkflow\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqOFF\(dq options.clean = \(dqalways\(dq hello_job = HelloWorld(\(dqWoot\(dq) with Toil(options) as toil: print(toil.start(hello_job)) .EE .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Do not include a \fI\&.\fP in the name of your python script (besides \fI\&.py\fP at the end). This is to allow toil to import the types and functions defined in your file while starting a new process. .UNINDENT .UNINDENT .sp This uses the \fI\%toil.common.Toil\fP class, which is used to run and resume Toil workflows. It is used as a context manager and allows for preliminary setup, such as staging of files into the job store on the leader node. An instance of the class is initialized by specifying an options object. The actual workflow is then invoked by calling the \fI\%toil.common.Toil.start()\fP method, passing the root job of the workflow, or, if a workflow is being restarted, \fI\%toil.common.Toil.restart()\fP should be used. Note that the context manager should have explicit if else branches addressing restart and non restart cases. The boolean value for these if else blocks is toil.options.restart. .sp For example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp class HelloWorld(Job): def __init__(self, message): Job.__init__(self) self.message = message def run(self, fileStore): return f\(dqHello, world!, I have a message: {self.message}\(dq if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_invokeworkflow2\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: if not toil.options.restart: job = HelloWorld(\(dqWoot!\(dq) output = toil.start(job) else: output = toil.restart() print(output) .EE .UNINDENT .UNINDENT .sp The call to \fI\%toil.job.Job.Runner.getDefaultOptions()\fP creates a set of default options for the workflow. The only argument is a description of how to store the workflow\(aqs state in what we call a \fIjob\-store\fP\&. Here the job\-store is contained in a directory within the current working directory called \(dqtoilWorkflowRun\(dq. Alternatively this string can encode other ways to store the necessary state, e.g. an S3 bucket object store location. By default the job\-store is deleted if the workflow completes successfully. .sp The workflow is executed in the final line, which creates an instance of HelloWorld and runs it as a workflow. Note all Toil workflows start from a single starting job, referred to as the \fIroot\fP job. The return value of the root job is returned as the result of the completed workflow (see promises below to see how this is a useful feature!). .SS Specifying Commandline Arguments .sp To allow command line control of the options we can use the \fI\%toil.job.Job.Runner.getDefaultArgumentParser()\fP method to create a \fI\%argparse.ArgumentParser\fP object which can be used to parse command line options for a Toil Python workflow. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX from toil.common import Toil from toil.job import Job class HelloWorld(Job): def __init__(self, message): Job.__init__(self) self.message = message def run(self, fileStore): return \(dqHello, world!, here\(aqs a message: %s\(dq % self.message if __name__ == \(dq__main__\(dq: parser = Job.Runner.getDefaultArgumentParser() options = parser.parse_args() options.logLevel = \(dqOFF\(dq options.clean = \(dqalways\(dq hello_job = HelloWorld(\(dqWoot\(dq) with Toil(options) as toil: print(toil.start(hello_job)) .EE .UNINDENT .UNINDENT .sp This creates a fully fledged Toil Python workflow with all the options Toil exposes as command line arguments. Running this program with \fB\-\-help\fP will print the full list of options. .sp Alternatively an existing \fI\%argparse.ArgumentParser\fP object can have Toil command line options added to it with the \fI\%toil.job.Job.Runner.addToilOptions()\fP method. .SS Resuming a Workflow .sp In the event that a workflow fails, either because of programmatic error within the jobs being run, or because of node failure, the workflow can be resumed. Workflows can only not be reliably resumed if the job\-store itself becomes corrupt. .sp Critical to resumption is that jobs can be rerun, even if they have apparently completed successfully. Put succinctly, a user defined job should not corrupt its input arguments. That way, regardless of node, network or leader failure the job can be restarted and the workflow resumed. .sp To resume a workflow specify the \(dqrestart\(dq option in the options object passed to \fI\%toil.common.Toil.start()\fP\&. If node failures are expected it can also be useful to use the integer \(dqretryCount\(dq option, which will attempt to rerun a job retryCount number of times before marking it fully failed. .sp In the common scenario that a small subset of jobs fail (including retry attempts) within a workflow Toil will continue to run other jobs until it can do no more, at which point \fI\%toil.common.Toil.start()\fP will raise a \fI\%toil.exceptions.FailedJobsException\fP exception. Typically at this point the user can decide to fix the script and resume the workflow or delete the job\-store manually and rerun the complete workflow. .SS Functions and Job Functions .sp Defining jobs by creating class definitions generally involves the boilerplate of creating a constructor. To avoid this the classes \fI\%toil.job.FunctionWrappingJob\fP and \fBtoil.job.JobFunctionWrappingTarget\fP allow functions to be directly converted to jobs. For example, the quick start example (repeated here): .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def helloWorld(message): return f\(dqHello, world!, here\(aqs a message: {message}\(dq if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_quickstart\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqOFF\(dq options.clean = \(dqalways\(dq hello_job = Job.wrapFn(helloWorld, \(dqWoot\(dq) with Toil(options) as toil: print(toil.start(hello_job)) # prints \(dqHello, world!, ...\(dq .EE .UNINDENT .UNINDENT .sp Is equivalent to the previous example, but using a function to define the job. .sp The function call: .INDENT 0.0 .INDENT 3.5 .sp .EX Job.wrapFn(helloWorld, \(dqWoot\(dq) .EE .UNINDENT .UNINDENT .sp Creates the instance of the \fBtoil.job.FunctionWrappingTarget\fP that wraps the function. .sp The keyword arguments \fImemory\fP, \fIcores\fP and \fIdisk\fP allow resource requirements to be specified as before. Even if they are not included as keyword arguments within a function header they can be passed as arguments when wrapping a function as a job and will be used to specify resource requirements. .sp We can also use the function wrapping syntax to a \fIjob function\fP, a function whose first argument is a reference to the wrapping job. Just like a \fIself\fP argument in a class, this allows access to the methods of the wrapping job, see \fBtoil.job.JobFunctionWrappingTarget\fP\&. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def helloWorld(job, message): job.log(f\(dqHello world, I have a message: {message}\(dq) if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_jobfunctions\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq hello_job = Job.wrapJobFn(helloWorld, \(dqWoot!\(dq) with Toil(options) as toil: toil.start(hello_job) .EE .UNINDENT .UNINDENT .sp Here \fBhelloWorld()\fP is a job function. It uses the \fI\%toil.job.Job.log()\fP to log a message that will be printed to the output console. Here the only subtle difference to note is the line: .INDENT 0.0 .INDENT 3.5 .sp .EX hello_job = Job.wrapJobFn(helloWorld, \(dqWoot\(dq) .EE .UNINDENT .UNINDENT .sp Which uses the function \fI\%toil.job.Job.wrapJobFn()\fP to wrap the job function instead of \fI\%toil.job.Job.wrapFn()\fP which wraps a vanilla function. .SS Workflows with Multiple Jobs .sp A \fIparent\fP job can have \fIchild\fP jobs and \fIfollow\-on\fP jobs. These relationships are specified by methods of the job class, e.g. \fI\%toil.job.Job.addChild()\fP and \fI\%toil.job.Job.addFollowOn()\fP\&. .sp Considering a set of jobs the nodes in a job graph and the child and follow\-on relationships the directed edges of the graph, we say that a job B that is on a directed path of child/follow\-on edges from a job \fBA\fP in the job graph is a \fIsuccessor\fP of \fBA\fP, similarly \fBA\fP is a \fIpredecessor\fP of \fBB\fP\&. .sp A parent job\(aqs child jobs are run directly after the parent job has completed, and in parallel. The follow\-on jobs of a job are run after its child jobs and their successors have completed. They are also run in parallel. Follow\-ons allow the easy specification of cleanup tasks that happen after a set of parallel child tasks. The following shows a simple example that uses the earlier \fBhelloWorld()\fP job function: .INDENT 0.0 .INDENT 3.5 .sp .EX from toil.common import Toil from toil.job import Job def helloWorld(job, message): job.log(f\(dqHello world, I have a message: {message}\(dq) if __name__ == \(dq__main__\(dq: parser = Job.Runner.getDefaultArgumentParser() options = parser.parse_args() options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq j1 = Job.wrapJobFn(helloWorld, \(dqfirst\(dq) j2 = Job.wrapJobFn(helloWorld, \(dqsecond or third\(dq) j3 = Job.wrapJobFn(helloWorld, \(dqsecond or third\(dq) j4 = Job.wrapJobFn(helloWorld, \(dqlast\(dq) j1.addChild(j2) j1.addChild(j3) j1.addFollowOn(j4) with Toil(options) as toil: toil.start(j1) .EE .UNINDENT .UNINDENT .sp In the example four jobs are created, first \fBj1\fP is run, then \fBj2\fP and \fBj3\fP are run in parallel as children of \fBj1\fP, finally \fBj4\fP is run as a follow\-on of \fBj1\fP\&. .sp There are multiple short hand functions to achieve the same workflow, for example: .INDENT 0.0 .INDENT 3.5 .sp .EX from toil.common import Toil from toil.job import Job def helloWorld(job, message): job.log(f\(dqHello world, I have a message: {message}\(dq) if __name__ == \(dq__main__\(dq: parser = Job.Runner.getDefaultArgumentParser() options = parser.parse_args() options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq j1 = Job.wrapJobFn(helloWorld, \(dqfirst\(dq) j2 = j1.addChildJobFn(helloWorld, \(dqsecond or third\(dq) j3 = j1.addChildJobFn(helloWorld, \(dqsecond or third\(dq) j4 = j1.addFollowOnJobFn(helloWorld, \(dqlast\(dq) with Toil(options) as toil: toil.start(j1) .EE .UNINDENT .UNINDENT .sp Equivalently defines the workflow, where the functions \fI\%toil.job.Job.addChildJobFn()\fP and \fI\%toil.job.Job.addFollowOnJobFn()\fP are used to create job functions as children or follow\-ons of an earlier job. .sp Jobs graphs are not limited to trees, and can express arbitrary directed acyclic graphs. For a precise definition of legal graphs see \fI\%toil.job.Job.checkJobGraphForDeadlocks()\fP\&. The previous example could be specified as a DAG as follows: .INDENT 0.0 .INDENT 3.5 .sp .EX from toil.common import Toil from toil.job import Job def helloWorld(job, message): job.log(f\(dqHello world, I have a message: {message}\(dq) if __name__ == \(dq__main__\(dq: parser = Job.Runner.getDefaultArgumentParser() options = parser.parse_args() options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq j1 = Job.wrapJobFn(helloWorld, \(dqfirst\(dq) j2 = j1.addChildJobFn(helloWorld, \(dqsecond or third\(dq) j3 = j1.addChildJobFn(helloWorld, \(dqsecond or third\(dq) j4 = j2.addChildJobFn(helloWorld, \(dqlast\(dq) j3.addChild(j4) with Toil(options) as toil: toil.start(j1) .EE .UNINDENT .UNINDENT .sp Note the use of an extra child edge to make \fBj4\fP a child of both \fBj2\fP and \fBj3\fP\&. .SS Dynamic Job Creation .sp The previous examples show a workflow being defined outside of a job. However, Toil also allows jobs to be created dynamically within jobs. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def binaryStringFn(job, depth, message=\(dq\(dq): if depth > 0: job.addChildJobFn(binaryStringFn, depth\-1, message + \(dq0\(dq) job.addChildJobFn(binaryStringFn, depth\-1, message + \(dq1\(dq) else: job.log(f\(dqBinary string: {message}\(dq) if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_dynamic\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: toil.start(Job.wrapJobFn(binaryStringFn, depth=5)) .EE .UNINDENT .UNINDENT .sp The job function \fBbinaryStringFn\fP logs all possible binary strings of length \fBn\fP (here \fBn=5\fP), creating a total of \fB2^(n+2) \- 1\fP jobs dynamically and recursively. Static and dynamic creation of jobs can be mixed in a Toil workflow, with jobs defined within a job or job function being created at run time. .SS Promises .sp The previous example of dynamic job creation shows variables from a parent job being passed to a child job. Such forward variable passing is naturally specified by recursive invocation of successor jobs within parent jobs. This can also be achieved statically by passing around references to the return variables of jobs. In Toil this is achieved with promises, as illustrated in the following example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def fn(job, i): job.log(\(dqi is: %s\(dq % i, level=100) return i + 1 if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_promises\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq j1 = Job.wrapJobFn(fn, 1) j2 = j1.addChildJobFn(fn, j1.rv()) j3 = j1.addFollowOnJobFn(fn, j2.rv()) with Toil(options) as toil: toil.start(j1) .EE .UNINDENT .UNINDENT .sp Running this workflow results in three log messages from the jobs: \fBi is 1\fP from \fBj1\fP, \fBi is 2\fP from \fBj2\fP and \fBi is 3\fP from \fBj3\fP\&. .sp The return value from the first job is \fIpromised\fP to the second job by the call to \fI\%toil.job.Job.rv()\fP in the following line: .INDENT 0.0 .INDENT 3.5 .sp .EX j2 = j1.addChildFn(fn, j1.rv()) .EE .UNINDENT .UNINDENT .sp The value of \fBj1.rv()\fP is a \fIpromise\fP, rather than the actual return value of the function, because \fBj1\fP for the given input has at that point not been evaluated. A promise (\fI\%toil.job.Promise\fP) is essentially a pointer to for the return value that is replaced by the actual return value once it has been evaluated. Therefore, when \fBj2\fP is run the promise becomes 2. .sp Promises also support indexing of return values: .INDENT 0.0 .INDENT 3.5 .sp .EX def parent(job): indexable = Job.wrapJobFn(fn) job.addChild(indexable) job.addFollowOnFn(raiseWrap, indexable.rv(2)) def raiseWrap(arg): raise RuntimeError(arg) # raises \(dq2\(dq def fn(job): return (0, 1, 2, 3) .EE .UNINDENT .UNINDENT .sp Promises can be quite useful. For example, we can combine dynamic job creation with promises to achieve a job creation process that mimics the functional patterns possible in many programming languages: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def binaryStrings(job, depth, message=\(dq\(dq): if depth > 0: s = [job.addChildJobFn(binaryStrings, depth \- 1, message + \(dq0\(dq).rv(), job.addChildJobFn(binaryStrings, depth \- 1, message + \(dq1\(dq).rv()] return job.addFollowOnFn(merge, s).rv() return [message] def merge(strings): return strings[0] + strings[1] if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_promises2\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.loglevel = \(dqOFF\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: print(toil.start(Job.wrapJobFn(binaryStrings, depth=5))) .EE .UNINDENT .UNINDENT .sp The return value \fBl\fP of the workflow is a list of all binary strings of length 10, computed recursively. Although a toy example, it demonstrates how closely Toil workflows can mimic typical programming patterns. .SS Promised Requirements .sp Promised requirements are a special case of \fI\%Promises\fP that allow a job\(aqs return value to be used as another job\(aqs resource requirements. .sp This is useful when, for example, a job\(aqs storage requirement is determined by a file staged to the job store by an earlier job: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job, PromisedRequirement from toil.lib.io import mkdtemp def parentJob(job): downloadJob = Job.wrapJobFn(stageFn, \(dqfile://\(dq + os.path.realpath(__file__), cores=0.1, memory=\(aq32M\(aq, disk=\(aq1M\(aq) job.addChild(downloadJob) analysis = Job.wrapJobFn(analysisJob, fileStoreID=downloadJob.rv(0), disk=PromisedRequirement(downloadJob.rv(1))) job.addFollowOn(analysis) def stageFn(job, url): importedFile = job.fileStore.import_file(url) return importedFile, importedFile.size def analysisJob(job, fileStoreID): # now do some analysis on the file pass if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_requirements\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: toil.start(Job.wrapJobFn(parentJob)) .EE .UNINDENT .UNINDENT .sp Note that this also makes use of the \fBsize\fP attribute of the \fI\%FileID\fP object. This promised requirements mechanism can also be used in combination with an aggregator for multiple jobs\(aq output values: .INDENT 0.0 .INDENT 3.5 .sp .EX def parentJob(job): aggregator = [] for fileNum in range(0, 10): downloadJob = Job.wrapJobFn(stageFn, \(dqfile://\(dq + os.path.realpath(__file__), cores=0.1, memory=\(aq32M\(aq, disk=\(aq1M\(aq) job.addChild(downloadJob) aggregator.append(downloadJob) analysis = Job.wrapJobFn(analysisJob, fileStoreID=downloadJob.rv(0), disk=PromisedRequirement(lambda xs: sum(xs), [j.rv(1) for j in aggregator])) job.addFollowOn(analysis) .EE .UNINDENT .UNINDENT .INDENT 0.0 .INDENT 3.5 .IP "Limitations" .sp Just like regular promises, the return value must be determined prior to scheduling any job that depends on the return value. In our example above, notice how the dependent jobs were follow ons to the parent while promising jobs are children of the parent. This ordering ensures that all promises are properly fulfilled. .UNINDENT .UNINDENT .SS FileID .sp The \fBtoil.fileStore.FileID\fP class is a small wrapper around Python\(aqs builtin string class. It is used to represent a file\(aqs ID in the file store, and has a \fBsize\fP attribute that is the file\(aqs size in bytes. This object is returned by \fBimportFile\fP and \fBwriteGlobalFile\fP\&. .SS Managing files within a workflow .sp It is frequently the case that a workflow will want to create files, both persistent and temporary, during its run. The \fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP class is used by jobs to manage these files in a manner that guarantees cleanup and resumption on failure. .sp The \fI\%toil.job.Job.run()\fP method has a file store instance as an argument. The following example shows how this can be used to create temporary files that persist for the length of the job, be placed in a specified local disk of the node and that will be cleaned up, regardless of failure, when the job finishes: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp class LocalFileStoreJob(Job): def run(self, fileStore): # self.tempDir will always contain the name of a directory within the allocated disk space reserved for the job scratchDir = self.tempDir # Similarly create a temporary file. scratchFile = fileStore.getLocalTempFile() if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_managing\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq # Create an instance of FooJob which will have at least 2 gigabytes of storage space. j = LocalFileStoreJob(disk=\(dq2G\(dq) # Run the workflow with Toil(options) as toil: toil.start(j) .EE .UNINDENT .UNINDENT .sp Job functions can also access the file store for the job. The equivalent of the \fBLocalFileStoreJob\fP class is .INDENT 0.0 .INDENT 3.5 .sp .EX def localFileStoreJobFn(job): scratchDir = job.tempDir scratchFile = job.fileStore.getLocalTempFile() .EE .UNINDENT .UNINDENT .sp Note that the \fBfileStore\fP attribute is accessed as an attribute of the \fBjob\fP argument. .sp In addition to temporary files that exist for the duration of a job, the file store allows the creation of files in a \fIglobal\fP store, which persists during the workflow and are globally accessible (hence the name) between jobs. For example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp def globalFileStoreJobFn(job): job.log(\(dqThe following example exercises all the methods provided \(dq \(dqby the toil.fileStores.abstractFileStore.AbstractFileStore class\(dq) # Create a local temporary file. scratchFile = job.fileStore.getLocalTempFile() # Write something in the scratch file. with open(scratchFile, \(aqw\(aq) as fH: fH.write(\(dqWhat a tangled web we weave\(dq) # Write a copy of the file into the file\-store; fileID is the key that can be used to retrieve the file. # This write is asynchronous by default fileID = job.fileStore.writeGlobalFile(scratchFile) # Write another file using a stream; fileID2 is the # key for this second file. with job.fileStore.writeGlobalFileStream(cleanup=True) as (fH, fileID2): fH.write(b\(dqOut brief candle\(dq) # Now read the first file; scratchFile2 is a local copy of the file that is read\-only by default. scratchFile2 = job.fileStore.readGlobalFile(fileID) # Read the second file to a desired location: scratchFile3. scratchFile3 = os.path.join(job.tempDir, \(dqfoo.txt\(dq) job.fileStore.readGlobalFile(fileID2, userPath=scratchFile3) # Read the second file again using a stream. with job.fileStore.readGlobalFileStream(fileID2) as fH: print(fH.read()) # This prints \(dqOut brief candle\(dq # Delete the first file from the global file\-store. job.fileStore.deleteGlobalFile(fileID) # It is unnecessary to delete the file keyed by fileID2 because we used the cleanup flag, # which removes the file after this job and all its successors have run (if the file still exists) if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_managing2\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: toil.start(Job.wrapJobFn(globalFileStoreJobFn)) .EE .UNINDENT .UNINDENT .sp The example demonstrates the global read, write and delete functionality of the file\-store, using both local copies of the files and streams to read and write the files. It covers all the methods provided by the file store interface. .sp What is obvious is that the file\-store provides no functionality to update an existing \(dqglobal\(dq file, meaning that files are, barring deletion, immutable. Also worth noting is that there is no file system hierarchy for files in the global file store. These limitations allow us to fairly easily support different object stores and to use caching to limit the amount of network file transfer between jobs. .SS Staging of Files into the Job Store .sp External files can be imported into or exported out of the job store prior to running a workflow when the \fI\%toil.common.Toil\fP context manager is used on the leader. The context manager provides methods \fI\%toil.common.Toil.importFile()\fP, and \fI\%toil.common.Toil.exportFile()\fP for this purpose. The destination and source locations of such files are described with URLs passed to the two methods. Local files can be imported and exported as relative paths, and should be relative to the directory where the toil workflow is initially run from. .sp Using absolute paths and appropriate schema where possible (prefixing with \(dq\fI\%file://\fP\(dq or \(dqs3:/\(dq for example), make imports and exports less ambiguous and is recommended. .sp A list of the currently supported URLs can be found at \fI\%toil.jobStores.abstractJobStore.AbstractJobStore.importFile()\fP\&. To import an external file into the job store as a shared file, pass the optional \fBsharedFileName\fP parameter to that method. .sp If a workflow fails for any reason an imported file acts as any other file in the job store. If the workflow was configured such that it not be cleaned up on a failed run, the file will persist in the job store and needs not be staged again when the workflow is resumed. .sp Example: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp class HelloWorld(Job): def __init__(self, id): Job.__init__(self) self.inputFileID = id def run(self, fileStore): with fileStore.readGlobalFileStream(self.inputFileID, encoding=\(aqutf\-8\(aq) as fi: with fileStore.writeGlobalFileStream(encoding=\(aqutf\-8\(aq) as (fo, outputFileID): fo.write(fi.read() + \(aqWorld!\(aq) return outputFileID if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_staging\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: if not toil.options.restart: ioFileDirectory = os.path.join(os.path.dirname(os.path.abspath(__file__)), \(dqstagingExampleFiles\(dq) inputFileID = toil.importFile(\(dqfile://\(dq + os.path.abspath(os.path.join(ioFileDirectory, \(dqin.txt\(dq))) outputFileID = toil.start(HelloWorld(inputFileID)) else: outputFileID = toil.restart() toil.exportFile(outputFileID, \(dqfile://\(dq + os.path.abspath(os.path.join(ioFileDirectory, \(dqout.txt\(dq))) .EE .UNINDENT .UNINDENT .SS Using Docker Containers in Toil .sp Docker containers are commonly used with Toil. The combination of Toil and Docker allows for pipelines to be fully portable between any platform that has both Toil and Docker installed. Docker eliminates the need for the user to do any other tool installation or environment setup. .sp In order to use Docker containers with Toil, Docker must be installed on all workers of the cluster. Instructions for installing Docker can be found on the \fI\%Docker\fP website. .sp When using Toil\-based autoscaling, Docker will be automatically set up on the cluster\(aqs worker nodes, so no additional installation steps are necessary. Further information on using Toil\-based autoscaling can be found in the \fI\%Running a Workflow with Autoscaling\fP documentation. .sp In order to use docker containers in a Toil workflow, the container can be built locally or downloaded in real time from an online docker repository like \fI\%Quay\fP\&. If the container is not in a repository, the container\(aqs layers must be accessible on each node of the cluster. .sp When invoking docker containers from within a Toil workflow, it is strongly recommended that you use \fBdockerCall()\fP, a toil job function provided in \fBtoil.lib.docker\fP\&. \fBdockerCall\fP leverages docker\(aqs own python API, and provides container cleanup on job failure. When docker containers are run without this feature, failed jobs can result in resource leaks. Docker\(aqs API can be found at \fI\%docker\-py\fP\&. .sp In order to use \fBdockerCall\fP, your installation of Docker must be set up to run without \fBsudo\fP\&. Instructions for setting this up can be found \fI\%here\fP\&. .sp An example of a basic \fBdockerCall\fP is below: .INDENT 0.0 .INDENT 3.5 .sp .EX dockerCall(job=job, tool=\(aqquay.io/ucsc_cgl/bwa\(aq, workDir=job.tempDir, parameters=[\(aqindex\(aq, \(aq/data/reference.fa\(aq]) .EE .UNINDENT .UNINDENT .sp Note the assumption that \fIreference.fa\fP file is located in \fI/data\fP\&. This is Toil\(aqs standard convention as a mount location to reduce boilerplate when calling \fIdockerCall\fP\&. Users can choose their own mount locations by supplying a \fIvolumes\fP kwarg to \fIdockerCall\fP, such as: \fIvolumes={working_dir: {\(aqbind\(aq: \(aq/data\(aq, \(aqmode\(aq: \(aqrw\(aq}}\fP, where \fIworking_dir\fP is an absolute path on the user\(aqs filesystem. .sp \fBdockerCall\fP can also be added to workflows like any other job function: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.docker import apiDockerCall from toil.lib.io import mkdtemp align = Job.wrapJobFn(apiDockerCall, image=\(aqubuntu\(aq, working_dir=os.getcwd(), parameters=[\(aqls\(aq, \(aq\-lha\(aq]) if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_docker\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: toil.start(align) .EE .UNINDENT .UNINDENT .sp \fI\%cgl\-docker\-lib\fP contains \fBdockerCall\fP\-compatible Dockerized tools that are commonly used in bioinformatics analysis. .sp The documentation provides guidelines for developing your own Docker containers that can be used with Toil and \fBdockerCall\fP\&. In order for a container to be compatible with \fBdockerCall\fP, it must have an \fBENTRYPOINT\fP set to a wrapper script, as described in cgl\-docker\-lib containerization standards. This can be set by passing in the optional keyword argument, \(aqentrypoint\(aq. Example: .INDENT 0.0 .INDENT 3.5 entrypoint=[\(dq/bin/bash\(dq,\(dq\-c\(dq] .UNINDENT .UNINDENT .sp dockerCall supports currently the 75 keyword arguments found in the python \fI\%Docker API\fP, under the \(aqrun\(aq command. .SS Services .sp It is sometimes desirable to run \fIservices\fP, such as a database or server, concurrently with a workflow. The \fI\%toil.job.Job.Service\fP class provides a simple mechanism for spawning such a service within a Toil workflow, allowing precise specification of the start and end time of the service, and providing start and end methods to use for initialization and cleanup. The following simple, conceptual example illustrates how services work: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp class DemoService(Job.Service): def start(self, fileStore): # Start up a database/service here # Return a value that enables another process to connect to the database return \(dqloginCredentials\(dq def check(self): # A function that if it returns False causes the service to quit # If it raises an exception the service is killed and an error is reported return True def stop(self, fileStore): # Cleanup the database here pass j = Job() s = DemoService() loginCredentialsPromise = j.addService(s) def dbFn(loginCredentials): # Use the login credentials returned from the service\(aqs start method to connect to the service pass j.addChildFn(dbFn, loginCredentialsPromise) if __name__ == \(dq__main__\(dq: jobstore: str = mkdtemp(\(dqtutorial_services\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: toil.start(j) .EE .UNINDENT .UNINDENT .sp In this example the DemoService starts a database in the start method, returning an object from the start method indicating how a client job would access the database. The service\(aqs stop method cleans up the database, while the service\(aqs check method is polled periodically to check the service is alive. .sp A DemoService instance is added as a service of the root job \fBj\fP, with resource requirements specified. The return value from \fI\%toil.job.Job.addService()\fP is a promise to the return value of the service\(aqs start method. When the promised is fulfilled it will represent how to connect to the database. The promise is passed to a child job of \fBj\fP, which uses it to make a database connection. The services of a job are started before any of its successors have been run and stopped after all the successors of the job have completed successfully. .sp Multiple services can be created per job, all run in parallel. Additionally, services can define sub\-services using \fBtoil.job.Job.Service.addChild()\fP\&. This allows complex networks of services to be created, e.g. Apache Spark clusters, within a workflow. .SS Checkpoints .sp Services complicate resuming a workflow after failure, because they can create complex dependencies between jobs. For example, consider a service that provides a database that multiple jobs update. If the database service fails and loses state, it is not clear that just restarting the service will allow the workflow to be resumed, because jobs that created that state may have already finished. To get around this problem Toil supports \fIcheckpoint\fP jobs, specified as the boolean keyword argument \fBcheckpoint\fP to a job or wrapped function, e.g.: .INDENT 0.0 .INDENT 3.5 .sp .EX j = Job(checkpoint=True) .EE .UNINDENT .UNINDENT .sp A checkpoint job is rerun if one or more of its successors fails its retry attempts, until it itself has exhausted its retry attempts. Upon restarting a checkpoint job all its existing successors are first deleted, and then the job is rerun to define new successors. By checkpointing a job that defines a service, upon failure of the service the database and the jobs that access the service can be redefined and rerun. .sp To make the implementation of checkpoint jobs simple, a job can only be a checkpoint if when first defined it has no successors, i.e. it can only define successors within its run method. .SS Encapsulation .sp Let \fBA\fP be a root job potentially with children and follow\-ons. Without an encapsulated job the simplest way to specify a job \fBB\fP which runs after \fBA\fP and all its successors is to create a parent of \fBA\fP, call it \fBAp\fP, and then make \fBB\fP a follow\-on of \fBAp\fP\&. e.g.: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp if __name__ == \(dq__main__\(dq: # A is a job with children and follow\-ons, for example: A = Job() A.addChild(Job()) A.addFollowOn(Job()) # B is a job which needs to run after A and its successors B = Job() # The way to do this without encapsulation is to make a parent of A, Ap, and make B a follow\-on of Ap. Ap = Job() Ap.addChild(A) Ap.addFollowOn(B) jobstore: str = mkdtemp(\(dqtutorial_encapsulations\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: print(toil.start(Ap)) .EE .UNINDENT .UNINDENT .sp An \fIencapsulated job\fP \fBE(A)\fP of \fBA\fP saves making \fBAp\fP, instead we can write: .INDENT 0.0 .INDENT 3.5 .sp .EX import os from toil.common import Toil from toil.job import Job from toil.lib.io import mkdtemp if __name__ == \(dq__main__\(dq: # A A = Job() A.addChild(Job()) A.addFollowOn(Job()) # Encapsulate A A = A.encapsulate() # B is a job which needs to run after A and its successors B = Job() # With encapsulation A and its successor subgraph appear to be a single job, hence: A.addChild(B) jobstore: str = mkdtemp(\(dqtutorial_encapsulations2\(dq) os.rmdir(jobstore) options = Job.Runner.getDefaultOptions(jobstore) options.logLevel = \(dqINFO\(dq options.clean = \(dqalways\(dq with Toil(options) as toil: print(toil.start(A)) .EE .UNINDENT .UNINDENT .sp Note the call to \fI\%toil.job.Job.encapsulate()\fP creates the \fBtoil.job.Job.EncapsulatedJob\fP\&. .SS Depending on Toil .sp If you are packing your workflow(s) as a pip\-installable distribution on PyPI, you might be tempted to declare Toil as a dependency in your \fBsetup.py\fP, via the \fBinstall_requires\fP keyword argument to \fBsetup()\fP\&. Unfortunately, this does not work, for two reasons: For one, Toil uses Setuptools\(aq \fIextra\fP mechanism to manage its own optional dependencies. If you explicitly declared a dependency on Toil, you would have to hard\-code a particular combination of extras (or no extras at all), robbing the user of the choice what Toil extras to install. Secondly, and more importantly, declaring a dependency on Toil would only lead to Toil being installed on the leader node of a cluster, but not the worker nodes. Auto\-deployment does not work here because Toil cannot auto\-deploy itself, the classic \(dqWhich came first, chicken or egg?\(dq problem. .sp In other words, you shouldn\(aqt explicitly depend on Toil. Document the dependency instead (as in \(dqThis workflow needs Toil version X.Y.Z to be installed\(dq) and optionally add a version check to your \fBsetup.py\fP\&. Refer to the \fBcheck_version()\fP function in the \fBtoil\-lib\fP project\(aqs \fI\%setup.py\fP for an example. Alternatively, you can also just depend on \fBtoil\-lib\fP and you\(aqll get that check for free. .sp If your workflow depends on a dependency of Toil, consider not making that dependency explicit either. If you do, you risk a version conflict between your project and Toil. The \fBpip\fP utility may silently ignore that conflict, breaking either Toil or your workflow. It is safest to simply assume that Toil installs that dependency for you. The only downside is that you are locked into the exact version of that dependency that Toil declares. But such is life with Python, which, unlike Java, has no means of dependencies belonging to different software components within the same process, and whose favored software distribution utility is \fI\%incapable\fP of properly resolving overlapping dependencies and detecting conflicts. .SS Best Practices for Dockerizing Toil Workflows .sp \fI\%Computational Genomics Lab\fP\(aqs \fI\%Dockstore\fP based production system provides workflow authors a way to run Dockerized versions of their pipeline in an automated, scalable fashion. To be compatible with this system of a workflow should meet the following requirements. In addition to the Docker container, a common workflow language \fI\%descriptor file\fP is needed. For inputs: .INDENT 0.0 .IP \(bu 2 Only command line arguments should be used for configuring the workflow. If the workflow relies on a configuration file, like \fI\%Toil\-RNAseq\fP or \fI\%ProTECT\fP, a wrapper script inside the Docker container can be used to parse the CLI and generate the necessary configuration file. .IP \(bu 2 All inputs to the pipeline should be explicitly enumerated rather than implicit. For example, don\(aqt rely on one FASTQ read\(aqs path to discover the location of its pair. This is necessary since all inputs are mapped to their own isolated directories when the Docker is called via Dockstore. .IP \(bu 2 All inputs must be documented in the CWL descriptor file. Examples of this file can be seen in both \fI\%Toil\-RNAseq\fP and \fI\%ProTECT\fP\&. .UNINDENT .sp For outputs: .INDENT 0.0 .IP \(bu 2 All outputs should be written to a local path rather than S3. .IP \(bu 2 Take care to package outputs in a local and user\-friendly way. For example, don\(aqt tar up all output if there are specific files that will care to see individually. .IP \(bu 2 All output file names should be deterministic and predictable. For example, don\(aqt prepend the name of an output file with PASS/FAIL depending on the outcome of the pipeline. .IP \(bu 2 All outputs must be documented in the CWL descriptor file. Examples of this file can be seen in both \fI\%Toil\-RNAseq\fP and \fI\%ProTECT\fP\&. .UNINDENT .SH TOIL CLASS API .sp The Toil class configures and starts a Toil run. .INDENT 0.0 .TP .B class toil.common.Toil(options) A context manager that represents a Toil workflow. .sp Specifically the batch system, job store, and its configuration. .INDENT 7.0 .TP .B Parameters \fBoptions\fP (\fI\%Namespace\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __init__(options) Initialize a Toil object from the given options. .sp Note that this is very light\-weight and that the bulk of the work is done when the context is entered. .INDENT 7.0 .TP .B Parameters \fBoptions\fP (\fI\%Namespace\fP) \-\- command line options specified by the user .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B start(rootJob) Invoke a Toil workflow with the given job as the root for an initial run. .sp This method must be called in the body of a \fBwith Toil(...) as toil:\fP statement. This method should not be called more than once for a workflow that has not finished. .INDENT 7.0 .TP .B Parameters \fBrootJob\fP (\fI\%Job\fP) \-\- The root job of the workflow .TP .B Return type \fI\%Any\fP .TP .B Returns The root job\(aqs return value .UNINDENT .UNINDENT .INDENT 7.0 .TP .B restart() Restarts a workflow that has been interrupted. .INDENT 7.0 .TP .B Return type \fI\%Any\fP .TP .B Returns The root job\(aqs return value .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod getJobStore(locator) Create an instance of the concrete job store implementation that matches the given locator. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocator\fP (\fI\%str\fP) \-\- The location of the job store to be represent by the instance .IP \(bu 2 \fBlocator\fP \-\- .UNINDENT .TP .B Return type \fI\%AbstractJobStore\fP .TP .B Returns an instance of a concrete subclass of AbstractJobStore .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static createBatchSystem(config) Create an instance of the batch system specified in the given config. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%Config\fP) \-\- the current configuration .TP .B Return type \fI\%AbstractBatchSystem\fP .TP .B Returns an instance of a concrete subclass of AbstractBatchSystem .UNINDENT .UNINDENT .INDENT 7.0 .TP .B import_file(src_uri, shared_file_name=None, symlink=True, check_existence=True) Import the file at the given URL into the job store. .sp By default, returns None if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcheck_existence\fP (\fI\%bool\fP) \-\- If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist. .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP\fI | \fP\fINone\fP) \-\- .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%Optional\fP[\fI\%FileID\fP] .UNINDENT .sp See \fI\%toil.jobStores.abstractJobStore.AbstractJobStore.importFile()\fP for a full description .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBshared_file_name\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBcheck_existence\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%FileID\fP | None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B export_file(file_id, dst_uri) Export file to destination pointed at by the destination URL. .sp See \fI\%toil.jobStores.abstractJobStore.AbstractJobStore.exportFile()\fP for a full description .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%FileID\fP) \-\- .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static normalize_uri(uri, check_existence=False) Given a URI, if it has no scheme, prepend \(dqfile:\(dq. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcheck_existence\fP (\fI\%bool\fP) \-\- If set, raise FileNotFoundError if a URI points to a local file that does not exist. .IP \(bu 2 \fBuri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static getToilWorkDir(configWorkDir=None) Return a path to a writable directory under which per\-workflow directories exist. .sp This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR. .INDENT 7.0 .TP .B Parameters \fBconfigWorkDir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Value passed to the program using the \-\-workDir flag .TP .B Return type \fI\%str\fP .TP .B Returns Path to the Toil work directory, constant across all machines .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir) Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig_work_dir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Value passed to the program using the \-\-workDir flag .IP \(bu 2 \fBconfig_coordination_dir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Value passed to the program using the \-\-coordinationDir flag .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static get_workflow_path_component(workflow_id) Get a safe filesystem path component for a workflow. .sp Will be consistent for all processes on a given machine, and different for all processes on different machines. .INDENT 7.0 .TP .B Parameters \fBworkflow_id\fP (\fI\%str\fP) \-\- The ID of the current Toil workflow. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod getLocalWorkflowDir(workflowID, configWorkDir=None) Return the directory where worker directories and the cache will be located for this workflow on this machine. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfigWorkDir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Value passed to the program using the \-\-workDir flag .IP \(bu 2 \fBworkflowID\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns Path to the local workflow directory on this machine .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir) Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine. .sp If an in\-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- Unique ID of the current workflow. .IP \(bu 2 \fBconfig_work_dir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Value used for the work directory in the current Toil Config. .IP \(bu 2 \fBconfig_coordination_dir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Value used for the coordination directory in the current Toil Config. .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns Path to the local workflow coordination directory on this machine. .UNINDENT .UNINDENT .UNINDENT .SH JOB STORE API .sp The job store interface is an abstraction layer that that hides the specific details of file storage, for example standard file systems, S3, etc. The \fI\%AbstractJobStore\fP API is implemented to support a give file store, e.g. S3. Implement this API to support a new file store. .INDENT 0.0 .TP .B class toil.jobStores.abstractJobStore.AbstractJobStore(locator) Represents the physical storage for the jobs and files in a Toil workflow. .sp JobStores are responsible for storing \fI\%toil.job.JobDescription\fP (which relate jobs to each other) and files. .sp Actual \fI\%toil.job.Job\fP objects are stored in files, referenced by JobDescriptions. All the non\-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs. .sp To actually get ahold of a \fI\%toil.job.Job\fP, use \fI\%toil.job.Job.loadJob()\fP with a JobStore and the relevant JobDescription. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __init__(locator) Create an instance of the job store. .sp The instance will not be fully functional until either \fI\%initialize()\fP or \fI\%resume()\fP is invoked. Note that the \fI\%destroy()\fP method may be invoked on the object with or without prior invocation of either of these two methods. .sp Takes and stores the locator string for the job store, which will be accessible via self.locator. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B initialize(config) Initialize this job store. .sp Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%Config\fP) \-\- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID. .TP .B Raises \fI\%JobStoreExistsException\fP \-\- if the physical storage for this job store already exists .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_config() Persists the value of the \fI\%AbstractJobStore.config\fP attribute to the job store, so that it can be retrieved later by other instances of this class. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resume() Connect this instance to the physical storage it represents and load the Toil configuration into the \fI\%AbstractJobStore.config\fP attribute. .INDENT 7.0 .TP .B Raises \fI\%NoSuchJobStoreException\fP \-\- if the physical storage for this job store doesn\(aqt exist .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property config: \fI\%Config\fP Return the Toil configuration associated with this job store. .INDENT 7.0 .TP .B Return type \fI\%toil.common.Config\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property locator: \fI\%str\fP Get the locator that defines the job store, which can be used to connect to it. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setRootJob(rootJobStoreID) Set the root job of the workflow backed by this job store. .INDENT 7.0 .TP .B Parameters \fBrootJobStoreID\fP (\fI\%FileID\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_root_job(job_id) Set the root job of the workflow backed by this job store. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%FileID\fP) \-\- The ID of the job to set as root .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_root_job() Loads the JobDescription for the root job in the current job store. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If no root job is set or if the root job doesn\(aqt exist in this job store .TP .B Return type \fI\%JobDescription\fP .TP .B Returns The root job. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create_root_job(job_description) Create the given JobDescription and set it as the root job in this job store. .INDENT 7.0 .TP .B Parameters \fBjob_description\fP (\fI\%JobDescription\fP) \-\- JobDescription to save and make the root job. .TP .B Return type \fI\%JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_root_job_return_value() Parse the return value from the root job. .sp Raises an exception if the root job hasn\(aqt fulfilled its promise yet. .INDENT 7.0 .TP .B Return type \fI\%Any\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B import_file(src_uri, shared_file_name=None, hardlink=False, symlink=True) Imports the file at the given URL into job store. The ID of the newly imported file is returned. If the name of a shared file name is provided, the file will be imported as such and None is returned. If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded. .sp Currently supported schemes are: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 .INDENT 2.0 .TP .B \(aqs3\(aq for objects in Amazon S3 e.g. s3://bucket/key .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqfile\(aq for local files e.g. \fI\%file:///local/file/path\fP .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqhttp\(aq e.g. \fI\%http://someurl.com/path\fP .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqgs\(aq e.g. gs://bucket/file .UNINDENT .UNINDENT .UNINDENT .UNINDENT .sp Raises FileNotFoundError if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. It must be a file, not a directory or prefix. .IP \(bu 2 \fBshared_file_name\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Optional name to assign to the imported file within the job store .IP \(bu 2 \fBsrc_uri\fP \-\- .IP \(bu 2 \fBshared_file_name\fP \-\- .IP \(bu 2 \fBhardlink\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Returns The jobStoreFileID of the imported file or None if shared_file_name was given .TP .B Return type \fI\%toil.fileStores.FileID\fP or None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B export_file(file_id, dst_uri) Exports file to destination pointed at by the destination URL. The exported file will be executable if and only if it was originally uploaded from an executable file on the local filesystem. .sp Refer to \fI\%AbstractJobStore.import_file()\fP documentation for currently supported URL schemes. .sp Note that the helper method _exportFile is used to read from the source and write to destination. To implement any optimizations that circumvent this, the _exportFile method should be overridden by subclasses of AbstractJobStore. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%FileID\fP) \-\- The id of the file in the job store that should be exported. .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. .IP \(bu 2 \fBfile_id\fP \-\- .IP \(bu 2 \fBdst_uri\fP \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod url_exists(src_uri) Return True if the file at the given URI exists, and False otherwise. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_size(src_uri) Get the size in bytes of the file at the given URL, or None if it cannot be obtained. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. .TP .B Return type \fI\%Optional\fP[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_is_directory(src_uri) Return True if the thing at the given URL is a directory, and False if it is a file. The URL may or may not end in \(aq/\(aq. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod list_url(src_uri) List the directory at the given URL. Returned path components can be joined with \(aq/\(aq onto the passed URL to form new URLs. Those that end in \(aq/\(aq correspond to directories. The provided URL may or may not end with \(aq/\(aq. .sp Currently supported schemes are: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 .INDENT 2.0 .TP .B \(aqs3\(aq for objects in Amazon S3 e.g. s3://bucket/prefix/ .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqfile\(aq for local files e.g. \fI\%file:///local/dir/path/\fP .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a directory or prefix in the storage mechanism of a supported URL scheme e.g. a prefix in an AWS s3 bucket. .IP \(bu 2 \fBsrc_uri\fP \-\- .UNINDENT .TP .B Return type \fI\%List\fP[\fI\%str\fP] .TP .B Returns A list of URL components in the given directory, already URL\-encoded. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod read_from_url(src_uri, writable) Read the given URL and write its content into the given writable stream. .sp Raises FileNotFoundError if the URL doesn\(aqt exist. .INDENT 7.0 .TP .B Return type \fI\%Tuple\fP[\fI\%int\fP, \fI\%bool\fP] .TP .B Returns The size of the file in bytes and whether the executable permission bit is set .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwritable\fP (\fI\%IO\fP[\fI\%bytes\fP]) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod open_url(src_uri) Read from the given URI. .sp Raises FileNotFoundError if the URL doesn\(aqt exist. .sp Has a readable stream interface, unlike \fI\%read_from_url()\fP which takes a writable stream. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%IO\fP[\fI\%bytes\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract destroy() The inverse of \fI\%initialize()\fP, this method deletes the physical storage represented by this instance. While not being atomic, this method \fIis\fP at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_env() Returns a dictionary of environment variables that this job store requires to be set in order to function properly on a worker. .INDENT 7.0 .TP .B Return type \fI\%dict\fP[\fI\%str\fP,\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clean(jobCache=None) Function to cleanup the state of a job store after a restart. .sp Fixes jobs that might have been partially updated. Resets the try counts and removes jobs that are not successors of the current root job. .INDENT 7.0 .TP .B Parameters \fBjobCache\fP (\fI\%Optional\fP[\fI\%Dict\fP[\fI\%Union\fP[\fI\%str\fP, \fI\%TemporaryID\fP], \fI\%JobDescription\fP]]) \-\- if a value it must be a dict from job ID keys to JobDescription object values. Jobs will be loaded from the cache (which can be downloaded from the job store in a batch) instead of piecemeal when recursed into. .TP .B Return type \fI\%JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract assign_job_id(job_description) Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription. .sp Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_description\fP (\fI\%JobDescription\fP) \-\- The JobDescription to give an ID to .IP \(bu 2 \fBjob_description\fP \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B batch() If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released. .INDENT 7.0 .TP .B Return type \fI\%Iterator\fP[\fI\%None\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract create_job(job_description) Writes the given JobDescription to the job store. The job must have an ID assigned already. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Returns The JobDescription passed. .TP .B Return type \fI\%toil.job.JobDescription\fP .TP .B Parameters \fBjob_description\fP (\fI\%JobDescription\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract job_exists(job_id) Indicates whether a description of the job with the specified jobStoreID exists in the job store .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_public_url(file_name) Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_name\fP (\fI\%str\fP) \-\- the jobStoreFileID of the file to generate a URL for .IP \(bu 2 \fBfile_name\fP \-\- .UNINDENT .TP .B Raises \fI\%NoSuchFileException\fP \-\- if the specified file does not exist in this job store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_shared_public_url(shared_file_name) Differs from \fI\%getPublicUrl()\fP in that this method is for generating URLs for shared files written by \fI\%writeSharedFileStream()\fP\&. .sp Returns a publicly accessible URL to the given file in the job store. The returned URL starts with \(aqhttp:\(aq, \(aqhttps:\(aq or \(aqfile:\(aq. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- The name of the shared file to generate a publically accessible url for. .IP \(bu 2 \fBshared_file_name\fP \-\- .UNINDENT .TP .B Raises \fI\%NoSuchFileException\fP \-\- raised if the specified file does not exist in the store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract load_job(job_id) Loads the description of the job referenced by the given ID, assigns it the job store\(aqs config, and returns it. .sp May declare the job to have failed (see \fI\%toil.job.JobDescription.setupJobAfterFailure()\fP) if there is evidence of a failed update attempt. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to load .TP .B Raises \fI\%NoSuchJobException\fP \-\- if there is no job with the given ID .TP .B Return type \fI\%JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract update_job(job_description) Persists changes to the state of the given JobDescription in this store atomically. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- the job to write to this job store .IP \(bu 2 \fBjob_description\fP (\fI\%JobDescription\fP) \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract delete_job(job_id) Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. .sp This operation is idempotent, i.e. deleting a job twice or deleting a non\-existent job will succeed silently. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to delete from this job store .IP \(bu 2 \fBjob_id\fP \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B jobs() Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object .INDENT 7.0 .TP .B Returns Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs .TP .B Return type Iterator[toil.job.jobDescription] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_file(local_path, job_id=None, cleanup=False) Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns an ID referencing the newly created file and can be used to read the file in the future. .TP .B Return type \fI\%str\fP .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_id\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .TP .B Return type \fI\%Iterator\fP[\fI\%Tuple\fP[\fI\%IO\fP[\fI\%bytes\fP], \fI\%str\fP]] .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future. .TP .B Return type Iterator[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBbasename\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_empty_file_store_id(job_id=None, cleanup=False, basename=None) Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .IP \(bu 2 \fBjob_id\fP \-\- .IP \(bu 2 \fBcleanup\fP \-\- .IP \(bu 2 \fBbasename\fP \-\- .UNINDENT .TP .B Returns a jobStoreFileID that references the newly created file and can be used to reference the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_file(file_id, local_path, symlink=False) Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation\-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed. .sp The file at the given local path may not be modified after this method returns! .sp Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to be copied .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path indicating where to place the contents of the given file in the job store .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link. .IP \(bu 2 \fBfile_id\fP \-\- .IP \(bu 2 \fBlocal_path\fP \-\- .IP \(bu 2 \fBsymlink\fP \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_file_stream(file_id, encoding=None, errors=None) Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%Union\fP[\fI\%FileID\fP, \fI\%str\fP]) \-\- ID of the file to get a readable file handle for .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBfile_id\fP \-\- .IP \(bu 2 \fBencoding\fP \-\- .IP \(bu 2 \fBerrors\fP \-\- .UNINDENT .TP .B Returns a context manager yielding a file handle which can be read from .TP .B Return type Iterator[Union[IO[\fI\%bytes\fP], IO[\fI\%str\fP]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract delete_file(file_id) Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non\-existent file will succeed silently. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to delete .IP \(bu 2 \fBfile_id\fP \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fileExists(jobStoreFileID) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract file_exists(file_id) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getFileSize(jobStoreFileID) Get the size of the given file in bytes. .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_file_size(file_id) Get the size of the given file in bytes, or 0 if it does not exist when queried. .sp Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .IP \(bu 2 \fBfile_id\fP \-\- .UNINDENT .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B updateFile(jobStoreFileID, localFilePath) Replaces the existing version of a file in the job store. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBlocalFilePath\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract update_file(file_id, local_path) Replaces the existing version of a file in the job store. .sp Throws an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path to a file that will overwrite the current version in the job store .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract update_file_stream(file_id, encoding=None, errors=None) Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBfile_id\fP \-\- .IP \(bu 2 \fBencoding\fP \-\- .IP \(bu 2 \fBerrors\fP \-\- .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .TP .B Return type \fI\%Iterator\fP[\fI\%IO\fP[\fI\%Any\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None) Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencrypted\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear. .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBshared_file_name\fP \-\- .IP \(bu 2 \fBencrypted\fP \-\- .IP \(bu 2 \fBencoding\fP \-\- .IP \(bu 2 \fBerrors\fP \-\- .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns a context manager yielding a writable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_shared_file_stream(shared_file_name, encoding=None, errors=None) Returns a context manager yielding a readable file handle to the global file referenced by the given name. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBshared_file_name\fP \-\- .IP \(bu 2 \fBencoding\fP \-\- .IP \(bu 2 \fBerrors\fP \-\- .UNINDENT .TP .B Returns a context manager yielding a readable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_logs(msg) Stores a message as a log in the jobstore. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmsg\fP (\fI\%str\fP) \-\- the string to be written .IP \(bu 2 \fBmsg\fP \-\- .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_logs(callback, read_all=False) Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages). .sp Only unread logs will be read unless the read_all parameter is set. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcallback\fP (\fI\%Callable\fP[\fI\%\&...\fP, \fI\%Any\fP]) \-\- a function to be applied to each of the stats file handles found .IP \(bu 2 \fBread_all\fP (\fI\%bool\fP) \-\- a boolean indicating whether to read the already processed stats files in addition to the unread stats files .IP \(bu 2 \fBcallback\fP \-\- .IP \(bu 2 \fBread_all\fP \-\- .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns the number of stats files processed .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_leader_pid() Write the pid of this process to a file in the job store. .sp Overwriting the current contents of pid.log is a feature, not a bug of this method. Other methods will rely on always having the most current pid available. So far there is no reason to store any old pids. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_leader_pid() Read the pid of the leader process to a file in the job store. .INDENT 7.0 .TP .B Raises \fI\%NoSuchFileException\fP \-\- If the PID file doesn\(aqt exist. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_leader_node_id() Write the leader node id to the job store. This should only be called by the leader. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_leader_node_id() Read the leader node id stored in the job store. .INDENT 7.0 .TP .B Raises \fI\%NoSuchFileException\fP \-\- If the node ID file doesn\(aqt exist. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_kill_flag(kill=False) Write a file inside the job store that serves as a kill flag. .sp The initialized file contains the characters \(dqNO\(dq. This should only be changed when the user runs the \(dqtoil kill\(dq command. .sp Changing this file to a \(dqYES\(dq triggers a kill of the leader process. The workers are expected to be cleaned up by the leader. .INDENT 7.0 .TP .B Parameters \fBkill\fP (\fI\%bool\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_kill_flag() Read the kill flag from the job store, and return True if the leader has been killed. False otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B default_caching() Jobstore\(aqs preference as to whether it likes caching or doesn\(aqt care about it. Some jobstores benefit from caching, however on some local configurations it can be flaky. .sp see \fI\%https://github.com/DataBiosphere/toil/issues/4218\fP .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .SH TOIL JOB API .sp Functions to wrap jobs and return values (promises). .SS FunctionWrappingJob .sp The subclass of Job for wrapping user functions. .INDENT 0.0 .TP .B class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs) Job used to wrap a function. In its \fIrun\fP method the wrapped function is called. .INDENT 7.0 .TP .B __init__(userFunction, *args, **kwargs) .INDENT 7.0 .TP .B Parameters \fBuserFunction\fP (\fIcallable\fP) \-\- The function to wrap. It will be called with \fB*args\fP and \fB**kwargs\fP as arguments. .UNINDENT .sp The keywords \fBmemory\fP, \fBcores\fP, \fBdisk\fP, \fBaccelerators\(ga, \(ga\(gapreemptible\fP and \fBcheckpoint\fP are reserved keyword arguments that if specified will be used to determine the resources required for the job, as \fBtoil.job.Job.__init__()\fP\&. If they are keyword arguments to the function they will be extracted from the function definition, but may be overridden by the user (as you would expect). .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS JobFunctionWrappingJob .sp The subclass of FunctionWrappingJob for wrapping user job functions. .INDENT 0.0 .TP .B class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs) A job function is a function whose first argument is a \fI\%Job\fP instance that is the wrapping job for the function. This can be used to add successor jobs for the function and perform all the functions the \fI\%Job\fP class provides. .sp To enable the job function to get access to the \fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP instance (see \fI\%toil.job.Job.run()\fP), it is made a variable of the wrapping job called fileStore. .sp To specify a job\(aqs resource requirements the following default keyword arguments can be specified: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 memory .IP \(bu 2 disk .IP \(bu 2 cores .IP \(bu 2 accelerators .IP \(bu 2 preemptible .UNINDENT .UNINDENT .UNINDENT .sp For example to wrap a function into a job we would call: .INDENT 7.0 .INDENT 3.5 .sp .EX Job.wrapJobFn(myJob, memory=\(aq100k\(aq, disk=\(aq1M\(aq, cores=0.1) .EE .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS EncapsulatedJob .sp The subclass of Job for \fIencapsulating\fP a job, allowing a subgraph of jobs to be treated as a single job. .INDENT 0.0 .TP .B class toil.job.EncapsulatedJob(job, unitName=None) A convenience Job class used to make a job subgraph appear to be a single job. .sp Let A be the root job of a job subgraph and B be another job we\(aqd like to run after A and all its successors have completed, for this use encapsulate: .INDENT 7.0 .INDENT 3.5 .sp .EX # Job A and subgraph, Job B A, B = A(), B() Aprime = A.encapsulate() Aprime.addChild(B) # B will run after A and all its successors have completed, A and its subgraph of # successors in effect appear to be just one job. .EE .UNINDENT .UNINDENT .sp If the job being encapsulated has predecessors (e.g. is not the root job), then the encapsulated job will inherit these predecessors. If predecessors are added to the job being encapsulated after the encapsulated job is created then the encapsulating job will NOT inherit these predecessors automatically. Care should be exercised to ensure the encapsulated job has the proper set of predecessors. .sp The return value of an encapsulated job (as accessed by the \fI\%toil.job.Job.rv()\fP function) is the return value of the root job, e.g. A().encapsulate().rv() and A().rv() will resolve to the same value after A or A.encapsulate() has been run. .INDENT 7.0 .TP .B __init__(job, unitName=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%toil.job.Job\fP) \-\- the job to encapsulate. .IP \(bu 2 \fBunitName\fP (\fI\%str\fP) \-\- human\-readable name to identify this job instance. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChild(childJob) Add a childJob to be run as child of this job. .sp Child jobs will be run directly after this job\(aqs \fI\%toil.job.Job.run()\fP method has completed. .INDENT 7.0 .TP .B Returns childJob: for call chaining .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addService(service, parentService=None) Add a service. .sp The \fI\%toil.job.Job.Service.start()\fP method of the service will be called after the run method has completed but before any successors are run. The service\(aqs \fI\%toil.job.Job.Service.stop()\fP method will be called once the successors of the job have been run. .sp Services allow things like databases and servers to be started and accessed by jobs in a workflow. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If service has already been made the child of a job or another service. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice\fP \-\- Service to add. .IP \(bu 2 \fBparentService\fP \-\- Service that will be started before \(aqservice\(aq is started. Allows trees of services to be established. parentService must be a service of this job. .UNINDENT .TP .B Returns a promise that will be replaced with the return value from \fI\%toil.job.Job.Service.start()\fP of service in any successor of the job. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOnJob) Add a follow\-on job. .sp Follow\-on jobs will be run after the child jobs and their successors have been run. .INDENT 7.0 .TP .B Returns followOnJob for call chaining .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rv(*path) Create a \fIpromise\fP (\fI\%toil.job.Promise\fP). .sp The \(dqpromise\(dq representing a return value of the job\(aqs run method, or, in case of a function\-wrapping job, the wrapped function\(aqs return value. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI(\fP\fIAny\fP\fI)\fP) \-\- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the \fI__getitem__()\fP magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is \fI[6,{\(aqa\(aq:42}]\fP, \fI\&.rv(0)\fP would select \fI6\fP , \fIrv(1)\fP would select \fI{\(aqa\(aq:3}\fP while \fIrv(1,\(aqa\(aq)\fP would select \fI3\fP\&. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a \fIslice\fP object. For example, assuming that the return value is \fI[6, 7, 8, 9]\fP then \fI\&.rv(slice(1, 3))\fP would select \fI[7, 8]\fP\&. Note that slicing really only makes sense at the end of path. .TP .B Return type \fI\%Promise\fP .TP .B Returns A promise representing the return value of this jobs \fI\%toil.job.Job.run()\fP method. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareForPromiseRegistration(jobStore) Set up to allow this job\(aqs promises to register themselves. .sp Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized. .sp The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor. .UNINDENT .UNINDENT .SS Promise .sp The class used to reference return values of jobs/services not yet run/started. .INDENT 0.0 .TP .B class toil.job.Promise(job, path) References a return value from a method as a \fIpromise\fP before the method itself is run. .sp References a return value from a \fI\%toil.job.Job.run()\fP or \fI\%toil.job.Job.Service.start()\fP method as a \fIpromise\fP before the method itself is run. .sp Let T be a job. Instances of \fI\%Promise\fP (termed a \fIpromise\fP) are returned by T.rv(), which is used to reference the return value of T\(aqs run function. When the promise is passed to the constructor (or as an argument to a wrapped function) of a different, successor job the promise will be replaced by the actual referenced return value. This mechanism allows a return values from one job\(aqs run method to be input argument to job before the former job\(aqs run function has been executed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%Job\fP) \-\- .IP \(bu 2 \fBpath\fP (\fI\%Any\fP) \-\- .UNINDENT .TP .B Return type \fI\%Promise\fP .UNINDENT .INDENT 7.0 .TP .B filesToDelete = {} A set of IDs of files containing promised values when we know we won\(aqt need them anymore .UNINDENT .INDENT 7.0 .TP .B __init__(job, path) Initialize this promise. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%Job\fP) \-\- the job whose return value this promise references .IP \(bu 2 \fBpath\fP (\fI\%Any\fP) \-\- see \fI\%Job.rv()\fP .IP \(bu 2 \fBjob\fP \-\- .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.PromisedRequirement(valueOrCallable, *args) Class for dynamically allocating job function resource requirements. .sp (involving \fI\%toil.job.Promise\fP instances.) .sp Use when resource requirements depend on the return value of a parent function. PromisedRequirements can be modified by passing a function that takes the \fI\%Promise\fP as input. .sp For example, let f, g, and h be functions. Then a Toil workflow can be defined as follows:: A = Job.wrapFn(f) B = A.addChildFn(g, cores=PromisedRequirement(A.rv()) C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv())) .INDENT 7.0 .TP .B __init__(valueOrCallable, *args) Initialize this Promised Requirement. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBvalueOrCallable\fP \-\- A single Promise instance or a function that takes args as input parameters. .IP \(bu 2 \fBargs\fP (\fI\%int\fP\fI or \fP\fI\&.Promise\fP) \-\- variable length argument list .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getValue() Return PromisedRequirement value. .UNINDENT .INDENT 7.0 .TP .B static convertPromises(kwargs) Return True if reserved resource keyword is a Promise or PromisedRequirement instance. .sp Converts Promise instance to PromisedRequirement. .INDENT 7.0 .TP .B Parameters \fBkwargs\fP (\fI\%Dict\fP[\fI\%str\fP, \fI\%Any\fP]) \-\- function keyword arguments .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .SH JOB METHODS API .sp Jobs are the units of work in Toil which are composed into workflows. .INDENT 0.0 .TP .B class toil.job.Job(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName=\(aq\(aq, checkpoint=False, displayName=\(aq\(aq, descriptionClass=None, local=None) Class represents a unit of work in toil. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmemory\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%None\fP]) \-\- .IP \(bu 2 \fBcores\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%float\fP, \fI\%None\fP]) \-\- .IP \(bu 2 \fBdisk\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%None\fP]) \-\- .IP \(bu 2 \fBaccelerators\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%Mapping\fP[\fI\%str\fP, \fI\%Any\fP], \fI\%AcceleratorRequirement\fP, \fI\%Sequence\fP[\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%Mapping\fP[\fI\%str\fP, \fI\%Any\fP], \fI\%AcceleratorRequirement\fP]], \fI\%None\fP]) \-\- .IP \(bu 2 \fBpreemptible\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%bool\fP, \fI\%None\fP]) \-\- .IP \(bu 2 \fBpreemptable\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%bool\fP, \fI\%None\fP]) \-\- .IP \(bu 2 \fBunitName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBcheckpoint\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- .IP \(bu 2 \fBdisplayName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBdescriptionClass\fP (\fI\%Optional\fP[\fI\%type\fP]) \-\- .IP \(bu 2 \fBlocal\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __init__(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName=\(aq\(aq, checkpoint=False, displayName=\(aq\(aq, descriptionClass=None, local=None) Job initializer. .sp This method must be called by any overriding constructor. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmemory\fP (\fI\%int\fP\fI or \fP\fIstring convertible by toil.lib.conversions.human2bytes to an int\fP) \-\- the maximum number of bytes of memory the job will require to run. .IP \(bu 2 \fBcores\fP (\fI\%float\fP\fI, \fP\fI\%int\fP\fI, or \fP\fIstring convertible by toil.lib.conversions.human2bytes to an int\fP) \-\- the number of CPU cores required. .IP \(bu 2 \fBdisk\fP (\fI\%int\fP\fI or \fP\fIstring convertible by toil.lib.conversions.human2bytes to an int\fP) \-\- the amount of local disk space required by the job, expressed in bytes. .IP \(bu 2 \fBaccelerators\fP (\fI\%int\fP\fI, \fP\fIstring\fP\fI, \fP\fI\%dict\fP\fI, or \fP\fI\%list\fP\fI of \fP\fIthose. Strings and dicts must be parseable by parse_accelerator.\fP) \-\- the computational accelerators required by the job. If a string, can be a string of a number, or a string specifying a model, brand, or API (with optional colon\-delimited count). .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP\fI, \fP\fIint in {0\fP\fI, \fP\fI1}\fP\fI, or \fP\fIstring in {\(aqfalse\(aq\fP\fI, \fP\fI\(aqtrue\(aq} in any case\fP) \-\- if the job can be run on a preemptible node. .IP \(bu 2 \fBpreemptable\fP (\fI\%Union\fP[\fI\%str\fP, \fI\%int\fP, \fI\%bool\fP, \fI\%None\fP]) \-\- legacy preemptible parameter, for backwards compatibility with workflows not using the preemptible keyword .IP \(bu 2 \fBunitName\fP (\fI\%str\fP) \-\- Human\-readable name for this instance of the job. .IP \(bu 2 \fBcheckpoint\fP (\fI\%bool\fP) \-\- if any of this job\(aqs successor jobs completely fails, exhausting all their retries, remove any successor jobs and rerun this job to restart the subtree. Job must be a leaf vertex in the job graph when initially defined, see \fBtoil.job.Job.checkNewCheckpointsAreCutVertices()\fP\&. .IP \(bu 2 \fBdisplayName\fP (\fI\%str\fP) \-\- Human\-readable job type display name. .IP \(bu 2 \fBdescriptionClass\fP (\fIclass\fP) \-\- Override for the JobDescription class used to describe the job. .IP \(bu 2 \fBlocal\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- if the job can be run on the leader. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property jobStoreID: \fI\%str\fP | \fI\%TemporaryID\fP Get the ID of this Job. .INDENT 7.0 .TP .B Return type Union[\fI\%str\fP, \fI\%TemporaryID\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property description: \fI\%JobDescription\fP Expose the JobDescription that describes this job. .INDENT 7.0 .TP .B Return type \fI\%JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property disk: \fI\%int\fP The maximum number of bytes of disk the job will require to run. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property memory The maximum number of bytes of memory the job will require to run. .UNINDENT .INDENT 7.0 .TP .B property cores: \fI\%int\fP | \fI\%float\fP The number of CPU cores required. .INDENT 7.0 .TP .B Return type Union[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property accelerators: \fI\%List\fP[\fI\%AcceleratorRequirement\fP] Any accelerators, such as GPUs, that are needed. .INDENT 7.0 .TP .B Return type List[\fI\%AcceleratorRequirement\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property preemptible: \fI\%bool\fP Whether the job can be run on a preemptible node. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property checkpoint: \fI\%bool\fP Determine if the job is a checkpoint job or not. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B assignConfig(config) Assign the given config object. .sp It will be used by various actions implemented inside the Job class. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%Config\fP) \-\- Config object to query .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP (\fI\%AbstractFileStore\fP) \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Return type \fI\%Any\fP .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChild(childJob) Add a childJob to be run as child of this job. .sp Child jobs will be run directly after this job\(aqs \fI\%toil.job.Job.run()\fP method has completed. .INDENT 7.0 .TP .B Return type \fI\%Job\fP .TP .B Returns childJob: for call chaining .TP .B Parameters \fBchildJob\fP (\fI\%Job\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasChild(childJob) Check if childJob is already a child of this job. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Returns True if childJob is a child of the job, else False. .TP .B Parameters \fBchildJob\fP (\fI\%Job\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOnJob) Add a follow\-on job. .sp Follow\-on jobs will be run after the child jobs and their successors have been run. .INDENT 7.0 .TP .B Return type \fI\%Job\fP .TP .B Returns followOnJob for call chaining .TP .B Parameters \fBfollowOnJob\fP (\fI\%Job\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasPredecessor(job) Check if a given job is already a predecessor of this job. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasFollowOn(followOnJob) Check if given job is already a follow\-on of this job. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Returns True if the followOnJob is a follow\-on of this job, else False. .TP .B Parameters \fBfollowOnJob\fP (\fI\%Job\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addService(service, parentService=None) Add a service. .sp The \fI\%toil.job.Job.Service.start()\fP method of the service will be called after the run method has completed but before any successors are run. The service\(aqs \fI\%toil.job.Job.Service.stop()\fP method will be called once the successors of the job have been run. .sp Services allow things like databases and servers to be started and accessed by jobs in a workflow. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If service has already been made the child of a job or another service. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice\fP (\fI\%Service\fP) \-\- Service to add. .IP \(bu 2 \fBparentService\fP (\fI\%Optional\fP[\fI\%Service\fP]) \-\- Service that will be started before \(aqservice\(aq is started. Allows trees of services to be established. parentService must be a service of this job. .UNINDENT .TP .B Return type \fI\%Promise\fP .TP .B Returns a promise that will be replaced with the return value from \fI\%toil.job.Job.Service.start()\fP of service in any successor of the job. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasService(service) Return True if the given Service is a service of this job, and False otherwise. .INDENT 7.0 .TP .B Parameters \fBservice\fP (\fI\%Service\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChildFn(fn, *args, **kwargs) Add a function as a child job. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%Callable\fP) \-\- Function to be run as a child job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Return type \fI\%FunctionWrappingJob\fP .TP .B Returns The new child job that wraps fn. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOnFn(fn, *args, **kwargs) Add a function as a follow\-on job. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%Callable\fP) \-\- Function to be run as a follow\-on job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Return type \fI\%FunctionWrappingJob\fP .TP .B Returns The new follow\-on job that wraps fn. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChildJobFn(fn, *args, **kwargs) Add a job function as a child job. .sp See \fI\%toil.job.JobFunctionWrappingJob\fP for a definition of a job function. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%Callable\fP) \-\- Job function to be run as a child job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Return type \fI\%FunctionWrappingJob\fP .TP .B Returns The new child job that wraps fn. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOnJobFn(fn, *args, **kwargs) Add a follow\-on job function. .sp See \fI\%toil.job.JobFunctionWrappingJob\fP for a definition of a job function. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%Callable\fP) \-\- Job function to be run as a follow\-on job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Return type \fI\%FunctionWrappingJob\fP .TP .B Returns The new follow\-on job that wraps fn. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property tempDir: \fI\%str\fP Shortcut to calling \fBjob.fileStore.getLocalTempDir()\fP\&. .sp Temp dir is created on first call and will be returned for first and future calls :return: Path to tempDir. See \fIjob.fileStore.getLocalTempDir\fP .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log(text, level=20) Log using \fBfileStore.log_to_leader()\fP\&. .INDENT 7.0 .TP .B Parameters \fBtext\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static wrapFn(fn, *args, **kwargs) Makes a Job out of a function. .sp Convenience function for constructor of \fI\%toil.job.FunctionWrappingJob\fP\&. .INDENT 7.0 .TP .B Parameters \fBfn\fP \-\- Function to be run with \fB*args\fP and \fB**kwargs\fP as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Return type \fI\%FunctionWrappingJob\fP .TP .B Returns The new function that wraps fn. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static wrapJobFn(fn, *args, **kwargs) Makes a Job out of a job function. .sp Convenience function for constructor of \fI\%toil.job.JobFunctionWrappingJob\fP\&. .INDENT 7.0 .TP .B Parameters \fBfn\fP \-\- Job function to be run with \fB*args\fP and \fB**kwargs\fP as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Return type \fI\%JobFunctionWrappingJob\fP .TP .B Returns The new job function that wraps fn. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B encapsulate(name=None) Encapsulates the job, see \fI\%toil.job.EncapsulatedJob\fP\&. Convenience function for constructor of \fI\%toil.job.EncapsulatedJob\fP\&. .INDENT 7.0 .TP .B Parameters \fBname\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Human\-readable name for the encapsulated job. .TP .B Return type \fI\%EncapsulatedJob\fP .TP .B Returns an encapsulated version of this job. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rv(*path) Create a \fIpromise\fP (\fI\%toil.job.Promise\fP). .sp The \(dqpromise\(dq representing a return value of the job\(aqs run method, or, in case of a function\-wrapping job, the wrapped function\(aqs return value. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI(\fP\fIAny\fP\fI)\fP) \-\- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the \fI__getitem__()\fP magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is \fI[6,{\(aqa\(aq:42}]\fP, \fI\&.rv(0)\fP would select \fI6\fP , \fIrv(1)\fP would select \fI{\(aqa\(aq:3}\fP while \fIrv(1,\(aqa\(aq)\fP would select \fI3\fP\&. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a \fIslice\fP object. For example, assuming that the return value is \fI[6, 7, 8, 9]\fP then \fI\&.rv(slice(1, 3))\fP would select \fI[7, 8]\fP\&. Note that slicing really only makes sense at the end of path. .TP .B Return type \fI\%Promise\fP .TP .B Returns A promise representing the return value of this jobs \fI\%toil.job.Job.run()\fP method. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareForPromiseRegistration(jobStore) Set up to allow this job\(aqs promises to register themselves. .sp Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized. .sp The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B checkJobGraphForDeadlocks() Ensures that a graph of Jobs (that hasn\(aqt yet been saved to the JobStore) doesn\(aqt contain any pathological relationships between jobs that would result in deadlocks if we tried to run the jobs. .sp See \fI\%toil.job.Job.checkJobGraphConnected()\fP, \fBtoil.job.Job.checkJobGraphAcyclic()\fP and \fI\%toil.job.Job.checkNewCheckpointsAreLeafVertices()\fP for more info. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if the job graph is cyclic, contains multiple roots or contains checkpoint jobs that are not leaf vertices when defined (see \fBtoil.job.Job.checkNewCheckpointsAreLeaves()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRootJobs() Return the set of root job objects that contain this job. .sp A root job is a job with no predecessors (i.e. which are not children, follow\-ons, or services). .sp Only deals with jobs created here, rather than loaded from the job store. .INDENT 7.0 .TP .B Return type \fI\%Set\fP[\fI\%Job\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B checkJobGraphConnected() .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if \fI\%toil.job.Job.getRootJobs()\fP does not contain exactly one root job. .UNINDENT .sp As execution always starts from one root job, having multiple root jobs will cause a deadlock to occur. .sp Only deals with jobs created here, rather than loaded from the job store. .UNINDENT .INDENT 7.0 .TP .B checkJobGraphAcylic() .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if the connected component of jobs containing this job contains any cycles of child/followOn dependencies in the \fIaugmented job graph\fP (see below). Such cycles are not allowed in valid job graphs. .UNINDENT .sp A follow\-on edge (A, B) between two jobs A and B is equivalent to adding a child edge to B from (1) A, (2) from each child of A, and (3) from the successors of each child of A. We call each such edge an edge an \(dqimplied\(dq edge. The augmented job graph is a job graph including all the implied edges. .sp For a job graph G = (V, E) the algorithm is \fBO(|V|^2)\fP\&. It is \fBO(|V| + |E|)\fP for a graph with no follow\-ons. The former follow\-on case could be improved! .sp Only deals with jobs created here, rather than loaded from the job store. .UNINDENT .INDENT 7.0 .TP .B checkNewCheckpointsAreLeafVertices() A checkpoint job is a job that is restarted if either it fails, or if any of its successors completely fails, exhausting their retries. .sp A job is a leaf it is has no successors. .sp A checkpoint job must be a leaf when initially added to the job graph. When its run method is invoked it can then create direct successors. This restriction is made to simplify implementation. .sp Only works on connected components of jobs not yet added to the JobStore. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if there exists a job being added to the graph for which checkpoint=True and which is not a leaf. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B defer(function, *args, **kwargs) Register a deferred function, i.e. a callable that will be invoked after the current attempt at running this job concludes. A job attempt is said to conclude when the job function (or the \fI\%toil.job.Job.run()\fP method for class\-based jobs) returns, raises an exception or after the process running it terminates abnormally. A deferred function will be called on the node that attempted to run the job, even if a subsequent attempt is made on another node. A deferred function should be idempotent because it may be called multiple times on the same node or even in the same process. More than one deferred function may be registered per job attempt by calling this method repeatedly with different arguments. If the same function is registered twice with the same or different arguments, it will be called twice per job attempt. .sp Examples for deferred functions are ones that handle cleanup of resources external to Toil, like Docker containers, files outside the work directory, etc. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfunction\fP (\fIcallable\fP) \-\- The function to be called after this job concludes. .IP \(bu 2 \fBargs\fP (\fI\%list\fP) \-\- The arguments to the function .IP \(bu 2 \fBkwargs\fP (\fI\%dict\fP) \-\- The keyword arguments to the function .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getTopologicalOrderingOfJobs() .INDENT 7.0 .TP .B Return type \fI\%List\fP[\fI\%Job\fP] .TP .B Returns a list of jobs such that for all pairs of indices i, j for which i < j, the job at index i can be run before the job at index j. .UNINDENT .sp Only considers jobs in this job\(aqs subgraph that are newly added, not loaded from the job store. .sp Ignores service jobs. .UNINDENT .INDENT 7.0 .TP .B saveBody(jobStore) Save the execution data for just this job to the JobStore, and fill in the JobDescription with the information needed to retrieve it. .sp The Job\(aqs JobDescription must have already had a real jobStoreID assigned to it. .sp Does not save the JobDescription. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- The job store to save the job body into. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B saveAsRootJob(jobStore) Save this job to the given jobStore as the root job of the workflow. .INDENT 7.0 .TP .B Return type \fI\%JobDescription\fP .TP .B Returns the JobDescription describing this job. .TP .B Parameters \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod loadJob(jobStore, jobDescription) Retrieves a \fI\%toil.job.Job\fP instance from a JobStore .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- The job store. .IP \(bu 2 \fBjobDescription\fP (\fI\%JobDescription\fP) \-\- the JobDescription of the job to retrieve. .UNINDENT .TP .B Return type \fI\%Job\fP .TP .B Returns The job referenced by the JobDescription. .UNINDENT .UNINDENT .UNINDENT .SS JobDescription .sp The class used to store all the information that the Toil Leader ever needs to know about a Job. .INDENT 0.0 .TP .B class toil.job.JobDescription(requirements, jobName, unitName=\(aq\(aq, displayName=\(aq\(aq, command=None, local=None) Stores all the information that the Toil Leader ever needs to know about a Job. .sp (requirements information, dependency information, commands to issue, etc.) .sp Can be obtained from an actual (i.e. executable) Job object, and can be used to obtain the Job object from the JobStore. .sp Never contains other Jobs or JobDescriptions: all reference is by ID. .sp Subclassed into variants for checkpoint jobs and service jobs that have their specific parameters. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirements\fP (\fI\%Mapping\fP[\fI\%str\fP, \fI\%Union\fP[\fI\%int\fP, \fI\%str\fP, \fI\%bool\fP]]) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBunitName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBdisplayName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBlocal\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __init__(requirements, jobName, unitName=\(aq\(aq, displayName=\(aq\(aq, command=None, local=None) Create a new JobDescription. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirements\fP (\fI\%Mapping\fP[\fI\%str\fP, \fI\%Union\fP[\fI\%int\fP, \fI\%str\fP, \fI\%bool\fP]]) \-\- Dict from string to number, string, or bool describing the resource requirements of the job. \(aqcores\(aq, \(aqmemory\(aq, \(aqdisk\(aq, and \(aqpreemptible\(aq fields, if set, are parsed and broken out into properties. If unset, the relevant property will be unspecified, and will be pulled from the assigned Config object if queried (see \fI\%toil.job.Requirer.assignConfig()\fP). .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- Name of the kind of job this is. May be used in job store IDs and logging. Also used to let the cluster scaler learn a model for how long the job will take. Ought to be the job class\(aqs name if no real user\-defined name is available. .IP \(bu 2 \fBunitName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Name of this instance of this kind of job. May appear with jobName in logging. .IP \(bu 2 \fBdisplayName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- A human\-readable name to identify this particular job instance. Ought to be the job class\(aqs name if no real user\-defined name is available. .IP \(bu 2 \fBlocal\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- If True, the job is meant to use minimal resources but is sensitive to execution latency, and so should be executed by the leader. .IP \(bu 2 \fBcommand\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_names() Get the names and ID of this job as a named tuple. .INDENT 7.0 .TP .B Return type \fI\%Names\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_chain() Get all the jobs that executed in this job\(aqs chain, in order. .sp For each job, produces a named tuple with its various names and its original job store ID. The jobs in the chain are in execution order. .sp If the job hasn\(aqt run yet or it didn\(aqt chain, produces a one\-item list. .INDENT 7.0 .TP .B Return type \fI\%List\fP[\fI\%Names\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B serviceHostIDsInBatches() Find all batches of service host job IDs that can be started at the same time. .sp (in the order they need to start in) .INDENT 7.0 .TP .B Return type \fI\%Iterator\fP[\fI\%List\fP[\fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B successorsAndServiceHosts() Get an iterator over all child, follow\-on, and service job IDs. .INDENT 7.0 .TP .B Return type \fI\%Iterator\fP[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B allSuccessors() Get an iterator over all child, follow\-on, and chained, inherited successor job IDs. .sp Follow\-ons will come before children. .INDENT 7.0 .TP .B Return type \fI\%Iterator\fP[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B successors_by_phase() Get an iterator over all child/follow\-on/chained inherited successor job IDs, along with their phase numbere on the stack. .sp Phases ececute higher numbers to lower numbers. .INDENT 7.0 .TP .B Return type \fI\%Iterator\fP[\fI\%Tuple\fP[\fI\%int\fP, \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property services Get a collection of the IDs of service host jobs for this job, in arbitrary order. .sp Will be empty if the job has no unfinished services. .UNINDENT .INDENT 7.0 .TP .B nextSuccessors() Return the collection of job IDs for the successors of this job that are ready to run. .sp If those jobs have multiple predecessor relationships, they may still be blocked on other jobs. .sp Returns None when at the final phase (all successors done), and an empty collection if there are more phases but they can\(aqt be entered yet (e.g. because we are waiting for the job itself to run). .INDENT 7.0 .TP .B Return type \fI\%Set\fP[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B filterSuccessors(predicate) Keep only successor jobs for which the given predicate function approves. .sp The predicate function is called with the job\(aqs ID. .sp Treats all other successors as complete and forgets them. .INDENT 7.0 .TP .B Parameters \fBpredicate\fP (\fI\%Callable\fP[[\fI\%str\fP], \fI\%bool\fP]) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B filterServiceHosts(predicate) Keep only services for which the given predicate approves. .sp The predicate function is called with the service host job\(aqs ID. .sp Treats all other services as complete and forgets them. .INDENT 7.0 .TP .B Parameters \fBpredicate\fP (\fI\%Callable\fP[[\fI\%str\fP], \fI\%bool\fP]) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clear_nonexistent_dependents(job_store) Remove all references to child, follow\-on, and associated service jobs that do not exist. .sp That is to say, all those that have been completed and removed. .INDENT 7.0 .TP .B Parameters \fBjob_store\fP (\fI\%AbstractJobStore\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clear_dependents() Remove all references to successor and service jobs. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_subtree_done() Check if the subtree is done. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Returns True if the job appears to be done, and all related child, follow\-on, and service jobs appear to be finished and removed. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B replace(other) Take on the ID of another JobDescription, retaining our own state and type. .sp When updated in the JobStore, we will save over the other JobDescription. .sp Useful for chaining jobs: the chained\-to job can replace the parent job. .sp Merges cleanup state and successors other than this job from the job being replaced into this one. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%JobDescription\fP) \-\- Job description to replace. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_new_version(other) Make sure a prospective new version of the JobDescription is actually moving forward in time and not backward. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%JobDescription\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChild(childID) Make the job with the given ID a child of the described job. .INDENT 7.0 .TP .B Parameters \fBchildID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOnID) Make the job with the given ID a follow\-on of the described job. .INDENT 7.0 .TP .B Parameters \fBfollowOnID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addServiceHostJob(serviceID, parentServiceID=None) Make the ServiceHostJob with the given ID a service of the described job. .sp If a parent ServiceHostJob ID is given, that parent service will be started first, and must have already been added. .UNINDENT .INDENT 7.0 .TP .B hasChild(childID) Return True if the job with the given ID is a child of the described job. .INDENT 7.0 .TP .B Parameters \fBchildID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasFollowOn(followOnID) Test if the job with the given ID is a follow\-on of the described job. .INDENT 7.0 .TP .B Parameters \fBfollowOnID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasServiceHostJob(serviceID) Test if the ServiceHostJob is a service of the described job. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B renameReferences(renames) Apply the given dict of ID renames to all references to jobs. .sp Does not modify our own ID or those of finished predecessors. IDs not present in the renames dict are left as\-is. .INDENT 7.0 .TP .B Parameters \fBrenames\fP (\fI\%Dict\fP[\fI\%TemporaryID\fP, \fI\%str\fP]) \-\- Rename operations to apply. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addPredecessor() Notify the JobDescription that a predecessor has been added to its Job. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B onRegistration(jobStore) Perform setup work that requires the JobStore. .sp Called by the Job saving logic when this JobDescription meets the JobStore and has its ID assigned. .sp Overridden to perform setup work (like hooking up flag files for service jobs) that requires the JobStore. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- The job store we are being placed into .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setupJobAfterFailure(exit_status=None, exit_reason=None) Configure job after a failure. .sp Reduce the remainingTryCount if greater than zero and set the memory to be at least as big as the default memory (in case of exhaustion of memory, which is common). .sp Requires a configuration to have been assigned (see \fI\%toil.job.Requirer.assignConfig()\fP). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexit_status\fP (\fI\%Optional\fP[\fI\%int\fP]) \-\- The exit code from the job. .IP \(bu 2 \fBexit_reason\fP (\fI\%Optional\fP[\fI\%BatchJobExitReason\fP]) \-\- The reason the job stopped, if available from the batch system. .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLogFileHandle(jobStore) Create a context manager that yields a file handle to the log file. .sp Assumes logJobStoreFileID is set. .UNINDENT .INDENT 7.0 .TP .B property remainingTryCount Get the number of tries remaining. .sp The try count set on the JobDescription, or the default based on the retry count from the config if none is set. .UNINDENT .INDENT 7.0 .TP .B clearRemainingTryCount() Clear remainingTryCount and set it back to its default value. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Returns True if a modification to the JobDescription was made, and False otherwise. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B reserve_versions(count) Reserve a job version number for later, for journaling asynchronously. .INDENT 7.0 .TP .B Parameters \fBcount\fP (\fI\%int\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pre_update_hook() Run before pickling and saving a created or updated version of this job. .sp Called by the job store. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .UNINDENT .SH JOB.RUNNER API .sp The Runner contains the methods needed to configure and start a Toil run. .INDENT 0.0 .TP .B class Job.Runner Used to setup and run Toil workflow. .INDENT 7.0 .TP .B static getDefaultArgumentParser(jobstore_as_flag=False) Get argument parser with added toil workflow options. .INDENT 7.0 .TP .B Parameters \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .TP .B Return type \fI\%ArgumentParser\fP .TP .B Returns The argument parser used by a toil workflow with added Toil options. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static getDefaultOptions(jobStore=None, jobstore_as_flag=False) Get default options for a toil workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- A string describing the jobStore for the workflow. .IP \(bu 2 \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .UNINDENT .TP .B Return type \fI\%Namespace\fP .TP .B Returns The options used by a toil workflow. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static addToilOptions(parser, jobstore_as_flag=False) Adds the default toil options to an \fI\%optparse\fP or \fI\%argparse\fP parser object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBparser\fP (\fI\%Union\fP[\fI\%OptionParser\fP, \fI\%ArgumentParser\fP]) \-\- Options object to add toil options to. .IP \(bu 2 \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static startToil(job, options) Run the toil workflow using the given options. .sp Deprecated by toil.common.Toil.start. .sp (see Job.Runner.getDefaultOptions and Job.Runner.addToilOptions) starting with this job. :type job: \fI\%Job\fP :param job: root job of the workflow :raises: toil.exceptions.FailedJobsException if at the end of function there remain failed jobs. :rtype: \fI\%Any\fP :return: The return value of the root job\(aqs run function. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%Any\fP .UNINDENT .UNINDENT .UNINDENT .SH JOB.FILESTORE API .sp The AbstractFileStore is an abstraction of a Toil run\(aqs shared storage. .INDENT 0.0 .TP .B class toil.fileStores.abstractFileStore.AbstractFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit) Interface used to allow user code run by Toil to read and write files. .sp Also provides the interface to other Toil facilities used by user code, including: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 normal (non\-real\-time) logging .IP \(bu 2 finding the correct temporary directory for scratch work .IP \(bu 2 importing and exporting files into and out of the workflow .UNINDENT .UNINDENT .UNINDENT .sp Stores user files in the jobStore, but keeps them separate from actual jobs. .sp May implement caching. .sp Passed as argument to the \fI\%toil.job.Job.run()\fP method. .sp Access to files is only permitted inside the context manager provided by \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.open()\fP\&. .sp Also responsible for committing completed jobs back to the job store with an update operation, and allowing that commit operation to be waited for. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- .IP \(bu 2 \fBjobDesc\fP (\fI\%JobDescription\fP) \-\- .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fI\%Callable\fP[[], \fI\%Any\fP]) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __init__(jobStore, jobDesc, file_store_dir, waitForPreviousCommit) Create a new file store object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- the job store in use for the current Toil run. .IP \(bu 2 \fBjobDesc\fP (\fI\%JobDescription\fP) \-\- the JobDescription object for the currently running job. .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- the per\-worker local temporary directory where the file store should store local files. Per\-job directories will be created under here by the file store. .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fI\%Callable\fP[[], \fI\%Any\fP]) \-\- the waitForCommit method of the previous job\(aqs file store, when jobs are running in sequence on the same worker. Used to prevent this file store\(aqs startCommit and the previous job\(aqs startCommit methods from running at the same time and racing. If they did race, it might be possible for the later job to be fully marked as completed in the job store before the eralier job was. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static createFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit, caching) Create a concreate FileStore. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%AbstractJobStore\fP) \-\- .IP \(bu 2 \fBjobDesc\fP (\fI\%JobDescription\fP) \-\- .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fI\%Callable\fP[[], \fI\%Any\fP]) \-\- .IP \(bu 2 \fBcaching\fP (\fI\%Optional\fP[\fI\%bool\fP]) \-\- .UNINDENT .TP .B Return type \fI\%Union\fP[\fI\%NonCachingFileStore\fP, \fI\%CachingFileStore\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static shutdownFileStore(workflowID, config_work_dir, config_coordination_dir) Carry out any necessary filestore\-specific cleanup. .sp This is a destructive operation and it is important to ensure that there are no other running processes on the system that are modifying or using the file store for this workflow. .sp This is the intended to be the last call to the file store in a Toil run, called by the batch system cleanup function upon batch system shutdown. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflowID\fP (\fI\%str\fP) \-\- The workflow ID for this invocation of the workflow .IP \(bu 2 \fBconfig_work_dir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- The path to the work directory in the Toil Config. .IP \(bu 2 \fBconfig_coordination_dir\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- The path to the coordination directory in the Toil Config. .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B open(job) Create the context manager around tasks prior and after a job has been run. .sp File operations are only permitted inside the context manager. .sp Implementations must only yield from within \fIwith super().open(job):\fP\&. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- The job instance of the toil job to run. .TP .B Return type \fI\%Generator\fP[\fI\%None\fP, \fI\%None\fP, \fI\%None\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_disk_usage() Get the number of bytes of disk used by the last job run under open(). .sp Disk usage is measured at the end of the job. TODO: Sample periodically and record peak usage. .INDENT 7.0 .TP .B Return type \fI\%Optional\fP[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLocalTempDir() Get a new local temporary directory in which to write files. .sp The directory will only persist for the duration of the job. .INDENT 7.0 .TP .B Return type \fI\%str\fP .TP .B Returns The absolute path to a new local temporary directory. This directory will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates, removing all files it contains recursively. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLocalTempFile(suffix=None, prefix=None) Get a new local temporary file that will persist for the duration of the job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- If not None, the file name will end with this string. Otherwise, default value \(dq.tmp\(dq will be used .IP \(bu 2 \fBprefix\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- If not None, the file name will start with this string. Otherwise, default value \(dqtmp\(dq will be used .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns The absolute path to a local temporary file. This file will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLocalTempFileName(suffix=None, prefix=None) Get a valid name for a new local file. Don\(aqt actually create a file at the path. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- If not None, the file name will end with this string. Otherwise, default value \(dq.tmp\(dq will be used .IP \(bu 2 \fBprefix\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- If not None, the file name will start with this string. Otherwise, default value \(dqtmp\(dq will be used .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns Path to valid file .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract writeGlobalFile(localFileName, cleanup=False) Upload a file (as a path) to the job store. .sp If the file is in a FileStore\-managed temporary directory (i.e. from \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()\fP), it will become a local copy of the file, eligible for deletion by \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile()\fP\&. .sp If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded again. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocalFileName\fP (\fI\%str\fP) \-\- The path to the local file to upload. The last path component (basename of the file) will remain associated with the file in the file store, if supported by the backing JobStore, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- if True then the copy of the global file will be deleted once the job and all its successors have completed running. If not the global file must be deleted manually. .UNINDENT .TP .B Return type \fI\%FileID\fP .TP .B Returns an ID that can be used to retrieve the file. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeGlobalFileStream(cleanup=False, basename=None, encoding=None, errors=None) Similar to writeGlobalFile, but allows the writing of a stream to the job store. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- The name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- Specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- is as in \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile()\fP\&. .IP \(bu 2 \fBbasename\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- If supported by the backing JobStore, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .UNINDENT .TP .B Return type \fI\%Iterator\fP[\fI\%Tuple\fP[\fI\%WriteWatchingStream\fP, \fI\%FileID\fP]] .TP .B Returns A context manager yielding a tuple of 1) a file handle which can be written to and 2) the toil.fileStores.FileID of the resulting file in the job store. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logAccess(fileStoreID, destination=None) Record that the given file was read by the job. .sp (to be announced if the job fails) .sp If destination is not None, it gives the path that the file was downloaded to. Otherwise, assumes that the file was streamed. .sp Must be called by \fI\%readGlobalFile()\fP and \fI\%readGlobalFileStream()\fP implementations. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%Union\fP[\fI\%FileID\fP, \fI\%str\fP]) \-\- .IP \(bu 2 \fBdestination\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False) Make the file associated with fileStoreID available locally. .sp If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem. .sp If a user path is specified, it is used as the destination. If a user path isn\(aqt specified, the file is stored in the local temp directory with an encoded name. .sp The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile. .sp Implementations must call \fI\%logAccess()\fP to report the download. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- job store id for the file .IP \(bu 2 \fBuserPath\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- a path to the name of file to which the global file will be copied or hard\-linked (see below). .IP \(bu 2 \fBcache\fP (\fI\%bool\fP) \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBmutable\fP (\fI\%bool\fP) \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- True if caller can accept symlink, False if caller can only accept a normal file or hardlink .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns An absolute path to a local, temporary copy of the file keyed by fileStoreID. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract readGlobalFileStream(fileStoreID, encoding=None, errors=None) Read a stream from the job store; similar to readGlobalFile. .sp The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBencoding\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%ContextManager\fP[\fI\%IO\fP[\fI\%bytes\fP] | \fI\%IO\fP[\fI\%str\fP]] .UNINDENT .sp Implementations must call \fI\%logAccess()\fP to report the download. .INDENT 7.0 .TP .B Return type \fI\%ContextManager\fP[\fI\%Union\fP[\fI\%IO\fP[\fI\%bytes\fP], \fI\%IO\fP[\fI\%str\fP]]] .TP .B Returns a context manager yielding a file handle which can be read from. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fI\%str\fP\fI | \fP\fINone\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fI\%str\fP\fI | \fP\fINone\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getGlobalFileSize(fileStoreID) Get the size of the file pointed to by the given ID, in bytes. .sp If a FileID or something else with a non\-None \(aqsize\(aq field, gets that. .sp Otherwise, asks the job store to poll the file\(aqs size. .sp Note that the job store may overestimate the file\(aqs size, for example if it is encrypted and had to be augmented with an IV or other encryption framing. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fI\%Union\fP[\fI\%FileID\fP, \fI\%str\fP]) \-\- File ID for the file .TP .B Return type \fI\%int\fP .TP .B Returns File\(aqs size in bytes, as stored in the job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract deleteLocalFile(fileStoreID) Delete local copies of files associated with the provided job store ID. .sp Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession. .sp The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job\(aqs file\-store\-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job\(aqs file\-store\-provided temp directory. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fI\%Union\fP[\fI\%FileID\fP, \fI\%str\fP]) \-\- File Store ID of the file to be deleted. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract deleteGlobalFile(fileStoreID) Delete local files and then permanently deletes them from the job store. .sp To ensure that the job can be restarted if necessary, the delete will not happen until after the job\(aqs run method has completed. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fI\%Union\fP[\fI\%FileID\fP, \fI\%str\fP]) \-\- the File Store ID of the file to be deleted. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log_to_leader(text, level=20) Send a logging message to the leader. The message will also be logged by the worker at the same level. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtext\fP (\fI\%str\fP) \-\- The string to log. .IP \(bu 2 \fBlevel\fP (\fI\%int\fP) \-\- The logging level. .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log_user_stream(name, stream) Send a stream of UTF\-8 text to the leader as a named log stream. .sp Useful for things like the error logs of Docker containers. The leader will show it to the user or organize it appropriately for user\-level log information. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- A hierarchical, .\-delimited string. .IP \(bu 2 \fBstream\fP (\fI\%IO\fP[\fI\%bytes\fP]) \-\- A stream of encoded text. Encoding errors will be tolerated. .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract startCommit(jobState=False) Update the status of the job on the disk. .sp May bump the version number of the job. .sp May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call. .INDENT 7.0 .TP .B Parameters \fBjobState\fP (\fI\%bool\fP) \-\- If True, commit the state of the FileStore\(aqs job, and file deletes. Otherwise, commit only file creates/updates. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract waitForCommit() Blocks while startCommit is running. .sp This function is called by this job\(aqs successor to ensure that it does not begin modifying the job store until after this job has finished doing so. .sp Might be called when startCommit is never called on a particular instance, in which case it does not block. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Returns Always returns True .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract classmethod shutdown(shutdown_info) Shutdown the filestore on this node. .sp This is intended to be called on batch system shutdown. .INDENT 7.0 .TP .B Parameters \fBshutdown_info\fP (\fI\%Any\fP) \-\- The implementation\-specific shutdown information, for shutting down the file store and removing all its state and all job local temp directories from the node. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.fileStores.FileID(fileStoreID, size, executable=False) A small wrapper around Python\(aqs builtin string class. .sp It is used to represent a file\(aqs ID in the file store, and has a size attribute that is the file\(aqs size in bytes. This object is returned by importFile and writeGlobalFile. .sp Calls into the file store can use bare strings; size will be queried from the job store if unavailable in the ID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsize\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBexecutable\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBargs\fP (\fI\%Any\fP) \-\- .UNINDENT .TP .B Return type \fI\%FileID\fP .UNINDENT .INDENT 7.0 .TP .B __init__(fileStoreID, size, executable=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsize\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBexecutable\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pack() Pack the FileID into a string so it can be passed through external code. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod unpack(packedFileStoreID) Unpack the result of pack() into a FileID object. .INDENT 7.0 .TP .B Parameters \fBpackedFileStoreID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%FileID\fP .UNINDENT .UNINDENT .UNINDENT .SH BATCH SYSTEM API .sp The batch system interface is used by Toil to abstract over different ways of running batches of jobs, for example on Slurm clusters, Kubernetes clusters, or a single node. The \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP API is implemented to run jobs using a given job management system. .SS Batch System Environment Variables .sp Environmental variables allow passing of scheduler specific parameters. .sp For SLURM there are two environment variables \- the first applies to all jobs, while the second defined the partition to use for parallel jobs: .INDENT 0.0 .INDENT 3.5 .sp .EX export TOIL_SLURM_ARGS=\(dq\-t 1:00:00 \-q fatq\(dq export TOIL_SLURM_PE=\(aqmulticore\(aq .EE .UNINDENT .UNINDENT .sp For TORQUE there are two environment variables \- one for everything but the resource requirements, and another \- for resources requirements (without the \fI\-l\fP prefix): .INDENT 0.0 .INDENT 3.5 .sp .EX export TOIL_TORQUE_ARGS=\(dq\-q fatq\(dq export TOIL_TORQUE_REQS=\(dqwalltime=1:00:00\(dq .EE .UNINDENT .UNINDENT .sp For GridEngine (SGE, UGE), there is an additional environmental variable to define the \fI\%parallel environment\fP for running multicore jobs: .INDENT 0.0 .INDENT 3.5 .sp .EX export TOIL_GRIDENGINE_PE=\(aqsmp\(aq export TOIL_GRIDENGINE_ARGS=\(aq\-q batch.q\(aq .EE .UNINDENT .UNINDENT .sp For HTCondor, additional parameters can be included in the submit file passed to condor_submit: .INDENT 0.0 .INDENT 3.5 .sp .EX export TOIL_HTCONDOR_PARAMS=\(aqrequirements = TARGET.has_sse4_2 == true; accounting_group = test\(aq .EE .UNINDENT .UNINDENT .sp The environment variable is parsed as a semicolon\-separated string of \fBparameter = value\fP pairs. .SS Batch System API .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem An abstract base class to represent the interface the batch system must provide to Toil. .INDENT 7.0 .TP .B abstract classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fI\%setUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract classmethod supportsWorkerCleanup() Whether this batch system supports worker cleanup. .sp Indicates whether this batch system invokes \fI\%BatchSystemSupport.workerCleanup()\fP after the last job for a particular workflow invocation finishes. Note that the term \fIworker\fP refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to \fIshut down\fP after the last worker process terminates. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setUserScript(userScript) Set the user script for this workflow. .sp This method must be called before the first job is issued to this batch system, and only if \fI\%supportsAutoDeployment()\fP returns True, otherwise it will raise an exception. .INDENT 7.0 .TP .B Parameters \fBuserScript\fP (\fI\%Resource\fP) \-\- the resource object representing the user script or module and the modules it depends on. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_message_bus(message_bus) Give the batch system an opportunity to connect directly to the message bus, so that it can send informational messages about the jobs it is running to other Toil components. .INDENT 7.0 .TP .B Parameters \fBmessage_bus\fP (\fI\%MessageBus\fP) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract issueBatchJob(jobDesc, job_environment=None) Issues a job with the specified command to the batch system and returns a unique jobID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP (\fI\%JobDescription\fP) \-\- a toil.job.JobDescription .IP \(bu 2 \fBjob_environment\fP (\fI\%Optional\fP[\fI\%Dict\fP[\fI\%str\fP, \fI\%str\fP]]) \-\- a collection of job\-specific environment variables to be set on the worker. .UNINDENT .TP .B Return type \fI\%int\fP .TP .B Returns a unique jobID that can be used to reference the newly issued job .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract killBatchJobs(jobIDs) Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob. .INDENT 7.0 .TP .B Parameters \fBjobIDs\fP (\fI\%List\fP[\fI\%int\fP]) \-\- list of IDs of jobs to kill .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getIssuedBatchJobIDs() Gets all currently issued jobs .INDENT 7.0 .TP .B Return type \fI\%List\fP[\fI\%int\fP] .TP .B Returns A list of jobs (as jobIDs) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getRunningBatchJobIDs() Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds. .INDENT 7.0 .TP .B Return type \fI\%Dict\fP[\fI\%int\fP, \fI\%float\fP] .TP .B Returns dictionary with currently running jobID keys and how many seconds they have been running as the value .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getUpdatedBatchJob(maxWait) Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once. .sp Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%int\fP) \-\- the number of seconds to block, waiting for a result .TP .B Return type \fI\%Optional\fP[\fI\%UpdatedBatchJobInfo\fP] .TP .B Returns If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall\-clock time the job ran for, or None if this batch system does not support tracking wall time. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getSchedulingStatusMessage() Get a log message fragment for the user about anything that might be going wrong in the batch system, if available. .sp If no useful message is available, return None. .sp This can be used to report what resource is the limiting factor when scheduling jobs, for example. If the leader thinks the workflow is stuck, the message can be displayed to the user to help them diagnose why it might be stuck. .INDENT 7.0 .TP .B Return type \fI\%Optional\fP[\fI\%str\fP] .TP .B Returns User\-directed message about scheduling state. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract shutdown() Called at the completion of a toil invocation. Should cleanly terminate all worker threads. .INDENT 7.0 .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setEnv(name, value=None) Set an environment variable for the worker process before it is launched. .sp The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job. .sp If no value is provided it will be looked up from the current environment. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .UNINDENT .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fI\%Union\fP[\fI\%ArgumentParser\fP, \fB_ArgumentGroup\fP]) \-\- .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getWorkerContexts() Get a list of picklable context manager objects to wrap worker work in, in order. .sp Can be used to ask the Toil worker to do things in\-process (such as configuring environment variables, hot\-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping \(dqexecutor\(dq process. .INDENT 7.0 .TP .B Return type \fI\%List\fP[\fI\%ContextManager\fP[\fI\%Any\fP]] .UNINDENT .UNINDENT .UNINDENT .SH JOB.SERVICE API .sp The Service class allows databases and servers to be spawned within a Toil workflow. .INDENT 0.0 .TP .B class Job.Service(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName=None) Abstract class used to define the interface to a service. .sp Should be subclassed by the user to define services. .sp Is not executed as a job; runs within a ServiceHostJob. .INDENT 7.0 .TP .B __init__(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName=None) Memory, core and disk requirements are specified identically to as in \fBtoil.job.Job.__init__()\fP\&. .UNINDENT .INDENT 7.0 .TP .B abstract start(job) Start the service. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Return type \fI\%Any\fP .TP .B Returns An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see \fI\%toil.job.Job.addService()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract stop(job) Stops the service. Function can block until complete. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Return type \fI\%None\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Checks the service is still running. .INDENT 7.0 .TP .B Raises \fBexceptions.RuntimeError\fP \-\- If the service failed, this will cause the service job to be labeled failed. .TP .B Return type \fI\%bool\fP .TP .B Returns True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! .UNINDENT .UNINDENT .UNINDENT .SH EXCEPTIONS API .sp Toil specific exceptions. .INDENT 0.0 .TP .B exception toil.job.JobException(message) General job exception. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .INDENT 7.0 .TP .B __init__(message) .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.job.JobGraphDeadlockException(string) An exception raised in the event that a workflow contains an unresolvable dependency, such as a cycle. See \fI\%toil.job.Job.checkJobGraphForDeadlocks()\fP\&. .INDENT 7.0 .TP .B __init__(string) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.ConcurrentFileModificationException(jobStoreFileID) Indicates that the file was attempted to be modified by multiple processes at once. .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%FileID\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __init__(jobStoreFileID) .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%FileID\fP) \-\- the ID of the file that was modified by multiple workers or processes concurrently .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.JobStoreExistsException(locator) Indicates that the specified job store already exists. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __init__(locator) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocator\fP (\fI\%str\fP) \-\- The location of the job store .IP \(bu 2 \fBlocator\fP \-\- .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.NoSuchFileException(jobStoreFileID, customName=None, *extra) Indicates that the specified file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%FileID\fP) \-\- .IP \(bu 2 \fBcustomName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- .IP \(bu 2 \fBextra\fP (\fI\%Any\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __init__(jobStoreFileID, customName=None, *extra) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%FileID\fP) \-\- the ID of the file that was mistakenly assumed to exist .IP \(bu 2 \fBcustomName\fP (\fI\%Optional\fP[\fI\%str\fP]) \-\- optionally, an alternate name for the nonexistent file .IP \(bu 2 \fBextra\fP (\fI\%Any\fP) \-\- optional extra information to add to the error message .IP \(bu 2 \fBextra\fP \-\- .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.NoSuchJobException(jobStoreID) Indicates that the specified job does not exist. .INDENT 7.0 .TP .B Parameters \fBjobStoreID\fP (\fI\%FileID\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __init__(jobStoreID) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreID\fP (\fI\%FileID\fP) \-\- the jobStoreID that was mistakenly assumed to exist .IP \(bu 2 \fBjobStoreID\fP \-\- .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.NoSuchJobStoreException(locator) Indicates that the specified job store does not exist. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __init__(locator) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocator\fP (\fI\%str\fP) \-\- The location of the job store .IP \(bu 2 \fBlocator\fP \-\- .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SH RUNNING TESTS .sp Test make targets, invoked as \fB$ make \fP, subject to which environment variables are set (see \fI\%Running Integration Tests\fP). .TS center; |l|l|. _ T{ TARGET T} T{ DESCRIPTION T} _ T{ test T} T{ Invokes all tests. T} _ T{ integration_test T} T{ Invokes only the integration tests. T} _ T{ test_offline T} T{ Skips building the Docker appliance and only invokes tests that have no docker dependencies. T} _ T{ integration_test_local T} T{ Makes integration tests easier to debug locally by running the integration tests serially and doesn\(aqt redirect output. This makes it appears on the terminal as expected. T} _ .TE .sp Before running tests for the first time, initialize your virtual environment following the steps in \fI\%Building from Source\fP\&. .sp Run all tests (including slow tests): .INDENT 0.0 .INDENT 3.5 .sp .EX $ make test .EE .UNINDENT .UNINDENT .sp Run only quick tests (as of Jul 25, 2018, this was ~ 20 minutes): .INDENT 0.0 .INDENT 3.5 .sp .EX $ export TOIL_TEST_QUICK=True; make test .EE .UNINDENT .UNINDENT .sp Run an individual test with: .INDENT 0.0 .INDENT 3.5 .sp .EX $ make test tests=src/toil/test/sort/sortTest.py::SortTest::testSort .EE .UNINDENT .UNINDENT .sp The default value for \fBtests\fP is \fB\(dqsrc\(dq\fP which includes all tests in the \fBsrc/\fP subdirectory of the project root. Tests that require a particular feature will be skipped implicitly. If you want to explicitly skip tests that depend on a currently installed \fIfeature\fP, use .INDENT 0.0 .INDENT 3.5 .sp .EX $ make test tests=\(dq\-m \(aqnot aws\(aq src\(dq .EE .UNINDENT .UNINDENT .sp This will run only the tests that don\(aqt depend on the \fBaws\fP extra, even if that extra is currently installed. Note the distinction between the terms \fIfeature\fP and \fIextra\fP\&. Every extra is a feature but there are features that are not extras, such as the \fBgridengine\fP feature. To skip tests involving both the \fBgridengine\fP feature and the \fBaws\fP extra, use the following: .INDENT 0.0 .INDENT 3.5 .sp .EX $ make test tests=\(dq\-m \(aqnot aws and not gridengine\(aq src\(dq .EE .UNINDENT .UNINDENT .SS Running Tests with pytest .sp Often it is simpler to use pytest directly, instead of calling the \fBmake\fP wrapper. This usually works as expected, but some tests need some manual preparation. To run a specific test with pytest, use the following: .INDENT 0.0 .INDENT 3.5 .sp .EX python3 \-m pytest src/toil/test/sort/sortTest.py::SortTest::testSort .EE .UNINDENT .UNINDENT .sp For more information, see the \fI\%pytest documentation\fP\&. .SS Running Integration Tests .sp These tests are generally only run using in our CI workflow due to their resource requirements and cost. However, they can be made available for local testing: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 Running tests that make use of Docker (e.g. autoscaling tests and Docker tests) require an appliance image to be hosted. First, make sure you have gone through the set up found in \fI\%Using Docker with Quay\fP\&. Then to build and host the appliance image run the \fBmake\fP target \fBpush_docker\fP\&. .INDENT 2.0 .INDENT 3.5 .sp .EX $ make push_docker .EE .UNINDENT .UNINDENT .IP \(bu 2 Running integration tests require activation via an environment variable as well as exporting information relevant to the desired tests. Enable the integration tests: .INDENT 2.0 .INDENT 3.5 .sp .EX $ export TOIL_TEST_INTEGRATIVE=True .EE .UNINDENT .UNINDENT .IP \(bu 2 Finally, set the environment variables for keyname and desired zone: .INDENT 2.0 .INDENT 3.5 .sp .EX $ export TOIL_X_KEYNAME=[Your Keyname] $ export TOIL_X_ZONE=[Desired Zone] .EE .UNINDENT .UNINDENT .sp Where \fBX\fP is one of our currently supported cloud providers (\fBGCE\fP, \fBAWS\fP). .IP \(bu 2 See the above sections for guidance on running tests. .UNINDENT .UNINDENT .UNINDENT .SS Test Environment Variables .TS center; |l|l|. _ T{ TOIL_TEST_TEMP T} T{ An absolute path to a directory where Toil tests will write their temporary files. Defaults to the system\(aqs \fI\%standard temporary directory\fP\&. T} _ T{ TOIL_TEST_INTEGRATIVE T} T{ If \fBTrue\fP, this allows the integration tests to run. Only valid when running the tests from the source directory via \fBmake test\fP or \fBmake test_parallel\fP\&. T} _ T{ TOIL_AWS_KEYNAME T} T{ An AWS keyname (see \fI\%Preparing your AWS environment\fP), which is required to run the AWS tests. T} _ T{ TOIL_GOOGLE_PROJECTID T} T{ A Google Cloud account projectID (see \fI\%Running in Google Compute Engine (GCE)\fP), which is required to to run the Google Cloud tests. T} _ T{ TOIL_TEST_QUICK T} T{ If \fBTrue\fP, long running tests are skipped. T} _ .TE .INDENT 0.0 .INDENT 3.5 .IP "Partial install and failing tests" .sp Some tests may fail with an ImportError if the required extras are not installed. Install Toil with all of the extras do prevent such errors. .UNINDENT .UNINDENT .SS Using Docker with Quay .sp \fI\%Docker\fP is needed for some of the tests. Follow the appropriate installation instructions for your system on their website to get started. .sp When running \fBmake test\fP you might still get the following error: .INDENT 0.0 .INDENT 3.5 .sp .EX $ make test Please set TOIL_DOCKER_REGISTRY, e.g. to quay.io/USER. .EE .UNINDENT .UNINDENT .sp To solve, make an account with \fI\%Quay\fP and specify it like so: .INDENT 0.0 .INDENT 3.5 .sp .EX $ TOIL_DOCKER_REGISTRY=quay.io/USER make test .EE .UNINDENT .UNINDENT .sp where \fBUSER\fP is your Quay username. .sp For convenience you may want to add this variable to your bashrc by running .INDENT 0.0 .INDENT 3.5 .sp .EX $ echo \(aqexport TOIL_DOCKER_REGISTRY=quay.io/USER\(aq >> $HOME/.bashrc .EE .UNINDENT .UNINDENT .SS Running Mesos Tests .sp If you\(aqre running Toil\(aqs Mesos tests, be sure to create the virtualenv with \fB\-\-system\-site\-packages\fP to include the Mesos Python bindings. Verify this by activating the virtualenv and running \fBpip list | grep mesos\fP\&. On macOS, this may come up empty. To fix it, run the following: .INDENT 0.0 .INDENT 3.5 .sp .EX for i in /usr/local/lib/python2.7/site\-packages/*mesos*; do ln \-snf $i venv/lib/python2.7/site\-packages/; done .EE .UNINDENT .UNINDENT .SH DEVELOPING WITH DOCKER .sp To develop on features reliant on the Toil Appliance (the docker image toil uses for AWS autoscaling), you should consider setting up a personal registry on \fI\%Quay\fP or \fI\%Docker Hub\fP\&. Because the Toil Appliance images are tagged with the Git commit they are based on and because only commits on our master branch trigger an appliance build on Quay, as soon as a developer makes a commit or dirties the working copy they will no longer be able to rely on Toil to automatically detect the proper Toil Appliance image. Instead, developers wishing to test any appliance changes in autoscaling should build and push their own appliance image to a personal Docker registry. This is described in the next section. .SS Making Your Own Toil Docker Image .sp \fBNote!\fP Toil checks if the docker image specified by TOIL_APPLIANCE_SELF exists prior to launching by using the docker v2 schema. This should be valid for any major docker repository, but there is an option to override this if desired using the option: \fB\-\-forceDockerAppliance\fP\&. .sp Here is a general workflow (similar instructions apply when using Docker Hub): .INDENT 0.0 .IP 1. 3 Make some changes to the provisioner of your local version of Toil .IP 2. 3 Go to the location where you installed the Toil source code and run .INDENT 3.0 .INDENT 3.5 .sp .EX $ make docker .EE .UNINDENT .UNINDENT .sp to automatically build a docker image that can now be uploaded to your personal \fI\%Quay\fP account. If you have not installed Toil source code yet see \fI\%Building from Source\fP\&. .IP 3. 3 If it\(aqs not already you will need Docker installed and need to \fI\%log into Quay\fP\&. Also you will want to make sure that your Quay account is public. .IP 4. 3 Set the environment variable \fBTOIL_DOCKER_REGISTRY\fP to your Quay account. If you find yourself doing this often you may want to add .INDENT 3.0 .INDENT 3.5 .sp .EX export TOIL_DOCKER_REGISTRY=quay.io/ .EE .UNINDENT .UNINDENT .sp to your \fB\&.bashrc\fP or equivalent. .IP 5. 3 Now you can run .INDENT 3.0 .INDENT 3.5 .sp .EX $ make push_docker .EE .UNINDENT .UNINDENT .sp which will upload the docker image to your Quay account. Take note of the image\(aqs tag for the next step. .IP 6. 3 Finally you will need to tell Toil from where to pull the Appliance image you\(aqve created (it uses the Toil release you have installed by default). To do this set the environment variable \fBTOIL_APPLIANCE_SELF\fP to the url of your image. For more info see \fI\%Environment Variables\fP\&. .IP 7. 3 Now you can launch your cluster! For more information see \fI\%Running a Workflow with Autoscaling\fP\&. .UNINDENT .SS Running a Cluster Locally .sp The Toil Appliance container can also be useful as a test environment since it can simulate a Toil cluster locally. An important caveat for this is autoscaling, since autoscaling will only work on an EC2 instance and cannot (at this time) be run on a local machine. .sp To spin up a local cluster, start by using the following Docker run command to launch a Toil leader container: .INDENT 0.0 .INDENT 3.5 .sp .EX docker run \e \-\-entrypoint=mesos\-master \e \-\-net=host \e \-d \e \-\-name=leader \e \-\-volume=/home/jobStoreParentDir:/jobStoreParentDir \e quay.io/ucsc_cgl/toil:3.6.0 \e \-\-registry=in_memory \e \-\-ip=127.0.0.1 \e \-\-port=5050 \e \-\-allocation_interval=500ms .EE .UNINDENT .UNINDENT .sp A couple notes on this command: the \fB\-d\fP flag tells Docker to run in daemon mode so the container will run in the background. To verify that the container is running you can run \fBdocker ps\fP to see all containers. If you want to run your own container rather than the official UCSC container you can simply replace the \fBquay.io/ucsc_cgl/toil:3.6.0\fP parameter with your own container name. .sp Also note that we are not mounting the job store directory itself, but rather the location where the job store will be written. Due to complications with running Docker on MacOS, I recommend only mounting directories within your home directory. The next command will launch the Toil worker container with similar parameters: .INDENT 0.0 .INDENT 3.5 .sp .EX docker run \e \-\-entrypoint=mesos\-slave \e \-\-net=host \e \-d \e \-\-name=worker \e \-\-volume=/home/jobStoreParentDir:/jobStoreParentDir \e quay.io/ucsc_cgl/toil:3.6.0 \e \-\-work_dir=/var/lib/mesos \e \-\-master=127.0.0.1:5050 \e \-\-ip=127.0.0.1 \e —\-attributes=preemptable:False \e \-\-resources=cpus:2 .EE .UNINDENT .UNINDENT .sp Note here that we are specifying 2 CPUs and a non\-preemptable worker. We can easily change either or both of these in a logical way. To change the number of cores we can change the 2 to whatever number you like, and to change the worker to be preemptable we change \fBpreemptable:False\fP to \fBpreemptable:True\fP\&. Also note that the same volume is mounted into the worker. This is needed since both the leader and worker write and read from the job store. Now that your cluster is running, you can run .INDENT 0.0 .INDENT 3.5 .sp .EX docker exec \-it leader bash .EE .UNINDENT .UNINDENT .sp to get a shell in your leader \(aqnode\(aq. You can also replace the \fBleader\fP parameter with \fBworker\fP to get shell access in your worker. .INDENT 0.0 .INDENT 3.5 .IP "Docker\-in\-Docker issues" .sp If you want to run Docker inside this Docker cluster (Dockerized tools, perhaps), you should also mount in the Docker socket via \fB\-v /var/run/docker.sock:/var/run/docker.sock\fP\&. This will give the Docker client inside the Toil Appliance access to the Docker engine on the host. Client/engine version mismatches have been known to cause issues, so we recommend using Docker version 1.12.3 on the host to be compatible with the Docker client installed in the Appliance. Finally, be careful where you write files inside the Toil Appliance \- \(aqchild\(aq Docker containers launched in the Appliance will actually be siblings to the Appliance since the Docker engine is located on the host. This means that the \(aqchild\(aq container can only mount in files from the Appliance if the files are located in a directory that was originally mounted into the Appliance from the host \- that way the files are accessible to the sibling container. Note: if Docker can\(aqt find the file/directory on the host it will silently fail and mount in an empty directory. .UNINDENT .UNINDENT .SH MAINTAINER'S GUIDELINES .sp In general, as developers and maintainers of the code, we adhere to the following guidelines: .INDENT 0.0 .IP \(bu 2 We strive to never break the build on master. All development should be done on branches, in either the main Toil repository or in developers\(aq forks. .IP \(bu 2 Pull requests should be used for any and all changes (except truly trivial ones). .IP \(bu 2 Pull requests should be in response to issues. If you find yourself making a pull request without an issue, you should create the issue first. .UNINDENT .SS Naming Conventions .INDENT 0.0 .IP \(bu 2 \fBCommit messages\fP \fIshould\fP be \fI\%great\fP\&. Most importantly, they \fImust\fP: .INDENT 2.0 .IP \(bu 2 Have a short subject line. If in need of more space, drop down \fBtwo\fP lines and write a body to explain what is changing and why it has to change. .IP \(bu 2 Write the subject line as a command: \fIDestroy all humans\fP, not \fIAll humans destroyed\fP\&. .IP \(bu 2 Reference the issue being fixed in a Github\-parseable format, such as \fI(resolves #1234)\fP at the end of the subject line, or \fIThis will fix #1234.\fP somewhere in the body. If no single commit on its own fixes the issue, the cross\-reference must appear in the pull request title or body instead. .UNINDENT .IP \(bu 2 \fBBranches\fP in the main Toil repository \fImust\fP start with \fBissues/\fP, followed by the issue number (or numbers, separated by a dash), followed by a short, lowercase, hyphenated description of the change. (There can be many open pull requests with their associated branches at any given point in time and this convention ensures that we can easily identify branches.) .sp Say there is an issue numbered #123 titled \fBFoo does not work\fP\&. The branch name would be \fBissues/123\-fix\-foo\fP and the title of the commit would be \fBFix foo in case of bar (resolves #123).\fP .UNINDENT .SS Pull Requests .INDENT 0.0 .IP \(bu 2 All pull requests must be reviewed by a person other than the request\(aqs author. Review the PR by following the \fI\%Reviewing Pull Requests\fP checklist. .IP \(bu 2 Modified pull requests must be re\-reviewed before merging. \fBNote that Github does not enforce this!\fP .IP \(bu 2 Merge pull requests by following the \fI\%Merging Pull Requests\fP checklist. .IP \(bu 2 When merging a pull request, make sure to update the \fI\%Draft Changelog\fP on the Github wiki, which we will use to produce the changelog for the next release. The PR template tells you to do this, so don\(aqt forget. New entries should go at the bottom. .IP \(bu 2 Pull requests will not be merged unless CI tests pass. Gitlab tests are only run on code in the main Toil repository on some branch, so it is the responsibility of the approving reviewer to make sure that pull requests from outside repositories are copied to branches in the main repository. This can be accomplished with (from a Toil clone): .INDENT 2.0 .INDENT 3.5 .sp .EX \&./contrib/admin/test\-pr theirusername their\-branch issues/123\-fix\-description\-here .EE .UNINDENT .UNINDENT .sp This must be repeated every time the PR submitter updates their PR, after checking to see that the update is not malicious. .sp If there is no issue corresponding to the PR, after which the branch can be named, the reviewer of the PR should first create the issue. .sp Developers who have push access to the main Toil repository are encouraged to make their pull requests from within the repository, to avoid this step. .IP \(bu 2 Prefer using \(dqSquash and marge\(dq when merging pull requests to master especially when the PR contains a \(dqsingle unit\(dq of work (i.e. if one were to rewrite the PR from scratch with all the fixes included, they would have one commit for the entire PR). This makes the commit history on master more readable and easier to debug in case of a breakage. .sp When squashing a PR from multiple authors, please add \fI\%Co\-authored\-by\fP to give credit to all contributing authors. .sp See \fI\%Issue #2816\fP for more details. .UNINDENT .SS Publishing a Release .sp These are the steps to take to publish a Toil release: .INDENT 0.0 .IP \(bu 2 Determine the release version \fBX.Y.Z\fP\&. This should follow \fI\%semantic versioning\fP; if user\-workflow\-breaking changes are made, \fBX\fP should be incremented, and \fBY\fP and \fBZ\fP should be zero. If non\-breaking changes are made but new functionality is added, \fBX\fP should remain the same as the last release, \fBY\fP should be incremented, and \fBZ\fP should be zero. If only patches are released, \fBX\fP and \fBY\fP should be the same as the last release and \fBZ\fP should be incremented. .IP \(bu 2 If it does not exist already, create a release branch in the Toil repo named \fBX.Y.x\fP, where \fBx\fP is a literal lower\-case \(dqx\(dq. For patch releases, find the existing branch and make sure it is up to date with the patch commits that are to be released. They may be \fI\%cherry\-picked over\fP from master. .IP \(bu 2 On the release branch, edit \fBversion_template.py\fP in the root of the repository. Find the line that looks like this (slightly different for patch releases): .INDENT 2.0 .INDENT 3.5 .sp .EX baseVersion = \(aqX.Y.0a1\(aq .EE .UNINDENT .UNINDENT .sp Make it look like this instead: .INDENT 2.0 .INDENT 3.5 .sp .EX baseVersion = \(aqX.Y.Z\(aq .EE .UNINDENT .UNINDENT .sp Commit your change to the branch. .IP \(bu 2 Tag the current state of the release branch as \fBreleases/X.Y.Z\fP\&. .IP \(bu 2 Make the Github release \fI\%here\fP, referencing that tag. For a non\-patch release, fill in the description with the changelog from \fI\%the wiki page\fP, which you should clear. For a patch release, just describe the patch. .IP \(bu 2 For a non\-patch release, set up the main branch so that development builds will declare themselves to be alpha versions of what the next release will probably be. Edit \fBversion_template.py\fP in the root of the repository on the main branch to set \fBbaseVersion\fP like this: .INDENT 2.0 .INDENT 3.5 .sp .EX baseVersion = \(aqX.Y+1.0a1\(aq .EE .UNINDENT .UNINDENT .sp Make sure to replace \fBX\fP and \fBY+1\fP with actual numbers. .UNINDENT .SS Using Git Hooks .sp In the \fBcontrib/hooks\fP directory, there are two scripts, \fBmypy\-after\-commit.py\fP and \fBmypy\-before\-push.py\fP, that can be set up as Git hooks to make sure you don\(aqt accidentally push commits that would immediately fail type\-checking. These are supposed to eliminate the need to run \fBmake mypy\fP constantly. You can install them into your Git working copy like this .INDENT 0.0 .INDENT 3.5 .sp .EX ln \-rs ./contrib/hooks/mypy\-after\-commit.py .git/hooks/post\-commit ln \-rs ./contrib/hooks/mypy\-before\-push.py .git/hooks/pre\-push .EE .UNINDENT .UNINDENT .sp After you make a commit, the post\-commit script will start type\-checking it, and if it takes too long re\-launch the process in the background. When you push, the pre\-push script will see if the commit you are pushing type\-checked successfully, and if it hasn\(aqt been type\-checked but is currently checked out, it will be type\-checked. If type\-checking fails, the push will be aborted. .sp Type\-checking will only be performed if you are in a Toil development virtual environment. If you aren\(aqt, the scripts won\(aqt do anything. .sp To bypass or override pre\-push hook, if it is wrong or if you need to push something that doesn\(aqt typecheck, you can \fBgit push \-\-no\-verify\fP\&. If the scripts get confused about whether a commit actually typechecks, you can clear out the type\-checking result cache, which is in \fB/var/run/user//.mypy_toil_result_cache\fP on Linux and in \fB\&.mypy_toil_result_cache\fP in the Toil repo on Mac. .sp To uninstall the scripts, delete \fB\&.git/hooks/post\-commit\fP and \fB\&.git/hooks/pre\-push\fP\&. .SS Adding Retries to a Function .sp See \fI\%toil.lib.retry\fP . .sp retry() can be used to decorate any function based on the list of errors one wishes to retry on. .sp This list of errors can contain normal Exception objects, and/or RetryCondition objects wrapping Exceptions to include additional conditions. .sp For example, retrying on a one Exception (HTTPError): .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[HTTPError]) def update_my_wallpaper(): return get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp Or: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[HTTPError, ValueError]) def update_my_wallpaper(): return get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp The examples above will retry for the default interval on any errors specified the \(dqerrors=\(dq arg list. .sp To retry on specifically 500/502/503/504 errors, you could specify an ErrorCondition object instead, for example: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[ ErrorCondition( error=HTTPError, error_codes=[500, 502, 503, 504] )]) def update_my_wallpaper(): return requests.get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp To retry on specifically errors containing the phrase \(dqNotFound\(dq: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[ ErrorCondition( error=HTTPError, error_message_must_include=\(dqNotFound\(dq )]) def update_my_wallpaper(): return requests.get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp To retry on all HTTPError errors EXCEPT an HTTPError containing the phrase \(dqNotFound\(dq: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[ HTTPError, ErrorCondition( error=HTTPError, error_message_must_include=\(dqNotFound\(dq, retry_on_this_condition=False )]) def update_my_wallpaper(): return requests.get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp To retry on boto3\(aqs specific status errors, an example of the implementation is: .INDENT 0.0 .INDENT 3.5 .sp .EX import boto3 from botocore.exceptions import ClientError @retry(errors=[ ErrorCondition( error=ClientError, boto_error_codes=[\(dqBucketNotFound\(dq] )]) def boto_bucket(bucket_name): boto_session = boto3.session.Session() s3_resource = boto_session.resource(\(aqs3\(aq) return s3_resource.Bucket(bucket_name) .EE .UNINDENT .UNINDENT .sp Any combination of these will also work, provided the codes are matched to the correct exceptions. A ValueError will not return a 404, for example. .sp The retry function as a decorator should make retrying functions easier and clearer. It also encourages smaller independent functions, as opposed to lumping many different things that may need to be retried on different conditions in the same function. .sp The ErrorCondition object tries to take some of the heavy lifting of writing specific retry conditions and boil it down to an API that covers all common use\-cases without the user having to write any new bespoke functions. .sp Use\-cases covered currently: .INDENT 0.0 .IP 1. 3 Retrying on a normal error, like a KeyError. .IP 2. 3 Retrying on HTTP error codes (use ErrorCondition). .IP 3. 3 Retrying on boto\(aqs specific status errors, like \(dqBucketNotFound\(dq (use ErrorCondition). .IP 4. 3 Retrying when an error message contains a certain phrase (use ErrorCondition). .IP 5. 3 Explicitly NOT retrying on a condition (use ErrorCondition). .UNINDENT .sp If new functionality is needed, it\(aqs currently best practice in Toil to add functionality to the ErrorCondition itself rather than making a new custom retry method. .SH PULL REQUEST CHECKLISTS .sp This document contains checklists for dealing with PRs. More general PR information is available at \fI\%Pull Requests\fP\&. .SS Reviewing Pull Requests .sp This checklist is to be kept in sync with the checklist in the pull request template. .sp When reviewing a PR, do the following: .INDENT 0.0 .IP \(bu 2 .INDENT 2.0 .TP .B Make sure it is coming from \fBissues/XXXX\-fix\-the\-thing\fP in the Toil repo, or from an external repo. .INDENT 7.0 .IP \(bu 2 If it is coming from an external repo, make sure to pull it in for CI with: .INDENT 2.0 .INDENT 3.5 .sp .EX contrib/admin/test\-pr otheruser theirbranchname issues/XXXX\-fix\-the\-thing .EE .UNINDENT .UNINDENT .IP \(bu 2 If there is no associated issue, \fI\%create one\fP\&. .UNINDENT .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B Read through the code changes. Make sure that it doesn\(aqt have: .INDENT 7.0 .IP \(bu 2 Addition of trailing whitespace. .IP \(bu 2 New variable or member names in \fBcamelCase\fP that want to be in \fBsnake_case\fP\&. .IP \(bu 2 New functions without \fI\%type hints\fP\&. .IP \(bu 2 New functions or classes without informative docstrings. .IP \(bu 2 Changes to semantics not reflected in the relevant docstrings. .IP \(bu 2 New or changed command line options for Toil workflows that are not reflected in \fBdocs/running/cliOptions.rst\fP .IP \(bu 2 New features without tests. .UNINDENT .UNINDENT .IP \(bu 2 Comment on the lines of code where problems exist with a review comment. You can shift\-click the line numbers in the diff to select multiple lines. .IP \(bu 2 Finish the review with an overall description of your opinion. .UNINDENT .SS Merging Pull Requests .sp This checklist is to be kept in sync with the checklist in the pull request template. .sp When merging a PR, do the following: .INDENT 0.0 .IP \(bu 2 Make sure the PR passes tests. .IP \(bu 2 Make sure the PR has been reviewed \fBsince its last modification\fP\&. If not, review it. .IP \(bu 2 .INDENT 2.0 .TP .B Merge with the Github \(dqSquash and merge\(dq feature. .INDENT 7.0 .IP \(bu 2 .INDENT 2.0 .TP .B If there are multiple authors\(aq commits, add \fI\%Co\-authored\-by\fP to give credit to all contributing authors. .UNINDENT .UNINDENT .UNINDENT .IP \(bu 2 Copy its recommended changelog entry to the \fI\%Draft Changelog\fP\&. .IP \(bu 2 Append the issue number in parentheses to the changelog entry. .UNINDENT .SH TOIL ARCHITECTURE .sp The following diagram layouts out the software architecture of Toil. .INDENT 0.0 .INDENT 2.5 [image: Toil's architecture is composed of the leader, the job store, the worker processes, the batch system, the node provisioner, and the stats and logging monitor.] [image] Figure 1: The basic components of Toil\(aqs architecture..UNINDENT .UNINDENT .INDENT 0.0 .TP .B These components are described below: .INDENT 7.0 .IP \(bu 2 .INDENT 2.0 .TP .B the leader: The leader is responsible for deciding which jobs should be run. To do this it traverses the job graph. Currently this is a single threaded process, but we make aggressive steps to prevent it becoming a bottleneck (see \fI\%Read\-only Leader\fP described below). .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B the job\-store: Handles all files shared between the components. Files in the job\-store are the means by which the state of the workflow is maintained. Each job is backed by a file in the job store, and atomic updates to this state are used to ensure the workflow can always be resumed upon failure. The job\-store can also store all user files, allowing them to be shared between jobs. The job\-store is defined by the \fI\%AbstractJobStore\fP class. Multiple implementations of this class allow Toil to support different back\-end file stores, e.g.: S3, network file systems, Google file store, etc. .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B workers: The workers are temporary processes responsible for running jobs, one at a time per worker. Each worker process is invoked with a job argument that it is responsible for running. The worker monitors this job and reports back success or failure to the leader by editing the job\(aqs state in the file\-store. If the job defines successor jobs the worker may choose to immediately run them (see \fI\%Job Chaining\fP below). .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B the batch\-system: Responsible for scheduling the jobs given to it by the leader, creating a worker command for each job. The batch\-system is defined by the \fI\%AbstractBatchSystem\fP class. Toil uses multiple existing batch systems to schedule jobs, including Apache Mesos, GridEngine and a multi\-process single node implementation that allows workflows to be run without any of these frameworks. Toil can therefore fairly easily be made to run a workflow using an existing cluster. .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B the node provisioner: Creates worker nodes in which the batch system schedules workers. It is defined by the \fI\%AbstractProvisioner\fP class. .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B the statistics and logging monitor: Monitors logging and statistics produced by the workers and reports them. Uses the job\-store to gather this information. .UNINDENT .UNINDENT .UNINDENT .SS Jobs and JobDescriptions .sp As noted in \fI\%Job Basics\fP, a job is the atomic unit of work in a Toil workflow. Workflows extend the \fI\%Job\fP class to define units of work. These jobs are pickled and stored in the job\-store by the leader, and are retrieved and un\-pickled by the worker when they are scheduled to run. .sp During scheduling, Toil does not work with the actual Job objects. Instead, \fI\%JobDescription\fP objects are used to store all the information that the Toil Leader ever needs to know about the Job. This includes requirements information, dependency information, commands to issue, etc. .sp Internally, the JobDescription object is referenced by its jobStoreID, which is often not human readable. However, the Job and JobDescription objects contain several human\-readable names that are useful for logging and identification: .TS center; |l|l|. _ T{ jobName T} T{ Name of the kind of job this is. This may be used in job store IDs and logging. Also used to let the cluster scaler learn a model for how long the job will take. Defaults to the job class\(aqs name if no real user\-defined name is available. .sp For a \fI\%FunctionWrappingJob\fP, the jobName is replaced by the wrapped function\(aqs name. .sp For a CWL workflow, the jobName is the class name of the internal job that is running the CWL workflow, such as \fB\(dqCWLJob\(dq\fP\&. T} _ T{ unitName T} T{ Name of this \fIinstance\fP of this kind of job. If set by the user, it will appear with the jobName in logging. .sp For a CWL workflow, the unitName is the dotted path from the workflow down to the task being run, including numbers for scatter steps. T} _ T{ displayName T} T{ A human\-readable name to identify this particular job instance. Used as an identifier of the job class in the stats report. Defaults to the job class\(aqs name if no real user\-defined name is available. .sp For CWL workflows, this includes the jobName and the unitName. T} _ .TE .SS Statistics and Logging .sp Toil\(aqs statistics and logging system is implemented in a joint class \fI\%StatsAndLogging\fP\&. The class can be instantiated and run as a thread on the leader, where it polls for new log files in the job store with the \fI\%read_logs()\fP method. These are JSON files, which contain structured data. Structured log messages from user Python code, stored under \fBworkers.logs_to_leader\fP, from the file store\(aqs \fI\%log_to_leader()\fP method, will be logged at the appropriate level. The text output that the worker captured for all its chained jobs, in \fBlogs.messages\fP, will be logged at debug level in the worker\(aqs output. If \fB\-\-writeLogs\fP or \fB\-\-writeLogsGzip\fP is provided, the received worker logs will also be stored by the StatsAndLogging thread into per\-job files inside the job store, using \fI\%writeLogFiles()\fP\&. .sp Note that the worker only fills this in if running with debug logging on, or if \fB\-\-writeLogsFromAllJobs\fP is set. Otherwise, logs from successful jobs are not persisted. Logs from failed jobs are persisted differently; they are written to the file store, and the log file is made available through \fI\%toil.job.JobDescription.getLogFileHandle()\fP\&. The leader thread retrieves these logs and calls back into \fI\%StatsAndLogging\fP to print or locally save them as appropriate. .sp The CWL and WDL interpreters use \fI\%log_user_stream()\fP to inject CWL and WDL task\-level logs into the stats and logging logging system. The full text of those logs gets stored in the JSON stats files, and when the StatsAndLogging thread sees them it reports and saves them, similarly to how it treats Toil job logs. .sp To ship the statistics and the non\-failed\-job logs around, the job store has a logs mailbox system: the \fI\%write_logs()\fP method deposits a string, and the \fI\%read_logs()\fP method on the leader passes the strings to a callback. It tracks a concept of new and old, based on whether the string has been read already by anyone, and one can read only the new values, or all values observed. The stats and logging system uses this to pass around structured JSON holding both log data and worker\-measured stats, and expects the \fI\%StatsAndLogging\fP thread to be the only live reader. .SS Optimizations .sp Toil implements lots of optimizations designed for scalability. Here we detail some of the key optimizations. .SS Read\-only leader .sp The leader process is currently implemented as a single thread. Most of the leader\(aqs tasks revolve around processing the state of jobs, each stored as a file within the job\-store. To minimise the load on this thread, each worker does as much work as possible to manage the state of the job it is running. As a result, with a couple of minor exceptions, the leader process never needs to write or update the state of a job within the job\-store. For example, when a job is complete and has no further successors the responsible worker deletes the job from the job\-store, marking it complete. The leader then only has to check for the existence of the file when it receives a signal from the batch\-system to know that the job is complete. This off\-loading of state management is orthogonal to future parallelization of the leader. .SS Job chaining .sp The scheduling of successor jobs is partially managed by the worker, reducing the number of individual jobs the leader needs to process. Currently this is very simple: if the there is a single next successor job to run and its resources fit within the resources of the current job and closely match the resources of the current job then the job is run immediately on the worker without returning to the leader. Further extensions of this strategy are possible, but for many workflows which define a series of serial successors (e.g. map sequencing reads, post\-process mapped reads, etc.) this pattern is very effective at reducing leader workload. .SS Preemptable node support .sp Critical to running at large\-scale is dealing with intermittent node failures. Toil is therefore designed to always be resumable providing the job\-store does not become corrupt. This robustness allows Toil to run on preemptible nodes, which are only available when others are not willing to pay more to use them. Designing workflows that divide into many short individual jobs that can use preemptable nodes allows for workflows to be efficiently scheduled and executed. .SS Caching .sp Running bioinformatic pipelines often require the passing of large datasets between jobs. Toil caches the results from jobs such that child jobs running on the same node can directly use the same file objects, thereby eliminating the need for an intermediary transfer to the job store. Caching also reduces the burden on the local disks, because multiple jobs can share a single file. The resulting drop in I/O allows pipelines to run faster, and, by the sharing of files, allows users to run more jobs in parallel by reducing overall disk requirements. .sp To demonstrate the efficiency of caching, we ran an experimental internal pipeline on 3 samples from the TCGA Lung Squamous Carcinoma (LUSC) dataset. The pipeline takes the tumor and normal exome fastqs, and the tumor rna fastq and input, and predicts MHC presented neoepitopes in the patient that are potential targets for T\-cell based immunotherapies. The pipeline was run individually on the samples on c3.8xlarge machines on AWS (60GB RAM,600GB SSD storage, 32 cores). The pipeline aligns the data to hg19\-based references, predicts MHC haplotypes using PHLAT, calls mutations using 2 callers (MuTect and RADIA) and annotates them using SnpEff, then predicts MHC:peptide binding using the IEDB suite of tools before running an in\-house rank boosting algorithm on the final calls. .sp To optimize time taken, The pipeline is written such that mutations are called on a per\-chromosome basis from the whole\-exome bams and are merged into a complete vcf. Running mutect in parallel on whole exome bams requires each mutect job to download the complete Tumor and Normal Bams to their working directories \-\- An operation that quickly fills the disk and limits the parallelizability of jobs. The workflow was run in Toil, with and without caching, and Figure 2 shows that the workflow finishes faster in the cached case while using less disk on average than the uncached run. We believe that benefits of caching arising from file transfers will be much higher on magnetic disk\-based storage systems as compared to the SSD systems we tested this on. .INDENT 0.0 .INDENT 2.5 [image: Graph outlining the efficiency gain from caching.] [image] Figure 2: Efficiency gain from caching. The lower half of each plot describes the disk used by the pipeline recorded every 10 minutes over the duration of the pipeline, and the upper half shows the corresponding stage of the pipeline that is being processed. Since jobs requesting the same file shared the same inode, the effective load on the disk is considerably lower than in the uncached case where every job downloads a personal copy of every file it needs. We see that in all cases, the uncached run uses almost 300\-400GB more that the cached run in the resource heavy mutation calling step. We also see a benefit in terms of wall time for each stage since we eliminate the time taken for file transfers..UNINDENT .UNINDENT .SS Toil support for Common Workflow Language .sp The CWL document and input document are loaded using the \(aqcwltool.load_tool\(aq module. This performs normalization and URI expansion (for example, relative file references are turned into absolute file URIs), validates the document against the CWL schema, initializes Python objects corresponding to major document elements (command line tools, workflows, workflow steps), and performs static type checking that sources and sinks have compatible types. .sp Input files referenced by the CWL document and input document are imported into the Toil file store. CWL documents may use any URI scheme supported by Toil file store, including local files and object storage. .sp The \(aqlocation\(aq field of File references are updated to reflect the import token returned by the Toil file store. .sp For directory inputs, the directory listing is stored in Directory object. Each individual files is imported into Toil file store. .sp An initial workflow Job is created from the toplevel CWL document. Then, control passes to the Toil engine which schedules the initial workflow job to run. .sp When the toplevel workflow job runs, it traverses the CWL workflow and creates a toil job for each step. The dependency graph is expressed by making downstream jobs children of upstream jobs, and initializing the child jobs with an input object containing the promises of output from upstream jobs. .sp Because Toil jobs have a single output, but CWL permits steps to have multiple output parameters that may feed into multiple other steps, the input to a CWLJob is expressed with an \(dqindirect dictionary\(dq. This is a dictionary of input parameters, where each entry value is a tuple of a promise and a promise key. When the job runs, the indirect dictionary is turned into a concrete input object by resolving each promise into its actual value (which is always a dict), and then looking up the promise key to get the actual value for the the input parameter. .sp If a workflow step specifies a scatter, then a scatter job is created and connected into the workflow graph as described above. When the scatter step runs, it creates child jobs for each parameterizations of the scatter. A gather job is added as a follow\-on to gather the outputs into arrays. .sp When running a command line tool, it first creates output and temporary directories under the Toil local temp dir. It runs the command line tool using the single_job_executor from CWLTool, providing a Toil\-specific constructor for filesystem access, and overriding the default PathMapper to use ToilPathMapper. .sp The ToilPathMapper keeps track of a file\(aqs symbolic identifier (the Toil FileID), its local path on the host (the value returned by readGlobalFile) and the the location of the file inside the Docker container. .sp After executing single_job_executor from CWLTool, it gets back the output object and status. If the underlying job failed, raise an exception. Files from the output object are added to the file store using writeGlobalFile and the \(aqlocation\(aq field of File references are updated to reflect the token returned by the Toil file store. .sp When the workflow completes, it returns an indirect dictionary linking to the outputs of the job steps that contribute to the final output. This is the value returned by toil.start() or toil.restart(). This is resolved to get the final output object. The files in this object are exported from the file store to \(aqoutdir\(aq on the host file system, and the \(aqlocation\(aq field of File references are updated to reflect the final exported location of the output files. .SH MINIMUM AWS IAM PERMISSIONS .sp Toil requires at least the following permissions in an IAM role to operate on a cluster. These are added by default when launching a cluster. However, ensure that they are present if creating a custom IAM role when \fI\%launching a cluster\fP with the \fB\-\-awsEc2ProfileArn\fP parameter. .INDENT 0.0 .INDENT 3.5 .sp .EX { \(dqVersion\(dq: \(dq2012\-10\-17\(dq, \(dqStatement\(dq: [ { \(dqEffect\(dq: \(dqAllow\(dq, \(dqAction\(dq: [ \(dqec2:*\(dq, \(dqs3:*\(dq, \(dqsdb:*\(dq, \(dqiam:PassRole\(dq ], \(dqResource\(dq: \(dq*\(dq } ] } .EE .UNINDENT .UNINDENT .SH AUTO-DEPLOYMENT .sp If you want to run a Toil Python workflow in a distributed environment, on multiple worker machines, either in the cloud or on a bare\-metal cluster, the Python code needs to be made available to those other machines. If the workflow\(aqs main module imports other modules, those modules also need to be made available on the workers. Toil can automatically do that for you, with a little help on your part. We call this feature \fIauto\-deployment\fP of a workflow. .sp Let\(aqs first examine various scenarios of auto\-deploying a workflow, which, as we\(aqll see shortly cannot be auto\-deployed. Lastly, we\(aqll deal with the issue of declaring \fI\%Toil as a dependency\fP of a workflow that is packaged as a setuptools distribution. .sp Toil can be easily deployed to a remote host. First, assuming you\(aqve followed our \fI\%Preparing your AWS environment\fP section to install Toil and use it to create a remote leader node on (in this example) AWS, you can now log into this into using \fI\%Ssh\-Cluster Command\fP and once on the remote host, create and activate a virtualenv (noting to make sure to use the \fB\-\-system\-site\-packages\fP option!): .INDENT 0.0 .INDENT 3.5 .sp .EX $ virtualenv \-\-system\-site\-packages venv $ . venv/bin/activate .EE .UNINDENT .UNINDENT .sp Note the \fB\-\-system\-site\-packages\fP option, which ensures that globally\-installed packages are accessible inside the virtualenv. Do not (re)install Toil after this! The \fB\-\-system\-site\-packages\fP option has already transferred Toil and the dependencies from your local installation of Toil for you. .sp From here, you can install a project and its dependencies: .INDENT 0.0 .INDENT 3.5 .sp .EX $ tree \&. ├── util │ ├── __init__.py │ └── sort │ ├── __init__.py │ └── quick.py └── workflow ├── __init__.py └── main.py 3 directories, 5 files $ pip install matplotlib $ cp \-R workflow util venv/lib/python3.9/site\-packages .EE .UNINDENT .UNINDENT .sp Ideally, your project would have a \fBsetup.py\fP file (see \fI\%setuptools\fP) which streamlines the installation process: .INDENT 0.0 .INDENT 3.5 .sp .EX $ tree \&. ├── util │ ├── __init__.py │ └── sort │ ├── __init__.py │ └── quick.py ├── workflow │ ├── __init__.py │ └── main.py └── setup.py 3 directories, 6 files $ pip install . .EE .UNINDENT .UNINDENT .sp Or, if your project has been published to PyPI: .INDENT 0.0 .INDENT 3.5 .sp .EX $ pip install my\-project .EE .UNINDENT .UNINDENT .sp In each case, we have created a virtualenv with the \fB\-\-system\-site\-packages\fP flag in the \fBvenv\fP subdirectory then installed the \fBmatplotlib\fP distribution from PyPI along with the two packages that our project consists of. (Again, both Python and Toil are assumed to be present on the leader and all worker nodes.) .sp We can now run our workflow: .INDENT 0.0 .INDENT 3.5 .sp .EX $ python3 main.py \-\-batchSystem=kubernetes … .EE .UNINDENT .UNINDENT .sp \fBIMPORTANT:\fP .INDENT 0.0 .INDENT 3.5 If workflow\(aqs external dependencies contain native code (i.e. are not pure Python) then they must be manually installed on each worker. .UNINDENT .UNINDENT .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 Neither \fBpython3 setup.py develop\fP nor \fBpip install \-e .\fP can be used in this process as, instead of copying the source files, they create \fB\&.egg\-link\fP files that Toil can\(aqt auto\-deploy. Similarly, \fBpython3 setup.py install\fP doesn\(aqt work either as it installs the project as a Python \fB\&.egg\fP which is also not currently supported by Toil (though it \fI\%could be\fP in the future). .sp Also note that using the \fB\-\-single\-version\-externally\-managed\fP flag with \fBsetup.py\fP will prevent the installation of your package as an \fB\&.egg\fP\&. It will also disable the automatic installation of your project\(aqs dependencies. .UNINDENT .UNINDENT .SS Auto Deployment with Sibling Python Files .sp This scenario applies if a Python workflow imports files that are its siblings: .INDENT 0.0 .INDENT 3.5 .sp .EX $ cd my_project $ ls userScript.py utilities.py $ ./userScript.py \-\-batchSystem=kubernetes … .EE .UNINDENT .UNINDENT .sp Here \fBuserScript.py\fP imports additional functionality from \fButilities.py\fP\&. Toil detects that \fBuserScript.py\fP has sibling Python files and copies them to the workers, alongside the main Python file. Note that sibling Python files will be auto\-deployed regardless of whether they are actually imported by the workflow: all \fB\&.py\fP files residing in the same directory as the main workflow Python file will automatically be auto\-deployed. .sp This structure is a suitable method of organizing the source code of reasonably complicated workflows. .SS Auto\-Deploying a Package Hierarchy .sp Recall that in Python, a \fI\%package\fP is a directory containing one or more \fB\&.py\fP files, one of which must be called \fB__init__.py\fP, and optionally other packages. For more involved workflows that contain a significant amount of code, this is the recommended way of organizing the source code. Because we use a package hierarchy, the main workflow file is actually a Python module. It is merely one of the modules in the package hierarchy. We need to inform Toil that we want to use a package hierarchy by invoking Python\(aqs \fB\-m\fP option. This enables Toil to identify the entire set of modules belonging to the workflow and copy all of them to each worker. Note that while using the \fB\-m\fP option is optional in the scenarios above, it is mandatory in this one. .sp The following shell session illustrates this: .INDENT 0.0 .INDENT 3.5 .sp .EX $ cd my_project $ tree \&. ├── utils │ ├── __init__.py │ └── sort │ ├── __init__.py │ └── quick.py └── workflow ├── __init__.py └── main.py 3 directories, 5 files $ python3 \-m workflow.main \-\-batchSystem=kubernetes … .EE .UNINDENT .UNINDENT .sp Here the workflow entry point module \fBmain.py\fP does not reside in the current directory, but is part of a package called \fButil\fP, in a subdirectory of the current directory. Additional functionality is in a separate module called \fButil.sort.quick\fP which corresponds to \fButil/sort/quick.py\fP\&. Because we invoke the workflow via \fBpython3 \-m workflow.main\fP, Toil can determine the root directory of the hierarchy–\fBmy_project\fP in this case–and copy all Python modules underneath it to each worker. The \fB\-m\fP option is documented \fI\%here\fP .sp When \fB\-m\fP is passed, Python adds the current working directory to \fBsys.path\fP, the list of root directories to be considered when resolving a module name like \fBworkflow.main\fP\&. Without that added convenience we\(aqd have to run the workflow as \fBPYTHONPATH=\(dq$PWD\(dq python3 \-m workflow.main\fP\&. This also means that Toil can detect the root directory of the invoked module\(aqs package hierarchy even if it isn\(aqt the current working directory. In other words we could do this: .INDENT 0.0 .INDENT 3.5 .sp .EX $ cd my_project $ export PYTHONPATH=\(dq$PWD\(dq $ cd /some/other/dir $ python3 \-m workflow.main \-\-batchSystem=kubernetes … .EE .UNINDENT .UNINDENT .sp Also note that the root directory itself must not be package, i.e. must not contain an \fB__init__.py\fP\&. .SS Relying on Shared Filesystems .sp Bare\-metal clusters typically mount a shared file system like NFS on each node. If every node has that file system mounted at the same path, you can place your project on that shared filesystem and run your Python workflow from there. Additionally, you can clone the Toil source tree into a directory on that shared file system and you won\(aqt even need to install Toil on every worker. Be sure to add both your project directory and the Toil clone to \fBPYTHONPATH\fP\&. Toil replicates \fBPYTHONPATH\fP from the leader to every worker. .INDENT 0.0 .INDENT 3.5 .IP "Using a shared filesystem" .sp Toil currently only supports a \fBtempdir\fP set to a local, non\-shared directory. .UNINDENT .UNINDENT .SS Toil Appliance .sp The term Toil Appliance refers to the Ubuntu\-based Docker image that Toil uses for the machines in Toil\-manages clusters, and for executing jobs on Kubernetes. It\(aqs easily deployed, only needs Docker, and allows a consistent environment on all Toil clusters. To specify a different image, see the Toil \fI\%Environment Variables\fP section. For more information on the Toil Appliance, see the \fI\%Running in AWS\fP section. .SH ENVIRONMENT VARIABLES .sp There are several environment variables that affect the way Toil runs. .TS center; |l|l|. _ T{ TOIL_CHECK_ENV T} T{ A flag that determines whether Toil will try to refer back to a Python virtual environment in which it is installed when composing commands that may be run on other hosts. If set to \fBTrue\fP, if Toil is installed in the current virtual environment, it will use absolute paths to its own executables (and the virtual environment must thus be available on at the same path on all nodes). Otherwise, Toil internal commands such as \fB_toil_worker\fP will be resolved according to the \fBPATH\fP on the node where they are executed. This setting can be useful in a shared HPC environment, where users may have their own Toil installations in virtual environments. T} _ T{ TOIL_WORKDIR T} T{ An absolute path to a directory where Toil will write its temporary files. This directory must exist on each worker node and may be set to a different value on each worker. The \fB\-\-workDir\fP command line option overrides this. When using the Toil docker container, such as on Kubernetes, this defaults to \fB/var/lib/toil\fP\&. When using Toil autoscaling with Mesos, this is somewhere inside the Mesos sandbox. In all other cases, the system\(aqs \fI\%standard temporary directory\fP is used. T} _ T{ TOIL_WORKDIR_OVERRIDE T} T{ An absolute path to a directory where Toil will write its temporary files. This overrides \fBTOIL_WORKDIR\fP and the \fB\-\-workDir\fP command line option. T} _ T{ TOIL_COORDINATION_DIR T} T{ An absolute path to a directory where Toil will write its lock files. This directory must exist on each worker node and may be set to a different value on each worker. The \fB\-\-coordinationDir\fP command line option overrides this. T} _ T{ TOIL_COORDINATION_DIR_OVERRIDE T} T{ An absolute path to a directory where Toil will write its lock files. This overrides \fBTOIL_COORDINATION_DIR\fP and the \fB\-\-coordinationDir\fP command line option. T} _ T{ TOIL_BATCH_LOGS_DIR T} T{ A directory to save batch system logs into, where the leader can access them. The \fB\-\-batchLogsDir\fP option overrides this. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf. T} _ T{ TOIL_KUBERNETES_HOST_PATH T} T{ A path on Kubernetes hosts that will be mounted as the Toil work directory in the workers, to allow for shared caching. Will be created if it doesn\(aqt already exist. T} _ T{ TOIL_KUBERNETES_OWNER T} T{ A name prefix for easy identification of Kubernetes jobs. If not set, Toil will use the current user name. T} _ T{ TOIL_KUBERNETES_SERVICE_ACCOUNT T} T{ A service account name to apply when creating Kubernetes pods. T} _ T{ TOIL_KUBERNETES_POD_TIMEOUT T} T{ Seconds to wait for a scheduled Kubernetes pod to start running. T} _ T{ KUBE_WATCH_ENABLED T} T{ A boolean variable that allows for users to utilize kubernetes watch stream feature instead of polling for running jobs. Default value is set to False. T} _ T{ TOIL_APPLIANCE_SELF T} T{ The fully qualified reference for the Toil Appliance you wish to use, in the form \fBREPO/IMAGE:TAG\fP\&. \fBquay.io/ucsc_cgl/toil:3.6.0\fP and \fBcket/toil:3.5.0\fP are both examples of valid options. Note that since Docker defaults to Dockerhub repos, only quay.io repos need to specify their registry. T} _ T{ TOIL_DOCKER_REGISTRY T} T{ The URL of the registry of the Toil Appliance image you wish to use. Docker will use Dockerhub by default, but the quay.io registry is also very popular and easily specifiable by setting this option to \fBquay.io\fP\&. T} _ T{ TOIL_DOCKER_NAME T} T{ The name of the Toil Appliance image you wish to use. Generally this is simply \fBtoil\fP but this option is provided to override this, since the image can be built with arbitrary names. T} _ T{ TOIL_AWS_SECRET_NAME T} T{ For the Kubernetes batch system, the name of a Kubernetes secret which contains a \fBcredentials\fP file granting access to AWS resources. Will be mounted as \fB~/.aws\fP inside Kubernetes\-managed Toil containers. Enables the AWSJobStore to be used with the Kubernetes batch system, if the credentials allow access to S3 and SimpleDB. T} _ T{ TOIL_AWS_ZONE T} T{ Zone to use when using AWS. Also determines region. Overrides TOIL_AWS_REGION. T} _ T{ TOIL_AWS_REGION T} T{ Region to use when using AWS. T} _ T{ TOIL_AWS_AMI T} T{ ID of the AMI to use in node provisioning. If in doubt, don\(aqt set this variable. T} _ T{ TOIL_AWS_NODE_DEBUG T} T{ Determines whether to preserve nodes that have failed health checks. If set to \fBTrue\fP, nodes that fail EC2 health checks won\(aqt immediately be terminated so they can be examined and the cause of failure determined. If any EC2 nodes are left behind in this manner, the security group will also be left behind by necessity as it cannot be deleted until all associated nodes have been terminated. T} _ T{ TOIL_AWS_BATCH_QUEUE T} T{ Name or ARN of an AWS Batch Queue to use with the AWS Batch batch system. T} _ T{ TOIL_AWS_BATCH_JOB_ROLE_ARN T} T{ ARN of an IAM role to run AWS Batch jobs as with the AWS Batch batch system. If the jobs are not run with an IAM role or on machines that have access to S3 and SimpleDB, the AWS job store will not be usable. T} _ T{ TOIL_GOOGLE_PROJECTID T} T{ The Google project ID to use when generating Google job store names for tests or CWL workflows. T} _ T{ TOIL_SLURM_ARGS T} T{ Arguments for sbatch for the slurm batch system. Do not pass CPU or memory specifications here. Instead, define resource requirements for the job. There is no default value for this variable. If neither \fB\-\-export\fP nor \fB\-\-export\-file\fP is in the argument list, \fB\-\-export=ALL\fP will be provided. T} _ T{ TOIL_SLURM_PE T} T{ Name of the slurm partition to use for parallel jobs. There is no default value for this variable. T} _ T{ TOIL_GRIDENGINE_ARGS T} T{ Arguments for qsub for the gridengine batch system. Do not pass CPU or memory specifications here. Instead, define resource requirements for the job. There is no default value for this variable. T} _ T{ TOIL_GRIDENGINE_PE T} T{ Parallel environment arguments for qsub and for the gridengine batch system. There is no default value for this variable. T} _ T{ TOIL_TORQUE_ARGS T} T{ Arguments for qsub for the Torque batch system. Do not pass CPU or memory specifications here. Instead, define extra parameters for the job such as queue. Example: \-q medium Use TOIL_TORQUE_REQS to pass extra values for the \-l resource requirements parameter. There is no default value for this variable. T} _ T{ TOIL_TORQUE_REQS T} T{ Arguments for the resource requirements for Torque batch system. Do not pass CPU or memory specifications here. Instead, define extra resource requirements as a string that goes after the \-l argument to qsub. Example: walltime=2:00:00,file=50gb There is no default value for this variable. T} _ T{ TOIL_LSF_ARGS T} T{ Additional arguments for the LSF\(aqs bsub command. Instead, define extra parameters for the job such as queue. Example: \-q medium. There is no default value for this variable. T} _ T{ TOIL_HTCONDOR_PARAMS T} T{ Additional parameters to include in the HTCondor submit file passed to condor_submit. Do not pass CPU or memory specifications here. Instead define extra parameters which may be required by HTCondor. This variable is parsed as a semicolon\-separated string of \fBparameter = value\fP pairs. Example: \fBrequirements = TARGET.has_sse4_2 == true; accounting_group = test\fP\&. There is no default value for this variable. T} _ T{ TOIL_CUSTOM_DOCKER_INIT_COMMAND T} T{ Any custom bash command to run in the Toil docker container prior to running the Toil services. Can be used for any custom initialization in the worker and/or primary nodes such as private docker docker authentication. Example for AWS ECR: \fBpip install awscli && eval $(aws ecr get\-login \-\-no\-include\-email \-\-region us\-east\-1)\fP\&. T} _ T{ TOIL_CUSTOM_INIT_COMMAND T} T{ Any custom bash command to run prior to starting the Toil appliance. Can be used for any custom initialization in the worker and/or primary nodes such as private docker authentication for the Toil appliance itself (i.e. from TOIL_APPLIANCE_SELF). T} _ T{ TOIL_S3_HOST T} T{ the IP address or hostname to use for connecting to S3. Example: \fBTOIL_S3_HOST=127.0.0.1\fP T} _ T{ TOIL_S3_PORT T} T{ a port number to use for connecting to S3. Example: \fBTOIL_S3_PORT=9001\fP T} _ T{ TOIL_S3_USE_SSL T} T{ enable or disable the usage of SSL for connecting to S3 (\fBTrue\fP by default). Example: \fBTOIL_S3_USE_SSL=False\fP T} _ T{ TOIL_WES_BROKER_URL T} T{ An optional broker URL to use to communicate between the WES server and Celery task queue. If unset, \fBamqp://guest:guest@localhost:5672//\fP is used. T} _ T{ TOIL_WES_JOB_STORE_TYPE T} T{ Type of job store to use by default for workflows run via the WES server. Can be \fBfile\fP, \fBaws\fP, or \fBgoogle\fP\&. T} _ T{ TOIL_OWNER_TAG T} T{ This will tag cloud resources with a tag reading: \(dqOwner: $TOIL_OWNER_TAG\(dq. This is used internally at UCSC to stop a bot we have that terminates untagged resources. T} _ T{ TOIL_AWS_PROFILE T} T{ The name of an AWS profile to run TOIL with. T} _ T{ TOIL_AWS_TAGS T} T{ This will tag cloud resources with any arbitrary tags given in a JSON format. These are overwritten in favor of CLI options when using launch cluster. For information on valid AWS tags, see \fI\%AWS Tags\fP\&. T} _ T{ SINGULARITY_DOCKER_HUB_MIRROR T} T{ An http or https URL for the Singularity wrapper in the Toil Docker container to use as a mirror for Docker Hub. T} _ T{ OMP_NUM_THREADS T} T{ The number of cores set for OpenMP applications in the workers. If not set, Toil will use the number of job threads. T} _ T{ GUNICORN_CMD_ARGS T} T{ Specify additional Gunicorn configurations for the Toil WES server. See \fI\%Gunicorn settings\fP\&. T} _ .TE .SH API REFERENCE .sp This page contains auto\-generated API reference documentation [1]\&. .SS \fI\%toil\fP .SS Subpackages .SS \fI\%toil.batchSystems\fP .SS Subpackages .SS \fI\%toil.batchSystems.mesos\fP .SS Subpackages .SS \fI\%toil.batchSystems.mesos.test\fP .SS Package Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ExceptionalThread\fP T} T{ A thread whose join() method re\-raises exceptions raised during run(). While join() is T} _ T{ \fI\%MesosTestSupport\fP T} T{ Mixin for test cases that need a running Mesos master and agent on the local host. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%retry\fP([intervals, infinite_retries, errors, ...]) T} T{ Retry a function if it fails with any Exception defined in \(dqerrors\(dq. T} _ T{ \fI\%cpu_count\fP() T} T{ Get the rounded\-up integer number of whole CPUs available. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.mesos.test.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None) Retry a function if it fails with any Exception defined in \(dqerrors\(dq. .sp Does so every x seconds, where x is defined by a list of numbers (ints or floats) in \(dqintervals\(dq. Also accepts ErrorCondition events for more detailed retry attempts. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBintervals\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI]\fP) \-\- A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back\-off before failing: 1s, 1s, 2s, 4s, 8s, 16s .IP \(bu 2 \fBinfinite_retries\fP (\fI\%bool\fP) \-\- If this is True, reset the intervals when they run out. Defaults to: False. .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fISequence\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%ErrorCondition\fP\fI, \fP\fIType\fP\fI[\fP\fI\%Exception\fP\fI]\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .sp A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: \- Exception (required) \- Error codes that must match to be retried (optional; defaults to not checking) \- A string that must be in the error message to be retried (optional; defaults to not checking) \- A bool that can be set to False to always error on this condition. .sp If not specified, this will default to a generic Exception. .IP \(bu 2 \fBlog_message\fP (\fIOptional\fP\fI[\fP\fITuple\fP\fI[\fP\fICallable\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Optional tuple of (\(dqlog/print function()\(dq, \(dqmessage string\(dq) that will precede each attempt. .IP \(bu 2 \fBprepare\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fICallable\fP\fI]\fP\fI]\fP) \-\- Optional list of functions to call, with the function\(aqs arguments, between retries, to reset state. .UNINDENT .TP .B Returns The result of the wrapped function or raise. .TP .B Return type Callable[[Any], Any] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.mesos.test.ExceptionalThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None) Bases: \fI\%threading.Thread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B exc_info .UNINDENT .INDENT 7.0 .TP .B run() Method representing the thread\(aqs activity. .sp You may override this method in a subclass. The standard run() method invokes the callable object passed to the object\(aqs constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tryRun() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B join(*args, **kwargs) Wait until the thread terminates. .sp This blocks the calling thread until the thread whose join() method is called terminates \-\- either normally or through an unhandled exception or until the optional timeout occurs. .sp When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened \-\- if the thread is still alive, the join() call timed out. .sp When the timeout argument is not present or None, the operation will block until the thread terminates. .sp A thread can be join()ed many times. .sp join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.mesos.test.cpu_count() Get the rounded\-up integer number of whole CPUs available. .sp Counts hyperthreads as CPUs. .sp Uses the system\(aqs actual CPU count, or the current v1 cgroup\(aqs quota per period, if the quota is set. .sp Ignores the cgroup\(aqs cpu shares value, because it\(aqs extremely difficult to interpret. See \fI\%https://github.com/kubernetes/kubernetes/issues/81021\fP\&. .sp Caches result for efficiency. .INDENT 7.0 .TP .B Returns Integer count of available CPUs, minimum 1. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.mesos.test.log .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.mesos.test.MesosTestSupport Mixin for test cases that need a running Mesos master and agent on the local host. .INDENT 7.0 .TP .B class MesosThread(numCores) Bases: \fI\%toil.lib.threading.ExceptionalThread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B lock .UNINDENT .INDENT 7.0 .TP .B abstract mesosCommand() .UNINDENT .INDENT 7.0 .TP .B tryRun() .UNINDENT .INDENT 7.0 .TP .B findMesosBinary(names) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class MesosMasterThread(numCores) Bases: \fBMesosThread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B mesosCommand() .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class MesosAgentThread(numCores) Bases: \fBMesosThread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B mesosCommand() .UNINDENT .UNINDENT .INDENT 7.0 .TP .B wait_for_master() .UNINDENT .UNINDENT .SS Submodules .SS \fI\%toil.batchSystems.mesos.batchSystem\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%MesosBatchSystem\fP T} T{ A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.mesos.batchSystem.log .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.mesos.batchSystem.MesosBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.local_support.BatchSystemLocalSupport\fP, \fI\%toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem\fP, \fBpymesos.Scheduler\fP .sp A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos tasks over a cluster of agent nodes. A Mesos framework consists of a scheduler and an executor. This class acts as the scheduler and is typically run on the master node that also runs the Mesos master process with which the scheduler communicates via a driver component. The executor is implemented in a separate class. It is run on each agent node and communicates with the Mesos agent process via another driver object. The scheduler may also be run on a separate node from the master, which we then call somewhat ambiguously the driver node. .INDENT 7.0 .TP .B class ExecutorInfo(nodeAddress, agentId, nodeInfo, lastSeen) .UNINDENT .INDENT 7.0 .TP .B userScript .INDENT 7.0 .TP .B Type \fI\%toil.resource.Resource\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fI\%setUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .UNINDENT .INDENT 7.0 .TP .B classmethod supportsWorkerCleanup() Whether this batch system supports worker cleanup. .sp Indicates whether this batch system invokes \fBBatchSystemSupport.workerCleanup()\fP after the last job for a particular workflow invocation finishes. Note that the term \fIworker\fP refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to \fIshut down\fP after the last worker process terminates. .UNINDENT .INDENT 7.0 .TP .B setUserScript(userScript) Set the user script for this workflow. .sp This method must be called before the first job is issued to this batch system, and only if \fI\%supportsAutoDeployment()\fP returns True, otherwise it will raise an exception. .INDENT 7.0 .TP .B Parameters \fBuserScript\fP \-\- the resource object representing the user script or module and the modules it depends on. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B ignoreNode(nodeAddress) Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished. .INDENT 7.0 .TP .B Parameters \fBnodeAddress\fP \-\- IP address of node to ignore. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B unignoreNode(nodeAddress) Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one. .UNINDENT .INDENT 7.0 .TP .B issueBatchJob(jobNode, job_environment=None) Issues the following command returning a unique jobID. Command is the string to run, memory is an int giving the number of bytes the job needs to run in and cores is the number of cpus needed for the job and error\-file is the path of the file to place any std\-err/std\-out in. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobNode\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killBatchJobs(jobIDs) Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob. .INDENT 7.0 .TP .B Parameters \fBjobIDs\fP \-\- list of IDs of jobs to kill .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getIssuedBatchJobIDs() Gets all currently issued jobs .INDENT 7.0 .TP .B Returns A list of jobs (as jobIDs) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningBatchJobIDs() Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds. .INDENT 7.0 .TP .B Returns dictionary with currently running jobID keys and how many seconds they have been running as the value .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUpdatedBatchJob(maxWait) Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once. .sp Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP \-\- the number of seconds to block, waiting for a result .TP .B Returns If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall\-clock time the job ran for, or None if this batch system does not support tracking wall time. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B nodeInUse(nodeIP) Can be used to determine if a worker node is running any tasks. If the node is doesn\(aqt exist, this function should simply return False. .INDENT 7.0 .TP .B Parameters \fBnodeIP\fP (\fI\%str\fP) \-\- The worker nodes private IP address .TP .B Returns True if the worker node has been issued any tasks, else False .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getWaitDuration() Gets the period of time to wait (floating point, in seconds) between checking for missing/overlong jobs. .UNINDENT .INDENT 7.0 .TP .B shutdown() Called at the completion of a toil invocation. Should cleanly terminate all worker threads. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B registered(driver, frameworkId, masterInfo) Invoked when the scheduler successfully registers with a Mesos master .UNINDENT .INDENT 7.0 .TP .B resourceOffers(driver, offers) Invoked when resources have been offered to this framework. .UNINDENT .INDENT 7.0 .TP .B statusUpdate(driver, update) Invoked when the status of a task has changed (e.g., a agent is lost and so the task is lost, a task finishes and an executor sends a status update saying so, etc). Note that returning from this callback _acknowledges_ receipt of this status update! If for whatever reason the scheduler aborts during this callback (or the process exits) another status update will be delivered (note, however, that this is currently not true if the agent sending the status update is lost/fails during that time). .UNINDENT .INDENT 7.0 .TP .B frameworkMessage(driver, executorId, agentId, message) Invoked when an executor sends a message. .UNINDENT .INDENT 7.0 .TP .B getNodes(preemptible=None, timeout=None) .INDENT 7.0 .TP .B Return all nodes that match: .INDENT 7.0 .IP \(bu 2 preemptible status (None includes all) .IP \(bu 2 timeout period (seen within the last # seconds, or None for all) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .IP \(bu 2 \fBtimeout\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Dict[\fI\%str\fP, \fI\%toil.batchSystems.abstractBatchSystem.NodeInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B reregistered(driver, masterInfo) Invoked when the scheduler re\-registers with a newly elected Mesos master. .UNINDENT .INDENT 7.0 .TP .B executorLost(driver, executorId, agentId, status) Invoked when an executor has exited/terminated abnormally. .UNINDENT .INDENT 7.0 .TP .B classmethod get_default_mesos_endpoint() Get the default IP/hostname and port that we will look for Mesos at. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%toil.batchSystems.options.OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.mesos.conftest\fP .SS Module Contents .INDENT 0.0 .TP .B toil.batchSystems.mesos.conftest.collect_ignore = [] .UNINDENT .SS \fI\%toil.batchSystems.mesos.executor\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%MesosExecutor\fP T} T{ Part of Toil\(aqs Mesos framework, runs on a Mesos agent. A Toil job is passed to it via the T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.mesos.executor.log .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.mesos.executor.MesosExecutor Bases: \fBpymesos.Executor\fP .sp Part of Toil\(aqs Mesos framework, runs on a Mesos agent. A Toil job is passed to it via the task.data field, and launched via call(toil.command). .INDENT 7.0 .TP .B registered(driver, executorInfo, frameworkInfo, agentInfo) Invoked once the executor driver has been able to successfully connect with Mesos. .UNINDENT .INDENT 7.0 .TP .B reregistered(driver, agentInfo) Invoked when the executor re\-registers with a restarted agent. .UNINDENT .INDENT 7.0 .TP .B disconnected(driver) Invoked when the executor becomes \(dqdisconnected\(dq from the agent (e.g., the agent is being restarted due to an upgrade). .UNINDENT .INDENT 7.0 .TP .B killTask(driver, taskId) Kill parent task process and all its spawned children .UNINDENT .INDENT 7.0 .TP .B shutdown(driver) .UNINDENT .INDENT 7.0 .TP .B error(driver, message) Invoked when a fatal error has occurred with the executor and/or executor driver. .UNINDENT .INDENT 7.0 .TP .B launchTask(driver, task) Invoked by SchedulerDriver when a Mesos task should be launched by this executor .UNINDENT .INDENT 7.0 .TP .B frameworkMessage(driver, message) Invoked when a framework message has arrived for this executor. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.mesos.executor.main() .UNINDENT .SS Package Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Shape\fP T} T{ Represents a job or a node\(aqs \(dqshape\(dq, in terms of the dimensions of memory, cores, disk and T} _ T{ \fI\%JobQueue\fP T} T{ T} _ T{ \fI\%MesosShape\fP T} T{ Represents a job or a node\(aqs \(dqshape\(dq, in terms of the dimensions of memory, cores, disk and T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%TaskData\fP T} T{ T} _ T{ \fI\%ToilJob\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.batchSystems.mesos.Shape(wallTime, memory, cores, disk, preemptible) Represents a job or a node\(aqs \(dqshape\(dq, in terms of the dimensions of memory, cores, disk and wall\-time allocation. .sp The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs? .sp The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwallTime\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcores\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __eq__(other) Return self==value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B greater_than(other) .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __gt__(other) Return self>value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __hash__() Return hash(self). .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.mesos.TaskData .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.mesos.JobQueue .INDENT 7.0 .TP .B insertJob(job, jobType) .UNINDENT .INDENT 7.0 .TP .B jobIDs() .UNINDENT .INDENT 7.0 .TP .B nextJobOfType(jobType) .UNINDENT .INDENT 7.0 .TP .B typeEmpty(jobType) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.mesos.MesosShape(wallTime, memory, cores, disk, preemptible) Bases: \fI\%toil.provisioners.abstractProvisioner.Shape\fP .sp Represents a job or a node\(aqs \(dqshape\(dq, in terms of the dimensions of memory, cores, disk and wall\-time allocation. .sp The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs? .sp The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwallTime\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcores\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __gt__(other) Inverted. Returns True if self is less than other, else returns False. .sp This is because jobTypes are sorted in decreasing order, and this was done to give expensive jobs priority. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.mesos.ToilJob .UNINDENT .SS Submodules .SS \fI\%toil.batchSystems.abstractBatchSystem\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BatchJobExitReason\fP T} T{ Enum where members are also (and must be) ints T} _ T{ \fI\%UpdatedBatchJobInfo\fP T} T{ T} _ T{ \fI\%WorkerCleanupInfo\fP T} T{ T} _ T{ \fI\%AbstractBatchSystem\fP T} T{ An abstract base class to represent the interface the batch system must provide to Toil. T} _ T{ \fI\%BatchSystemSupport\fP T} T{ Partial implementation of AbstractBatchSystem, support methods. T} _ T{ \fI\%NodeInfo\fP T} T{ The coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores T} _ T{ \fI\%AbstractScalableBatchSystem\fP T} T{ A batch system that supports a variable number of worker nodes. T} _ T{ \fI\%ResourcePool\fP T} T{ Represents an integral amount of a resource (such as memory bytes). T} _ T{ \fI\%ResourceSet\fP T} T{ Represents a collection of distinct resources (such as accelerators). T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%EXIT_STATUS_UNAVAILABLE_VALUE\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.abstractBatchSystem.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.abstractBatchSystem.EXIT_STATUS_UNAVAILABLE_VALUE = 255 .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.BatchJobExitReason Bases: \fI\%enum.IntEnum\fP .sp Enum where members are also (and must be) ints .INDENT 7.0 .TP .B FINISHED: \fI\%int\fP = 1 Successfully finished. .UNINDENT .INDENT 7.0 .TP .B FAILED: \fI\%int\fP = 2 Job finished, but failed. .UNINDENT .INDENT 7.0 .TP .B LOST: \fI\%int\fP = 3 Preemptable failure (job\(aqs executing host went away). .UNINDENT .INDENT 7.0 .TP .B KILLED: \fI\%int\fP = 4 Job killed before finishing. .UNINDENT .INDENT 7.0 .TP .B ERROR: \fI\%int\fP = 5 Internal error. .UNINDENT .INDENT 7.0 .TP .B MEMLIMIT: \fI\%int\fP = 6 Job hit batch system imposed memory limit. .UNINDENT .INDENT 7.0 .TP .B classmethod to_string(value) Convert to human\-readable string. .sp Given an int that may be or may be equal to a value from the enum, produce the string value of its matching enum entry, or a stringified int. .INDENT 7.0 .TP .B Parameters \fBvalue\fP (\fI\%int\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo Bases: \fBNamedTuple\fP .INDENT 7.0 .TP .B jobID: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B exitStatus: \fI\%int\fP The exit status (integer value) of the job. 0 implies successful. .sp EXIT_STATUS_UNAVAILABLE_VALUE is used when the exit status is not available (e.g. job is lost, or otherwise died but actual exit code was not reported). .UNINDENT .INDENT 7.0 .TP .B exitReason: \fI\%BatchJobExitReason\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B wallTime: \fI\%float\fP | \fI\%int\fP | \fI\%None\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo Bases: \fBNamedTuple\fP .INDENT 7.0 .TP .B work_dir: \fI\%str\fP | \fI\%None\fP Work directory path (where the cache would go) if specified by user .UNINDENT .INDENT 7.0 .TP .B coordination_dir: \fI\%str\fP | \fI\%None\fP Coordination directory path (where lock files would go) if specified by user .UNINDENT .INDENT 7.0 .TP .B workflow_id: \fI\%str\fP Used to identify files specific to this workflow .UNINDENT .INDENT 7.0 .TP .B clean_work_dir: \fI\%str\fP When to clean up the work and coordination directories for a job (\(aqalways\(aq, \(aqonSuccess\(aq, \(aqonError\(aq, \(aqnever\(aq) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem Bases: \fI\%abc.ABC\fP .sp An abstract base class to represent the interface the batch system must provide to Toil. .INDENT 7.0 .TP .B abstract classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fI\%setUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract classmethod supportsWorkerCleanup() Whether this batch system supports worker cleanup. .sp Indicates whether this batch system invokes \fI\%BatchSystemSupport.workerCleanup()\fP after the last job for a particular workflow invocation finishes. Note that the term \fIworker\fP refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to \fIshut down\fP after the last worker process terminates. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract setUserScript(userScript) Set the user script for this workflow. .sp This method must be called before the first job is issued to this batch system, and only if \fI\%supportsAutoDeployment()\fP returns True, otherwise it will raise an exception. .INDENT 7.0 .TP .B Parameters \fBuserScript\fP (\fI\%toil.resource.Resource\fP) \-\- the resource object representing the user script or module and the modules it depends on. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_message_bus(message_bus) Give the batch system an opportunity to connect directly to the message bus, so that it can send informational messages about the jobs it is running to other Toil components. .INDENT 7.0 .TP .B Parameters \fBmessage_bus\fP (\fI\%toil.bus.MessageBus\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract issueBatchJob(jobDesc, job_environment=None) Issues a job with the specified command to the batch system and returns a unique jobID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- a toil.job.JobDescription .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- a collection of job\-specific environment variables to be set on the worker. .UNINDENT .TP .B Returns a unique jobID that can be used to reference the newly issued job .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract killBatchJobs(jobIDs) Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob. .INDENT 7.0 .TP .B Parameters \fBjobIDs\fP (\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- list of IDs of jobs to kill .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getIssuedBatchJobIDs() Gets all currently issued jobs .INDENT 7.0 .TP .B Returns A list of jobs (as jobIDs) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon. .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getRunningBatchJobIDs() Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds. .INDENT 7.0 .TP .B Returns dictionary with currently running jobID keys and how many seconds they have been running as the value .TP .B Return type Dict[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getUpdatedBatchJob(maxWait) Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once. .sp Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%int\fP) \-\- the number of seconds to block, waiting for a result .TP .B Returns If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall\-clock time the job ran for, or None if this batch system does not support tracking wall time. .TP .B Return type Optional[\fI\%UpdatedBatchJobInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getSchedulingStatusMessage() Get a log message fragment for the user about anything that might be going wrong in the batch system, if available. .sp If no useful message is available, return None. .sp This can be used to report what resource is the limiting factor when scheduling jobs, for example. If the leader thinks the workflow is stuck, the message can be displayed to the user to help them diagnose why it might be stuck. .INDENT 7.0 .TP .B Returns User\-directed message about scheduling state. .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract shutdown() Called at the completion of a toil invocation. Should cleanly terminate all worker threads. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract setEnv(name, value=None) Set an environment variable for the worker process before it is launched. .sp The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job. .sp If no value is provided it will be looked up from the current environment. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%toil.batchSystems.options.OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getWorkerContexts() Get a list of picklable context manager objects to wrap worker work in, in order. .sp Can be used to ask the Toil worker to do things in\-process (such as configuring environment variables, hot\-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping \(dqexecutor\(dq process. .INDENT 7.0 .TP .B Return type List[ContextManager[Any]] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.BatchSystemSupport(config, maxCores, maxMemory, maxDisk) Bases: \fI\%AbstractBatchSystem\fP .sp Partial implementation of AbstractBatchSystem, support methods. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBmaxCores\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmaxMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxDisk\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_resource_request(requirer) Check resource request is not greater than that available or allowed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirer\fP (\fI\%toil.job.Requirer\fP) \-\- Object whose requirements are being checked .IP \(bu 2 \fBjob_name\fP (\fI\%str\fP) \-\- Name of the job being checked, for generating a useful error report. .IP \(bu 2 \fBdetail\fP (\fI\%str\fP) \-\- Batch\-system\-specific message to include in the error. .UNINDENT .TP .B Raises \fI\%InsufficientSystemResources\fP \-\- raised when a resource is requested in an amount greater than allowed .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setEnv(name, value=None) Set an environment variable for the worker process before it is launched. The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job. .sp If no value is provided it will be looked up from the current environment. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- the environment variable to be set on the worker. .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- if given, the environment variable given by name will be set to this value. If None, the variable\(aqs current value will be used as the value on the worker .UNINDENT .TP .B Raises \fI\%RuntimeError\fP \-\- if value is None and the name cannot be found in the environment .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_message_bus(message_bus) Give the batch system an opportunity to connect directly to the message bus, so that it can send informational messages about the jobs it is running to other Toil components. .INDENT 7.0 .TP .B Parameters \fBmessage_bus\fP (\fI\%toil.bus.MessageBus\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_batch_logs_dir() Get the directory where the backing batch system should save its logs. .sp Only really makes sense if the backing batch system actually saves logs to a filesystem; Kubernetes for example does not. Ought to be a directory shared between the leader and the workers, if the backing batch system writes logs onto the worker\(aqs view of the filesystem, like many HPC schedulers do. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B format_std_out_err_path(toil_job_id, cluster_job_id, std) Format path for batch system standard output/error and other files generated by the batch system itself. .sp Files will be written to the batch logs directory (\-\-batchLogsDir, defaulting to the Toil work directory) with names containing both the Toil and batch system job IDs, for ease of debugging job failures. .INDENT 7.0 .TP .B Param int toil_job_id : The unique id that Toil gives a job. .TP .B Param cluster_job_id : What the cluster, for example, GridEngine, uses as its internal job id. .TP .B Param string std : The provenance of the stream (for example: \(aqerr\(aq for \(aqstderr\(aq or \(aqout\(aq for \(aqstdout\(aq) .TP .B Return type string : Formatted filename; however if self.config.noStdOutErr is true, returns \(aq/dev/null\(aq or equivalent. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtoil_job_id\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcluster_job_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBstd\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B format_std_out_err_glob(toil_job_id) Get a glob string that will match all file paths generated by format_std_out_err_path for a job. .INDENT 7.0 .TP .B Parameters \fBtoil_job_id\fP (\fI\%int\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static workerCleanup(info) Cleans up the worker node on batch system shutdown. .sp Also see \fBsupportsWorkerCleanup()\fP\&. .INDENT 7.0 .TP .B Parameters \fBinfo\fP (\fI\%WorkerCleanupInfo\fP) \-\- A named tuple consisting of all the relevant information for cleaning up the worker. .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.NodeInfo(coresUsed, memoryUsed, coresTotal, memoryTotal, requestedCores, requestedMemory, workers) The coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores busy), reflecting the CPU load of the node. .sp The memoryUsed attribute is a floating point value between 0 (no memory used) and 1 (all memory used), reflecting the memory pressure on the node. .sp The coresTotal and memoryTotal attributes are the node\(aqs resources, not just the used resources .sp The requestedCores and requestedMemory attributes are all the resources that Toil Jobs have reserved on the node, regardless of whether the resources are actually being used by the Jobs. .sp The workers attribute is an integer reflecting the number of workers currently active workers on the node. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcoresUsed\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmemoryUsed\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBcoresTotal\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmemoryTotal\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBrequestedCores\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBrequestedMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBworkers\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem Bases: \fI\%AbstractBatchSystem\fP .sp A batch system that supports a variable number of worker nodes. .sp Used by \fI\%toil.provisioners.clusterScaler.ClusterScaler\fP to scale the number of worker nodes in the cluster up or down depending on overall load. .INDENT 7.0 .TP .B abstract getNodes(preemptible=None, timeout=600) Returns a dictionary mapping node identifiers of preemptible or non\-preemptible nodes to NodeInfo objects, one for each node. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- If True (False) only (non\-)preemptible nodes will be returned. If None, all nodes will be returned. .IP \(bu 2 \fBtimeout\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type Dict[\fI\%str\fP, \fI\%NodeInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract nodeInUse(nodeIP) Can be used to determine if a worker node is running any tasks. If the node is doesn\(aqt exist, this function should simply return False. .INDENT 7.0 .TP .B Parameters \fBnodeIP\fP (\fI\%str\fP) \-\- The worker nodes private IP address .TP .B Returns True if the worker node has been issued any tasks, else False .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract ignoreNode(nodeAddress) Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished. .INDENT 7.0 .TP .B Parameters \fBnodeAddress\fP (\fI\%str\fP) \-\- IP address of node to ignore. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract unignoreNode(nodeAddress) Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one. .INDENT 7.0 .TP .B Parameters \fBnodeAddress\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.batchSystems.abstractBatchSystem.InsufficientSystemResources(requirer, resource, available=None, batch_system=None, source=None, details=[]) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirer\fP (\fI\%toil.job.Requirer\fP) \-\- .IP \(bu 2 \fBresource\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBavailable\fP (\fIOptional\fP\fI[\fP\fItoil.job.ParsedRequirement\fP\fI]\fP) \-\- .IP \(bu 2 \fBbatch_system\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBsource\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdetails\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Explain the exception. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.batchSystems.abstractBatchSystem.AcquisitionTimeoutException(resource, requested, available) Bases: \fI\%Exception\fP .sp To be raised when a resource request times out. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBresource\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBrequested\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI, \fP\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBavailable\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI, \fP\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.ResourcePool(initial_value, resource_type, timeout=5) Represents an integral amount of a resource (such as memory bytes). Amounts can be acquired immediately or with a timeout, and released. Provides a context manager to do something with an amount of resource acquired. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinitial_value\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBresource_type\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtimeout\fP (\fI\%float\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B acquireNow(amount) Reserve the given amount of the given resource. Returns True if successful and False if this is not possible immediately. .INDENT 7.0 .TP .B Parameters \fBamount\fP (\fI\%int\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B acquire(amount) Reserve the given amount of the given resource. Raises AcquisitionTimeoutException if this is not possible in under self.timeout time. .INDENT 7.0 .TP .B Parameters \fBamount\fP (\fI\%int\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B release(amount) .INDENT 7.0 .TP .B Parameters \fBamount\fP (\fI\%int\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B acquisitionOf(amount) .INDENT 7.0 .TP .B Parameters \fBamount\fP (\fI\%int\fP) \-\- .TP .B Return type Iterator[None] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractBatchSystem.ResourceSet(initial_value, resource_type, timeout=5) Represents a collection of distinct resources (such as accelerators). Subsets can be acquired immediately or with a timeout, and released. Provides a context manager to do something with a set of of resources acquired. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinitial_value\fP (\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBresource_type\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtimeout\fP (\fI\%float\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B acquireNow(subset) Reserve the given amount of the given resource. Returns True if successful and False if this is not possible immediately. .INDENT 7.0 .TP .B Parameters \fBsubset\fP (\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B acquire(subset) Reserve the given amount of the given resource. Raises AcquisitionTimeoutException if this is not possible in under self.timeout time. .INDENT 7.0 .TP .B Parameters \fBsubset\fP (\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B release(subset) .INDENT 7.0 .TP .B Parameters \fBsubset\fP (\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_free_snapshot() Get a snapshot of what items are free right now. May be stale as soon as you get it, but you will need some kind of hint to try and do an acquire. .INDENT 7.0 .TP .B Return type Set[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B acquisitionOf(subset) .INDENT 7.0 .TP .B Parameters \fBsubset\fP (\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type Iterator[None] .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.abstractGridEngineBatchSystem\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AbstractGridEngineBatchSystem\fP T} T{ A partial implementation of BatchSystemSupport for batch systems run on a T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%JobTuple\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.abstractGridEngineBatchSystem.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.abstractGridEngineBatchSystem.JobTuple .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.cleanup_support.BatchSystemCleanupSupport\fP .sp A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto\-deployment is not implemented. .INDENT 7.0 .TP .B class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) Bases: \fI\%threading.Thread\fP .sp A class that represents a thread of control. .sp This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnewJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBupdatedJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkillQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkilledJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBboss\fP (\fI\%AbstractGridEngineBatchSystem\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getBatchSystemID(jobID) Get batch system\-specific job ID .sp Note: for the moment this is the only consistent way to cleanly get the batch system job ID .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fI\%int\fP) \-\- Toil BatchSystem numerical job ID .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B forgetJob(jobID) Remove jobID passed .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fI\%int\fP) \-\- toil job ID .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B createJobs(newJob) Create a new job with the given attributes. .sp Implementation\-specific; called by AbstractGridEngineWorker.run() .INDENT 7.0 .TP .B Parameters \fBnewJob\fP (\fIJobTuple\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killJobs() Kill any running jobs within worker .UNINDENT .INDENT 7.0 .TP .B checkOnJobs() Check and update status of all running jobs. .sp Respects statePollingWait and will return cached results if not within time period to talk with the scheduler. .UNINDENT .INDENT 7.0 .TP .B run() Run any new jobs .UNINDENT .INDENT 7.0 .TP .B abstract coalesce_job_exit_codes(batch_job_id_list) Returns exit codes and possibly exit reasons for a list of jobs, or None if they are running. .sp Called by AbstractGridEngineWorker.checkOnJobs(). .sp This is an optional part of the interface. It should raise NotImplementedError if not actually implemented for a particular scheduler. .INDENT 7.0 .TP .B Parameters \fBbatch_job_id_list\fP (\fIstring\fP) \-\- List of batch system job ID .TP .B Return type List[Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) Preparation in putting together a command\-line string for submitting to batch system (via submitJob().) .INDENT 7.0 .TP .B Param int cpu .TP .B Param int memory .TP .B Param int jobID: Toil job ID .TP .B Param string subLine: the command line string to be called .TP .B Param string jobName: the name of the Toil job, to provide metadata to batch systems if desired .TP .B Param dict job_environment: the environment variables to be set on the worker .TP .B Return type List[\fI\%str\fP] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBgpus\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract submitJob(subLine) Wrapper routine for submitting the actual command\-line call, then processing the output to get the batch system job ID .INDENT 7.0 .TP .B Param string subLine: the literal command line string to be called .TP .B Return type string: batch system job ID, which will be stored internally .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getRunningJobIDs() Get a list of running job IDs. Implementation\-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() .INDENT 7.0 .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract killJob(jobID) Kill specific job with the Toil job ID. Implementation\-specific; called by AbstractGridEngineWorker.killJobs() .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fIstring\fP) \-\- Toil job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getJobExitCode(batchJobID) Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason. .sp Returns None if the job is still running. .sp If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation\-specific; called by AbstractGridEngineWorker.checkOnJobs(). .sp The exit code will only be 0 if the job affirmatively succeeded. .INDENT 7.0 .TP .B Parameters \fBbatchjobID\fP (\fIstring\fP) \-\- batch system job ID .TP .B Return type Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None] .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod supportsWorkerCleanup() Whether this batch system supports worker cleanup. .sp Indicates whether this batch system invokes \fBBatchSystemSupport.workerCleanup()\fP after the last job for a particular workflow invocation finishes. Note that the term \fIworker\fP refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to \fIshut down\fP after the last worker process terminates. .UNINDENT .INDENT 7.0 .TP .B classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fBsetUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .UNINDENT .INDENT 7.0 .TP .B issueBatchJob(jobDesc, job_environment=None) Issues a job with the specified command to the batch system and returns a unique jobID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP \-\- a toil.job.JobDescription .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- a collection of job\-specific environment variables to be set on the worker. .UNINDENT .TP .B Returns a unique jobID that can be used to reference the newly issued job .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killBatchJobs(jobIDs) Kills the given jobs, represented as Job ids, then checks they are dead by checking they are not in the list of issued jobs. .UNINDENT .INDENT 7.0 .TP .B getIssuedBatchJobIDs() Gets the list of issued jobs .UNINDENT .INDENT 7.0 .TP .B getRunningBatchJobIDs() Retrieve running job IDs from local and batch scheduler. .sp Respects statePollingWait and will return cached results if not within time period to talk with the scheduler. .UNINDENT .INDENT 7.0 .TP .B getUpdatedBatchJob(maxWait) Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once. .sp Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP \-\- the number of seconds to block, waiting for a result .TP .B Returns If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall\-clock time the job ran for, or None if this batch system does not support tracking wall time. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Signals worker to shutdown (via sentinel) then cleanly joins the thread .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setEnv(name, value=None) Set an environment variable for the worker process before it is launched. The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job. .sp If no value is provided it will be looked up from the current environment. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP \-\- the environment variable to be set on the worker. .IP \(bu 2 \fBvalue\fP \-\- if given, the environment variable given by name will be set to this value. If None, the variable\(aqs current value will be used as the value on the worker .UNINDENT .TP .B Raises \fI\%RuntimeError\fP \-\- if value is None and the name cannot be found in the environment .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod getWaitDuration() .UNINDENT .INDENT 7.0 .TP .B sleepSeconds(sleeptime=1) Helper function to drop on all state\-querying functions to avoid over\-querying. .UNINDENT .INDENT 7.0 .TP .B with_retries(operation, *args, **kwargs) Call operation with args and kwargs. If one of the calls to an SGE command fails, sleep and try again for a set number of times. .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.awsBatch\fP .sp Batch system for running Toil workflows on AWS Batch. .sp Useful with the AWS job store. .sp AWS Batch has no means for scheduling based on disk usage, so the backing machines need to have \(dqenough\(dq disk and other constraints need to guarantee that disk does not fill. .sp Assumes that an AWS Batch Queue name or ARN is already provided. .sp Handles creating and destroying a JobDefinition for the workflow run. .sp Additional containers should be launched with Singularity, not Docker. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AWSBatchBatchSystem\fP T} T{ Adds cleanup support when the last running job leaves a node, for batch T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%STATE_TO_EXIT_REASON\fP T} T{ T} _ T{ \fI\%MAX_POLL_COUNT\fP T} T{ T} _ T{ \fI\%MIN_REQUESTABLE_MIB\fP T} T{ T} _ T{ \fI\%MIN_REQUESTABLE_CORES\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.awsBatch.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.awsBatch.STATE_TO_EXIT_REASON: Dict[\fI\%str\fP, \fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP] .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.awsBatch.MAX_POLL_COUNT = 100 .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.awsBatch.MIN_REQUESTABLE_MIB = 4 .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.awsBatch.MIN_REQUESTABLE_CORES = 1 .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.awsBatch.AWSBatchBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.cleanup_support.BatchSystemCleanupSupport\fP .sp Adds cleanup support when the last running job leaves a node, for batch systems that can\(aqt provide it using the backing scheduler. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBmaxCores\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmaxMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxDisk\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fI\%setUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setUserScript(user_script) Set the user script for this workflow. .sp This method must be called before the first job is issued to this batch system, and only if \fI\%supportsAutoDeployment()\fP returns True, otherwise it will raise an exception. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBuserScript\fP \-\- the resource object representing the user script or module and the modules it depends on. .IP \(bu 2 \fBuser_script\fP (\fI\%toil.resource.Resource\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueBatchJob(job_desc, job_environment=None) Issues a job with the specified command to the batch system and returns a unique jobID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP \-\- a toil.job.JobDescription .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- a collection of job\-specific environment variables to be set on the worker. .IP \(bu 2 \fBjob_desc\fP (\fI\%toil.job.JobDescription\fP) \-\- .UNINDENT .TP .B Returns a unique jobID that can be used to reference the newly issued job .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUpdatedBatchJob(maxWait) Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once. .sp Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%int\fP) \-\- the number of seconds to block, waiting for a result .TP .B Returns If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall\-clock time the job ran for, or None if this batch system does not support tracking wall time. .TP .B Return type Optional[\fI\%toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Called at the completion of a toil invocation. Should cleanly terminate all worker threads. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getIssuedBatchJobIDs() Gets all currently issued jobs .INDENT 7.0 .TP .B Returns A list of jobs (as jobIDs) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon. .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningBatchJobIDs() Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds. .INDENT 7.0 .TP .B Returns dictionary with currently running jobID keys and how many seconds they have been running as the value .TP .B Return type Dict[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killBatchJobs(job_ids) Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobIDs\fP \-\- list of IDs of jobs to kill .IP \(bu 2 \fBjob_ids\fP (\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%toil.batchSystems.options.OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.cleanup_support\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BatchSystemCleanupSupport\fP T} T{ Adds cleanup support when the last running job leaves a node, for batch T} _ T{ \fI\%WorkerCleanupContext\fP T} T{ Context manager used by \fI\%BatchSystemCleanupSupport\fP to implement T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.cleanup_support.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.cleanup_support.BatchSystemCleanupSupport(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.local_support.BatchSystemLocalSupport\fP .sp Adds cleanup support when the last running job leaves a node, for batch systems that can\(aqt provide it using the backing scheduler. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBmaxCores\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmaxMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxDisk\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod supportsWorkerCleanup() Whether this batch system supports worker cleanup. .sp Indicates whether this batch system invokes \fBBatchSystemSupport.workerCleanup()\fP after the last job for a particular workflow invocation finishes. Note that the term \fIworker\fP refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to \fIshut down\fP after the last worker process terminates. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getWorkerContexts() Get a list of picklable context manager objects to wrap worker work in, in order. .sp Can be used to ask the Toil worker to do things in\-process (such as configuring environment variables, hot\-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping \(dqexecutor\(dq process. .INDENT 7.0 .TP .B Return type List[ContextManager[Any]] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.cleanup_support.WorkerCleanupContext(workerCleanupInfo) Context manager used by \fI\%BatchSystemCleanupSupport\fP to implement cleanup on a node after the last worker is done working. .sp Gets wrapped around the worker\(aqs work. .INDENT 7.0 .TP .B Parameters \fBworkerCleanupInfo\fP (\fI\%toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __enter__() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __exit__(type, value, traceback) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtype\fP (\fIOptional\fP\fI[\fP\fIType\fP\fI[\fP\fI\%BaseException\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%BaseException\fP\fI]\fP) \-\- .IP \(bu 2 \fBtraceback\fP (\fIOptional\fP\fI[\fP\fI\%types.TracebackType\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.contained_executor\fP .sp Executor for running inside a container. .sp Useful for Kubernetes batch system and TES batch system plugin. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%pack_job\fP(job_desc[, user_script, environment]) T} T{ Create a command that, when run, will execute the given job. T} _ T{ \fI\%executor\fP() T} T{ Main function of the _toil_contained_executor entrypoint. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.contained_executor.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.contained_executor.pack_job(job_desc, user_script=None, environment=None) Create a command that, when run, will execute the given job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_desc\fP (\fI\%toil.job.JobDescription\fP) \-\- Job description for the job to run. .IP \(bu 2 \fBuser_script\fP (\fIOptional\fP\fI[\fP\fI\%toil.resource.Resource\fP\fI]\fP) \-\- User script that will be loaded before the job is run. .IP \(bu 2 \fBenvironment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Environment variable dict that will be applied before the job is run. .UNINDENT .TP .B Returns Command to run the job, as an argument list that can be run inside the Toil appliance container. .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.contained_executor.executor() Main function of the _toil_contained_executor entrypoint. .sp Runs inside the Toil container. .sp Responsible for setting up the user script and running the command for the job (which may in turn invoke the Toil worker entrypoint). .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.gridengine\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%GridEngineBatchSystem\fP T} T{ A partial implementation of BatchSystemSupport for batch systems run on a T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.gridengine.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.gridengine.GridEngineBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem\fP .sp A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto\-deployment is not implemented. .INDENT 7.0 .TP .B class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker\fP .sp Grid Engine\-specific AbstractGridEngineWorker methods .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnewJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBupdatedJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkillQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkilledJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBboss\fP (\fI\%AbstractGridEngineBatchSystem\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningJobIDs() Get a list of running job IDs. Implementation\-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() .INDENT 7.0 .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killJob(jobID) Kill specific job with the Toil job ID. Implementation\-specific; called by AbstractGridEngineWorker.killJobs() .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fIstring\fP) \-\- Toil job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) Preparation in putting together a command\-line string for submitting to batch system (via submitJob().) .INDENT 7.0 .TP .B Param int cpu .TP .B Param int memory .TP .B Param int jobID: Toil job ID .TP .B Param string subLine: the command line string to be called .TP .B Param string jobName: the name of the Toil job, to provide metadata to batch systems if desired .TP .B Param dict job_environment: the environment variables to be set on the worker .TP .B Return type List[\fI\%str\fP] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBgpus\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B submitJob(subLine) Wrapper routine for submitting the actual command\-line call, then processing the output to get the batch system job ID .INDENT 7.0 .TP .B Param string subLine: the literal command line string to be called .TP .B Return type string: batch system job ID, which will be stored internally .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobExitCode(sgeJobID) Get job exist code, checking both qstat and qacct. Return None if still running. Higher level should retry on CalledProcessErrorStderr, for the case the job has finished and qacct result is stale. .UNINDENT .INDENT 7.0 .TP .B prepareQsub(cpu, mem, jobID, job_environment=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmem\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod getWaitDuration() .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.htcondor\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%HTCondorBatchSystem\fP T} T{ A partial implementation of BatchSystemSupport for batch systems run on a T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%JobTuple\fP T} T{ T} _ T{ \fI\%schedd_lock\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.htcondor.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.htcondor.JobTuple .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.htcondor.schedd_lock .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.htcondor.HTCondorBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem\fP .sp A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto\-deployment is not implemented. .INDENT 7.0 .TP .B class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker\fP .sp A class that represents a thread of control. .sp This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnewJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBupdatedJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkillQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkilledJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBboss\fP (\fI\%AbstractGridEngineBatchSystem\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B createJobs(newJob) Create a new job with the given attributes. .sp Implementation\-specific; called by AbstractGridEngineWorker.run() .INDENT 7.0 .TP .B Parameters \fBnewJob\fP (\fIJobTuple\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareSubmission(cpu, memory, disk, jobID, jobName, command, environment) Preparation in putting together a command\-line string for submitting to batch system (via submitJob().) .INDENT 7.0 .TP .B Param int cpu .TP .B Param int memory .TP .B Param int jobID: Toil job ID .TP .B Param string subLine: the command line string to be called .TP .B Param string jobName: the name of the Toil job, to provide metadata to batch systems if desired .TP .B Param dict job_environment: the environment variables to be set on the worker .TP .B Return type List[\fI\%str\fP] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBenvironment\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B submitJob(submitObj) Wrapper routine for submitting the actual command\-line call, then processing the output to get the batch system job ID .INDENT 7.0 .TP .B Param string subLine: the literal command line string to be called .TP .B Return type string: batch system job ID, which will be stored internally .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningJobIDs() Get a list of running job IDs. Implementation\-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() .INDENT 7.0 .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killJob(jobID) Kill specific job with the Toil job ID. Implementation\-specific; called by AbstractGridEngineWorker.killJobs() .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fIstring\fP) \-\- Toil job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobExitCode(batchJobID) Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason. .sp Returns None if the job is still running. .sp If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation\-specific; called by AbstractGridEngineWorker.checkOnJobs(). .sp The exit code will only be 0 if the job affirmatively succeeded. .INDENT 7.0 .TP .B Parameters \fBbatchjobID\fP (\fIstring\fP) \-\- batch system job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B connectSchedd() Connect to HTCondor Schedd and yield a Schedd object. .sp You can only use it inside the context. Handles locking to make sure that only one thread is trying to do this at a time. .UNINDENT .INDENT 7.0 .TP .B duplicate_quotes(value) Escape a string by doubling up all single and double quotes. .sp This is used for arguments we pass to htcondor that need to be inside both double and single quote enclosures. .INDENT 7.0 .TP .B Parameters \fBvalue\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getEnvString(overrides) Build an environment string that a HTCondor Submit object can use. .sp For examples of valid strings, see: \fI\%http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html#man\-condor\-submit\-environment\fP .INDENT 7.0 .TP .B Parameters \fBoverrides\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueBatchJob(jobNode, job_environment=None) Issues a job with the specified command to the batch system and returns a unique jobID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP \-\- a toil.job.JobDescription .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- a collection of job\-specific environment variables to be set on the worker. .UNINDENT .TP .B Returns a unique jobID that can be used to reference the newly issued job .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.kubernetes\fP .sp Batch system for running Toil workflows on Kubernetes. .sp Ony useful with network\-based job stores, like AWSJobStore. .sp Within non\-privileged Kubernetes containers, additional Docker containers cannot yet be launched. That functionality will need to wait for user\-mode Docker .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%KubernetesBatchSystem\fP T} T{ Adds cleanup support when the last running job leaves a node, for batch T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%is_retryable_kubernetes_error\fP(e) T} T{ A function that determines whether or not Toil should retry or stop given T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%retryable_kubernetes_errors\fP T} T{ T} _ T{ \fI\%KeyValuesList\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.kubernetes.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.kubernetes.retryable_kubernetes_errors: List[Type[\fI\%Exception\fP] | \fI\%toil.lib.retry.ErrorCondition\fP] .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.kubernetes.is_retryable_kubernetes_error(e) A function that determines whether or not Toil should retry or stop given exceptions thrown by Kubernetes. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.kubernetes.KeyValuesList .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.kubernetes.KubernetesBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.cleanup_support.BatchSystemCleanupSupport\fP .sp Adds cleanup support when the last running job leaves a node, for batch systems that can\(aqt provide it using the backing scheduler. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBmaxCores\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxDisk\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class DecoratorWrapper(to_wrap, decorator) Class to wrap an object so all its methods are decorated. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBto_wrap\fP (\fIAny\fP) \-\- .IP \(bu 2 \fBdecorator\fP (\fICallable\fP\fI[\fP\fI[\fP\fICallable\fP\fI[\fP\fIP\fP\fI, \fP\fIAny\fP\fI]\fP\fI]\fP\fI, \fP\fICallable\fP\fI[\fP\fIP\fP\fI, \fP\fIAny\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B P .UNINDENT .INDENT 7.0 .TP .B __getattr__(name) Get a member as if we are actually the wrapped object. If it looks callable, we will decorate it. .INDENT 7.0 .TP .B Parameters \fBname\fP (\fI\%str\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class Placement Internal format for pod placement constraints and preferences. .INDENT 7.0 .TP .B required_labels: KeyValuesList = [] Labels which are required to be present (with these values). .UNINDENT .INDENT 7.0 .TP .B desired_labels: KeyValuesList = [] Labels which are optional, but preferred to be present (with these values). .UNINDENT .INDENT 7.0 .TP .B prohibited_labels: KeyValuesList = [] Labels which are not allowed to be present (with these values). .UNINDENT .INDENT 7.0 .TP .B tolerated_taints: KeyValuesList = [] Taints which are allowed to be present (with these values). .UNINDENT .INDENT 7.0 .TP .B set_preemptible(preemptible) Add constraints for a job being preemptible or not. .sp Preemptible jobs will be able to run on preemptible or non\-preemptible nodes, and will prefer preemptible nodes if available. .sp Non\-preemptible jobs will not be allowed to run on nodes that are marked as preemptible. .sp Understands the labeling scheme used by EKS, and the taint scheme used by GCE. The Toil\-managed Kubernetes setup will mimic at least one of these. .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B apply(pod_spec) Set \fBaffinity\fP and/or \fBtolerations\fP fields on pod_spec, so that it runs on the right kind of nodes for the constraints we represent. .INDENT 7.0 .TP .B Parameters \fBpod_spec\fP (\fIkubernetes.client.V1PodSpec\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class KubernetesConfig Bases: \fBProtocol\fP .sp Type\-enforcing protocol for Toil configs that have the extra Kubernetes batch system fields. .sp TODO: Until MyPY lets protocols inherit form non\-protocols, we will have to let the fact that this also has to be a Config just be manually enforced. .INDENT 7.0 .TP .B kubernetes_host_path: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_owner: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_service_account: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_pod_timeout: \fI\%float\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B ItemT .UNINDENT .INDENT 7.0 .TP .B CovItemT .UNINDENT .INDENT 7.0 .TP .B P .UNINDENT .INDENT 7.0 .TP .B R .UNINDENT .INDENT 7.0 .TP .B OptionType .UNINDENT .INDENT 7.0 .TP .B classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fI\%setUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setUserScript(userScript) Set the user script for this workflow. .sp This method must be called before the first job is issued to this batch system, and only if \fI\%supportsAutoDeployment()\fP returns True, otherwise it will raise an exception. .INDENT 7.0 .TP .B Parameters \fBuserScript\fP (\fI\%toil.resource.Resource\fP) \-\- the resource object representing the user script or module and the modules it depends on. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueBatchJob(job_desc, job_environment=None) Issues a job with the specified command to the batch system and returns a unique jobID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP \-\- a toil.job.JobDescription .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- a collection of job\-specific environment variables to be set on the worker. .IP \(bu 2 \fBjob_desc\fP (\fI\%toil.job.JobDescription\fP) \-\- .UNINDENT .TP .B Returns a unique jobID that can be used to reference the newly issued job .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUpdatedBatchJob(maxWait) Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once. .sp Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%float\fP) \-\- the number of seconds to block, waiting for a result .TP .B Returns If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall\-clock time the job ran for, or None if this batch system does not support tracking wall time. .TP .B Return type Optional[\fI\%toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Called at the completion of a toil invocation. Should cleanly terminate all worker threads. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getIssuedBatchJobIDs() Gets all currently issued jobs .INDENT 7.0 .TP .B Returns A list of jobs (as jobIDs) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon. .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningBatchJobIDs() Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds. .INDENT 7.0 .TP .B Returns dictionary with currently running jobID keys and how many seconds they have been running as the value .TP .B Return type Dict[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killBatchJobs(jobIDs) Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob. .INDENT 7.0 .TP .B Parameters \fBjobIDs\fP (\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- list of IDs of jobs to kill .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_default_kubernetes_owner() Get the default Kubernetes\-acceptable username string to tack onto jobs. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%toil.batchSystems.options.OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.local_support\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BatchSystemLocalSupport\fP T} T{ Adds a local queue for helper jobs, useful for CWL & others. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.local_support.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.local_support.BatchSystemLocalSupport(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.abstractBatchSystem.BatchSystemSupport\fP .sp Adds a local queue for helper jobs, useful for CWL & others. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBmaxCores\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmaxMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxDisk\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B handleLocalJob(jobDesc) To be called by issueBatchJobs. .sp Returns the jobID if the jobDesc has been submitted to the local queue, otherwise returns None .INDENT 7.0 .TP .B Parameters \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .TP .B Return type Optional[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killLocalJobs(jobIDs) Will kill all local jobs that match the provided jobIDs. .sp To be called by killBatchJobs. .INDENT 7.0 .TP .B Parameters \fBjobIDs\fP (\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getIssuedLocalJobIDs() To be called by getIssuedBatchJobIDs. .INDENT 7.0 .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningLocalJobIDs() To be called by getRunningBatchJobIDs(). .INDENT 7.0 .TP .B Return type Dict[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUpdatedLocalJob(maxWait) To be called by getUpdatedBatchJob(). .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%int\fP) \-\- .TP .B Return type Optional[\fI\%toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getNextJobID() Must be used to get job IDs so that the local and batch jobs do not conflict. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdownLocal() To be called from shutdown(). .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.lsf\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%LSFBatchSystem\fP T} T{ A partial implementation of BatchSystemSupport for batch systems run on a T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.lsf.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.lsf.LSFBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem\fP .sp A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto\-deployment is not implemented. .INDENT 7.0 .TP .B class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker\fP .sp LSF specific AbstractGridEngineWorker methods. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnewJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBupdatedJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkillQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkilledJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBboss\fP (\fI\%AbstractGridEngineBatchSystem\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningJobIDs() Get a list of running job IDs. Implementation\-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() .INDENT 7.0 .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fallbackRunningJobIDs(currentjobs) .UNINDENT .INDENT 7.0 .TP .B killJob(jobID) Kill specific job with the Toil job ID. Implementation\-specific; called by AbstractGridEngineWorker.killJobs() .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fIstring\fP) \-\- Toil job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) Preparation in putting together a command\-line string for submitting to batch system (via submitJob().) .INDENT 7.0 .TP .B Param int cpu .TP .B Param int memory .TP .B Param int jobID: Toil job ID .TP .B Param string subLine: the command line string to be called .TP .B Param string jobName: the name of the Toil job, to provide metadata to batch systems if desired .TP .B Param dict job_environment: the environment variables to be set on the worker .TP .B Return type List[\fI\%str\fP] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBgpus\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B submitJob(subLine) Wrapper routine for submitting the actual command\-line call, then processing the output to get the batch system job ID .INDENT 7.0 .TP .B Param string subLine: the literal command line string to be called .TP .B Return type string: batch system job ID, which will be stored internally .UNINDENT .UNINDENT .INDENT 7.0 .TP .B coalesce_job_exit_codes(batch_job_id_list) Returns exit codes and possibly exit reasons for a list of jobs, or None if they are running. .sp Called by AbstractGridEngineWorker.checkOnJobs(). .sp This is an optional part of the interface. It should raise NotImplementedError if not actually implemented for a particular scheduler. .INDENT 7.0 .TP .B Parameters \fBbatch_job_id_list\fP (\fIstring\fP) \-\- List of batch system job ID .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobExitCode(lsfJobID) Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason. .sp Returns None if the job is still running. .sp If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation\-specific; called by AbstractGridEngineWorker.checkOnJobs(). .sp The exit code will only be 0 if the job affirmatively succeeded. .INDENT 7.0 .TP .B Parameters \fBbatchjobID\fP (\fIstring\fP) \-\- batch system job ID .TP .B Return type Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B parse_bjobs_record(bjobs_record, job) Helper functions for getJobExitCode and to parse the bjobs status record .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbjobs_record\fP (\fI\%dict\fP) \-\- .IP \(bu 2 \fBjob\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobExitCodeBACCT(job) .INDENT 7.0 .TP .B Return type Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fallbackGetJobExitCode(job) .INDENT 7.0 .TP .B Return type Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareBsub(cpu, mem, jobID) Make a bsub commandline to execute. .INDENT 7.0 .TP .B params: cpu: number of cores needed mem: number of bytes of memory needed jobID: ID number of the job .UNINDENT .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmem\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B parseBjobs(bjobs_output_str) Parse records from bjobs json type output .INDENT 7.0 .TP .B Params bjobs_output_str stdout of bjobs json type output .UNINDENT .UNINDENT .INDENT 7.0 .TP .B parseMaxMem(jobID) Parse the maximum memory from job. .INDENT 7.0 .TP .B Parameters \fBjobID\fP \-\- ID number of the job .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getWaitDuration() We give LSF a second to catch its breath (in seconds) .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.lsfHelper\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%find\fP(basedir, string) T} T{ walk basedir and return all files matching string T} _ T{ \fI\%find_first_match\fP(basedir, string) T} T{ return the first file that matches string starting from basedir T} _ T{ \fI\%get_conf_file\fP(filename, env) T} T{ T} _ T{ \fI\%apply_conf_file\fP(fn, conf_filename) T} T{ T} _ T{ \fI\%per_core_reserve_from_stream\fP(stream) T} T{ T} _ T{ \fI\%get_lsf_units_from_stream\fP(stream) T} T{ T} _ T{ \fI\%tokenize_conf_stream\fP(conf_handle) T} T{ convert the key=val pairs in a LSF config stream to tuples of tokens T} _ T{ \fI\%apply_bparams\fP(fn) T} T{ apply fn to each line of bparams, returning the result T} _ T{ \fI\%apply_lsadmin\fP(fn) T} T{ apply fn to each line of lsadmin, returning the result T} _ T{ \fI\%get_lsf_units\fP([resource]) T} T{ check if we can find LSF_UNITS_FOR_LIMITS in lsadmin and lsf.conf T} _ T{ \fI\%parse_mem_and_cmd_from_output\fP(output) T} T{ Use regex to find \(dqMAX MEM\(dq and \(dqCommand\(dq inside of an output. T} _ T{ \fI\%get_lsf_version\fP() T} T{ Get current LSF version T} _ T{ \fI\%check_lsf_json_output_supported\fP() T} T{ Check if the current LSF system supports bjobs json output. T} _ T{ \fI\%parse_memory\fP(mem) T} T{ Parse memory parameter. T} _ T{ \fI\%per_core_reservation\fP() T} T{ returns True if the cluster is configured for reservations to be per core, T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%LSB_PARAMS_FILENAME\fP T} T{ T} _ T{ \fI\%LSF_CONF_FILENAME\fP T} T{ T} _ T{ \fI\%LSF_CONF_ENV\fP T} T{ T} _ T{ \fI\%DEFAULT_LSF_UNITS\fP T} T{ T} _ T{ \fI\%DEFAULT_RESOURCE_UNITS\fP T} T{ T} _ T{ \fI\%LSF_JSON_OUTPUT_MIN_VERSION\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.LSB_PARAMS_FILENAME = \(aqlsb.params\(aq .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.LSF_CONF_FILENAME = \(aqlsf.conf\(aq .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.LSF_CONF_ENV = [\(aqLSF_CONFDIR\(aq, \(aqLSF_ENVDIR\(aq] .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.DEFAULT_LSF_UNITS = \(aqKB\(aq .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.DEFAULT_RESOURCE_UNITS = \(aqMB\(aq .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.LSF_JSON_OUTPUT_MIN_VERSION = \(aq10.1.0.2\(aq .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.find(basedir, string) walk basedir and return all files matching string .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.find_first_match(basedir, string) return the first file that matches string starting from basedir .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.get_conf_file(filename, env) .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.apply_conf_file(fn, conf_filename) .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.per_core_reserve_from_stream(stream) .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.get_lsf_units_from_stream(stream) .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.tokenize_conf_stream(conf_handle) convert the key=val pairs in a LSF config stream to tuples of tokens .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.apply_bparams(fn) apply fn to each line of bparams, returning the result .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.apply_lsadmin(fn) apply fn to each line of lsadmin, returning the result .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.get_lsf_units(resource=False) check if we can find LSF_UNITS_FOR_LIMITS in lsadmin and lsf.conf files, preferring the value in bparams, then lsadmin, then the lsf.conf file .INDENT 7.0 .TP .B Parameters \fBresource\fP (\fI\%bool\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.parse_mem_and_cmd_from_output(output) Use regex to find \(dqMAX MEM\(dq and \(dqCommand\(dq inside of an output. .INDENT 7.0 .TP .B Parameters \fBoutput\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.get_lsf_version() Get current LSF version .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.check_lsf_json_output_supported() Check if the current LSF system supports bjobs json output. .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.parse_memory(mem) Parse memory parameter. .INDENT 7.0 .TP .B Parameters \fBmem\fP (\fI\%float\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.lsfHelper.per_core_reservation() returns True if the cluster is configured for reservations to be per core, False if it is per job .UNINDENT .SS \fI\%toil.batchSystems.options\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%OptionSetter\fP T} T{ Protocol for the setOption function we get to let us set up CLI options for T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%set_batchsystem_options\fP(batch_system, set_option) T} T{ Call set_option for all the options for the given named batch system, or T} _ T{ \fI\%add_all_batchsystem_options\fP(parser) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.options.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.options.OptionSetter Bases: \fBProtocol\fP .sp Protocol for the setOption function we get to let us set up CLI options for each batch system. .sp Actual functionality is defined in the Config class. .INDENT 7.0 .TP .B OptionType .UNINDENT .INDENT 7.0 .TP .B __call__(option_name, parsing_function=None, check_function=None, default=None, env=None, old_names=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBoption_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBparsing_function\fP (\fIOptional\fP\fI[\fP\fICallable\fP\fI[\fP\fI[\fP\fIAny\fP\fI]\fP\fI, \fP\fIOptionType\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBcheck_function\fP (\fIOptional\fP\fI[\fP\fICallable\fP\fI[\fP\fI[\fP\fIOptionType\fP\fI]\fP\fI, \fP\fIUnion\fP\fI[\fP\fINone\fP\fI, \fP\fI\%bool\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBdefault\fP (\fIOptional\fP\fI[\fP\fIOptionType\fP\fI]\fP) \-\- .IP \(bu 2 \fBenv\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBold_names\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.options.set_batchsystem_options(batch_system, set_option) Call set_option for all the options for the given named batch system, or all batch systems if no name is provided. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbatch_system\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBset_option\fP (\fI\%OptionSetter\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.options.add_all_batchsystem_options(parser) .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.registry\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%add_batch_system_factory\fP(key, class_factory) T} T{ Adds a batch system to the registry for workflow or plugin\-supplied batch systems. T} _ T{ \fI\%get_batch_systems\fP() T} T{ Get the names of all the availsble batch systems. T} _ T{ \fI\%get_batch_system\fP(key) T} T{ Get a batch system class by name. T} _ T{ \fI\%aws_batch_batch_system_factory\fP() T} T{ T} _ T{ \fI\%gridengine_batch_system_factory\fP() T} T{ T} _ T{ \fI\%lsf_batch_system_factory\fP() T} T{ T} _ T{ \fI\%single_machine_batch_system_factory\fP() T} T{ T} _ T{ \fI\%mesos_batch_system_factory\fP() T} T{ T} _ T{ \fI\%slurm_batch_system_factory\fP() T} T{ T} _ T{ \fI\%torque_batch_system_factory\fP() T} T{ T} _ T{ \fI\%htcondor_batch_system_factory\fP() T} T{ T} _ T{ \fI\%kubernetes_batch_system_factory\fP() T} T{ T} _ T{ \fI\%__getattr__\fP(name) T} T{ Implement a fallback attribute getter to handle deprecated constants. T} _ T{ \fI\%addBatchSystemFactory\fP(key, batchSystemFactory) T} T{ Deprecated method to add a batch system. T} _ T{ \fI\%save_batch_system_plugin_state\fP() T} T{ Return a snapshot of the plugin registry that can be restored to remove T} _ T{ \fI\%restore_batch_system_plugin_state\fP(snapshot) T} T{ Restore the batch system registry state to a snapshot from T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%DEFAULT_BATCH_SYSTEM\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.registry.logger .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.add_batch_system_factory(key, class_factory) Adds a batch system to the registry for workflow or plugin\-supplied batch systems. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBclass_factory\fP (\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIType\fP\fI[\fP\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP\fI]\fP\fI]\fP) \-\- A function that returns a batch system class (NOT an instance), which implements \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP\&. .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.get_batch_systems() Get the names of all the availsble batch systems. .INDENT 7.0 .TP .B Return type Sequence[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.get_batch_system(key) Get a batch system class by name. .INDENT 7.0 .TP .B Raises KeyError if the key is not the name of a batch system, and ImportError if the batch system\(aqs class cannot be loaded. .TP .B Parameters \fBkey\fP (\fI\%str\fP) \-\- .TP .B Return type Type[\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.DEFAULT_BATCH_SYSTEM = \(aqsingle_machine\(aq .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.aws_batch_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.gridengine_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.lsf_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.single_machine_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.mesos_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.slurm_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.torque_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.htcondor_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.kubernetes_batch_system_factory() .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.__getattr__(name) Implement a fallback attribute getter to handle deprecated constants. .sp See <\fI\%https://stackoverflow.com/a/48242860\fP>. .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.addBatchSystemFactory(key, batchSystemFactory) Deprecated method to add a batch system. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBbatchSystemFactory\fP (\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIType\fP\fI[\fP\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.save_batch_system_plugin_state() Return a snapshot of the plugin registry that can be restored to remove added plugins. Useful for testing the plugin system in\-process with other tests. .INDENT 7.0 .TP .B Return type Tuple[List[\fI\%str\fP], Dict[\fI\%str\fP, Callable[[], Type[\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP]]]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.batchSystems.registry.restore_batch_system_plugin_state(snapshot) Restore the batch system registry state to a snapshot from save_batch_system_plugin_state(). .INDENT 7.0 .TP .B Parameters \fBsnapshot\fP (\fITuple\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIType\fP\fI[\fP\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP\fI]\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.singleMachine\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%SingleMachineBatchSystem\fP T} T{ The interface for running jobs on a single machine, runs all the jobs you T} _ T{ \fI\%Info\fP T} T{ Record for a running job. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.singleMachine.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.singleMachine.SingleMachineBatchSystem(config, maxCores, maxMemory, maxDisk, max_jobs=None) Bases: \fI\%toil.batchSystems.abstractBatchSystem.BatchSystemSupport\fP .sp The interface for running jobs on a single machine, runs all the jobs you give it as they come in, but in parallel. .sp Uses a single \(dqdaddy\(dq thread to manage a fleet of child processes. .sp Communication with the daddy thread happens via two queues: one queue of jobs waiting to be run (the input queue), and one queue of jobs that are finished/stopped and need to be returned by getUpdatedBatchJob (the output queue). .sp When the batch system is shut down, the daddy thread is stopped. .sp If running in debug\-worker mode, jobs are run immediately as they are sent to the batch system, in the sending thread, and the daddy thread is not run. But the queues are still used. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBmaxCores\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmaxMemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmaxDisk\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmax_jobs\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B numCores .UNINDENT .INDENT 7.0 .TP .B minCores = 0.1 The minimal fractional CPU. Tasks with a smaller core requirement will be rounded up to this value. .UNINDENT .INDENT 7.0 .TP .B physicalMemory .UNINDENT .INDENT 7.0 .TP .B classmethod supportsAutoDeployment() Whether this batch system supports auto\-deployment of the user script itself. .sp If it does, the \fBsetUserScript()\fP can be invoked to set the resource object representing the user script. .sp Note to implementors: If your implementation returns True here, it should also override .UNINDENT .INDENT 7.0 .TP .B classmethod supportsWorkerCleanup() Whether this batch system supports worker cleanup. .sp Indicates whether this batch system invokes \fBBatchSystemSupport.workerCleanup()\fP after the last job for a particular workflow invocation finishes. Note that the term \fIworker\fP refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to \fIshut down\fP after the last worker process terminates. .UNINDENT .INDENT 7.0 .TP .B daddy() Be the \(dqdaddy\(dq thread. .sp Our job is to look at jobs from the input queue. .sp If a job fits in the available resources, we allocate resources for it and kick off a child process. .sp We also check on our children. .sp When a child finishes, we reap it, release its resources, and put its information in the output queue. .UNINDENT .INDENT 7.0 .TP .B getSchedulingStatusMessage() Get a log message fragment for the user about anything that might be going wrong in the batch system, if available. .sp If no useful message is available, return None. .sp This can be used to report what resource is the limiting factor when scheduling jobs, for example. If the leader thinks the workflow is stuck, the message can be displayed to the user to help them diagnose why it might be stuck. .INDENT 7.0 .TP .B Returns User\-directed message about scheduling state. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_resource_request(requirer) Check resource request is not greater than that available or allowed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirer\fP (\fI\%toil.job.Requirer\fP) \-\- Object whose requirements are being checked .IP \(bu 2 \fBjob_name\fP (\fI\%str\fP) \-\- Name of the job being checked, for generating a useful error report. .IP \(bu 2 \fBdetail\fP (\fI\%str\fP) \-\- Batch\-system\-specific message to include in the error. .UNINDENT .TP .B Raises \fI\%InsufficientSystemResources\fP \-\- raised when a resource is requested in an amount greater than allowed .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueBatchJob(jobDesc, job_environment=None) Adds the command and resources to a queue to be run. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killBatchJobs(jobIDs) Kills jobs by ID. .INDENT 7.0 .TP .B Parameters \fBjobIDs\fP (\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getIssuedBatchJobIDs() Just returns all the jobs that have been run, but not yet returned as updated. .INDENT 7.0 .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningBatchJobIDs() Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds. .INDENT 7.0 .TP .B Returns dictionary with currently running jobID keys and how many seconds they have been running as the value .TP .B Return type Dict[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Terminate cleanly and join daddy thread. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUpdatedBatchJob(maxWait) Returns a tuple of a no\-longer\-running job, the return value of its process, and its runtime, or None. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%int\fP) \-\- .TP .B Return type Optional[\fI\%toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%toil.batchSystems.options.OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.singleMachine.Info(startTime, popen, resources, killIntended) Record for a running job. .sp Stores the start time of the job, the Popen object representing its child (or None), the tuple of (coreFractions, memory, disk) it is using (or None), and whether the job is supposed to be being killed. .UNINDENT .SS \fI\%toil.batchSystems.slurm\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%SlurmBatchSystem\fP T} T{ A partial implementation of BatchSystemSupport for batch systems run on a T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.slurm.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.slurm.SlurmBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem\fP .sp A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto\-deployment is not implemented. .INDENT 7.0 .TP .B class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker\fP .sp A class that represents a thread of control. .sp This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnewJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBupdatedJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkillQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBkilledJobsQueue\fP (\fI\%queue.Queue\fP) \-\- .IP \(bu 2 \fBboss\fP (\fI\%AbstractGridEngineBatchSystem\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRunningJobIDs() Get a list of running job IDs. Implementation\-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() .INDENT 7.0 .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killJob(jobID) Kill specific job with the Toil job ID. Implementation\-specific; called by AbstractGridEngineWorker.killJobs() .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fIstring\fP) \-\- Toil job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) Preparation in putting together a command\-line string for submitting to batch system (via submitJob().) .INDENT 7.0 .TP .B Param int cpu .TP .B Param int memory .TP .B Param int jobID: Toil job ID .TP .B Param string subLine: the command line string to be called .TP .B Param string jobName: the name of the Toil job, to provide metadata to batch systems if desired .TP .B Param dict job_environment: the environment variables to be set on the worker .TP .B Return type List[\fI\%str\fP] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBgpus\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B submitJob(subLine) Wrapper routine for submitting the actual command\-line call, then processing the output to get the batch system job ID .INDENT 7.0 .TP .B Param string subLine: the literal command line string to be called .TP .B Return type string: batch system job ID, which will be stored internally .UNINDENT .UNINDENT .INDENT 7.0 .TP .B coalesce_job_exit_codes(batch_job_id_list) Collect all job exit codes in a single call. :param batch_job_id_list: list of Job ID strings, where each string has the form \(dq[.]\(dq. :return: list of job exit codes or exit code, exit reason pairs associated with the list of job IDs. .INDENT 7.0 .TP .B Parameters \fBbatch_job_id_list\fP (\fI\%list\fP) \-\- .TP .B Return type List[Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobExitCode(batchJobID) Get job exit code for given batch job ID. :param batchJobID: string of the form \(dq[.]\(dq. :return: integer job exit code. .INDENT 7.0 .TP .B Parameters \fBbatchJobID\fP (\fI\%str\fP) \-\- .TP .B Return type Union[\fI\%int\fP, Tuple[\fI\%int\fP, Optional[\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP]], None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareSbatch(cpu, mem, jobID, jobName, job_environment, gpus) Returns the sbatch command line to run to queue the job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmem\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBgpus\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B parse_elapsed(elapsed) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B OptionType .UNINDENT .INDENT 7.0 .TP .B classmethod add_options(parser) If this batch system provides any command line options, add them to the given parser. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%argparse.ArgumentParser\fP\fI, \fP\fIargparse._ArgumentGroup\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setOptions(setOption) Process command line or configuration options relevant to this batch system. .INDENT 7.0 .TP .B Parameters \fBsetOption\fP (\fI\%toil.batchSystems.options.OptionSetter\fP) \-\- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect. .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.batchSystems.torque\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%TorqueBatchSystem\fP T} T{ A partial implementation of BatchSystemSupport for batch systems run on a T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.batchSystems.torque.logger .UNINDENT .INDENT 0.0 .TP .B class toil.batchSystems.torque.TorqueBatchSystem(config, maxCores, maxMemory, maxDisk) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem\fP .sp A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto\-deployment is not implemented. .INDENT 7.0 .TP .B class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss) Bases: \fI\%toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker\fP .sp A class that represents a thread of control. .sp This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass. .INDENT 7.0 .TP .B getRunningJobIDs() Get a list of running job IDs. Implementation\-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs() .INDENT 7.0 .TP .B Return type \fI\%list\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUpdatedBatchJob(maxWait) .UNINDENT .INDENT 7.0 .TP .B killJob(jobID) Kill specific job with the Toil job ID. Implementation\-specific; called by AbstractGridEngineWorker.killJobs() .INDENT 7.0 .TP .B Parameters \fBjobID\fP (\fIstring\fP) \-\- Toil job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None) Preparation in putting together a command\-line string for submitting to batch system (via submitJob().) .INDENT 7.0 .TP .B Param int cpu .TP .B Param int memory .TP .B Param int jobID: Toil job ID .TP .B Param string subLine: the command line string to be called .TP .B Param string jobName: the name of the Toil job, to provide metadata to batch systems if desired .TP .B Param dict job_environment: the environment variables to be set on the worker .TP .B Return type List[\fI\%str\fP] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBgpus\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B submitJob(subLine) Wrapper routine for submitting the actual command\-line call, then processing the output to get the batch system job ID .INDENT 7.0 .TP .B Param string subLine: the literal command line string to be called .TP .B Return type string: batch system job ID, which will be stored internally .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobExitCode(torqueJobID) Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason. .sp Returns None if the job is still running. .sp If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation\-specific; called by AbstractGridEngineWorker.checkOnJobs(). .sp The exit code will only be 0 if the job affirmatively succeeded. .INDENT 7.0 .TP .B Parameters \fBbatchjobID\fP (\fIstring\fP) \-\- batch system job ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareQsub(cpu, mem, jobID, job_environment) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpu\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmem\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjobID\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBjob_environment\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B generateTorqueWrapper(command, jobID) A very simple script generator that just wraps the command given; for now this goes to default tempdir .UNINDENT .UNINDENT .UNINDENT .SS Package Contents .INDENT 0.0 .TP .B exception toil.batchSystems.DeadlockException(msg) Bases: \fI\%Exception\fP .sp Exception thrown by the Leader or BatchSystem when a deadlock is encountered due to insufficient resources to run the workflow .INDENT 7.0 .TP .B __str__() Stringify the exception, including the message. .UNINDENT .UNINDENT .SS \fI\%toil.cwl\fP .SS Submodules .SS \fI\%toil.cwl.conftest\fP .SS Module Contents .INDENT 0.0 .TP .B toil.cwl.conftest.collect_ignore = [] .UNINDENT .SS \fI\%toil.cwl.cwltoil\fP .sp Implemented support for Common Workflow Language (CWL) for Toil. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%UnresolvedDict\fP T} T{ Tag to indicate a dict contains promises that must be resolved. T} _ T{ \fI\%SkipNull\fP T} T{ Internal sentinel object. T} _ T{ \fI\%Conditional\fP T} T{ Object holding conditional expression until we are ready to evaluate it. T} _ T{ \fI\%ResolveSource\fP T} T{ Apply linkMerge and pickValue operators to values coming into a port. T} _ T{ \fI\%StepValueFrom\fP T} T{ A workflow step input which has a valueFrom expression attached to it. T} _ T{ \fI\%DefaultWithSource\fP T} T{ A workflow step input that has both a source and a default value. T} _ T{ \fI\%JustAValue\fP T} T{ A simple value masquerading as a \(aqresolve\(aq\-able object. T} _ T{ \fI\%ToilPathMapper\fP T} T{ Keeps track of files in a Toil way. T} _ T{ \fI\%ToilSingleJobExecutor\fP T} T{ A SingleJobExecutor that does not assume it is at the top level of the workflow. T} _ T{ \fI\%ToilTool\fP T} T{ Mixin to hook Toil into a cwltool tool type. T} _ T{ \fI\%ToilCommandLineTool\fP T} T{ Subclass the cwltool command line tool to provide the custom ToilPathMapper. T} _ T{ \fI\%ToilExpressionTool\fP T} T{ Subclass the cwltool expression tool to provide the custom ToilPathMapper. T} _ T{ \fI\%ToilFsAccess\fP T} T{ Custom filesystem access class which handles toil filestore references. T} _ T{ \fI\%CWLNamedJob\fP T} T{ Base class for all CWL jobs that do user work, to give them useful names. T} _ T{ \fI\%ResolveIndirect\fP T} T{ Helper Job. T} _ T{ \fI\%CWLJobWrapper\fP T} T{ Wrap a CWL job that uses dynamic resources requirement. T} _ T{ \fI\%CWLJob\fP T} T{ Execute a CWL tool using cwltool.executors.SingleJobExecutor. T} _ T{ \fI\%CWLScatter\fP T} T{ Implement workflow scatter step. T} _ T{ \fI\%CWLGather\fP T} T{ Follows on to a scatter Job. T} _ T{ \fI\%SelfJob\fP T} T{ Fake job object to facilitate implementation of CWLWorkflow.run(). T} _ T{ \fI\%CWLWorkflow\fP T} T{ Toil Job to convert a CWL workflow graph into a Toil job graph. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%cwltoil_was_removed\fP() T} T{ Complain about deprecated entrypoint. T} _ T{ \fI\%filter_skip_null\fP(name, value) T} T{ Recursively filter out SkipNull objects from \(aqvalue\(aq. T} _ T{ \fI\%ensure_no_collisions\fP(directory[, dir_description]) T} T{ Make sure no items in the given CWL Directory have the same name. T} _ T{ \fI\%resolve_dict_w_promises\fP(dict_w_promises[, file_store]) T} T{ Resolve a dictionary of promises evaluate expressions to produce the actual values. T} _ T{ \fI\%simplify_list\fP(maybe_list) T} T{ Turn a length one list loaded by cwltool into a scalar. T} _ T{ \fI\%toil_make_tool\fP(toolpath_object, loadingContext) T} T{ Emit custom ToilCommandLineTools. T} _ T{ \fI\%check_directory_dict_invariants\fP(contents) T} T{ Make sure a directory structure dict makes sense. Throws an error T} _ T{ \fI\%decode_directory\fP(dir_path) T} T{ Decode a directory from a \(dqtoildir:\(dq path to a directory (or a file in it). T} _ T{ \fI\%encode_directory\fP(contents) T} T{ Encode a directory from a \(dqtoildir:\(dq path to a directory (or a file in it). T} _ T{ \fI\%toil_get_file\fP(file_store, index, existing, uri[, ...]) T} T{ Set up the given file or directory from the Toil jobstore at a file URI T} _ T{ \fI\%write_file\fP(writeFunc, index, existing, file_uri) T} T{ Write a file into the Toil jobstore. T} _ T{ \fI\%path_to_loc\fP(obj) T} T{ Make a path into a location. T} _ T{ \fI\%import_files\fP(import_function, fs_access, fileindex, ...) T} T{ Prepare all files and directories. T} _ T{ \fI\%upload_directory\fP(directory_metadata, directory_contents) T} T{ Upload a Directory object. T} _ T{ \fI\%upload_file\fP(uploadfunc, fileindex, existing, file_metadata) T} T{ Update a file object so that the file will be accessible from another machine. T} _ T{ \fI\%writeGlobalFileWrapper\fP(file_store, fileuri) T} T{ Wrap writeGlobalFile to accept \fI\%file://\fP URIs. T} _ T{ \fI\%remove_empty_listings\fP(rec) T} T{ T} _ T{ \fI\%toilStageFiles\fP(toil, cwljob, outdir[, destBucket, ...]) T} T{ Copy input files out of the global file store and update location and path. T} _ T{ \fI\%get_container_engine\fP(runtime_context) T} T{ T} _ T{ \fI\%makeJob\fP(tool, jobobj, runtime_context, parent_name, ...) T} T{ Create the correct Toil Job object for the CWL tool. T} _ T{ \fI\%remove_pickle_problems\fP(obj) T} T{ Doc_loader does not pickle correctly, causing Toil errors, remove from objects. T} _ T{ \fI\%visitSteps\fP(cmdline_tool, op) T} T{ Iterate over a CWL Process object, running the op on each tool description T} _ T{ \fI\%rm_unprocessed_secondary_files\fP(job_params) T} T{ T} _ T{ \fI\%filtered_secondary_files\fP(unfiltered_secondary_files) T} T{ Remove unprocessed secondary files. T} _ T{ \fI\%scan_for_unsupported_requirements\fP(tool[, ...]) T} T{ Scan the given CWL tool for any unsupported optional features. T} _ T{ \fI\%determine_load_listing\fP(tool) T} T{ Determine the directory.listing feature in CWL. T} _ T{ \fI\%generate_default_job_store\fP(batch_system_name, ...) T} T{ Choose a default job store appropriate to the requested batch system and T} _ T{ \fI\%get_options\fP(args) T} T{ Parse given args and properly add non\-Toil arguments into the cwljob of the Namespace. T} _ T{ \fI\%main\fP([args, stdout]) T} T{ Run the main loop for toil\-cwl\-runner. T} _ T{ \fI\%find_default_container\fP(args, builder) T} T{ Find the default constructor by consulting a Toil.options object. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%DEFAULT_TMPDIR\fP T} T{ T} _ T{ \fI\%DEFAULT_TMPDIR_PREFIX\fP T} T{ T} _ T{ \fI\%DirectoryContents\fP T} T{ T} _ T{ \fI\%ProcessType\fP T} T{ T} _ T{ \fI\%usage_message\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.cwl.cwltoil.logger .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.DEFAULT_TMPDIR .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.DEFAULT_TMPDIR_PREFIX .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.cwltoil_was_removed() Complain about deprecated entrypoint. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.UnresolvedDict Bases: \fBDict\fP[\fBAny\fP, \fBAny\fP] .sp Tag to indicate a dict contains promises that must be resolved. .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.SkipNull Internal sentinel object. .sp Indicates a null value produced by each port of a skipped conditional step. The CWL 1.2 specification calls for treating this the exactly the same as a null value. .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.filter_skip_null(name, value) Recursively filter out SkipNull objects from \(aqvalue\(aq. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- Name of port producing this value. Only used when we find an unhandled null from a conditional step and we print out a warning. The name allows the user to better localize which step/port was responsible for the unhandled null. .IP \(bu 2 \fBvalue\fP (\fIAny\fP) \-\- port output value object .UNINDENT .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.ensure_no_collisions(directory, dir_description=None) Make sure no items in the given CWL Directory have the same name. .sp If any do, raise a WorkflowException about a \(dqFile staging conflict\(dq. .sp Does not recurse into subdirectories. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdirectory\fP (\fIcwltool.utils.DirectoryType\fP) \-\- .IP \(bu 2 \fBdir_description\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.Conditional(expression=None, outputs=None, requirements=None, container_engine=\(aqdocker\(aq) Object holding conditional expression until we are ready to evaluate it. .sp Evaluation occurs at the moment the encloses step is ready to run. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexpression\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBoutputs\fP (\fIUnion\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIcwltool.utils.CWLOutputType\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBrequirements\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBcontainer_engine\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_false(job) Determine if expression evaluates to False given completed step inputs. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- job output object .TP .B Returns bool .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B skipped_outputs() Generate a dict of SkipNull objects corresponding to the output structure. .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, \fI\%SkipNull\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ResolveSource(name, input, source_key, promises) Apply linkMerge and pickValue operators to values coming into a port. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBinput\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- .IP \(bu 2 \fBsource_key\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBpromises\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%toil.job.Job\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B promise_tuples: List[Tuple[\fI\%str\fP, \fI\%toil.job.Promise\fP]] | Tuple[\fI\%str\fP, \fI\%toil.job.Promise\fP] .UNINDENT .INDENT 7.0 .TP .B __repr__() Allow for debug printing. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resolve() First apply linkMerge then pickValue if either present. .INDENT 7.0 .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B link_merge(values) Apply linkMerge operator to \fIvalues\fP object. .INDENT 7.0 .TP .B Parameters \fBvalues\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- result of step .TP .B Return type Union[List[cwltool.utils.CWLOutputType], cwltool.utils.CWLOutputType] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pick_value(values) Apply pickValue operator to \fIvalues\fP object. .INDENT 7.0 .TP .B Parameters \fBvalues\fP (\fIUnion\fP\fI[\fP\fIList\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%SkipNull\fP\fI]\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- Intended to be a list, but other types will be returned without modification. .TP .B Returns .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.StepValueFrom(expr, source, req, container_engine) A workflow step input which has a valueFrom expression attached to it. .sp The valueFrom expression will be evaluated to produce the actual input object for the step. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexpr\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsource\fP (\fIAny\fP) \-\- .IP \(bu 2 \fBreq\fP (\fIList\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- .IP \(bu 2 \fBcontainer_engine\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Allow for debug printing. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B eval_prep(step_inputs, file_store) Resolve the contents of any file in a set of inputs. .sp The inputs must be associated with the StepValueFrom object\(aqs self.source. .sp Called when loadContents is specified. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstep_inputs\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- Workflow step inputs. .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- A toil file store, needed to resolve toilfile:// paths. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resolve() Resolve the promise in the valueFrom expression\(aqs context. .INDENT 7.0 .TP .B Returns object that will serve as expression context .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B do_eval(inputs) Evaluate the valueFrom expression with the given input object. .INDENT 7.0 .TP .B Parameters \fBinputs\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .TP .B Returns object .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.DefaultWithSource(default, source) A workflow step input that has both a source and a default value. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdefault\fP (\fIAny\fP) \-\- .IP \(bu 2 \fBsource\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Allow for debug printing. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resolve() Determine the final input value when the time is right. .sp (when the source can be resolved) .INDENT 7.0 .TP .B Returns dict .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.JustAValue(val) A simple value masquerading as a \(aqresolve\(aq\-able object. .INDENT 7.0 .TP .B Parameters \fBval\fP (\fIAny\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __repr__() Allow for debug printing. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resolve() Return the value. .INDENT 7.0 .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.resolve_dict_w_promises(dict_w_promises, file_store=None) Resolve a dictionary of promises evaluate expressions to produce the actual values. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdict_w_promises\fP (\fIUnion\fP\fI[\fP\fI\%UnresolvedDict\fP\fI, \fP\fIcwltool.utils.CWLObjectType\fP\fI, \fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%StepValueFrom\fP\fI]\fP\fI]\fP\fI]\fP) \-\- input dict for these values .IP \(bu 2 \fBfile_store\fP (\fIOptional\fP\fI[\fP\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns dictionary of actual values .TP .B Return type cwltool.utils.CWLObjectType .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.simplify_list(maybe_list) Turn a length one list loaded by cwltool into a scalar. .sp Anything else is passed as\-is, by reference. .INDENT 7.0 .TP .B Parameters \fBmaybe_list\fP (\fIAny\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ToilPathMapper(referenced_files, basedir, stagedir, separateDirs=True, get_file=None, stage_listing=False, streaming_allowed=True) Bases: \fBcwltool.pathmapper.PathMapper\fP .sp Keeps track of files in a Toil way. .sp Maps between the symbolic identifier of a file (the Toil FileID), its local path on the host (the value returned by readGlobalFile) and the location of the file inside the software container. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBreferenced_files\fP (\fIList\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- .IP \(bu 2 \fBbasedir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBstagedir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBseparateDirs\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBget_file\fP (\fIUnion\fP\fI[\fP\fIAny\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBstage_listing\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBstreaming_allowed\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B visit(obj, stagedir, basedir, copy=False, staged=False) Iterate over a CWL object, resolving File and Directory path references. .sp This is called on each File or Directory CWL object. The Files and Directories all have \(dqlocation\(dq fields. For the Files, these are from upload_file(), and for the Directories, these are from upload_directory() or cwltool internally. With upload_directory(), they and their children will be assigned locations based on listing the Directories using ToilFsAccess. With cwltool, locations will be set as absolute paths. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBobj\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- The CWL File or Directory to process .IP \(bu 2 \fBstagedir\fP (\fI\%str\fP) \-\- The base path for target paths to be generated under, except when a File or Directory has an overriding parent directory in dirname .IP \(bu 2 \fBbasedir\fP (\fI\%str\fP) \-\- The directory from which relative paths should be resolved; used as the base directory for the StdFsAccess that generated the listing being processed. .IP \(bu 2 \fBcopy\fP (\fI\%bool\fP) \-\- If set, use writable types for Files and Directories. .IP \(bu 2 \fBstaged\fP (\fI\%bool\fP) \-\- Starts as True at the top of the recursion. Set to False when entering a directory that we can actually download, so we don\(aqt stage files and subdirectories separately from the directory as a whole. Controls the staged flag on generated mappings, and therefore whether files and directories are actually placed at their mapped\-to target locations. If stage_listing is True, we will leave this True throughout and stage everything. .UNINDENT .TP .B Return type None .UNINDENT .sp Produces one MapperEnt for every unique location for a File or Directory. These MapperEnt objects are instructions to cwltool\(aqs stage_files function: \fI\%https://github.com/common\-workflow\-language/cwltool/blob/a3e3a5720f7b0131fa4f9c0b3f73b62a347278a6/cwltool/process.py#L254\fP .sp The MapperEnt has fields: .sp resolved: An absolute local path anywhere on the filesystem where the file/directory can be found, or the contents of a file to populate it with if type is CreateWritableFile or CreateFile. Or, a URI understood by the StdFsAccess in use (for example, toilfile:). .sp target: An absolute path under stagedir that the file or directory will then be placed at by cwltool. Except if a File or Directory has a dirname field, giving its parent path, that is used instead. .sp type: One of: .INDENT 7.0 .INDENT 3.5 File: cwltool will copy or link the file from resolved to target, if possible. .sp CreateFile: cwltool will create the file at target, treating resolved as the contents. .sp WritableFile: cwltool will copy the file from resolved to target, making it writable. .sp CreateWritableFile: cwltool will create the file at target, treating resolved as the contents, and make it writable. .sp Directory: cwltool will copy or link the directory from resolved to target, if possible. Otherwise, cwltool will make the directory at target if resolved starts with \(dq_:\(dq. Otherwise it will do nothing. .sp WritableDirectory: cwltool will copy the directory from resolved to target, if possible. Otherwise, cwltool will make the directory at target if resolved starts with \(dq_:\(dq. Otherwise it will do nothing. .UNINDENT .UNINDENT .sp staged: if set to False, cwltool will not make or copy anything for this entry .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ToilSingleJobExecutor Bases: \fBcwltool.executors.SingleJobExecutor\fP .sp A SingleJobExecutor that does not assume it is at the top level of the workflow. .sp We need this because otherwise every job thinks it is top level and tries to discover secondary files, which may exist when they haven\(aqt actually been passed at the top level and thus aren\(aqt supposed to be visible. .INDENT 7.0 .TP .B run_jobs(process, job_order_object, logger, runtime_context) run_jobs from SingleJobExecutor, but not in a top level runtime context. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprocess\fP (\fIcwltool.process.Process\fP) \-\- .IP \(bu 2 \fBjob_order_object\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBlogger\fP (\fI\%logging.Logger\fP) \-\- .IP \(bu 2 \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ToilTool Mixin to hook Toil into a cwltool tool type. .INDENT 7.0 .TP .B make_path_mapper(reffiles, stagedir, runtimeContext, separateDirs) Create the appropriate PathMapper for the situation. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBreffiles\fP (\fIList\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBstagedir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBruntimeContext\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .IP \(bu 2 \fBseparateDirs\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type cwltool.pathmapper.PathMapper .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Return string representation of this tool type. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ToilCommandLineTool(toolpath_object, loadingContext) Bases: \fI\%ToilTool\fP, \fBcwltool.command_line_tool.CommandLineTool\fP .sp Subclass the cwltool command line tool to provide the custom ToilPathMapper. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtoolpath_object\fP (\fIruamel.yaml.comments.CommentedMap\fP) \-\- .IP \(bu 2 \fBloadingContext\fP (\fIcwltool.context.LoadingContext\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ToilExpressionTool(toolpath_object, loadingContext) Bases: \fI\%ToilTool\fP, \fBcwltool.command_line_tool.ExpressionTool\fP .sp Subclass the cwltool expression tool to provide the custom ToilPathMapper. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtoolpath_object\fP (\fIruamel.yaml.comments.CommentedMap\fP) \-\- .IP \(bu 2 \fBloadingContext\fP (\fIcwltool.context.LoadingContext\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.toil_make_tool(toolpath_object, loadingContext) Emit custom ToilCommandLineTools. .sp This factory function is meant to be passed to cwltool.load_tool(). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtoolpath_object\fP (\fIruamel.yaml.comments.CommentedMap\fP) \-\- .IP \(bu 2 \fBloadingContext\fP (\fIcwltool.context.LoadingContext\fP) \-\- .UNINDENT .TP .B Return type cwltool.process.Process .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.DirectoryContents .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.check_directory_dict_invariants(contents) Make sure a directory structure dict makes sense. Throws an error otherwise. .sp Currently just checks to make sure no empty\-string keys exist. .INDENT 7.0 .TP .B Parameters \fBcontents\fP (\fIDirectoryContents\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.decode_directory(dir_path) Decode a directory from a \(dqtoildir:\(dq path to a directory (or a file in it). .sp Returns the decoded directory dict, the remaining part of the path (which may be None), and the deduplication key string that uniquely identifies the directory. .INDENT 7.0 .TP .B Parameters \fBdir_path\fP (\fI\%str\fP) \-\- .TP .B Return type Tuple[DirectoryContents, Optional[\fI\%str\fP], \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.encode_directory(contents) Encode a directory from a \(dqtoildir:\(dq path to a directory (or a file in it). .sp Takes the directory dict, which is a dict from name to URI for a file or dict for a subdirectory. .INDENT 7.0 .TP .B Parameters \fBcontents\fP (\fIDirectoryContents\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ToilFsAccess(basedir, file_store=None) Bases: \fBcwltool.stdfsaccess.StdFsAccess\fP .sp Custom filesystem access class which handles toil filestore references. .sp Normal file paths will be resolved relative to basedir, but \(aqtoilfile:\(aq and \(aqtoildir:\(aq URIs will be fulfilled from the Toil file store. .sp Also supports URLs supported by Toil job store implementations. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbasedir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfile_store\fP (\fIOptional\fP\fI[\fP\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B glob(pattern) .INDENT 7.0 .TP .B Parameters \fBpattern\fP (\fI\%str\fP) \-\- .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B open(fn, mode) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfn\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBmode\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type IO[Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exists(path) Test for file existence. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B size(path) .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B isfile(fn) .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B isdir(fn) .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B listdir(fn) .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%str\fP) \-\- .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B join(path, *paths) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBpaths\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B realpath(fn) .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.toil_get_file(file_store, index, existing, uri, streamable=False, streaming_allowed=True, pipe_threads=None) Set up the given file or directory from the Toil jobstore at a file URI where it can be accessed locally. .sp Run as part of the tool setup, inside jobs on the workers. Also used as part of reorganizing files to get them uploaded at the end of a tool. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- The Toil file store to download from. .IP \(bu 2 \fBindex\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Maps from downloaded file path back to input Toil URI. .IP \(bu 2 \fBexisting\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Maps from URI to downloaded file path. .IP \(bu 2 \fBuri\fP (\fI\%str\fP) \-\- The URI for the file to download. .IP \(bu 2 \fBstreamable\fP (\fI\%bool\fP) \-\- If the file is has \(aqstreamable\(aq flag set .IP \(bu 2 \fBstreaming_allowed\fP (\fI\%bool\fP) \-\- If streaming is allowed .IP \(bu 2 \fBpipe_threads\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fITuple\fP\fI[\fP\fI\%threading.Thread\fP\fI, \fP\fI\%int\fP\fI]\fP\fI]\fP\fI]\fP) \-\- List of threads responsible for streaming the data and open file descriptors corresponding to those files. Caller is responsible to close the file descriptors (to break the pipes) and join the threads .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.write_file(writeFunc, index, existing, file_uri) Write a file into the Toil jobstore. .sp \(aqexisting\(aq is a set of files retrieved as inputs from toil_get_file. This ensures they are mapped back as the same name if passed through. .sp Returns a toil uri path to the object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwriteFunc\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fI\%toil.fileStores.FileID\fP\fI]\fP) \-\- .IP \(bu 2 \fBindex\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBexisting\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBfile_uri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.path_to_loc(obj) Make a path into a location. .sp (If a CWL object has a \(dqpath\(dq and not a \(dqlocation\(dq) .INDENT 7.0 .TP .B Parameters \fBobj\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.import_files(import_function, fs_access, fileindex, existing, cwl_object, skip_broken=False, skip_remote=False, bypass_file_store=False, log_level=logging.DEBUG) Prepare all files and directories. .sp Will be executed from the leader or worker in the context of the given CWL tool, order, or output object to be used on the workers. Make sure their sizes are set and import all the files. .sp Recurses inside directories using the fs_access to find files to upload and subdirectory structure to encode, even if their listings are not set or not recursive. .sp Preserves any listing fields. .sp If a file cannot be found (like if it is an optional secondary file that doesn\(aqt exist), fails, unless skip_broken is set, in which case it leaves the location it was supposed to have been at. .sp Also does some miscelaneous normalization. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBimport_function\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fI\%toil.fileStores.FileID\fP\fI]\fP) \-\- The function used to upload a URI and get a Toil FileID for it. .IP \(bu 2 \fBfs_access\fP (\fIcwltool.stdfsaccess.StdFsAccess\fP) \-\- the CWL FS access object we use to access the filesystem to find files to import. Needs to support the URI schemes used. .IP \(bu 2 \fBfileindex\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Forward map to fill in from file URI to Toil storage location, used by write_file to deduplicate writes. .IP \(bu 2 \fBexisting\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Reverse map to fill in from Toil storage location to file URI. Not read from. .IP \(bu 2 \fBcwl_object\fP (\fIOptional\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- CWL tool (or workflow order) we are importing files for .IP \(bu 2 \fBskip_broken\fP (\fI\%bool\fP) \-\- If True, when files can\(aqt be imported because they e.g. don\(aqt exist, leave their locations alone rather than failing with an error. .IP \(bu 2 \fBskp_remote\fP \-\- If True, leave remote URIs in place instead of importing files. .IP \(bu 2 \fBbypass_file_store\fP (\fI\%bool\fP) \-\- If True, leave \fI\%file://\fP URIs in place instead of importing files and directories. .IP \(bu 2 \fBlog_level\fP (\fI\%int\fP) \-\- Log imported files at the given level. .IP \(bu 2 \fBskip_remote\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.upload_directory(directory_metadata, directory_contents, skip_broken=False) Upload a Directory object. .sp Ignores the listing (which may not be recursive and isn\(aqt safe or efficient to touch), and instead uses directory_contents, which is a recursive dict structure from filename to file URI or subdirectory contents dict. .sp Makes sure the directory actually exists, and rewrites its location to be something we can use on another machine. .sp We can\(aqt rely on the directory\(aqs listing as visible to the next tool as a complete recursive description of the files we will need to present to the tool, since some tools require it to be cleared or single\-level but still expect to see its contents in the filesystem. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdirectory_metadata\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBdirectory_contents\fP (\fIDirectoryContents\fP) \-\- .IP \(bu 2 \fBskip_broken\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.upload_file(uploadfunc, fileindex, existing, file_metadata, skip_broken=False, skip_remote=False) Update a file object so that the file will be accessible from another machine. .sp Uploads local files to the Toil file store, and sets their location to a reference to the toil file store. .sp Unless skip_remote is set, downloads remote files into the file store and sets their locations to references into the file store as well. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBuploadfunc\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fI\%toil.fileStores.FileID\fP\fI]\fP) \-\- .IP \(bu 2 \fBfileindex\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBexisting\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBfile_metadata\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBskip_broken\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBskip_remote\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.writeGlobalFileWrapper(file_store, fileuri) Wrap writeGlobalFile to accept \fI\%file://\fP URIs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .IP \(bu 2 \fBfileuri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%toil.fileStores.FileID\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.remove_empty_listings(rec) .INDENT 7.0 .TP .B Parameters \fBrec\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.CWLNamedJob(cores=1, memory=\(aq1GiB\(aq, disk=\(aq1MiB\(aq, accelerators=None, preemptible=None, tool_id=None, parent_name=None, subjob_name=None, local=None) Bases: \fI\%toil.job.Job\fP .sp Base class for all CWL jobs that do user work, to give them useful names. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcores\fP (\fIUnion\fP\fI[\fP\fI\%float\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%str\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%str\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBaccelerators\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%toil.job.AcceleratorRequirement\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .IP \(bu 2 \fBtool_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBsubjob_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBlocal\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.ResolveIndirect(cwljob, parent_name=None) Bases: \fI\%CWLNamedJob\fP .sp Helper Job. .sp Accepts an unresolved dict (containing promises) and produces a dictionary of actual values. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcwljob\fP (\fItoil.job.Promised\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Evaluate the promises and return their values. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type cwltool.utils.CWLObjectType .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.toilStageFiles(toil, cwljob, outdir, destBucket=None, log_level=logging.DEBUG) Copy input files out of the global file store and update location and path. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdestBucket\fP (\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fINone\fP\fI]\fP) \-\- If set, export to this base URL instead of to the local filesystem. .IP \(bu 2 \fBlog_level\fP (\fI\%int\fP) \-\- Log each file transfered at the given level. .IP \(bu 2 \fBtoil\fP (\fI\%toil.common.Toil\fP) \-\- .IP \(bu 2 \fBcwljob\fP (\fIUnion\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI, \fP\fIList\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBoutdir\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.CWLJobWrapper(tool, cwljob, runtime_context, parent_name, conditional=None) Bases: \fI\%CWLNamedJob\fP .sp Wrap a CWL job that uses dynamic resources requirement. .sp When executed, this creates a new child job which has the correct resource requirement set. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtool\fP (\fIcwltool.process.Process\fP) \-\- .IP \(bu 2 \fBcwljob\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconditional\fP (\fIUnion\fP\fI[\fP\fI\%Conditional\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Create a child job with the correct resource requirements set. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.CWLJob(tool, cwljob, runtime_context, parent_name=None, conditional=None) Bases: \fI\%CWLNamedJob\fP .sp Execute a CWL tool using cwltool.executors.SingleJobExecutor. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtool\fP (\fIcwltool.process.Process\fP) \-\- .IP \(bu 2 \fBcwljob\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconditional\fP (\fIUnion\fP\fI[\fP\fI\%Conditional\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B required_env_vars(cwljob) Yield environment variables from EnvVarRequirement. .INDENT 7.0 .TP .B Parameters \fBcwljob\fP (\fIAny\fP) \-\- .TP .B Return type Iterator[Tuple[\fI\%str\fP, \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B populate_env_vars(cwljob) Prepare environment variables necessary at runtime for the job. .sp Env vars specified in the CWL \(dqrequirements\(dq section should already be loaded in self.cwltool.requirements, however those specified with \(dqEnvVarRequirement\(dq take precedence and are only populated here. Therefore, this not only returns a dictionary with all evaluated \(dqEnvVarRequirement\(dq env vars, but checks self.cwltool.requirements for any env vars with the same name and replaces their value with that found in the \(dqEnvVarRequirement\(dq env var if it exists. .INDENT 7.0 .TP .B Parameters \fBcwljob\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Execute the CWL document. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.get_container_engine(runtime_context) .INDENT 7.0 .TP .B Parameters \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.makeJob(tool, jobobj, runtime_context, parent_name, conditional) Create the correct Toil Job object for the CWL tool. .sp Types: workflow, job, or job wrapper for dynamic resource requirements. .INDENT 7.0 .TP .B Returns \(dqwfjob, followOn\(dq if the input tool is a workflow, and \(dqjob, job\(dq otherwise .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtool\fP (\fIcwltool.process.Process\fP) \-\- .IP \(bu 2 \fBjobobj\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconditional\fP (\fIUnion\fP\fI[\fP\fI\%Conditional\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Union[Tuple[\fI\%CWLWorkflow\fP, \fI\%ResolveIndirect\fP], Tuple[\fI\%CWLJob\fP, \fI\%CWLJob\fP], Tuple[\fI\%CWLJobWrapper\fP, \fI\%CWLJobWrapper\fP]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.CWLScatter(step, cwljob, runtime_context, parent_name, conditional) Bases: \fI\%toil.job.Job\fP .sp Implement workflow scatter step. .sp When run, this creates a child job for each parameterization of the scatter. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstep\fP (\fIcwltool.workflow.WorkflowStep\fP) \-\- .IP \(bu 2 \fBcwljob\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconditional\fP (\fIUnion\fP\fI[\fP\fI\%Conditional\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B flat_crossproduct_scatter(joborder, scatter_keys, outputs, postScatterEval) Cartesian product of the inputs, then flattened. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjoborder\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBscatter_keys\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBoutputs\fP (\fIList\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBpostScatterEval\fP (\fICallable\fP\fI[\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI, \fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B nested_crossproduct_scatter(joborder, scatter_keys, postScatterEval) Cartesian product of the inputs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjoborder\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBscatter_keys\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBpostScatterEval\fP (\fICallable\fP\fI[\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI, \fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type List[toil.job.Promised[cwltool.utils.CWLObjectType]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Generate the follow on scatter jobs. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type List[toil.job.Promised[cwltool.utils.CWLObjectType]] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.CWLGather(step, outputs) Bases: \fI\%toil.job.Job\fP .sp Follows on to a scatter Job. .sp This gathers the outputs of each job in the scatter into an array for each output parameter. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstep\fP (\fIcwltool.workflow.WorkflowStep\fP) \-\- .IP \(bu 2 \fBoutputs\fP (\fItoil.job.Promised\fP\fI[\fP\fIUnion\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI, \fP\fIList\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static extract(obj, k) Extract the given key from the obj. .sp If the object is a list, extract it from all members of the list. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBobj\fP (\fIUnion\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI, \fP\fIList\fP\fI[\fP\fIcwltool.utils.CWLObjectType\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBk\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Union[cwltool.utils.CWLOutputType, List[cwltool.utils.CWLObjectType]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Gather all the outputs of the scatter. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.SelfJob(j, v) Bases: \fI\%toil.job.Job\fP .sp Fake job object to facilitate implementation of CWLWorkflow.run(). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBj\fP (\fI\%CWLWorkflow\fP) \-\- .IP \(bu 2 \fBv\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rv(*path) Return our properties dictionary. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fIAny\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChild(c) Add a child to our workflow. .INDENT 7.0 .TP .B Parameters \fBc\fP (\fI\%toil.job.Job\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasChild(c) Check if the given child is in our workflow. .INDENT 7.0 .TP .B Parameters \fBc\fP (\fI\%toil.job.Job\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.ProcessType .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.remove_pickle_problems(obj) Doc_loader does not pickle correctly, causing Toil errors, remove from objects. .INDENT 7.0 .TP .B Parameters \fBobj\fP (\fIProcessType\fP) \-\- .TP .B Return type ProcessType .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.cwl.cwltoil.CWLWorkflow(cwlwf, cwljob, runtime_context, parent_name=None, conditional=None) Bases: \fI\%CWLNamedJob\fP .sp Toil Job to convert a CWL workflow graph into a Toil job graph. .sp The Toil job graph will include the appropriate dependencies. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcwlwf\fP (\fIcwltool.workflow.Workflow\fP) \-\- .IP \(bu 2 \fBcwljob\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .IP \(bu 2 \fBruntime_context\fP (\fIcwltool.context.RuntimeContext\fP) \-\- .IP \(bu 2 \fBparent_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconditional\fP (\fIUnion\fP\fI[\fP\fI\%Conditional\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Convert a CWL Workflow graph into a Toil job graph. .sp Always runs on the leader, because the batch system knows to schedule it as a local job. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type Union[\fI\%UnresolvedDict\fP, Dict[\fI\%str\fP, \fI\%SkipNull\fP]] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.visitSteps(cmdline_tool, op) Iterate over a CWL Process object, running the op on each tool description CWL object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcmdline_tool\fP (\fIcwltool.process.Process\fP) \-\- .IP \(bu 2 \fBop\fP (\fICallable\fP\fI[\fP\fI[\fP\fIruamel.yaml.comments.CommentedMap\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.rm_unprocessed_secondary_files(job_params) .INDENT 7.0 .TP .B Parameters \fBjob_params\fP (\fIAny\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.filtered_secondary_files(unfiltered_secondary_files) Remove unprocessed secondary files. .sp Interpolated strings and optional inputs in secondary files were added to CWL in version 1.1. .sp The CWL libraries we call do successfully resolve the interpolated strings, but add the resolved fields to the list of unresolved fields so we remove them here after the fact. .sp We keep secondary files using the \(aqtoildir:\(aq, or \(aq_:\(aq protocols, or using the \(aqfile:\(aq protocol and indicating files or directories that actually exist. The \(aqrequired\(aq logic seems to be handled deeper in cwltool.builder.Builder(), and correctly determines which files should be imported. Therefore we remove the files here and if this file is SUPPOSED to exist, it will still give the appropriate file does not exist error, but just a bit further down the track. .INDENT 7.0 .TP .B Parameters \fBunfiltered_secondary_files\fP (\fIcwltool.utils.CWLObjectType\fP) \-\- .TP .B Return type List[cwltool.utils.CWLObjectType] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.scan_for_unsupported_requirements(tool, bypass_file_store=False) Scan the given CWL tool for any unsupported optional features. .sp If it has them, raise an informative UnsupportedRequirement. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtool\fP (\fIcwltool.process.Process\fP) \-\- The CWL tool to check for unsupported requirements. .IP \(bu 2 \fBbypass_file_store\fP (\fI\%bool\fP) \-\- True if the Toil file store is not being used to transport files between nodes, and raw origin node \fI\%file://\fP URIs are exposed to tools instead. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.determine_load_listing(tool) Determine the directory.listing feature in CWL. .sp In CWL, any input directory can have a DIRECTORY_NAME.listing (where DIRECTORY_NAME is any variable name) set to one of the following three options: .INDENT 7.0 .IP 1. 3 .INDENT 3.0 .TP .B no_listing: DIRECTORY_NAME.listing will be undefined. e.g. .INDENT 7.0 .INDENT 3.5 inputs.DIRECTORY_NAME.listing == unspecified .UNINDENT .UNINDENT .UNINDENT .IP 2. 3 .INDENT 3.0 .TP .B shallow_listing: DIRECTORY_NAME.listing will return a list one level deep of DIRECTORY_NAME\(aqs contents. e.g. .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .TP .B inputs.DIRECTORY_NAME.listing == [items in directory] inputs.DIRECTORY_NAME.listing[0].listing == undefined inputs.DIRECTORY_NAME.listing.length == # of items in directory .UNINDENT .UNINDENT .UNINDENT .UNINDENT .IP 3. 3 .INDENT 3.0 .TP .B deep_listing: DIRECTORY_NAME.listing will return a list of the entire contents of DIRECTORY_NAME. e.g. .INDENT 7.0 .INDENT 3.5 inputs.DIRECTORY_NAME.listing == [items in directory] inputs.DIRECTORY_NAME.listing[0].listing == [items in subdirectory if it exists and is the first item listed] inputs.DIRECTORY_NAME.listing.length == # of items in directory .UNINDENT .UNINDENT .UNINDENT .UNINDENT .sp See \fI\%https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingRequirement\fP and \fI\%https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingEnum\fP .sp DIRECTORY_NAME.listing should be determined first from loadListing. If that\(aqs not specified, from LoadListingRequirement. Else, default to \(dqno_listing\(dq if unspecified. .INDENT 7.0 .TP .B Parameters \fBtool\fP (\fIcwltool.process.Process\fP) \-\- ToilCommandLineTool .TP .B Return str One of \(aqno_listing\(aq, \(aqshallow_listing\(aq, or \(aqdeep_listing\(aq. .TP .B Return type typing_extensions.Literal[no_listing, shallow_listing, deep_listing] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.cwl.cwltoil.NoAvailableJobStoreException Bases: \fI\%Exception\fP .sp Indicates that no job store name is available. .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.generate_default_job_store(batch_system_name, provisioner_name, local_directory) Choose a default job store appropriate to the requested batch system and provisioner, and installed modules. Raises an error if no good default is available and the user must choose manually. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbatch_system_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Registry name of the batch system the user has requested, if any. If no name has been requested, should be None. .IP \(bu 2 \fBprovisioner_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Name of the provisioner the user has requested, if any. Recognized provisioners include \(aqaws\(aq and \(aqgce\(aq. None indicates that no provisioner is in use. .IP \(bu 2 \fBlocal_directory\fP (\fI\%str\fP) \-\- Path to a nonexistent local directory suitable for use as a file job store. .UNINDENT .TP .B Return str Job store specifier for a usable job store. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.usage_message .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.get_options(args) Parse given args and properly add non\-Toil arguments into the cwljob of the Namespace. :param args: List of args from command line :return: options namespace .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type configargparse.Namespace .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.main(args=None, stdout=sys.stdout) Run the main loop for toil\-cwl\-runner. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBstdout\fP (\fITextIO\fP) \-\- .UNINDENT .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.cwltoil.find_default_container(args, builder) Find the default constructor by consulting a Toil.options object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fIconfigargparse.Namespace\fP) \-\- .IP \(bu 2 \fBbuilder\fP (\fIcwltool.builder.Builder\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .SS \fI\%toil.cwl.utils\fP .sp Utility functions used for Toil\(aqs CWL interpreter. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%visit_top_cwl_class\fP(rec, classes, op) T} T{ Apply the given operation to all top\-level CWL objects with the given named CWL class. T} _ T{ \fI\%visit_cwl_class_and_reduce\fP(rec, classes, op_down, op_up) T} T{ Apply the given operations to all CWL objects with the given named CWL class. T} _ T{ \fI\%get_from_structure\fP(dir_dict, path) T} T{ Given a relative path, follow it in the given directory structure. T} _ T{ \fI\%download_structure\fP(file_store, index, existing, ...) T} T{ Download nested dictionary from the Toil file store to a local path. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE\fP T} T{ T} _ T{ \fI\%CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION\fP T} T{ T} _ T{ \fI\%DownReturnType\fP T} T{ T} _ T{ \fI\%UpReturnType\fP T} T{ T} _ T{ \fI\%DirectoryStructure\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.cwl.utils.logger .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE = 33 .UNINDENT .INDENT 0.0 .TP .B exception toil.cwl.utils.CWLUnsupportedException Bases: \fI\%Exception\fP .sp Fallback exception. .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION: Type[cwltool.errors.UnsupportedRequirement] | Type[\fI\%CWLUnsupportedException\fP] .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.visit_top_cwl_class(rec, classes, op) Apply the given operation to all top\-level CWL objects with the given named CWL class. .sp Like cwltool\(aqs visit_class but doesn\(aqt look inside any object visited. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrec\fP (\fIAny\fP) \-\- .IP \(bu 2 \fBclasses\fP (\fIIterable\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBop\fP (\fICallable\fP\fI[\fP\fI[\fP\fIAny\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.DownReturnType .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.UpReturnType .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.visit_cwl_class_and_reduce(rec, classes, op_down, op_up) Apply the given operations to all CWL objects with the given named CWL class. .sp Applies the down operation top\-down, and the up operation bottom\-up, and passes the down operation\(aqs result and a list of the up operation results for all child keys (flattening across lists and collapsing nodes of non\-matching classes) to the up operation. .INDENT 7.0 .TP .B Returns The flattened list of up operation results from all calls. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrec\fP (\fIAny\fP) \-\- .IP \(bu 2 \fBclasses\fP (\fIIterable\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBop_down\fP (\fICallable\fP\fI[\fP\fI[\fP\fIAny\fP\fI]\fP\fI, \fP\fIDownReturnType\fP\fI]\fP) \-\- .IP \(bu 2 \fBop_up\fP (\fICallable\fP\fI[\fP\fI[\fP\fIAny\fP\fI, \fP\fIDownReturnType\fP\fI, \fP\fIList\fP\fI[\fP\fIUpReturnType\fP\fI]\fP\fI]\fP\fI, \fP\fIUpReturnType\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type List[UpReturnType] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.DirectoryStructure .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.get_from_structure(dir_dict, path) Given a relative path, follow it in the given directory structure. .sp Return the string URI for files, the directory dict for subdirectories, or None for nonexistent things. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdir_dict\fP (\fIDirectoryStructure\fP) \-\- .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Union[\fI\%str\fP, DirectoryStructure, None] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cwl.utils.download_structure(file_store, index, existing, dir_dict, into_dir) Download nested dictionary from the Toil file store to a local path. .sp Guaranteed to fill the structure with real files, and not symlinks out of it to elsewhere. File URIs may be toilfile: URIs or any other URI that Toil\(aqs job store system can read. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- The Toil file store to download from. .IP \(bu 2 \fBindex\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Maps from downloaded file path back to input URI. .IP \(bu 2 \fBexisting\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Maps from file_store_id URI to downloaded file path. .IP \(bu 2 \fBdir_dict\fP (\fIDirectoryStructure\fP) \-\- a dict from string to string (for files) or dict (for subdirectories) describing a directory structure. .IP \(bu 2 \fBinto_dir\fP (\fI\%str\fP) \-\- The directory to download the top\-level dict\(aqs files into. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .SS Package Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%check_cwltool_version\fP() T} T{ Check if the installed cwltool version matches Toil\(aqs expected version. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%cwltool_version\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.cwl.cwltool_version = \(aq3.1.20240112164112\(aq .UNINDENT .INDENT 0.0 .TP .B toil.cwl.check_cwltool_version() Check if the installed cwltool version matches Toil\(aqs expected version. .sp A warning is printed to standard error if the versions differ. We do not assume that logging is set up already. Safe to call repeatedly; only one warning will be printed. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.fileStores\fP .SS Submodules .SS \fI\%toil.fileStores.abstractFileStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AbstractFileStore\fP T} T{ Interface used to allow user code run by Toil to read and write files. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.fileStores.abstractFileStore.logger .UNINDENT .INDENT 0.0 .TP .B class toil.fileStores.abstractFileStore.AbstractFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit) Bases: \fI\%abc.ABC\fP .sp Interface used to allow user code run by Toil to read and write files. .sp Also provides the interface to other Toil facilities used by user code, including: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 normal (non\-real\-time) logging .IP \(bu 2 finding the correct temporary directory for scratch work .IP \(bu 2 importing and exporting files into and out of the workflow .UNINDENT .UNINDENT .UNINDENT .sp Stores user files in the jobStore, but keeps them separate from actual jobs. .sp May implement caching. .sp Passed as argument to the \fI\%toil.job.Job.run()\fP method. .sp Access to files is only permitted inside the context manager provided by \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.open()\fP\&. .sp Also responsible for committing completed jobs back to the job store with an update operation, and allowing that commit operation to be waited for. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static createFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit, caching) Create a concreate FileStore. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBcaching\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Union[\fI\%toil.fileStores.nonCachingFileStore.NonCachingFileStore\fP, \fI\%toil.fileStores.cachingFileStore.CachingFileStore\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static shutdownFileStore(workflowID, config_work_dir, config_coordination_dir) Carry out any necessary filestore\-specific cleanup. .sp This is a destructive operation and it is important to ensure that there are no other running processes on the system that are modifying or using the file store for this workflow. .sp This is the intended to be the last call to the file store in a Toil run, called by the batch system cleanup function upon batch system shutdown. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflowID\fP (\fI\%str\fP) \-\- The workflow ID for this invocation of the workflow .IP \(bu 2 \fBconfig_work_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The path to the work directory in the Toil Config. .IP \(bu 2 \fBconfig_coordination_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The path to the coordination directory in the Toil Config. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B open(job) Create the context manager around tasks prior and after a job has been run. .sp File operations are only permitted inside the context manager. .sp Implementations must only yield from within \fIwith super().open(job):\fP\&. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.Job\fP) \-\- The job instance of the toil job to run. .TP .B Return type Generator[None, None, None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_disk_usage() Get the number of bytes of disk used by the last job run under open(). .sp Disk usage is measured at the end of the job. TODO: Sample periodically and record peak usage. .INDENT 7.0 .TP .B Return type Optional[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLocalTempDir() Get a new local temporary directory in which to write files. .sp The directory will only persist for the duration of the job. .INDENT 7.0 .TP .B Returns The absolute path to a new local temporary directory. This directory will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates, removing all files it contains recursively. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLocalTempFile(suffix=None, prefix=None) Get a new local temporary file that will persist for the duration of the job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If not None, the file name will end with this string. Otherwise, default value \(dq.tmp\(dq will be used .IP \(bu 2 \fBprefix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If not None, the file name will start with this string. Otherwise, default value \(dqtmp\(dq will be used .UNINDENT .TP .B Returns The absolute path to a local temporary file. This file will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLocalTempFileName(suffix=None, prefix=None) Get a valid name for a new local file. Don\(aqt actually create a file at the path. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If not None, the file name will end with this string. Otherwise, default value \(dq.tmp\(dq will be used .IP \(bu 2 \fBprefix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If not None, the file name will start with this string. Otherwise, default value \(dqtmp\(dq will be used .UNINDENT .TP .B Returns Path to valid file .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract writeGlobalFile(localFileName, cleanup=False) Upload a file (as a path) to the job store. .sp If the file is in a FileStore\-managed temporary directory (i.e. from \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()\fP), it will become a local copy of the file, eligible for deletion by \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile()\fP\&. .sp If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded again. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocalFileName\fP (\fI\%str\fP) \-\- The path to the local file to upload. The last path component (basename of the file) will remain associated with the file in the file store, if supported by the backing JobStore, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- if True then the copy of the global file will be deleted once the job and all its successors have completed running. If not the global file must be deleted manually. .UNINDENT .TP .B Returns an ID that can be used to retrieve the file. .TP .B Return type \fI\%toil.fileStores.FileID\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeGlobalFileStream(cleanup=False, basename=None, encoding=None, errors=None) Similar to writeGlobalFile, but allows the writing of a stream to the job store. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- is as in \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile()\fP\&. .IP \(bu 2 \fBbasename\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If supported by the backing JobStore, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .UNINDENT .TP .B Returns A context manager yielding a tuple of 1) a file handle which can be written to and 2) the toil.fileStores.FileID of the resulting file in the job store. .TP .B Return type Iterator[Tuple[\fI\%toil.lib.io.WriteWatchingStream\fP, \fI\%toil.fileStores.FileID\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logAccess(fileStoreID, destination=None) Record that the given file was read by the job. .sp (to be announced if the job fails) .sp If destination is not None, it gives the path that the file was downloaded to. Otherwise, assumes that the file was streamed. .sp Must be called by \fI\%readGlobalFile()\fP and \fI\%readGlobalFileStream()\fP implementations. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fIUnion\fP\fI[\fP\fI\%toil.fileStores.FileID\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdestination\fP (\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False) Make the file associated with fileStoreID available locally. .sp If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem. .sp If a user path is specified, it is used as the destination. If a user path isn\(aqt specified, the file is stored in the local temp directory with an encoded name. .sp The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile. .sp Implementations must call \fI\%logAccess()\fP to report the download. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- job store id for the file .IP \(bu 2 \fBuserPath\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- a path to the name of file to which the global file will be copied or hard\-linked (see below). .IP \(bu 2 \fBcache\fP (\fI\%bool\fP) \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBmutable\fP (\fI\%bool\fP) \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- True if caller can accept symlink, False if caller can only accept a normal file or hardlink .UNINDENT .TP .B Returns An absolute path to a local, temporary copy of the file keyed by fileStoreID. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readGlobalFileStream(fileStoreID: \fI\%str\fP, encoding: Literal[None] = None, errors: \fI\%str\fP | \fI\%None\fP = None) -> ContextManager[IO[\fI\%bytes\fP]] .TP .B readGlobalFileStream(fileStoreID: \fI\%str\fP, encoding: \fI\%str\fP, errors: \fI\%str\fP | \fI\%None\fP = None) -> ContextManager[IO[\fI\%str\fP]] Read a stream from the job store; similar to readGlobalFile. .sp The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBencoding\fP \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .UNINDENT .sp Implementations must call \fI\%logAccess()\fP to report the download. .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be read from. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getGlobalFileSize(fileStoreID) Get the size of the file pointed to by the given ID, in bytes. .sp If a FileID or something else with a non\-None \(aqsize\(aq field, gets that. .sp Otherwise, asks the job store to poll the file\(aqs size. .sp Note that the job store may overestimate the file\(aqs size, for example if it is encrypted and had to be augmented with an IV or other encryption framing. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fIUnion\fP\fI[\fP\fI\%toil.fileStores.FileID\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- File ID for the file .TP .B Returns File\(aqs size in bytes, as stored in the job store .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract deleteLocalFile(fileStoreID) Delete local copies of files associated with the provided job store ID. .sp Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession. .sp The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job\(aqs file\-store\-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job\(aqs file\-store\-provided temp directory. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fIUnion\fP\fI[\fP\fI\%toil.fileStores.FileID\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- File Store ID of the file to be deleted. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract deleteGlobalFile(fileStoreID) Delete local files and then permanently deletes them from the job store. .sp To ensure that the job can be restarted if necessary, the delete will not happen until after the job\(aqs run method has completed. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fIUnion\fP\fI[\fP\fI\%toil.fileStores.FileID\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- the File Store ID of the file to be deleted. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B importFile(srcUrl, sharedFileName=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrcUrl\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsharedFileName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%toil.fileStores.FileID\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B import_file(src_uri, shared_file_name=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBshared_file_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%toil.fileStores.FileID\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exportFile(jobStoreFileID, dstUrl) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdstUrl\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract export_file(file_id, dst_uri) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log_to_leader(text, level=logging.INFO) Send a logging message to the leader. The message will also be logged by the worker at the same level. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtext\fP (\fI\%str\fP) \-\- The string to log. .IP \(bu 2 \fBlevel\fP (\fI\%int\fP) \-\- The logging level. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logToMaster(text, level=logging.INFO) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtext\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBlevel\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log_user_stream(name, stream) Send a stream of UTF\-8 text to the leader as a named log stream. .sp Useful for things like the error logs of Docker containers. The leader will show it to the user or organize it appropriately for user\-level log information. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- A hierarchical, .\-delimited string. .IP \(bu 2 \fBstream\fP (\fIIO\fP\fI[\fP\fI\%bytes\fP\fI]\fP) \-\- A stream of encoded text. Encoding errors will be tolerated. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract startCommit(jobState=False) Update the status of the job on the disk. .sp May bump the version number of the job. .sp May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call. .INDENT 7.0 .TP .B Parameters \fBjobState\fP (\fI\%bool\fP) \-\- If True, commit the state of the FileStore\(aqs job, and file deletes. Otherwise, commit only file creates/updates. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract waitForCommit() Blocks while startCommit is running. .sp This function is called by this job\(aqs successor to ensure that it does not begin modifying the job store until after this job has finished doing so. .sp Might be called when startCommit is never called on a particular instance, in which case it does not block. .INDENT 7.0 .TP .B Returns Always returns True .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract classmethod shutdown(shutdown_info) Shutdown the filestore on this node. .sp This is intended to be called on batch system shutdown. .INDENT 7.0 .TP .B Parameters \fBshutdown_info\fP (\fIAny\fP) \-\- The implementation\-specific shutdown information, for shutting down the file store and removing all its state and all job local temp directories from the node. .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.fileStores.cachingFileStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%CachingFileStore\fP T} T{ A cache\-enabled file store. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%SQLITE_TIMEOUT_SECS\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.fileStores.cachingFileStore.logger .UNINDENT .INDENT 0.0 .TP .B toil.fileStores.cachingFileStore.SQLITE_TIMEOUT_SECS = 60.0 .UNINDENT .INDENT 0.0 .TP .B exception toil.fileStores.cachingFileStore.CacheError(message) Bases: \fI\%Exception\fP .sp Error Raised if the user attempts to add a non\-local file to cache .UNINDENT .INDENT 0.0 .TP .B exception toil.fileStores.cachingFileStore.CacheUnbalancedError Bases: \fI\%CacheError\fP .sp Raised if file store can\(aqt free enough space for caching .INDENT 7.0 .TP .B message = \(aqUnable unable to free enough space for caching. This error frequently arises due to jobs using...\(aq .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.fileStores.cachingFileStore.IllegalDeletionCacheError(deletedFile) Bases: \fI\%CacheError\fP .sp Error raised if the caching code discovers a file that represents a reference to a cached file to have gone missing. .sp This can be a big problem if a hard link is moved, because then the cache will be unable to evict the file it links to. .sp Remember that files read with readGlobalFile may not be deleted by the user and need to be deleted with deleteLocalFile. .UNINDENT .INDENT 0.0 .TP .B exception toil.fileStores.cachingFileStore.InvalidSourceCacheError(message) Bases: \fI\%CacheError\fP .sp Error raised if the user attempts to add a non\-local file to cache .UNINDENT .INDENT 0.0 .TP .B class toil.fileStores.cachingFileStore.CachingFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit) Bases: \fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP .sp A cache\-enabled file store. .sp Provides files that are read out as symlinks or hard links into a cache directory for the node, if permitted by the workflow. .sp Also attempts to write files back to the backing JobStore asynchronously, after quickly taking them into the cache. Writes are only required to finish when the job\(aqs actual state after running is committed back to the job store. .sp Internaly, manages caching using a database. Each node has its own database, shared between all the workers on the node. The database contains several tables: .sp files contains one entry for each file in the cache. Each entry knows the path to its data on disk. It also knows its global file ID, its state, and its owning worker PID. If the owning worker dies, another worker will pick it up. It also knows its size. .sp File states are: .INDENT 7.0 .IP \(bu 2 \(dqcached\(dq: happily stored in the cache. Reads can happen immediately. Owner is null. May be adopted and moved to state \(dqdeleting\(dq by anyone, if it has no outstanding immutable references. .IP \(bu 2 \(dqdownloading\(dq: in the process of being saved to the cache by a non\-null owner. Reads must wait for the state to become \(dqcached\(dq. If the worker dies, goes to state \(dqdeleting\(dq, because we don\(aqt know if it was fully downloaded or if anyone still needs it. No references can be created to a \(dqdownloading\(dq file except by the worker responsible for downloading it. .IP \(bu 2 \(dquploadable\(dq: stored in the cache and ready to be written to the job store by a non\-null owner. Transitions to \(dquploading\(dq when a (thread of) the owning worker process picks it up and begins uploading it, to free cache space or to commit a completed job. If the worker dies, goes to state \(dqcached\(dq, because it may have outstanding immutable references from the dead\-but\-not\-cleaned\-up job that was going to write it. .IP \(bu 2 \(dquploading\(dq: stored in the cache and being written to the job store by a non\-null owner. Transitions to \(dqcached\(dq when successfully uploaded. If the worker dies, goes to state \(dqcached\(dq, because it may have outstanding immutable references from the dead\-but\-not\-cleaned\-up job that was writing it. .IP \(bu 2 \(dqdeleting\(dq: in the process of being removed from the cache by a non\-null owner. Will eventually be removed from the database. .UNINDENT .sp refs contains one entry for each outstanding reference to a cached file (hard link, symlink, or full copy). The table name is refs instead of references because references is an SQL reserved word. It remembers what job ID has the reference, and the path the reference is at. References have three states: .INDENT 7.0 .IP \(bu 2 \(dqimmutable\(dq: represents a hardlink or symlink to a file in the cache. Dedicates the file\(aqs size in bytes of the job\(aqs disk requirement to the cache, to be used to cache this file or to keep around other files without references. May be upgraded to \(dqcopying\(dq if the link can\(aqt actually be created. .IP \(bu 2 \(dqcopying\(dq: records that a file in the cache is in the process of being copied to a path. Will be upgraded to a mutable reference eventually. .IP \(bu 2 \(dqmutable\(dq: records that a file from the cache was copied to a certain path. Exist only to support deleteLocalFile\(aqs API. Only files with only mutable references (or no references) are eligible for eviction. .UNINDENT .sp jobs contains one entry for each job currently running. It keeps track of the job\(aqs ID, the worker that is supposed to be running the job, the job\(aqs disk requirement, and the job\(aqs local temp dir path that will need to be cleaned up. When workers check for jobs whose workers have died, they null out the old worker, and grab ownership of and clean up jobs and their references until the null\-worker jobs are gone. .sp properties contains key, value pairs for tracking total space available, and whether caching is free for this run. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property con: \fI\%sqlite3.Connection\fP Get the database connection to be used for the current thread. .INDENT 7.0 .TP .B Return type \fI\%sqlite3.Connection\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property cur: \fI\%sqlite3.Cursor\fP Get the main cursor to be used for the current thread. .INDENT 7.0 .TP .B Return type \fI\%sqlite3.Cursor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B as_process() Assume the process\(aqs identity to act on the caching database. .sp Yields the process\(aqs name in the caching database, and holds onto a lock while your thread has it. .INDENT 7.0 .TP .B Return type Generator[\fI\%str\fP, None, None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getCacheLimit() Return the total number of bytes to which the cache is limited. .sp If no limit is available, raises an error. .UNINDENT .INDENT 7.0 .TP .B getCacheUsed() Return the total number of bytes used in the cache. .sp If no value is available, raises an error. .UNINDENT .INDENT 7.0 .TP .B getCacheExtraJobSpace() Return the total number of bytes of disk space requested by jobs running against this cache but not yet used. .sp We can get into a situation where the jobs on the node take up all its space, but then they want to write to or read from the cache. So when that happens, we need to debit space from them somehow... .sp If no value is available, raises an error. .UNINDENT .INDENT 7.0 .TP .B getCacheAvailable() Return the total number of free bytes available for caching, or, if negative, the total number of bytes of cached files that need to be evicted to free up enough space for all the currently scheduled jobs. .sp If no value is available, raises an error. .UNINDENT .INDENT 7.0 .TP .B getSpaceUsableForJobs() Return the total number of bytes that are not taken up by job requirements, ignoring files and file usage. We can\(aqt ever run more jobs than we actually have room for, even with caching. .sp If not retrievable, raises an error. .UNINDENT .INDENT 7.0 .TP .B getCacheUnusedJobRequirement() Return the total number of bytes of disk space requested by the current job and not used by files the job is using in the cache. .sp Mutable references don\(aqt count, but immutable/uploading ones do. .sp If no value is available, raises an error. .UNINDENT .INDENT 7.0 .TP .B adjustCacheLimit(newTotalBytes) Adjust the total cache size limit to the given number of bytes. .UNINDENT .INDENT 7.0 .TP .B fileIsCached(fileID) Return true if the given file is currently cached, and false otherwise. .sp Note that this can\(aqt really be relied upon because a file may go cached \-> deleting after you look at it. If you need to do something with the file you need to do it in a transaction. .UNINDENT .INDENT 7.0 .TP .B getFileReaderCount(fileID) Return the number of current outstanding reads of the given file. .sp Counts mutable references too. .UNINDENT .INDENT 7.0 .TP .B cachingIsFree() Return true if files can be cached for free, without taking up space. Return false otherwise. .sp This will be true when working with certain job stores in certain configurations, most notably the FileJobStore. .UNINDENT .INDENT 7.0 .TP .B open(job) This context manager decorated method allows cache\-specific operations to be conducted before and after the execution of a job in worker.py .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.Job\fP) \-\- .TP .B Return type Generator[None, None, None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeGlobalFile(localFileName, cleanup=False, executable=False) Creates a file in the jobstore and returns a FileID reference. .UNINDENT .INDENT 7.0 .TP .B readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False) Make the file associated with fileStoreID available locally. .sp If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem. .sp If a user path is specified, it is used as the destination. If a user path isn\(aqt specified, the file is stored in the local temp directory with an encoded name. .sp The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile. .sp Implementations must call \fBlogAccess()\fP to report the download. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP \-\- job store id for the file .IP \(bu 2 \fBuserPath\fP \-\- a path to the name of file to which the global file will be copied or hard\-linked (see below). .IP \(bu 2 \fBcache\fP \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBmutable\fP \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBsymlink\fP \-\- True if caller can accept symlink, False if caller can only accept a normal file or hardlink .UNINDENT .TP .B Returns An absolute path to a local, temporary copy of the file keyed by fileStoreID. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readGlobalFileStream(fileStoreID, encoding=None, errors=None) Read a stream from the job store; similar to readGlobalFile. .sp The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBencoding\fP \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .UNINDENT .sp Implementations must call \fBlogAccess()\fP to report the download. .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be read from. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B deleteLocalFile(fileStoreID) Delete local copies of files associated with the provided job store ID. .sp Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession. .sp The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job\(aqs file\-store\-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job\(aqs file\-store\-provided temp directory. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP \-\- File Store ID of the file to be deleted. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B deleteGlobalFile(fileStoreID) Delete local files and then permanently deletes them from the job store. .sp To ensure that the job can be restarted if necessary, the delete will not happen until after the job\(aqs run method has completed. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP \-\- the File Store ID of the file to be deleted. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exportFile(jobStoreFileID, dstUrl) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdstUrl\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B export_file(file_id, dst_uri) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B waitForCommit() Blocks while startCommit is running. .sp This function is called by this job\(aqs successor to ensure that it does not begin modifying the job store until after this job has finished doing so. .sp Might be called when startCommit is never called on a particular instance, in which case it does not block. .INDENT 7.0 .TP .B Returns Always returns True .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B startCommit(jobState=False) Update the status of the job on the disk. .sp May bump the version number of the job. .sp May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call. .INDENT 7.0 .TP .B Parameters \fBjobState\fP \-\- If True, commit the state of the FileStore\(aqs job, and file deletes. Otherwise, commit only file creates/updates. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B startCommitThread(state_to_commit) Run in a thread to actually commit the current job. .INDENT 7.0 .TP .B Parameters \fBstate_to_commit\fP (\fIOptional\fP\fI[\fP\fI\%toil.job.JobDescription\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod shutdown(shutdown_info) .INDENT 7.0 .TP .B Parameters \fBshutdown_info\fP (\fITuple\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Tuple of the coordination directory (where the cache database is) and the cache directory (where the cached data is). .TP .B Return type None .UNINDENT .sp Job local temp directories will be removed due to their appearance in the database. .UNINDENT .INDENT 7.0 .TP .B __del__() Cleanup function that is run when destroying the class instance that ensures that all the file writing threads exit. .UNINDENT .UNINDENT .SS \fI\%toil.fileStores.nonCachingFileStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%NonCachingFileStore\fP T} T{ Interface used to allow user code run by Toil to read and write files. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.fileStores.nonCachingFileStore.logger: \fI\%logging.Logger\fP .UNINDENT .INDENT 0.0 .TP .B class toil.fileStores.nonCachingFileStore.NonCachingFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit) Bases: \fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP .sp Interface used to allow user code run by Toil to read and write files. .sp Also provides the interface to other Toil facilities used by user code, including: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 normal (non\-real\-time) logging .IP \(bu 2 finding the correct temporary directory for scratch work .IP \(bu 2 importing and exporting files into and out of the workflow .UNINDENT .UNINDENT .UNINDENT .sp Stores user files in the jobStore, but keeps them separate from actual jobs. .sp May implement caching. .sp Passed as argument to the \fI\%toil.job.Job.run()\fP method. .sp Access to files is only permitted inside the context manager provided by \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.open()\fP\&. .sp Also responsible for committing completed jobs back to the job store with an update operation, and allowing that commit operation to be waited for. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBjobDesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBfile_store_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwaitForPreviousCommit\fP (\fICallable\fP\fI[\fP\fI[\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static check_for_coordination_corruption(coordination_dir) Make sure the coordination directory hasn\(aqt been deleted unexpectedly. .sp Slurm has been known to delete XDG_RUNTIME_DIR out from under processes it was promised to, so it is possible that in certain misconfigured environments the coordination directory and everything in it could go away unexpectedly. We are going to regularly make sure that the things we think should exist actually exist, and we are going to abort if they do not. .INDENT 7.0 .TP .B Parameters \fBcoordination_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_for_state_corruption() Make sure state tracking information hasn\(aqt been deleted unexpectedly. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B open(job) Create the context manager around tasks prior and after a job has been run. .sp File operations are only permitted inside the context manager. .sp Implementations must only yield from within \fIwith super().open(job):\fP\&. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.Job\fP) \-\- The job instance of the toil job to run. .TP .B Return type Generator[None, None, None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeGlobalFile(localFileName, cleanup=False) Upload a file (as a path) to the job store. .sp If the file is in a FileStore\-managed temporary directory (i.e. from \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()\fP), it will become a local copy of the file, eligible for deletion by \fI\%toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile()\fP\&. .sp If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded again. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocalFileName\fP (\fI\%str\fP) \-\- The path to the local file to upload. The last path component (basename of the file) will remain associated with the file in the file store, if supported by the backing JobStore, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- if True then the copy of the global file will be deleted once the job and all its successors have completed running. If not the global file must be deleted manually. .UNINDENT .TP .B Returns an ID that can be used to retrieve the file. .TP .B Return type \fI\%toil.fileStores.FileID\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False) Make the file associated with fileStoreID available locally. .sp If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem. .sp If a user path is specified, it is used as the destination. If a user path isn\(aqt specified, the file is stored in the local temp directory with an encoded name. .sp The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile. .sp Implementations must call \fBlogAccess()\fP to report the download. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- job store id for the file .IP \(bu 2 \fBuserPath\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- a path to the name of file to which the global file will be copied or hard\-linked (see below). .IP \(bu 2 \fBcache\fP (\fI\%bool\fP) \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBmutable\fP (\fI\%bool\fP) \-\- Described in \fBtoil.fileStores.CachingFileStore.readGlobalFile()\fP .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- True if caller can accept symlink, False if caller can only accept a normal file or hardlink .UNINDENT .TP .B Returns An absolute path to a local, temporary copy of the file keyed by fileStoreID. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readGlobalFileStream(fileStoreID: \fI\%str\fP, encoding: Literal[None] = None, errors: \fI\%str\fP | \fI\%None\fP = None) -> ContextManager[IO[\fI\%bytes\fP]] .TP .B readGlobalFileStream(fileStoreID: \fI\%str\fP, encoding: \fI\%str\fP, errors: \fI\%str\fP | \fI\%None\fP = None) -> ContextManager[IO[\fI\%str\fP]] Read a stream from the job store; similar to readGlobalFile. .sp The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBencoding\fP \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .UNINDENT .sp Implementations must call \fBlogAccess()\fP to report the download. .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be read from. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exportFile(jobStoreFileID, dstUrl) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdstUrl\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B export_file(file_id, dst_uri) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B deleteLocalFile(fileStoreID) Delete local copies of files associated with the provided job store ID. .sp Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession. .sp The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job\(aqs file\-store\-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job\(aqs file\-store\-provided temp directory. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fI\%str\fP) \-\- File Store ID of the file to be deleted. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B deleteGlobalFile(fileStoreID) Delete local files and then permanently deletes them from the job store. .sp To ensure that the job can be restarted if necessary, the delete will not happen until after the job\(aqs run method has completed. .INDENT 7.0 .TP .B Parameters \fBfileStoreID\fP (\fI\%str\fP) \-\- the File Store ID of the file to be deleted. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B waitForCommit() Blocks while startCommit is running. .sp This function is called by this job\(aqs successor to ensure that it does not begin modifying the job store until after this job has finished doing so. .sp Might be called when startCommit is never called on a particular instance, in which case it does not block. .INDENT 7.0 .TP .B Returns Always returns True .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B startCommit(jobState=False) Update the status of the job on the disk. .sp May bump the version number of the job. .sp May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call. .INDENT 7.0 .TP .B Parameters \fBjobState\fP (\fI\%bool\fP) \-\- If True, commit the state of the FileStore\(aqs job, and file deletes. Otherwise, commit only file creates/updates. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __del__() Cleanup function that is run when destroying the class instance. Nothing to do since there are no async write events. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod shutdown(shutdown_info) .INDENT 7.0 .TP .B Parameters \fBshutdown_info\fP (\fI\%str\fP) \-\- The coordination directory. .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS Package Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%FileID\fP T} T{ A small wrapper around Python\(aqs builtin string class. T} _ .TE .INDENT 0.0 .TP .B class toil.fileStores.FileID(fileStoreID, size, executable=False) Bases: \fI\%str\fP .sp A small wrapper around Python\(aqs builtin string class. .sp It is used to represent a file\(aqs ID in the file store, and has a size attribute that is the file\(aqs size in bytes. This object is returned by importFile and writeGlobalFile. .sp Calls into the file store can use bare strings; size will be queried from the job store if unavailable in the ID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsize\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBexecutable\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pack() Pack the FileID into a string so it can be passed through external code. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod forPath(fileStoreID, filePath) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfileStoreID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfilePath\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%FileID\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod unpack(packedFileStoreID) Unpack the result of pack() into a FileID object. .INDENT 7.0 .TP .B Parameters \fBpackedFileStoreID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%FileID\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.jobStores\fP .SS Subpackages .SS \fI\%toil.jobStores.aws\fP .SS Submodules .SS \fI\%toil.jobStores.aws.jobStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AWSJobStore\fP T} T{ A job store that uses Amazon\(aqs S3 for file storage and SimpleDB for storing job info and T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%boto3_session\fP T} T{ T} _ T{ \fI\%s3_boto3_resource\fP T} T{ T} _ T{ \fI\%s3_boto3_client\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%CONSISTENCY_TICKS\fP T} T{ T} _ T{ \fI\%CONSISTENCY_TIME\fP T} T{ T} _ T{ \fI\%aRepr\fP T} T{ T} _ T{ \fI\%custom_repr\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.boto3_session .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.s3_boto3_resource .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.s3_boto3_client .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.logger .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.CONSISTENCY_TICKS = 5 .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.CONSISTENCY_TIME = 1 .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.aws.jobStore.ChecksumError Bases: \fI\%Exception\fP .sp Raised when a download from AWS does not contain the correct data. .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.aws.jobStore.AWSJobStore(locator, partSize=50 << 20) Bases: \fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP .sp A job store that uses Amazon\(aqs S3 for file storage and SimpleDB for storing job info and enforcing strong consistency on the S3 file storage. There will be SDB domains for jobs and files and a versioned S3 bucket for file contents. Job objects are pickled, compressed, partitioned into chunks of 1024 bytes and each chunk is stored as a an attribute of the SDB item representing the job. UUIDs are used to identify jobs and files. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocator\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBpartSize\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class FileInfo(fileID, ownerID, encrypted, version=None, content=None, numContentChunks=0, checksum=None) Bases: \fI\%toil.jobStores.aws.utils.SDBHelper\fP .sp Represents a file in this job store. .INDENT 7.0 .TP .B property fileID .UNINDENT .INDENT 7.0 .TP .B property ownerID .UNINDENT .INDENT 7.0 .TP .B property version .UNINDENT .INDENT 7.0 .TP .B property previousVersion .UNINDENT .INDENT 7.0 .TP .B property content .UNINDENT .INDENT 7.0 .TP .B property checksum .UNINDENT .INDENT 7.0 .TP .B outer .INDENT 7.0 .TP .B Type \fI\%AWSJobStore\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod create(ownerID) .UNINDENT .INDENT 7.0 .TP .B classmethod presenceIndicator() The key that is guaranteed to be present in the return value of binaryToAttributes(). Assuming that binaryToAttributes() is used with SDB\(aqs PutAttributes, the return value of this method could be used to detect the presence/absence of an item in SDB. .UNINDENT .INDENT 7.0 .TP .B classmethod exists(jobStoreFileID) .UNINDENT .INDENT 7.0 .TP .B classmethod load(jobStoreFileID) .UNINDENT .INDENT 7.0 .TP .B classmethod loadOrCreate(jobStoreFileID, ownerID, encrypted) .UNINDENT .INDENT 7.0 .TP .B classmethod loadOrFail(jobStoreFileID, customName=None) .INDENT 7.0 .TP .B Return type \fI\%AWSJobStore.FileInfo\fP .TP .B Returns an instance of this class representing the file with the given ID .TP .B Raises \fI\%NoSuchFileException\fP \-\- if given file does not exist .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod fromItem(item) Convert an SDB item to an instance of this class. .INDENT 7.0 .UNINDENT .UNINDENT .INDENT 7.0 .TP .B toItem() Convert this instance to an attribute dictionary suitable for SDB put_attributes(). .INDENT 7.0 .TP .B Return type (\fI\%dict\fP,\fI\%int\fP) .TP .B Returns the attributes dict and an integer specifying the the number of chunk attributes in the dictionary that are used for storing inlined content. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static maxInlinedSize() .UNINDENT .INDENT 7.0 .TP .B save() .UNINDENT .INDENT 7.0 .TP .B upload(localFilePath, calculateChecksum=True) .UNINDENT .INDENT 7.0 .TP .B uploadStream(multipart=True, allowInlining=True, encoding=None, errors=None) Context manager that gives out a binary or text mode upload stream to upload data. .UNINDENT .INDENT 7.0 .TP .B copyFrom(srcObj) Copies contents of source key into this file. .INDENT 7.0 .TP .B Parameters \fBsrcObj\fP (\fIS3.Object\fP) \-\- The key (object) that will be copied from .UNINDENT .UNINDENT .INDENT 7.0 .TP .B copyTo(dstObj) Copies contents of this file to the given key. .INDENT 7.0 .TP .B Parameters \fBdstObj\fP (\fIS3.Object\fP) \-\- The key (object) to copy this file\(aqs content to .UNINDENT .UNINDENT .INDENT 7.0 .TP .B download(localFilePath, verifyChecksum=True) .UNINDENT .INDENT 7.0 .TP .B downloadStream(verifyChecksum=True, encoding=None, errors=None) Context manager that gives out a download stream to download data. .UNINDENT .INDENT 7.0 .TP .B delete() .UNINDENT .INDENT 7.0 .TP .B getSize() Return the size of the referenced item in bytes. .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property sseKeyPath .UNINDENT .INDENT 7.0 .TP .B bucketNameRe .UNINDENT .INDENT 7.0 .TP .B minBucketNameLen = 3 .UNINDENT .INDENT 7.0 .TP .B maxBucketNameLen = 63 .UNINDENT .INDENT 7.0 .TP .B maxNameLen = 10 .UNINDENT .INDENT 7.0 .TP .B nameSeparator = \(aq\-\-\(aq .UNINDENT .INDENT 7.0 .TP .B jobsPerBatchInsert = 25 .UNINDENT .INDENT 7.0 .TP .B itemsPerBatchDelete = 25 .UNINDENT .INDENT 7.0 .TP .B sharedFileOwnerID .UNINDENT .INDENT 7.0 .TP .B statsFileOwnerID .UNINDENT .INDENT 7.0 .TP .B readStatsFileOwnerID .UNINDENT .INDENT 7.0 .TP .B versionings .UNINDENT .INDENT 7.0 .TP .B initialize(config) Initialize this job store. .sp Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store. .INDENT 7.0 .TP .B Parameters \fBconfig\fP \-\- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID. .TP .B Raises \fI\%JobStoreExistsException\fP \-\- if the physical storage for this job store already exists .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resume() Connect this instance to the physical storage it represents and load the Toil configuration into the \fBAbstractJobStore.config\fP attribute. .INDENT 7.0 .TP .B Raises \fI\%NoSuchJobStoreException\fP \-\- if the physical storage for this job store doesn\(aqt exist .UNINDENT .UNINDENT .INDENT 7.0 .TP .B batch() If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released. .UNINDENT .INDENT 7.0 .TP .B assign_job_id(job_description) Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription. .sp Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated. .INDENT 7.0 .TP .B Parameters \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- The JobDescription to give an ID to .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create_job(job_description) Writes the given JobDescription to the job store. The job must have an ID assigned already. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Returns The JobDescription passed. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B job_exists(job_id) Indicates whether a description of the job with the specified jobStoreID exists in the job store .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B jobs() Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object .INDENT 7.0 .TP .B Returns Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs .TP .B Return type Iterator[toil.job.jobDescription] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_job(job_id) Loads the description of the job referenced by the given ID, assigns it the job store\(aqs config, and returns it. .sp May declare the job to have failed (see \fI\%toil.job.JobDescription.setupJobAfterFailure()\fP) if there is evidence of a failed update attempt. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP \-\- the ID of the job to load .TP .B Raises \fI\%NoSuchJobException\fP \-\- if there is no job with the given ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_job(job_description) Persists changes to the state of the given JobDescription in this store atomically. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- the job to write to this job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_job(job_id) Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. .sp This operation is idempotent, i.e. deleting a job twice or deleting a non\-existent job will succeed silently. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to delete from this job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_empty_file_store_id(jobStoreID=None, cleanup=False, basename=None) Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .UNINDENT .TP .B Returns a jobStoreFileID that references the newly created file and can be used to reference the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_file(local_path, job_id=None, cleanup=False) Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns an ID referencing the newly created file and can be used to read the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future. .TP .B Return type Iterator[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None) Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencrypted\fP (\fI\%bool\fP) \-\- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns a context manager yielding a writable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_file(file_id, local_path) Replaces the existing version of a file in the job store. .sp Throws an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBlocal_path\fP \-\- the local path to a file that will overwrite the current version in the job store .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_file_stream(file_id, encoding=None, errors=None) Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B file_exists(file_id) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP \-\- an ID referencing the file to be checked .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_file_size(file_id) Get the size of the given file in bytes, or 0 if it does not exist when queried. .sp Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file(file_id, local_path, symlink=False) Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation\-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed. .sp The file at the given local path may not be modified after this method returns! .sp Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to be copied .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path indicating where to place the contents of the given file in the job store .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file_stream(file_id, encoding=None, errors=None) Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to get a readable file handle for .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a file handle which can be read from .TP .B Return type Iterator[Union[IO[\fI\%bytes\fP], IO[\fI\%str\fP]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_shared_file_stream(shared_file_name, encoding=None, errors=None) Returns a context manager yielding a readable file handle to the global file referenced by the given name. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a readable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_file(file_id) Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non\-existent file will succeed silently. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to delete .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_logs(msg) Stores a message as a log in the jobstore. .INDENT 7.0 .TP .B Parameters \fBmsg\fP (\fI\%str\fP) \-\- the string to be written .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_logs(callback, read_all=False) Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages). .sp Only unread logs will be read unless the read_all parameter is set. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcallback\fP (\fICallable\fP) \-\- a function to be applied to each of the stats file handles found .IP \(bu 2 \fBread_all\fP (\fI\%bool\fP) \-\- a boolean indicating whether to read the already processed stats files in addition to the unread stats files .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns the number of stats files processed .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_public_url(jobStoreFileID) Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBfile_name\fP (\fI\%str\fP) \-\- the jobStoreFileID of the file to generate a URL for .TP .B Raises \fI\%NoSuchFileException\fP \-\- if the specified file does not exist in this job store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_shared_public_url(shared_file_name) Differs from \fBgetPublicUrl()\fP in that this method is for generating URLs for shared files written by \fBwriteSharedFileStream()\fP\&. .sp Returns a publicly accessible URL to the given file in the job store. The returned URL starts with \(aqhttp:\(aq, \(aqhttps:\(aq or \(aqfile:\(aq. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBshared_file_name\fP (\fI\%str\fP) \-\- The name of the shared file to generate a publically accessible url for. .TP .B Raises \fI\%NoSuchFileException\fP \-\- raised if the specified file does not exist in the store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B destroy() The inverse of \fI\%initialize()\fP, this method deletes the physical storage represented by this instance. While not being atomic, this method \fIis\fP at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.aRepr .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.jobStore.custom_repr .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.aws.jobStore.BucketLocationConflictException(bucketRegion) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .UNINDENT .SS \fI\%toil.jobStores.aws.utils\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%SDBHelper\fP T} T{ A mixin with methods for storing limited amounts of binary data in an SDB item T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%fileSizeAndTime\fP(localFilePath) T} T{ T} _ T{ \fI\%uploadFromPath\fP(localFilePath, resource, bucketName, fileID) T} T{ Uploads a file to s3, using multipart uploading if applicable T} _ T{ \fI\%uploadFile\fP(readable, resource, bucketName, fileID[, ...]) T} T{ Upload a readable object to s3, using multipart uploading if applicable. T} _ T{ \fI\%copyKeyMultipart\fP(resource, srcBucketName, srcKeyName, ...) T} T{ Copies a key from a source key to a destination key in multiple parts. Note that if the T} _ T{ \fI\%monkeyPatchSdbConnection\fP(sdb) T} T{ .INDENT 0.0 .TP .B type sdb SDBConnection .UNINDENT T} _ T{ \fI\%sdb_unavailable\fP(e) T} T{ T} _ T{ \fI\%no_such_sdb_domain\fP(e) T} T{ T} _ T{ \fI\%retryable_ssl_error\fP(e) T} T{ T} _ T{ \fI\%retryable_sdb_errors\fP(e) T} T{ T} _ T{ \fI\%retry_sdb\fP([delays, timeout, predicate]) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%DIAL_SPECIFIC_REGION_CONFIG\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.jobStores.aws.utils.logger .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.DIAL_SPECIFIC_REGION_CONFIG .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.aws.utils.SDBHelper A mixin with methods for storing limited amounts of binary data in an SDB item .sp .EX >>> import os >>> H=SDBHelper >>> H.presenceIndicator() u\(aqnumChunks\(aq >>> H.binaryToAttributes(None)[\(aqnumChunks\(aq] 0 >>> H.attributesToBinary({u\(aqnumChunks\(aq: 0}) (None, 0) >>> H.binaryToAttributes(b\(aq\(aq) {u\(aq000\(aq: b\(aqVQ==\(aq, u\(aqnumChunks\(aq: 1} >>> H.attributesToBinary({u\(aqnumChunks\(aq: 1, u\(aq000\(aq: b\(aqVQ==\(aq}) (b\(aq\(aq, 1) .EE .sp Good pseudo\-random data is very likely smaller than its bzip2ed form. Subtract 1 for the type character, i.e \(aqC\(aq or \(aqU\(aq, with which the string is prefixed. We should get one full chunk: .sp .EX >>> s = os.urandom(H.maxRawValueSize\-1) >>> d = H.binaryToAttributes(s) >>> len(d), len(d[\(aq000\(aq]) (2, 1024) >>> H.attributesToBinary(d) == (s, 1) True .EE .sp One byte more and we should overflow four bytes into the second chunk, two bytes for base64\-encoding the additional character and two bytes for base64\-padding to the next quartet. .sp .EX >>> s += s[0:1] >>> d = H.binaryToAttributes(s) >>> len(d), len(d[\(aq000\(aq]), len(d[\(aq001\(aq]) (3, 1024, 4) >>> H.attributesToBinary(d) == (s, 2) True .EE .INDENT 7.0 .TP .B maxAttributesPerItem = 256 .UNINDENT .INDENT 7.0 .TP .B maxValueSize = 1024 .UNINDENT .INDENT 7.0 .TP .B maxRawValueSize .UNINDENT .INDENT 7.0 .TP .B classmethod maxBinarySize(extraReservedChunks=0) .UNINDENT .INDENT 7.0 .TP .B classmethod binaryToAttributes(binary) Turn a bytestring, or None, into SimpleDB attributes. .UNINDENT .INDENT 7.0 .TP .B classmethod presenceIndicator() The key that is guaranteed to be present in the return value of binaryToAttributes(). Assuming that binaryToAttributes() is used with SDB\(aqs PutAttributes, the return value of this method could be used to detect the presence/absence of an item in SDB. .UNINDENT .INDENT 7.0 .TP .B classmethod attributesToBinary(attributes) .INDENT 7.0 .TP .B Return type (\fI\%str\fP|None,\fI\%int\fP) .TP .B Returns the binary data and the number of chunks it was composed from .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.fileSizeAndTime(localFilePath) .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.uploadFromPath(localFilePath, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20) Uploads a file to s3, using multipart uploading if applicable .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocalFilePath\fP (\fI\%str\fP) \-\- Path of the file to upload to s3 .IP \(bu 2 \fBresource\fP (\fIS3.Resource\fP) \-\- boto3 resource .IP \(bu 2 \fBbucketName\fP (\fI\%str\fP) \-\- name of the bucket to upload to .IP \(bu 2 \fBfileID\fP (\fI\%str\fP) \-\- the name of the file to upload to .IP \(bu 2 \fBheaderArgs\fP (\fI\%dict\fP) \-\- http headers to use when uploading \- generally used for encryption purposes .IP \(bu 2 \fBpartSize\fP (\fI\%int\fP) \-\- max size of each part in the multipart upload, in bytes .UNINDENT .TP .B Returns version of the newly uploaded file .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.uploadFile(readable, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20) Upload a readable object to s3, using multipart uploading if applicable. :param readable: a readable stream or a file path to upload to s3 :param S3.Resource resource: boto3 resource :param str bucketName: name of the bucket to upload to :param str fileID: the name of the file to upload to :param dict headerArgs: http headers to use when uploading \- generally used for encryption purposes :param int partSize: max size of each part in the multipart upload, in bytes :return: version of the newly uploaded file .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbucketName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfileID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBheaderArgs\fP (\fIOptional\fP\fI[\fP\fI\%dict\fP\fI]\fP) \-\- .IP \(bu 2 \fBpartSize\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.aws.utils.ServerSideCopyProhibitedError Bases: \fI\%RuntimeError\fP .sp Raised when AWS refuses to perform a server\-side copy between S3 keys, and insists that you pay to download and upload the data yourself instead. .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.copyKeyMultipart(resource, srcBucketName, srcKeyName, srcKeyVersion, dstBucketName, dstKeyName, sseAlgorithm=None, sseKey=None, copySourceSseAlgorithm=None, copySourceSseKey=None) Copies a key from a source key to a destination key in multiple parts. Note that if the destination key exists it will be overwritten implicitly, and if it does not exist a new key will be created. If the destination bucket does not exist an error will be raised. .sp This function will always do a fast, server\-side copy, at least until/unless <\fI\%https://github.com/boto/boto3/issues/3270\fP> is fixed. In some situations, a fast, server\-side copy is not actually possible. For example, when residing in an AWS VPC with an S3 VPC Endpoint configured, copying from a bucket in another region to a bucket in your own region cannot be performed server\-side. This is because the VPC Endpoint S3 API servers refuse to perform server\-side copies between regions, the source region\(aqs API servers refuse to initiate the copy and refer you to the destination bucket\(aqs region\(aqs API servers, and the VPC routing tables are configured to redirect all access to the current region\(aqs S3 API servers to the S3 Endpoint API servers instead. .sp If a fast server\-side copy is not actually possible, a ServerSideCopyProhibitedError will be raised. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBresource\fP (\fImypy_boto3_s3.S3ServiceResource\fP) \-\- boto3 resource .IP \(bu 2 \fBsrcBucketName\fP (\fI\%str\fP) \-\- The name of the bucket to be copied from. .IP \(bu 2 \fBsrcKeyName\fP (\fI\%str\fP) \-\- The name of the key to be copied from. .IP \(bu 2 \fBsrcKeyVersion\fP (\fI\%str\fP) \-\- The version of the key to be copied from. .IP \(bu 2 \fBdstBucketName\fP (\fI\%str\fP) \-\- The name of the destination bucket for the copy. .IP \(bu 2 \fBdstKeyName\fP (\fI\%str\fP) \-\- The name of the destination key that will be created or overwritten. .IP \(bu 2 \fBsseAlgorithm\fP (\fI\%str\fP) \-\- Server\-side encryption algorithm for the destination. .IP \(bu 2 \fBsseKey\fP (\fI\%str\fP) \-\- Server\-side encryption key for the destination. .IP \(bu 2 \fBcopySourceSseAlgorithm\fP (\fI\%str\fP) \-\- Server\-side encryption algorithm for the source. .IP \(bu 2 \fBcopySourceSseKey\fP (\fI\%str\fP) \-\- Server\-side encryption key for the source. .UNINDENT .TP .B Return type \fI\%str\fP .TP .B Returns The version of the copied file (or None if versioning is not enabled for dstBucket). .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.monkeyPatchSdbConnection(sdb) .INDENT 7.0 .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.sdb_unavailable(e) .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.no_such_sdb_domain(e) .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.retryable_ssl_error(e) .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.retryable_sdb_errors(e) .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.aws.utils.retry_sdb(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_sdb_errors) .UNINDENT .SS Submodules .SS \fI\%toil.jobStores.abstractJobStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AbstractJobStore\fP T} T{ Represents the physical storage for the jobs and files in a Toil workflow. T} _ T{ \fI\%JobStoreSupport\fP T} T{ A mostly fake JobStore to access URLs not really associated with real job T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.jobStores.abstractJobStore.logger .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.ProxyConnectionError Bases: \fI\%BaseException\fP .sp Dummy class. .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.InvalidImportExportUrlException(url) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .INDENT 7.0 .TP .B Parameters \fBurl\fP (\fI\%urllib.parse.ParseResult\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.UnimplementedURLException(url, operation) Bases: \fI\%RuntimeError\fP .sp Unspecified run\-time error. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBurl\fP (\fI\%urllib.parse.ParseResult\fP) \-\- .IP \(bu 2 \fBoperation\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.NoSuchJobException(jobStoreID) Bases: \fI\%Exception\fP .sp Indicates that the specified job does not exist. .INDENT 7.0 .TP .B Parameters \fBjobStoreID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.ConcurrentFileModificationException(jobStoreFileID) Bases: \fI\%Exception\fP .sp Indicates that the file was attempted to be modified by multiple processes at once. .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.NoSuchFileException(jobStoreFileID, customName=None, *extra) Bases: \fI\%Exception\fP .sp Indicates that the specified file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBcustomName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBextra\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.NoSuchJobStoreException(locator) Bases: \fI\%Exception\fP .sp Indicates that the specified job store does not exist. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.abstractJobStore.JobStoreExistsException(locator) Bases: \fI\%Exception\fP .sp Indicates that the specified job store already exists. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.abstractJobStore.AbstractJobStore(locator) Bases: \fI\%abc.ABC\fP .sp Represents the physical storage for the jobs and files in a Toil workflow. .sp JobStores are responsible for storing \fI\%toil.job.JobDescription\fP (which relate jobs to each other) and files. .sp Actual \fI\%toil.job.Job\fP objects are stored in files, referenced by JobDescriptions. All the non\-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs. .sp To actually get ahold of a \fI\%toil.job.Job\fP, use \fI\%toil.job.Job.loadJob()\fP with a JobStore and the relevant JobDescription. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B property config: \fI\%toil.common.Config\fP Return the Toil configuration associated with this job store. .INDENT 7.0 .TP .B Return type \fI\%toil.common.Config\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property locator: \fI\%str\fP Get the locator that defines the job store, which can be used to connect to it. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rootJobStoreIDFileName = \(aqrootJobStoreID\(aq .UNINDENT .INDENT 7.0 .TP .B publicUrlExpiration .UNINDENT .INDENT 7.0 .TP .B sharedFileNameRegex .UNINDENT .INDENT 7.0 .TP .B initialize(config) Initialize this job store. .sp Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID. .TP .B Raises \fI\%JobStoreExistsException\fP \-\- if the physical storage for this job store already exists .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeConfig() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_config() Persists the value of the \fI\%AbstractJobStore.config\fP attribute to the job store, so that it can be retrieved later by other instances of this class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resume() Connect this instance to the physical storage it represents and load the Toil configuration into the \fI\%AbstractJobStore.config\fP attribute. .INDENT 7.0 .TP .B Raises \fI\%NoSuchJobStoreException\fP \-\- if the physical storage for this job store doesn\(aqt exist .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setRootJob(rootJobStoreID) Set the root job of the workflow backed by this job store. .INDENT 7.0 .TP .B Parameters \fBrootJobStoreID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_root_job(job_id) Set the root job of the workflow backed by this job store. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%toil.fileStores.FileID\fP) \-\- The ID of the job to set as root .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B loadRootJob() .INDENT 7.0 .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_root_job() Loads the JobDescription for the root job in the current job store. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If no root job is set or if the root job doesn\(aqt exist in this job store .TP .B Returns The root job. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B createRootJob(desc) .INDENT 7.0 .TP .B Parameters \fBdesc\fP (\fI\%toil.job.JobDescription\fP) \-\- .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create_root_job(job_description) Create the given JobDescription and set it as the root job in this job store. .INDENT 7.0 .TP .B Parameters \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- JobDescription to save and make the root job. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRootJobReturnValue() .INDENT 7.0 .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_root_job_return_value() Parse the return value from the root job. .sp Raises an exception if the root job hasn\(aqt fulfilled its promise yet. .INDENT 7.0 .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B importFile(srcUrl: \fI\%str\fP, sharedFileName: \fI\%str\fP, hardlink: \fI\%bool\fP = False, symlink: \fI\%bool\fP = True) -> \fI\%None\fP .TP .B importFile(srcUrl: \fI\%str\fP, sharedFileName: \fI\%None\fP = None, hardlink: \fI\%bool\fP = False, symlink: \fI\%bool\fP = True) -> \fI\%toil.fileStores.FileID\fP .UNINDENT .INDENT 7.0 .TP .B import_file(src_uri: \fI\%str\fP, shared_file_name: \fI\%str\fP, hardlink: \fI\%bool\fP = False, symlink: \fI\%bool\fP = True) -> \fI\%None\fP .TP .B import_file(src_uri: \fI\%str\fP, shared_file_name: \fI\%None\fP = None, hardlink: \fI\%bool\fP = False, symlink: \fI\%bool\fP = True) -> \fI\%toil.fileStores.FileID\fP Imports the file at the given URL into job store. The ID of the newly imported file is returned. If the name of a shared file name is provided, the file will be imported as such and None is returned. If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded. .sp Currently supported schemes are: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 .INDENT 2.0 .TP .B \(aqs3\(aq for objects in Amazon S3 e.g. s3://bucket/key .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqfile\(aq for local files e.g. \fI\%file:///local/file/path\fP .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqhttp\(aq e.g. \fI\%http://someurl.com/path\fP .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqgs\(aq e.g. gs://bucket/file .UNINDENT .UNINDENT .UNINDENT .UNINDENT .sp Raises FileNotFoundError if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. It must be a file, not a directory or prefix. .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- Optional name to assign to the imported file within the job store .UNINDENT .TP .B Returns The jobStoreFileID of the imported file or None if shared_file_name was given .TP .B Return type \fI\%toil.fileStores.FileID\fP or None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exportFile(jobStoreFileID, dstUrl) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdstUrl\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B export_file(file_id, dst_uri) Exports file to destination pointed at by the destination URL. The exported file will be executable if and only if it was originally uploaded from an executable file on the local filesystem. .sp Refer to \fI\%AbstractJobStore.import_file()\fP documentation for currently supported URL schemes. .sp Note that the helper method _exportFile is used to read from the source and write to destination. To implement any optimizations that circumvent this, the _exportFile method should be overridden by subclasses of AbstractJobStore. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- The id of the file in the job store that should be exported. .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod url_exists(src_uri) Return True if the file at the given URI exists, and False otherwise. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_size(src_uri) Get the size in bytes of the file at the given URL, or None if it cannot be obtained. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. .TP .B Return type Optional[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_is_directory(src_uri) Return True if the thing at the given URL is a directory, and False if it is a file. The URL may or may not end in \(aq/\(aq. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod list_url(src_uri) List the directory at the given URL. Returned path components can be joined with \(aq/\(aq onto the passed URL to form new URLs. Those that end in \(aq/\(aq correspond to directories. The provided URL may or may not end with \(aq/\(aq. .sp Currently supported schemes are: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 .INDENT 2.0 .TP .B \(aqs3\(aq for objects in Amazon S3 e.g. s3://bucket/prefix/ .UNINDENT .IP \(bu 2 .INDENT 2.0 .TP .B \(aqfile\(aq for local files e.g. \fI\%file:///local/dir/path/\fP .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- URL that points to a directory or prefix in the storage mechanism of a supported URL scheme e.g. a prefix in an AWS s3 bucket. .TP .B Returns A list of URL components in the given directory, already URL\-encoded. .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod read_from_url(src_uri, writable) Read the given URL and write its content into the given writable stream. .sp Raises FileNotFoundError if the URL doesn\(aqt exist. .INDENT 7.0 .TP .B Returns The size of the file in bytes and whether the executable permission bit is set .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_uri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwritable\fP (\fIIO\fP\fI[\fP\fI\%bytes\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Tuple[\fI\%int\fP, \fI\%bool\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod open_url(src_uri) Read from the given URI. .sp Raises FileNotFoundError if the URL doesn\(aqt exist. .sp Has a readable stream interface, unlike \fI\%read_from_url()\fP which takes a writable stream. .INDENT 7.0 .TP .B Parameters \fBsrc_uri\fP (\fI\%str\fP) \-\- .TP .B Return type IO[\fI\%bytes\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract destroy() The inverse of \fI\%initialize()\fP, this method deletes the physical storage represented by this instance. While not being atomic, this method \fIis\fP at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getEnv() .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_env() Returns a dictionary of environment variables that this job store requires to be set in order to function properly on a worker. .INDENT 7.0 .TP .B Return type \fI\%dict\fP[\fI\%str\fP,\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clean(jobCache=None) Function to cleanup the state of a job store after a restart. .sp Fixes jobs that might have been partially updated. Resets the try counts and removes jobs that are not successors of the current root job. .INDENT 7.0 .TP .B Parameters \fBjobCache\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%toil.job.TemporaryID\fP\fI]\fP\fI, \fP\fI\%toil.job.JobDescription\fP\fI]\fP\fI]\fP) \-\- if a value it must be a dict from job ID keys to JobDescription object values. Jobs will be loaded from the cache (which can be downloaded from the job store in a batch) instead of piecemeal when recursed into. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B assignID(jobDescription) .INDENT 7.0 .TP .B Parameters \fBjobDescription\fP (\fI\%toil.job.JobDescription\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract assign_job_id(job_description) Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription. .sp Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated. .INDENT 7.0 .TP .B Parameters \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- The JobDescription to give an ID to .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B batch() If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released. .INDENT 7.0 .TP .B Return type Iterator[None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create(jobDescription) .INDENT 7.0 .TP .B Parameters \fBjobDescription\fP (\fI\%toil.job.JobDescription\fP) \-\- .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract create_job(job_description) Writes the given JobDescription to the job store. The job must have an ID assigned already. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Returns The JobDescription passed. .TP .B Return type \fI\%toil.job.JobDescription\fP .TP .B Parameters \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exists(jobStoreID) .INDENT 7.0 .TP .B Parameters \fBjobStoreID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract job_exists(job_id) Indicates whether a description of the job with the specified jobStoreID exists in the job store .INDENT 7.0 .TP .B Return type \fI\%bool\fP .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getPublicUrl(fileName) .INDENT 7.0 .TP .B Parameters \fBfileName\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_public_url(file_name) Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBfile_name\fP (\fI\%str\fP) \-\- the jobStoreFileID of the file to generate a URL for .TP .B Raises \fI\%NoSuchFileException\fP \-\- if the specified file does not exist in this job store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getSharedPublicUrl(sharedFileName) .INDENT 7.0 .TP .B Parameters \fBsharedFileName\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_shared_public_url(shared_file_name) Differs from \fI\%getPublicUrl()\fP in that this method is for generating URLs for shared files written by \fI\%writeSharedFileStream()\fP\&. .sp Returns a publicly accessible URL to the given file in the job store. The returned URL starts with \(aqhttp:\(aq, \(aqhttps:\(aq or \(aqfile:\(aq. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBshared_file_name\fP (\fI\%str\fP) \-\- The name of the shared file to generate a publically accessible url for. .TP .B Raises \fI\%NoSuchFileException\fP \-\- raised if the specified file does not exist in the store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load(jobStoreID) .INDENT 7.0 .TP .B Parameters \fBjobStoreID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract load_job(job_id) Loads the description of the job referenced by the given ID, assigns it the job store\(aqs config, and returns it. .sp May declare the job to have failed (see \fI\%toil.job.JobDescription.setupJobAfterFailure()\fP) if there is evidence of a failed update attempt. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to load .TP .B Raises \fI\%NoSuchJobException\fP \-\- if there is no job with the given ID .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update(jobDescription) .INDENT 7.0 .TP .B Parameters \fBjobDescription\fP (\fI\%toil.job.JobDescription\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract update_job(job_description) Persists changes to the state of the given JobDescription in this store atomically. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- the job to write to this job store .IP \(bu 2 \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete(jobStoreID) .INDENT 7.0 .TP .B Parameters \fBjobStoreID\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract delete_job(job_id) Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. .sp This operation is idempotent, i.e. deleting a job twice or deleting a non\-existent job will succeed silently. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to delete from this job store .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract jobs() Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object .INDENT 7.0 .TP .B Returns Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs .TP .B Return type Iterator[toil.job.jobDescription] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeFile(localFilePath, jobStoreID=None, cleanup=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocalFilePath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobStoreID\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_file(local_path, job_id=None, cleanup=False) Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns an ID referencing the newly created file and can be used to read the file in the future. .TP .B Return type \fI\%str\fP .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjob_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeFileStream(jobStoreID=None, cleanup=False, basename=None, encoding=None, errors=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreID\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBbasename\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type ContextManager[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .TP .B Return type Iterator[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future. .TP .B Return type Iterator[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBbasename\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getEmptyFileStoreID(jobStoreID=None, cleanup=False, basename=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreID\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBbasename\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_empty_file_store_id(job_id=None, cleanup=False, basename=None) Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .UNINDENT .TP .B Returns a jobStoreFileID that references the newly created file and can be used to reference the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readFile(jobStoreFileID, localFilePath, symlink=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBlocalFilePath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_file(file_id, local_path, symlink=False) Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation\-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed. .sp The file at the given local path may not be modified after this method returns! .sp Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to be copied .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path indicating where to place the contents of the given file in the job store .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readFileStream(jobStoreFileID, encoding=None, errors=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Union[ContextManager[IO[\fI\%bytes\fP]], ContextManager[IO[\fI\%str\fP]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file_stream(file_id: \fI\%toil.fileStores.FileID\fP | \fI\%str\fP, encoding: Literal[None] = None, errors: \fI\%str\fP | \fI\%None\fP = None) -> ContextManager[IO[\fI\%bytes\fP]] .TP .B read_file_stream(file_id: \fI\%toil.fileStores.FileID\fP | \fI\%str\fP, encoding: \fI\%str\fP, errors: \fI\%str\fP | \fI\%None\fP = None) -> ContextManager[IO[\fI\%str\fP]] Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to get a readable file handle for .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a file handle which can be read from .TP .B Return type Iterator[Union[IO[\fI\%bytes\fP], IO[\fI\%str\fP]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B deleteFile(jobStoreFileID) .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract delete_file(file_id) Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non\-existent file will succeed silently. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to delete .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fileExists(jobStoreFileID) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract file_exists(file_id) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getFileSize(jobStoreFileID) Get the size of the given file in bytes. .INDENT 7.0 .TP .B Parameters \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_file_size(file_id) Get the size of the given file in bytes, or 0 if it does not exist when queried. .sp Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B updateFile(jobStoreFileID, localFilePath) Replaces the existing version of a file in the job store. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBlocalFilePath\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract update_file(file_id, local_path) Replaces the existing version of a file in the job store. .sp Throws an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path to a file that will overwrite the current version in the job store .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B updateFileStream(jobStoreFileID, encoding=None, errors=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type ContextManager[IO[Any]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract update_file_stream(file_id, encoding=None, errors=None) Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .TP .B Return type Iterator[IO[Any]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeSharedFileStream(sharedFileName, isProtected=None, encoding=None, errors=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsharedFileName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBisProtected\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type ContextManager[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None) Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencrypted\fP (\fI\%bool\fP) \-\- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns a context manager yielding a writable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readSharedFileStream(sharedFileName, encoding=None, errors=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsharedFileName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBencoding\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type ContextManager[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_shared_file_stream(shared_file_name, encoding=None, errors=None) Returns a context manager yielding a readable file handle to the global file referenced by the given name. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a readable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeStatsAndLogging(statsAndLoggingString) .INDENT 7.0 .TP .B Parameters \fBstatsAndLoggingString\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract write_logs(msg) Stores a message as a log in the jobstore. .INDENT 7.0 .TP .B Parameters \fBmsg\fP (\fI\%str\fP) \-\- the string to be written .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B readStatsAndLogging(callback, readAll=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcallback\fP (\fICallable\fP\fI[\fP\fIEllipsis\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBreadAll\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract read_logs(callback, read_all=False) Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages). .sp Only unread logs will be read unless the read_all parameter is set. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcallback\fP (\fICallable\fP) \-\- a function to be applied to each of the stats file handles found .IP \(bu 2 \fBread_all\fP (\fI\%bool\fP) \-\- a boolean indicating whether to read the already processed stats files in addition to the unread stats files .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns the number of stats files processed .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_leader_pid() Write the pid of this process to a file in the job store. .sp Overwriting the current contents of pid.log is a feature, not a bug of this method. Other methods will rely on always having the most current pid available. So far there is no reason to store any old pids. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_leader_pid() Read the pid of the leader process to a file in the job store. .INDENT 7.0 .TP .B Raises \fI\%NoSuchFileException\fP \-\- If the PID file doesn\(aqt exist. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_leader_node_id() Write the leader node id to the job store. This should only be called by the leader. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_leader_node_id() Read the leader node id stored in the job store. .INDENT 7.0 .TP .B Raises \fI\%NoSuchFileException\fP \-\- If the node ID file doesn\(aqt exist. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_kill_flag(kill=False) Write a file inside the job store that serves as a kill flag. .sp The initialized file contains the characters \(dqNO\(dq. This should only be changed when the user runs the \(dqtoil kill\(dq command. .sp Changing this file to a \(dqYES\(dq triggers a kill of the leader process. The workers are expected to be cleaned up by the leader. .INDENT 7.0 .TP .B Parameters \fBkill\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_kill_flag() Read the kill flag from the job store, and return True if the leader has been killed. False otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B default_caching() Jobstore\(aqs preference as to whether it likes caching or doesn\(aqt care about it. Some jobstores benefit from caching, however on some local configurations it can be flaky. .sp see \fI\%https://github.com/DataBiosphere/toil/issues/4218\fP .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.abstractJobStore.JobStoreSupport(locator) Bases: \fI\%AbstractJobStore\fP .sp A mostly fake JobStore to access URLs not really associated with real job stores. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .SS \fI\%toil.jobStores.conftest\fP .SS Module Contents .INDENT 0.0 .TP .B toil.jobStores.conftest.collect_ignore = [] .UNINDENT .SS \fI\%toil.jobStores.fileJobStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%FileJobStore\fP T} T{ A job store that uses a directory on a locally attached file system. To be compatible with T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.jobStores.fileJobStore.logger .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.fileJobStore.FileJobStore(path, fanOut=1000) Bases: \fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP .sp A job store that uses a directory on a locally attached file system. To be compatible with distributed batch systems, that file system must be shared by all worker nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfanOut\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B validDirs = \(aqabcdefghijklmnopqrstuvwxyz0123456789\(aq .UNINDENT .INDENT 7.0 .TP .B validDirsSet .UNINDENT .INDENT 7.0 .TP .B JOB_DIR_PREFIX = \(aqinstance\-\(aq .UNINDENT .INDENT 7.0 .TP .B JOB_NAME_DIR_PREFIX = \(aqkind\-\(aq .UNINDENT .INDENT 7.0 .TP .B BUFFER_SIZE = 10485760 .UNINDENT .INDENT 7.0 .TP .B default_caching() Jobstore\(aqs preference as to whether it likes caching or doesn\(aqt care about it. Some jobstores benefit from caching, however on some local configurations it can be flaky. .sp see \fI\%https://github.com/DataBiosphere/toil/issues/4218\fP .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .UNINDENT .INDENT 7.0 .TP .B initialize(config) Initialize this job store. .sp Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store. .INDENT 7.0 .TP .B Parameters \fBconfig\fP \-\- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID. .TP .B Raises \fI\%JobStoreExistsException\fP \-\- if the physical storage for this job store already exists .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resume() Connect this instance to the physical storage it represents and load the Toil configuration into the \fBAbstractJobStore.config\fP attribute. .INDENT 7.0 .TP .B Raises \fI\%NoSuchJobStoreException\fP \-\- if the physical storage for this job store doesn\(aqt exist .UNINDENT .UNINDENT .INDENT 7.0 .TP .B destroy() The inverse of \fI\%initialize()\fP, this method deletes the physical storage represented by this instance. While not being atomic, this method \fIis\fP at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow. .UNINDENT .INDENT 7.0 .TP .B assign_job_id(job_description) Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription. .sp Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated. .INDENT 7.0 .TP .B Parameters \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- The JobDescription to give an ID to .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create_job(job_description) Writes the given JobDescription to the job store. The job must have an ID assigned already. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Returns The JobDescription passed. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B batch() If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released. .UNINDENT .INDENT 7.0 .TP .B job_exists(job_id) Indicates whether a description of the job with the specified jobStoreID exists in the job store .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_public_url(jobStoreFileID) Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBfile_name\fP (\fI\%str\fP) \-\- the jobStoreFileID of the file to generate a URL for .TP .B Raises \fI\%NoSuchFileException\fP \-\- if the specified file does not exist in this job store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_shared_public_url(sharedFileName) Differs from \fBgetPublicUrl()\fP in that this method is for generating URLs for shared files written by \fBwriteSharedFileStream()\fP\&. .sp Returns a publicly accessible URL to the given file in the job store. The returned URL starts with \(aqhttp:\(aq, \(aqhttps:\(aq or \(aqfile:\(aq. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBshared_file_name\fP (\fI\%str\fP) \-\- The name of the shared file to generate a publically accessible url for. .TP .B Raises \fI\%NoSuchFileException\fP \-\- raised if the specified file does not exist in the store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_job(job_id) Loads the description of the job referenced by the given ID, assigns it the job store\(aqs config, and returns it. .sp May declare the job to have failed (see \fI\%toil.job.JobDescription.setupJobAfterFailure()\fP) if there is evidence of a failed update attempt. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP \-\- the ID of the job to load .TP .B Raises \fI\%NoSuchJobException\fP \-\- if there is no job with the given ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_job(job) Persists changes to the state of the given JobDescription in this store atomically. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- the job to write to this job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_job(job_id) Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. .sp This operation is idempotent, i.e. deleting a job twice or deleting a non\-existent job will succeed silently. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to delete from this job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B jobs() Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object .INDENT 7.0 .TP .B Returns Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs .TP .B Return type Iterator[toil.job.jobDescription] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B optional_hard_copy(hardlink) .UNINDENT .INDENT 7.0 .TP .B write_file(local_path, job_id=None, cleanup=False) Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns an ID referencing the newly created file and can be used to read the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future. .TP .B Return type Iterator[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_empty_file_store_id(jobStoreID=None, cleanup=False, basename=None) Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .UNINDENT .TP .B Returns a jobStoreFileID that references the newly created file and can be used to reference the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_file(file_id, local_path) Replaces the existing version of a file in the job store. .sp Throws an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBlocal_path\fP \-\- the local path to a file that will overwrite the current version in the job store .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file(file_id, local_path, symlink=False) Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation\-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed. .sp The file at the given local path may not be modified after this method returns! .sp Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to be copied .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path indicating where to place the contents of the given file in the job store .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_file(file_id) Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non\-existent file will succeed silently. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to delete .UNINDENT .UNINDENT .INDENT 7.0 .TP .B file_exists(file_id) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP \-\- an ID referencing the file to be checked .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_file_size(file_id) Get the size of the given file in bytes, or 0 if it does not exist when queried. .sp Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_file_stream(file_id, encoding=None, errors=None) Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file_stream(file_id: \fI\%str\fP | \fI\%toil.fileStores.FileID\fP, encoding: Literal[None] = None, errors: \fI\%str\fP | \fI\%None\fP = None) -> Iterator[IO[\fI\%bytes\fP]] .TP .B read_file_stream(file_id: \fI\%str\fP | \fI\%toil.fileStores.FileID\fP, encoding: \fI\%str\fP, errors: \fI\%str\fP | \fI\%None\fP = None) -> Iterator[IO[\fI\%str\fP]] .TP .B read_file_stream(file_id: \fI\%str\fP | \fI\%toil.fileStores.FileID\fP, encoding: \fI\%str\fP | \fI\%None\fP = None, errors: \fI\%str\fP | \fI\%None\fP = None) -> Iterator[IO[\fI\%bytes\fP]] | Iterator[IO[\fI\%str\fP]] Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to get a readable file handle for .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a file handle which can be read from .TP .B Return type Iterator[Union[IO[\fI\%bytes\fP], IO[\fI\%str\fP]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None) Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencrypted\fP (\fI\%bool\fP) \-\- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns a context manager yielding a writable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_shared_file_stream(shared_file_name, encoding=None, errors=None) Returns a context manager yielding a readable file handle to the global file referenced by the given name. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a readable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B list_all_file_names(for_job=None) Get all the file names (not file IDs) of files stored in the job store. .sp Used for debugging. .INDENT 7.0 .TP .B Parameters \fBfor_job\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If set, restrict the list to files for a particular job. .TP .B Return type Iterable[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_logs(msg) Stores a message as a log in the jobstore. .INDENT 7.0 .TP .B Parameters \fBmsg\fP (\fI\%str\fP) \-\- the string to be written .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_logs(callback, read_all=False) Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages). .sp Only unread logs will be read unless the read_all parameter is set. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcallback\fP (\fICallable\fP) \-\- a function to be applied to each of the stats file handles found .IP \(bu 2 \fBread_all\fP (\fI\%bool\fP) \-\- a boolean indicating whether to read the already processed stats files in addition to the unread stats files .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns the number of stats files processed .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.jobStores.googleJobStore\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%GoogleJobStore\fP T} T{ Represents the physical storage for the jobs and files in a Toil workflow. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%google_retry_predicate\fP(e) T} T{ necessary because under heavy load google may throw T} _ T{ \fI\%google_retry\fP(f) T} T{ This decorator retries the wrapped function if google throws any angry service T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ T{ \fI\%GOOGLE_STORAGE\fP T} T{ T} _ T{ \fI\%MAX_BATCH_SIZE\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.jobStores.googleJobStore.log .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.googleJobStore.GOOGLE_STORAGE = \(aqgs\(aq .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.googleJobStore.MAX_BATCH_SIZE = 1000 .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.googleJobStore.google_retry_predicate(e) .INDENT 7.0 .TP .B necessary because under heavy load google may throw TooManyRequests: 429 The project exceeded the rate limit for creating and deleting buckets. .UNINDENT .sp or numerous other server errors which need to be retried. .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.googleJobStore.google_retry(f) This decorator retries the wrapped function if google throws any angry service errors. .sp It should wrap any function that makes use of the Google Client API .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.googleJobStore.GoogleJobStore(locator) Bases: \fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP .sp Represents the physical storage for the jobs and files in a Toil workflow. .sp JobStores are responsible for storing \fI\%toil.job.JobDescription\fP (which relate jobs to each other) and files. .sp Actual \fI\%toil.job.Job\fP objects are stored in files, referenced by JobDescriptions. All the non\-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs. .sp To actually get ahold of a \fI\%toil.job.Job\fP, use \fI\%toil.job.Job.loadJob()\fP with a JobStore and the relevant JobDescription. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B nodeServiceAccountJson = \(aq/root/service_account.json\(aq .UNINDENT .INDENT 7.0 .TP .B classmethod create_client() Produce a client for Google Sotrage with the highest level of access we can get. .sp Fall back to anonymous access if no project is available, unlike the Google Storage module\(aqs behavior. .sp Warn if GOOGLE_APPLICATION_CREDENTIALS is set but not actually present. .INDENT 7.0 .TP .B Return type google.cloud.storage.Client .UNINDENT .UNINDENT .INDENT 7.0 .TP .B initialize(config=None) Initialize this job store. .sp Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store. .INDENT 7.0 .TP .B Parameters \fBconfig\fP \-\- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID. .TP .B Raises \fI\%JobStoreExistsException\fP \-\- if the physical storage for this job store already exists .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resume() Connect this instance to the physical storage it represents and load the Toil configuration into the \fBAbstractJobStore.config\fP attribute. .INDENT 7.0 .TP .B Raises \fI\%NoSuchJobStoreException\fP \-\- if the physical storage for this job store doesn\(aqt exist .UNINDENT .UNINDENT .INDENT 7.0 .TP .B destroy() The inverse of \fI\%initialize()\fP, this method deletes the physical storage represented by this instance. While not being atomic, this method \fIis\fP at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow. .UNINDENT .INDENT 7.0 .TP .B assign_job_id(job_description) Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription. .sp Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated. .INDENT 7.0 .TP .B Parameters \fBjob_description\fP (\fI\%toil.job.JobDescription\fP) \-\- The JobDescription to give an ID to .UNINDENT .UNINDENT .INDENT 7.0 .TP .B batch() If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released. .UNINDENT .INDENT 7.0 .TP .B create_job(job_description) Writes the given JobDescription to the job store. The job must have an ID assigned already. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Returns The JobDescription passed. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B job_exists(job_id) Indicates whether a description of the job with the specified jobStoreID exists in the job store .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_public_url(fileName) Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBfile_name\fP (\fI\%str\fP) \-\- the jobStoreFileID of the file to generate a URL for .TP .B Raises \fI\%NoSuchFileException\fP \-\- if the specified file does not exist in this job store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_shared_public_url(sharedFileName) Differs from \fBgetPublicUrl()\fP in that this method is for generating URLs for shared files written by \fBwriteSharedFileStream()\fP\&. .sp Returns a publicly accessible URL to the given file in the job store. The returned URL starts with \(aqhttp:\(aq, \(aqhttps:\(aq or \(aqfile:\(aq. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBshared_file_name\fP (\fI\%str\fP) \-\- The name of the shared file to generate a publically accessible url for. .TP .B Raises \fI\%NoSuchFileException\fP \-\- raised if the specified file does not exist in the store .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_job(job_id) Loads the description of the job referenced by the given ID, assigns it the job store\(aqs config, and returns it. .sp May declare the job to have failed (see \fI\%toil.job.JobDescription.setupJobAfterFailure()\fP) if there is evidence of a failed update attempt. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP \-\- the ID of the job to load .TP .B Raises \fI\%NoSuchJobException\fP \-\- if there is no job with the given ID .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_job(job) Persists changes to the state of the given JobDescription in this store atomically. .sp Must call jobDescription.pre_update_hook() .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- the job to write to this job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_job(job_id) Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it. .sp This operation is idempotent, i.e. deleting a job twice or deleting a non\-existent job will succeed silently. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- the ID of the job to delete from this job store .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_env() Return a dict of environment variables to send out to the workers so they can load the job store. .UNINDENT .INDENT 7.0 .TP .B jobs() Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object .INDENT 7.0 .TP .B Returns Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs .TP .B Return type Iterator[toil.job.jobDescription] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_file(local_path, job_id=None, cleanup=False) Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob. .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns an ID referencing the newly created file and can be used to read the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None) Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchJobException\fP \-\- if the job specified via jobStoreID does not exist .UNINDENT .UNINDENT .sp FIXME: some implementations may not raise this .INDENT 7.0 .TP .B Returns a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future. .TP .B Return type Iterator[Tuple[IO[\fI\%bytes\fP], \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_empty_file_store_id(jobStoreID=None, cleanup=False, basename=None) Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- the id of a job, or None. If specified, the may be associated with that job in a job\-store\-specific way. This may influence the returned ID. .IP \(bu 2 \fBcleanup\fP (\fI\%bool\fP) \-\- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing. .IP \(bu 2 \fBbasename\fP (\fI\%str\fP) \-\- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected. .UNINDENT .TP .B Returns a jobStoreFileID that references the newly created file and can be used to reference the file in the future. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file(file_id, local_path, symlink=False) Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation\-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed. .sp The file at the given local path may not be modified after this method returns! .sp Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to be copied .IP \(bu 2 \fBlocal_path\fP (\fI\%str\fP) \-\- the local path indicating where to place the contents of the given file in the job store .IP \(bu 2 \fBsymlink\fP (\fI\%bool\fP) \-\- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_file_stream(file_id, encoding=None, errors=None) Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to get a readable file handle for .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a file handle which can be read from .TP .B Return type Iterator[Union[IO[\fI\%bytes\fP], IO[\fI\%str\fP]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_file(file_id) Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non\-existent file will succeed silently. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- ID of the file to delete .UNINDENT .UNINDENT .INDENT 7.0 .TP .B file_exists(file_id) Determine whether a file exists in this job store. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP \-\- an ID referencing the file to be checked .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_file_size(file_id) Get the size of the given file in bytes, or 0 if it does not exist when queried. .sp Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc. .INDENT 7.0 .TP .B Parameters \fBfile_id\fP (\fI\%str\fP) \-\- an ID referencing the file to be checked .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_file(file_id, local_path) Replaces the existing version of a file in the job store. .sp Throws an exception if the file does not exist. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBlocal_path\fP \-\- the local path to a file that will overwrite the current version in the job store .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B update_file_stream(file_id, encoding=None, errors=None) Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- the ID of the file in the job store to be updated .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .IP \(bu 2 \fI\%NoSuchFileException\fP \-\- if the specified file does not exist .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_shared_file_stream(shared_file_name, encrypted=True, encoding=None, errors=None) Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencrypted\fP (\fI\%bool\fP) \-\- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear. .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns a context manager yielding a writable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_shared_file_stream(shared_file_name, isProtected=True, encoding=None, errors=None) Returns a context manager yielding a readable file handle to the global file referenced by the given name. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBshared_file_name\fP (\fI\%str\fP) \-\- A file name matching AbstractJobStore.fileNameRegex, unique within this job store .IP \(bu 2 \fBencoding\fP (\fI\%str\fP) \-\- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode. .IP \(bu 2 \fBerrors\fP (\fI\%str\fP) \-\- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to \(aqstrict\(aq when an encoding is specified. .UNINDENT .TP .B Returns a context manager yielding a readable file handle .TP .B Return type Iterator[IO[\fI\%bytes\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_logs(msg) Stores a message as a log in the jobstore. .INDENT 7.0 .TP .B Parameters \fBmsg\fP (\fI\%str\fP) \-\- the string to be written .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_logs(callback, read_all=False) Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages). .sp Only unread logs will be read unless the read_all parameter is set. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcallback\fP (\fICallable\fP) \-\- a function to be applied to each of the stats file handles found .IP \(bu 2 \fBread_all\fP (\fI\%bool\fP) \-\- a boolean indicating whether to read the already processed stats files in addition to the unread stats files .UNINDENT .TP .B Raises \fI\%ConcurrentFileModificationException\fP \-\- if the file was modified concurrently during an invocation of this method .TP .B Returns the number of stats files processed .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.jobStores.utils\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%WritablePipe\fP T} T{ An object\-oriented wrapper for os.pipe. Clients should subclass it, implement T} _ T{ \fI\%ReadablePipe\fP T} T{ An object\-oriented wrapper for os.pipe. Clients should subclass it, implement T} _ T{ \fI\%ReadableTransformingPipe\fP T} T{ A pipe which is constructed around a readable stream, and which provides a T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%generate_locator\fP(job_store_type[, local_suggestion, ...]) T} T{ Generate a random locator for a job store of the given type. Raises an T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.jobStores.utils.log .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.utils.WritablePipe(encoding=None, errors=None) Bases: \fI\%abc.ABC\fP .sp An object\-oriented wrapper for os.pipe. Clients should subclass it, implement \fI\%readFrom()\fP to consume the readable end of the pipe, then instantiate the class as a context manager to get the writable end. See the example below. .sp .EX >>> import sys, shutil >>> class MyPipe(WritablePipe): \&... def readFrom(self, readable): \&... shutil.copyfileobj(codecs.getreader(\(aqutf\-8\(aq)(readable), sys.stdout) >>> with MyPipe() as writable: \&... _ = writable.write(\(aqHello, world!\en\(aq.encode(\(aqutf\-8\(aq)) Hello, world! .EE .sp Each instance of this class creates a thread and invokes the readFrom method in that thread. The thread will be join()ed upon normal exit from the context manager, i.e. the body of the \fIwith\fP statement. If an exception occurs, the thread will not be joined but a well\-behaved \fI\%readFrom()\fP implementation will terminate shortly thereafter due to the pipe having been closed. .sp Now, exceptions in the reader thread will be reraised in the main thread: .sp .EX >>> class MyPipe(WritablePipe): \&... def readFrom(self, readable): \&... raise RuntimeError(\(aqHello, world!\(aq) >>> with MyPipe() as writable: \&... pass Traceback (most recent call last): \&... RuntimeError: Hello, world! .EE .sp More complicated, less illustrative tests: .sp Same as above, but proving that handles are closed: .sp .EX >>> x = os.dup(0); os.close(x) >>> class MyPipe(WritablePipe): \&... def readFrom(self, readable): \&... raise RuntimeError(\(aqHello, world!\(aq) >>> with MyPipe() as writable: \&... pass Traceback (most recent call last): \&... RuntimeError: Hello, world! >>> y = os.dup(0); os.close(y); x == y True .EE .sp Exceptions in the body of the with statement aren\(aqt masked, and handles are closed: .sp .EX >>> x = os.dup(0); os.close(x) >>> class MyPipe(WritablePipe): \&... def readFrom(self, readable): \&... pass >>> with MyPipe() as writable: \&... raise RuntimeError(\(aqHello, world!\(aq) Traceback (most recent call last): \&... RuntimeError: Hello, world! >>> y = os.dup(0); os.close(y); x == y True .EE .INDENT 7.0 .TP .B abstract readFrom(readable) Implement this method to read data from the pipe. This method should support both binary and text mode output. .INDENT 7.0 .TP .B Parameters \fBreadable\fP (\fIfile\fP) \-\- the file object representing the readable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __enter__() .UNINDENT .INDENT 7.0 .TP .B __exit__(exc_type, exc_val, exc_tb) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.utils.ReadablePipe(encoding=None, errors=None) Bases: \fI\%abc.ABC\fP .sp An object\-oriented wrapper for os.pipe. Clients should subclass it, implement \fI\%writeTo()\fP to place data into the writable end of the pipe, then instantiate the class as a context manager to get the writable end. See the example below. .sp .EX >>> import sys, shutil >>> class MyPipe(ReadablePipe): \&... def writeTo(self, writable): \&... writable.write(\(aqHello, world!\en\(aq.encode(\(aqutf\-8\(aq)) >>> with MyPipe() as readable: \&... shutil.copyfileobj(codecs.getreader(\(aqutf\-8\(aq)(readable), sys.stdout) Hello, world! .EE .sp Each instance of this class creates a thread and invokes the \fI\%writeTo()\fP method in that thread. The thread will be join()ed upon normal exit from the context manager, i.e. the body of the \fIwith\fP statement. If an exception occurs, the thread will not be joined but a well\-behaved \fI\%writeTo()\fP implementation will terminate shortly thereafter due to the pipe having been closed. .sp Now, exceptions in the reader thread will be reraised in the main thread: .sp .EX >>> class MyPipe(ReadablePipe): \&... def writeTo(self, writable): \&... raise RuntimeError(\(aqHello, world!\(aq) >>> with MyPipe() as readable: \&... pass Traceback (most recent call last): \&... RuntimeError: Hello, world! .EE .sp More complicated, less illustrative tests: .sp Same as above, but proving that handles are closed: .sp .EX >>> x = os.dup(0); os.close(x) >>> class MyPipe(ReadablePipe): \&... def writeTo(self, writable): \&... raise RuntimeError(\(aqHello, world!\(aq) >>> with MyPipe() as readable: \&... pass Traceback (most recent call last): \&... RuntimeError: Hello, world! >>> y = os.dup(0); os.close(y); x == y True .EE .sp Exceptions in the body of the with statement aren\(aqt masked, and handles are closed: .sp .EX >>> x = os.dup(0); os.close(x) >>> class MyPipe(ReadablePipe): \&... def writeTo(self, writable): \&... pass >>> with MyPipe() as readable: \&... raise RuntimeError(\(aqHello, world!\(aq) Traceback (most recent call last): \&... RuntimeError: Hello, world! >>> y = os.dup(0); os.close(y); x == y True .EE .INDENT 7.0 .TP .B abstract writeTo(writable) Implement this method to write data from the pipe. This method should support both binary and text mode input. .INDENT 7.0 .TP .B Parameters \fBwritable\fP (\fIfile\fP) \-\- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __enter__() .UNINDENT .INDENT 7.0 .TP .B __exit__(exc_type, exc_val, exc_tb) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.jobStores.utils.ReadableTransformingPipe(source, encoding=None, errors=None) Bases: \fI\%ReadablePipe\fP .sp A pipe which is constructed around a readable stream, and which provides a context manager that gives a readable stream. .sp Useful as a base class for pipes which have to transform or otherwise visit bytes that flow through them, instead of just consuming or producing data. .sp Clients should subclass it and implement \fI\%transform()\fP, like so: .sp .EX >>> import sys, shutil >>> class MyPipe(ReadableTransformingPipe): \&... def transform(self, readable, writable): \&... writable.write(readable.read().decode(\(aqutf\-8\(aq).upper().encode(\(aqutf\-8\(aq)) >>> class SourcePipe(ReadablePipe): \&... def writeTo(self, writable): \&... writable.write(\(aqHello, world!\en\(aq.encode(\(aqutf\-8\(aq)) >>> with SourcePipe() as source: \&... with MyPipe(source) as transformed: \&... shutil.copyfileobj(codecs.getreader(\(aqutf\-8\(aq)(transformed), sys.stdout) HELLO, WORLD! .EE .sp The \fI\%transform()\fP method runs in its own thread, and should move data chunk by chunk instead of all at once. It should finish normally if it encounters either an EOF on the readable, or a \fI\%BrokenPipeError\fP on the writable. This means that it should make sure to actually catch a \fI\%BrokenPipeError\fP when writing. .sp See also: \fBtoil.lib.misc.WriteWatchingStream\fP\&. .INDENT 7.0 .TP .B abstract transform(readable, writable) Implement this method to ship data through the pipe. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBreadable\fP (\fIfile\fP) \-\- the input stream file object to transform. .IP \(bu 2 \fBwritable\fP (\fIfile\fP) \-\- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeTo(writable) Implement this method to write data from the pipe. This method should support both binary and text mode input. .INDENT 7.0 .TP .B Parameters \fBwritable\fP (\fIfile\fP) \-\- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.jobStores.utils.JobStoreUnavailableException Bases: \fI\%RuntimeError\fP .sp Raised when a particular type of job store is requested but can\(aqt be used. .UNINDENT .INDENT 0.0 .TP .B toil.jobStores.utils.generate_locator(job_store_type, local_suggestion=None, decoration=None) Generate a random locator for a job store of the given type. Raises an JobStoreUnavailableException if that job store cannot be used. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_store_type\fP (\fI\%str\fP) \-\- Registry name of the job store to use. .IP \(bu 2 \fBlocal_suggestion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Path to a nonexistent local directory suitable for use as a file job store. .IP \(bu 2 \fBdecoration\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Extra string to add to the job store locator, if convenient. .UNINDENT .TP .B Return str Job store locator for a usable job store. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .SS \fI\%toil.lib\fP .SS Subpackages .SS \fI\%toil.lib.aws\fP .SS Submodules .SS \fI\%toil.lib.aws.ami\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%get_flatcar_ami\fP(ec2_client[, architecture]) T} T{ Retrieve the flatcar AMI image to use as the base for all Toil autoscaling instances. T} _ T{ \fI\%flatcar_release_feed_amis\fP(region[, architecture, source]) T} T{ Yield AMI IDs for the given architecture from the Flatcar release feed. T} _ T{ \fI\%feed_flatcar_ami_release\fP(ec2_client[, architecture, ...]) T} T{ Check a Flatcar release feed for the latest flatcar AMI. T} _ T{ \fI\%aws_marketplace_flatcar_ami_search\fP(ec2_client[, ...]) T} T{ Query AWS for all AMI names matching \fBFlatcar\-stable\-*\fP and return the most recent one. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.aws.ami.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.ami.get_flatcar_ami(ec2_client, architecture=\(aqamd64\(aq) Retrieve the flatcar AMI image to use as the base for all Toil autoscaling instances. .sp AMI must be available to the user on AWS (attempting to launch will return a 403 otherwise). .INDENT 7.0 .TP .B Priority is: .INDENT 7.0 .IP 1. 3 User specified AMI via TOIL_AWS_AMI .IP 2. 3 Official AMI from stable.release.flatcar\-linux.net .IP 3. 3 Search the AWS Marketplace .UNINDENT .UNINDENT .sp If all of these sources fail, we raise an error to complain. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBec2_client\fP (\fIbotocore.client.BaseClient\fP) \-\- Boto3 EC2 Client .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- The architecture type for the new AWS machine. Can be either amd64 or arm64 .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.ami.flatcar_release_feed_amis(region, architecture=\(aqamd64\(aq, source=\(aqstable\(aq) Yield AMI IDs for the given architecture from the Flatcar release feed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsource\fP (\fI\%str\fP) \-\- can be set to a Flatcar release channel (\(aqstable\(aq, \(aqbeta\(aq, or \(aqalpha\(aq), \(aqarchive\(aq to check the Internet Archive for a feed, and \(aqtoil\(aq to check if the Toil project has put up a feed. .IP \(bu 2 \fBregion\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Iterator[\fI\%str\fP] .UNINDENT .sp Retries if the release feed cannot be fetched. If the release feed has a permanent error, yields nothing. If some entries in the release feed are unparseable, yields the others. .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.ami.feed_flatcar_ami_release(ec2_client, architecture=\(aqamd64\(aq, source=\(aqstable\(aq) Check a Flatcar release feed for the latest flatcar AMI. .sp Verify it\(aqs on AWS. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBec2_client\fP (\fIbotocore.client.BaseClient\fP) \-\- Boto3 EC2 Client .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- The architecture type for the new AWS machine. Can be either amd64 or arm64 .IP \(bu 2 \fBsource\fP (\fI\%str\fP) \-\- can be set to a Flatcar release channel (\(aqstable\(aq, \(aqbeta\(aq, or \(aqalpha\(aq), \(aqarchive\(aq to check the Internet Archive for a feed, and \(aqtoil\(aq to check if the Toil project has put up a feed. .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.ami.aws_marketplace_flatcar_ami_search(ec2_client, architecture=\(aqamd64\(aq) Query AWS for all AMI names matching \fBFlatcar\-stable\-*\fP and return the most recent one. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBec2_client\fP (\fIbotocore.client.BaseClient\fP) \-\- .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .SS \fI\%toil.lib.aws.iam\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%init_action_collection\fP() T} T{ Initialization of an action collection, an action collection contains allowed Actions and NotActions T} _ T{ \fI\%add_to_action_collection\fP(a, b) T} T{ Combines two action collections T} _ T{ \fI\%policy_permissions_allow\fP(given_permissions[, ...]) T} T{ Check whether given set of actions are a subset of another given set of actions, returns true if they are T} _ T{ \fI\%permission_matches_any\fP(perm, list_perms) T} T{ Takes a permission and checks whether it\(aqs contained within a list of given permissions T} _ T{ \fI\%get_actions_from_policy_document\fP(policy_doc) T} T{ Given a policy document, go through each statement and create an AllowedActionCollection representing the T} _ T{ \fI\%allowed_actions_attached\fP(iam, attached_policies) T} T{ Go through all attached policy documents and create an AllowedActionCollection representing granted permissions. T} _ T{ \fI\%allowed_actions_roles\fP(iam, policy_names, role_name) T} T{ Returns a dictionary containing a list of all aws actions allowed for a given role. T} _ T{ \fI\%collect_policy_actions\fP(policy_documents) T} T{ Collect all of the actions allowed by the given policy documents into one AllowedActionCollection. T} _ T{ \fI\%allowed_actions_user\fP(iam, policy_names, user_name) T} T{ Gets all allowed actions for a user given by user_name, returns a dictionary, keyed by resource, T} _ T{ \fI\%allowed_actions_group\fP(iam, policy_names, group_name) T} T{ Gets all allowed actions for a group given by group_name, returns a dictionary, keyed by resource, T} _ T{ \fI\%get_policy_permissions\fP(region) T} T{ Returns an action collection containing lists of all permission grant patterns keyed by resource T} _ T{ \fI\%get_aws_account_num\fP() T} T{ Returns AWS account num T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%CLUSTER_LAUNCHING_PERMISSIONS\fP T} T{ T} _ T{ \fI\%AllowedActionCollection\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.aws.iam.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.CLUSTER_LAUNCHING_PERMISSIONS = [\(aqiam:CreateRole\(aq, \(aqiam:CreateInstanceProfile\(aq, \(aqiam:TagInstanceProfile\(aq, \(aqiam:DeleteRole\(aq,... .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.AllowedActionCollection .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.init_action_collection() Initialization of an action collection, an action collection contains allowed Actions and NotActions by resource, these are patterns containing wildcards, an Action explicitly allows a matched pattern, eg ec2:* will explicitly allow all ec2 permissions .sp A NotAction will explicitly allow all actions that don\(aqt match a specific pattern eg iam:* allows all non iam actions .INDENT 7.0 .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.add_to_action_collection(a, b) Combines two action collections .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBa\fP (\fIAllowedActionCollection\fP) \-\- .IP \(bu 2 \fBb\fP (\fIAllowedActionCollection\fP) \-\- .UNINDENT .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.policy_permissions_allow(given_permissions, required_permissions=[]) Check whether given set of actions are a subset of another given set of actions, returns true if they are otherwise false and prints a warning. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequired_permissions\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Dictionary containing actions required, keyed by resource .IP \(bu 2 \fBgiven_permissions\fP (\fIAllowedActionCollection\fP) \-\- Set of actions that are granted to a user or role .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.permission_matches_any(perm, list_perms) Takes a permission and checks whether it\(aqs contained within a list of given permissions Returns True if it is otherwise False .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBperm\fP (\fI\%str\fP) \-\- Permission to check in string form .IP \(bu 2 \fBlist_perms\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Permission list to check against .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.get_actions_from_policy_document(policy_doc) Given a policy document, go through each statement and create an AllowedActionCollection representing the permissions granted in the policy document. .INDENT 7.0 .TP .B Parameters \fBpolicy_doc\fP (\fImypy_boto3_iam.type_defs.PolicyDocumentDictTypeDef\fP) \-\- A policy document to examine .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.allowed_actions_attached(iam, attached_policies) Go through all attached policy documents and create an AllowedActionCollection representing granted permissions. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBiam\fP (\fImypy_boto3_iam.IAMClient\fP) \-\- IAM client to use .IP \(bu 2 \fBattached_policies\fP (\fIList\fP\fI[\fP\fImypy_boto3_iam.type_defs.AttachedPolicyTypeDef\fP\fI]\fP) \-\- Attached policies .UNINDENT .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.allowed_actions_roles(iam, policy_names, role_name) Returns a dictionary containing a list of all aws actions allowed for a given role. This dictionary is keyed by resource and gives a list of policies allowed on that resource. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBiam\fP (\fImypy_boto3_iam.IAMClient\fP) \-\- IAM client to use .IP \(bu 2 \fBpolicy_names\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Name of policy document associated with a role .IP \(bu 2 \fBrole_name\fP (\fI\%str\fP) \-\- Name of role to get associated policies .UNINDENT .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.collect_policy_actions(policy_documents) Collect all of the actions allowed by the given policy documents into one AllowedActionCollection. .INDENT 7.0 .TP .B Parameters \fBpolicy_documents\fP (\fIList\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fImypy_boto3_iam.type_defs.PolicyDocumentDictTypeDef\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.allowed_actions_user(iam, policy_names, user_name) Gets all allowed actions for a user given by user_name, returns a dictionary, keyed by resource, with a list of permissions allowed for each given resource. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBiam\fP (\fImypy_boto3_iam.IAMClient\fP) \-\- IAM client to use .IP \(bu 2 \fBpolicy_names\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Name of policy document associated with a user .IP \(bu 2 \fBuser_name\fP (\fI\%str\fP) \-\- Name of user to get associated policies .UNINDENT .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.allowed_actions_group(iam, policy_names, group_name) Gets all allowed actions for a group given by group_name, returns a dictionary, keyed by resource, with a list of permissions allowed for each given resource. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBiam\fP (\fImypy_boto3_iam.IAMClient\fP) \-\- IAM client to use .IP \(bu 2 \fBpolicy_names\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Name of policy document associated with a user .IP \(bu 2 \fBgroup_name\fP (\fI\%str\fP) \-\- Name of group to get associated policies .UNINDENT .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.get_policy_permissions(region) Returns an action collection containing lists of all permission grant patterns keyed by resource that they are allowed upon. Requires AWS credentials to be associated with a user or assumed role. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBzone\fP \-\- AWS zone to connect to .IP \(bu 2 \fBregion\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type AllowedActionCollection .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.iam.get_aws_account_num() Returns AWS account num .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .SS \fI\%toil.lib.aws.session\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AWSConnectionManager\fP T} T{ Class that represents a connection to AWS. Caches Boto 3 and Boto 2 objects T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%establish_boto3_session\fP([region_name]) T} T{ Get a Boto 3 session usable by the current thread. T} _ T{ \fI\%client\fP(service_name[, region_name, endpoint_url, config]) T} T{ Get a Boto 3 client for a particular AWS service, usable by the current thread. T} _ T{ \fI\%resource\fP(service_name[, region_name, endpoint_url]) T} T{ Get a Boto 3 resource for a particular AWS service, usable by the current thread. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.aws.session.logger .UNINDENT .INDENT 0.0 .TP .B class toil.lib.aws.session.AWSConnectionManager Class that represents a connection to AWS. Caches Boto 3 and Boto 2 objects by region. .sp Access to any kind of item goes through the particular method for the thing you want (session, resource, service, Boto2 Context), and then you pass the region you want to work in, and possibly the type of thing you want, as arguments. .sp This class is intended to eventually enable multi\-region clusters, where connections to multiple regions may need to be managed in the same provisioner. .sp We also support None for a region, in which case no region will be passed to Boto/Boto3. The caller is responsible for implementing e.g. TOIL_AWS_REGION support. .sp Since connection objects may not be thread safe (see <\fI\%https://boto3.amazonaws.com/v1/documentation/api/1.14.31/guide/session.html#multithreading\-or\-multiprocessing\-with\-sessions\fP>), one is created for each thread that calls the relevant lookup method. .INDENT 7.0 .TP .B session(region) Get the Boto3 Session to use for the given region. .INDENT 7.0 .TP .B Parameters \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type boto3.session.Session .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resource(region, service_name, endpoint_url=None) Get the Boto3 Resource to use with the given service (like \(aqec2\(aq) in the given region. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBendpoint_url\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- AWS endpoint URL to use for the client. If not specified, a default is used. .IP \(bu 2 \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBservice_name\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type boto3.resources.base.ServiceResource .UNINDENT .UNINDENT .INDENT 7.0 .TP .B client(region, service_name, endpoint_url=None, config=None) Get the Boto3 Client to use with the given service (like \(aqec2\(aq) in the given region. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBendpoint_url\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- AWS endpoint URL to use for the client. If not specified, a default is used. .IP \(bu 2 \fBconfig\fP (\fIOptional\fP\fI[\fP\fIbotocore.client.Config\fP\fI]\fP) \-\- Custom configuration to use for the client. .IP \(bu 2 \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBservice_name\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type botocore.client.BaseClient .UNINDENT .UNINDENT .INDENT 7.0 .TP .B boto2(region, service_name) Get the connected boto2 connection for the given region and service. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBservice_name\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type boto.connection.AWSAuthConnection .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.session.establish_boto3_session(region_name=None) Get a Boto 3 session usable by the current thread. .sp This function may not always establish a \fInew\fP session; it can be memoized. .INDENT 7.0 .TP .B Parameters \fBregion_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type boto3.Session .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.session.client(service_name, region_name=None, endpoint_url=None, config=None) Get a Boto 3 client for a particular AWS service, usable by the current thread. .sp Global alternative to AWSConnectionManager. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBendpoint_url\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconfig\fP (\fIOptional\fP\fI[\fP\fIbotocore.client.Config\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type botocore.client.BaseClient .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.session.resource(service_name, region_name=None, endpoint_url=None) Get a Boto 3 resource for a particular AWS service, usable by the current thread. .sp Global alternative to AWSConnectionManager. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion_name\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBendpoint_url\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type boto3.resources.base.ServiceResource .UNINDENT .UNINDENT .SS \fI\%toil.lib.aws.utils\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%delete_iam_role\fP(role_name[, region, quiet]) T} T{ T} _ T{ \fI\%delete_iam_instance_profile\fP(instance_profile_name[, ...]) T} T{ T} _ T{ \fI\%delete_sdb_domain\fP(sdb_domain_name[, region, quiet]) T} T{ T} _ T{ \fI\%connection_reset\fP(e) T} T{ Return true if an error is a connection reset error. T} _ T{ \fI\%retryable_s3_errors\fP(e) T} T{ Return true if this is an error from S3 that looks like we ought to retry our request. T} _ T{ \fI\%retry_s3\fP([delays, timeout, predicate]) T} T{ Retry iterator of context managers specifically for S3 operations. T} _ T{ \fI\%delete_s3_bucket\fP(s3_resource, bucket[, quiet]) T} T{ Delete the given S3 bucket. T} _ T{ \fI\%create_s3_bucket\fP(s3_resource, bucket_name, region) T} T{ Create an AWS S3 bucket, using the given Boto3 S3 session, with the T} _ T{ \fI\%enable_public_objects\fP(bucket_name) T} T{ Enable a bucket to contain objects which are public. T} _ T{ \fI\%get_bucket_region\fP(bucket_name[, endpoint_url, ...]) T} T{ Get the AWS region name associated with the given S3 bucket. T} _ T{ \fI\%region_to_bucket_location\fP(region) T} T{ T} _ T{ \fI\%bucket_location_to_region\fP(location) T} T{ T} _ T{ \fI\%get_object_for_url\fP(url[, existing]) T} T{ Extracts a key (object) from a given parsed s3:// URL. T} _ T{ \fI\%list_objects_for_url\fP(url) T} T{ Extracts a key (object) from a given parsed s3:// URL. The URL will be T} _ T{ \fI\%flatten_tags\fP(tags) T} T{ Convert tags from a key to value dict into a list of \(aqKey\(aq: xxx, \(aqValue\(aq: xxx dicts. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%BotoServerError\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%THROTTLED_ERROR_CODES\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.aws.utils.BotoServerError .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.THROTTLED_ERROR_CODES = [\(aqThrottling\(aq, \(aqThrottlingException\(aq, \(aqThrottledException\(aq, \(aqRequestThrottledException\(aq,... .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.delete_iam_role(role_name, region=None, quiet=True) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrole_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBquiet\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinstance_profile_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBquiet\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsdb_domain_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBquiet\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.connection_reset(e) Return true if an error is a connection reset error. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.retryable_s3_errors(e) Return true if this is an error from S3 that looks like we ought to retry our request. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors) Retry iterator of context managers specifically for S3 operations. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdelays\fP (\fIIterable\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBtimeout\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBpredicate\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%Exception\fP\fI]\fP\fI, \fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Iterator[ContextManager[None]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True) Delete the given S3 bucket. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBs3_resource\fP (\fImypy_boto3_s3.S3ServiceResource\fP) \-\- .IP \(bu 2 \fBbucket\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBquiet\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region) Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region. .sp Supports the us\-east\-1 region, where bucket creation is special. .sp \fIALL\fP S3 bucket creation should use this function. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBs3_resource\fP (\fImypy_boto3_s3.S3ServiceResource\fP) \-\- .IP \(bu 2 \fBbucket_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion\fP (\fIUnion\fP\fI[\fP\fImypy_boto3_s3.literals.BucketLocationConstraintType\fP\fI, \fP\fILiteral\fP\fI[\fP\fIus\-east\-1\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type mypy_boto3_s3.service_resource.Bucket .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.enable_public_objects(bucket_name) Enable a bucket to contain objects which are public. .sp This adjusts the bucket\(aqs Public Access Block setting to not block all public access, and also adjusts the bucket\(aqs Object Ownership setting to a setting which enables object ACLs. .sp Does \fInot\fP touch the \fIaccount\fP\(aqs Public Access Block setting, which can also interfere here. That is probably best left to the account administrator. .sp This configuration used to be the default, and is what most of Toil\(aqs code is written to expect, but it was changed so that new buckets default to the more restrictive setting <\fI\%https://aws.amazon.com/about\-aws/whats\-new/2022/12/amazon\-s3\-automatically\-enable\-block\-public\-access\-disable\-access\-control\-lists\-buckets\-april\-2023/\fP>, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior. .INDENT 7.0 .TP .B Parameters \fBbucket_name\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None, only_strategies=None) Get the AWS region name associated with the given S3 bucket. .sp Takes an optional S3 API URL override. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBonly_strategies\fP (\fIOptional\fP\fI[\fP\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP\fI]\fP) \-\- For testing, use only strategies with 1\-based numbers in this set. .IP \(bu 2 \fBbucket_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBendpoint_url\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.region_to_bucket_location(region) .INDENT 7.0 .TP .B Parameters \fBregion\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.bucket_location_to_region(location) .INDENT 7.0 .TP .B Parameters \fBlocation\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.get_object_for_url(url, existing=None) Extracts a key (object) from a given parsed s3:// URL. .sp If existing is true and the object does not exist, raises FileNotFoundError. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexisting\fP (\fI\%bool\fP) \-\- If True, key is expected to exist. If False, key is expected not to exists and it will be created. If None, the key will be created if it doesn\(aqt exist. .IP \(bu 2 \fBurl\fP (\fI\%urllib.parse.ParseResult\fP) \-\- .UNINDENT .TP .B Return type mypy_boto3_s3.service_resource.Object .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.list_objects_for_url(url) Extracts a key (object) from a given parsed s3:// URL. The URL will be supplemented with a trailing slash if it is missing. .INDENT 7.0 .TP .B Parameters \fBurl\fP (\fI\%urllib.parse.ParseResult\fP) \-\- .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.utils.flatten_tags(tags) Convert tags from a key to value dict into a list of \(aqKey\(aq: xxx, \(aqValue\(aq: xxx dicts. .INDENT 7.0 .TP .B Parameters \fBtags\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type List[Dict[\fI\%str\fP, \fI\%str\fP]] .UNINDENT .UNINDENT .SS Package Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%get_current_aws_region\fP() T} T{ Return the AWS region that the currently configured AWS zone (see T} _ T{ \fI\%get_aws_zone_from_environment\fP() T} T{ Get the AWS zone from TOIL_AWS_ZONE if set. T} _ T{ \fI\%get_aws_zone_from_metadata\fP() T} T{ Get the AWS zone from instance metadata, if on EC2 and the boto module is T} _ T{ \fI\%get_aws_zone_from_boto\fP() T} T{ Get the AWS zone from the Boto config file, if it is configured and the T} _ T{ \fI\%get_aws_zone_from_environment_region\fP() T} T{ Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set. T} _ T{ \fI\%get_current_aws_zone\fP() T} T{ Get the currently configured or occupied AWS zone to use. T} _ T{ \fI\%zone_to_region\fP(zone) T} T{ Get a region (e.g. us\-west\-2) from a zone (e.g. us\-west\-1c). T} _ T{ \fI\%running_on_ec2\fP() T} T{ Return True if we are currently running on EC2, and false otherwise. T} _ T{ \fI\%running_on_ecs\fP() T} T{ Return True if we are currently running on Amazon ECS, and false otherwise. T} _ T{ \fI\%build_tag_dict_from_env\fP([environment]) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.aws.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.get_current_aws_region() Return the AWS region that the currently configured AWS zone (see get_current_aws_zone()) is in. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.get_aws_zone_from_environment() Get the AWS zone from TOIL_AWS_ZONE if set. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.get_aws_zone_from_metadata() Get the AWS zone from instance metadata, if on EC2 and the boto module is available. Otherwise, gets the AWS zone from ECS task metadata, if on ECS. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.get_aws_zone_from_boto() Get the AWS zone from the Boto config file, if it is configured and the boto module is available. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.get_aws_zone_from_environment_region() Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.get_current_aws_zone() Get the currently configured or occupied AWS zone to use. .sp Reports the TOIL_AWS_ZONE environment variable if set. .sp Otherwise, if we have boto and are running on EC2, or if we are on ECS, reports the zone we are running in. .sp Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region. .sp Finally, if we have boto2, and a default region is configured in Boto 2, chooses a zone in that region. .sp Returns None if no method can produce a zone to use. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.zone_to_region(zone) Get a region (e.g. us\-west\-2) from a zone (e.g. us\-west\-1c). .INDENT 7.0 .TP .B Parameters \fBzone\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.running_on_ec2() Return True if we are currently running on EC2, and false otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.running_on_ecs() Return True if we are currently running on Amazon ECS, and false otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.aws.build_tag_dict_from_env(environment=os.environ) .INDENT 7.0 .TP .B Parameters \fBenvironment\fP (\fIMutableMapping\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .SS \fI\%toil.lib.encryption\fP .SS Submodules .SS \fI\%toil.lib.encryption.conftest\fP .SS Module Contents .INDENT 0.0 .TP .B toil.lib.encryption.conftest.collect_ignore = [] .UNINDENT .SS Submodules .SS \fI\%toil.lib.accelerators\fP .sp Accelerator (i.e. GPU) utilities for Toil .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%have_working_nvidia_smi\fP() T} T{ Return True if the nvidia\-smi binary, from nvidia\(aqs CUDA userspace T} _ T{ \fI\%get_host_accelerator_numbers\fP() T} T{ Work out what accelerator is what. T} _ T{ \fI\%have_working_nvidia_docker_runtime\fP() T} T{ Return True if Docker exists and can handle an \(dqnvidia\(dq runtime and the \(dq\-\-gpus\(dq option. T} _ T{ \fI\%count_nvidia_gpus\fP() T} T{ Return the number of nvidia GPUs seen by nvidia\-smi, or 0 if it is not working. T} _ T{ \fI\%get_individual_local_accelerators\fP() T} T{ Determine all the local accelerators available. Report each with count 1, T} _ T{ \fI\%get_restrictive_environment_for_local_accelerators\fP(...) T} T{ Get environment variables which can be applied to a process to restrict it T} _ .TE .INDENT 0.0 .TP .B toil.lib.accelerators.have_working_nvidia_smi() Return True if the nvidia\-smi binary, from nvidia\(aqs CUDA userspace utilities, is installed and can be run successfully. .sp TODO: This isn\(aqt quite the same as the check that cwltool uses to decide if it can fulfill a CUDARequirement. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.accelerators.get_host_accelerator_numbers() Work out what accelerator is what. .sp For each accelerator visible to us, returns the host\-side (for example, outside\-of\-Slurm\-job) number for that accelerator. It is often the same as the apparent number. .sp Can be used with Docker\(aqs \-\-gpus=\(aq\(dqdevice=#,#,#\(dq\(aq option to forward the right GPUs as seen from a Docker daemon. .INDENT 7.0 .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.accelerators.have_working_nvidia_docker_runtime() Return True if Docker exists and can handle an \(dqnvidia\(dq runtime and the \(dq\-\-gpus\(dq option. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.accelerators.count_nvidia_gpus() Return the number of nvidia GPUs seen by nvidia\-smi, or 0 if it is not working. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.accelerators.get_individual_local_accelerators() Determine all the local accelerators available. Report each with count 1, in the order of the number that can be used to assign them. .sp TODO: How will numbers work with multiple types of accelerator? We need an accelerator assignment API. .INDENT 7.0 .TP .B Return type List[\fI\%toil.job.AcceleratorRequirement\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.accelerators.get_restrictive_environment_for_local_accelerators(accelerator_numbers) Get environment variables which can be applied to a process to restrict it to using only the given accelerator numbers. .sp The numbers are in the space of accelerators returned by get_individual_local_accelerators(). .INDENT 7.0 .TP .B Parameters \fBaccelerator_numbers\fP (\fIUnion\fP\fI[\fP\fISet\fP\fI[\fP\fI\%int\fP\fI]\fP\fI, \fP\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .SS \fI\%toil.lib.bioio\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%system\fP(command) T} T{ A convenience wrapper around subprocess.check_call that logs the command before passing it T} _ T{ \fI\%getLogLevelString\fP([logger]) T} T{ T} _ T{ \fI\%setLoggingFromOptions\fP(options) T} T{ T} _ T{ \fI\%getTempFile\fP([suffix, rootDir]) T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.bioio.system(command) A convenience wrapper around subprocess.check_call that logs the command before passing it on. The command can be either a string or a sequence of strings. If it is a string shell=True will be passed to subprocess.check_call. :type command: str | sequence[string] .UNINDENT .INDENT 0.0 .TP .B toil.lib.bioio.getLogLevelString(logger=None) .UNINDENT .INDENT 0.0 .TP .B toil.lib.bioio.setLoggingFromOptions(options) .UNINDENT .INDENT 0.0 .TP .B toil.lib.bioio.getTempFile(suffix=\(aq\(aq, rootDir=None) .UNINDENT .SS \fI\%toil.lib.compatibility\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%deprecated\fP(new_function_name) T} T{ T} _ T{ \fI\%compat_bytes\fP(s) T} T{ T} _ T{ \fI\%compat_bytes_recursive\fP(data) T} T{ Convert a tree of objects over bytes to objects over strings. T} _ .TE .INDENT 0.0 .TP .B toil.lib.compatibility.deprecated(new_function_name) .INDENT 7.0 .TP .B Parameters \fBnew_function_name\fP (\fI\%str\fP) \-\- .TP .B Return type Callable[Ellipsis, Any] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.compatibility.compat_bytes(s) .INDENT 7.0 .TP .B Parameters \fBs\fP (\fIUnion\fP\fI[\fP\fI\%bytes\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.compatibility.compat_bytes_recursive(data) Convert a tree of objects over bytes to objects over strings. .INDENT 7.0 .TP .B Parameters \fBdata\fP (\fIAny\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .SS \fI\%toil.lib.conversions\fP .sp Conversion utilities for mapping memory, disk, core declarations from strings to numbers and vice versa. Also contains general conversion functions .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%bytes_in_unit\fP([unit]) T} T{ T} _ T{ \fI\%convert_units\fP(num, src_unit[, dst_unit]) T} T{ Returns a float representing the converted input in dst_units. T} _ T{ \fI\%parse_memory_string\fP(string) T} T{ Given a string representation of some memory (i.e. \(aq1024 Mib\(aq), return the T} _ T{ \fI\%human2bytes\fP(string) T} T{ Given a string representation of some memory (i.e. \(aq1024 Mib\(aq), return the T} _ T{ \fI\%bytes2human\fP(n) T} T{ Return a binary value as a human readable string with units. T} _ T{ \fI\%b_to_mib\fP(n) T} T{ Convert a number from bytes to mibibytes. T} _ T{ \fI\%mib_to_b\fP(n) T} T{ Convert a number from mibibytes to bytes. T} _ T{ \fI\%hms_duration_to_seconds\fP(hms) T} T{ Parses a given time string in hours:minutes:seconds, T} _ T{ \fI\%strtobool\fP(val) T} T{ Make a human\-readable string into a bool. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%BINARY_PREFIXES\fP T} T{ T} _ T{ \fI\%DECIMAL_PREFIXES\fP T} T{ T} _ T{ \fI\%VALID_PREFIXES\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.conversions.BINARY_PREFIXES = [\(aqki\(aq, \(aqmi\(aq, \(aqgi\(aq, \(aqti\(aq, \(aqpi\(aq, \(aqei\(aq, \(aqkib\(aq, \(aqmib\(aq, \(aqgib\(aq, \(aqtib\(aq, \(aqpib\(aq, \(aqeib\(aq] .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.DECIMAL_PREFIXES = [\(aqb\(aq, \(aqk\(aq, \(aqm\(aq, \(aqg\(aq, \(aqt\(aq, \(aqp\(aq, \(aqe\(aq, \(aqkb\(aq, \(aqmb\(aq, \(aqgb\(aq, \(aqtb\(aq, \(aqpb\(aq, \(aqeb\(aq] .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.VALID_PREFIXES .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.bytes_in_unit(unit=\(aqB\(aq) .INDENT 7.0 .TP .B Parameters \fBunit\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.convert_units(num, src_unit, dst_unit=\(aqB\(aq) Returns a float representing the converted input in dst_units. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnum\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBsrc_unit\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdst_unit\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.parse_memory_string(string) Given a string representation of some memory (i.e. \(aq1024 Mib\(aq), return the number and unit. .INDENT 7.0 .TP .B Parameters \fBstring\fP (\fI\%str\fP) \-\- .TP .B Return type Tuple[\fI\%float\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.human2bytes(string) Given a string representation of some memory (i.e. \(aq1024 Mib\(aq), return the integer number of bytes. .INDENT 7.0 .TP .B Parameters \fBstring\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.bytes2human(n) Return a binary value as a human readable string with units. .INDENT 7.0 .TP .B Parameters \fBn\fP (\fISupportsInt\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.b_to_mib(n) Convert a number from bytes to mibibytes. .INDENT 7.0 .TP .B Parameters \fBn\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.mib_to_b(n) Convert a number from mibibytes to bytes. .INDENT 7.0 .TP .B Parameters \fBn\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.hms_duration_to_seconds(hms) Parses a given time string in hours:minutes:seconds, returns an equivalent total seconds value .INDENT 7.0 .TP .B Parameters \fBhms\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.conversions.strtobool(val) Make a human\-readable string into a bool. .sp Convert a string along the lines of \(dqy\(dq, \(dq1\(dq, \(dqON\(dq, \(dqTrUe\(dq, or \(dqYes\(dq to True, and the corresponding false\-ish values to False. .INDENT 7.0 .TP .B Parameters \fBval\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .SS \fI\%toil.lib.docker\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%dockerCheckOutput\fP(*args, **kwargs) T} T{ T} _ T{ \fI\%dockerCall\fP(*args, **kwargs) T} T{ T} _ T{ \fI\%subprocessDockerCall\fP(*args, **kwargs) T} T{ T} _ T{ \fI\%apiDockerCall\fP(job, image[, parameters, deferParam, ...]) T} T{ A toil wrapper for the python docker API. T} _ T{ \fI\%dockerKill\fP(container_name[, gentleKill, remove, timeout]) T} T{ Immediately kills a container. Equivalent to \(dqdocker kill\(dq: T} _ T{ \fI\%dockerStop\fP(container_name[, remove]) T} T{ Gracefully kills a container. Equivalent to \(dqdocker stop\(dq: T} _ T{ \fI\%containerIsRunning\fP(container_name[, timeout]) T} T{ Checks whether the container is running or not. T} _ T{ \fI\%getContainerName\fP(job) T} T{ Create a random string including the job name, and return it. Name will T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%FORGO\fP T} T{ T} _ T{ \fI\%STOP\fP T} T{ T} _ T{ \fI\%RM\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.docker.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.FORGO = 0 .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.STOP = 1 .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.RM = 2 .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.dockerCheckOutput(*args, **kwargs) .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.dockerCall(*args, **kwargs) .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.subprocessDockerCall(*args, **kwargs) .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.apiDockerCall(job, image, parameters=None, deferParam=None, volumes=None, working_dir=None, containerName=None, entrypoint=None, detach=False, log_config=None, auto_remove=None, remove=False, user=None, environment=None, stdout=None, stderr=False, stream=False, demux=False, streamfile=None, accelerators=None, timeout=365 * 24 * 60 * 60, **kwargs) A toil wrapper for the python docker API. .sp Docker API Docs: \fI\%https://docker\-py.readthedocs.io/en/stable/index.html\fP Docker API Code: \fI\%https://github.com/docker/docker\-py\fP .sp This implements docker\(aqs python API within toil so that calls are run as jobs, with the intention that failed/orphaned docker jobs be handled appropriately. .sp Example of using dockerCall in toil to index a FASTA file with SAMtools: .INDENT 7.0 .INDENT 3.5 .sp .EX def toil_job(job): working_dir = job.fileStore.getLocalTempDir() path = job.fileStore.readGlobalFile(ref_id, os.path.join(working_dir, \(aqref.fasta\(aq) parameters = [\(aqfaidx\(aq, path] apiDockerCall(job, image=\(aqquay.io/ucgc_cgl/samtools:latest\(aq, working_dir=working_dir, parameters=parameters) .EE .UNINDENT .UNINDENT .sp Note that when run with detach=False, or with detach=True and stdout=True or stderr=True, this is a blocking call. When run with detach=True and without output capture, the container is started and returned without waiting for it to finish. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fItoil.Job.job\fP) \-\- The Job instance for the calling function. .IP \(bu 2 \fBimage\fP (\fI\%str\fP) \-\- Name of the Docker image to be used. (e.g. \(aqquay.io/ucsc_cgl/samtools:latest\(aq) .IP \(bu 2 \fBparameters\fP (\fI\%list\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- A list of string elements. If there are multiple elements, these will be joined with spaces. This handling of multiple elements provides backwards compatibility with previous versions which called docker using subprocess.check_call(). If list of lists: list[list[str]], then treat as successive commands chained with pipe. .IP \(bu 2 \fBworking_dir\fP (\fI\%str\fP) \-\- The working directory. .IP \(bu 2 \fBdeferParam\fP (\fI\%int\fP) \-\- Action to take on the container upon job completion. FORGO (0) leaves the container untouched and running. STOP (1) Sends SIGTERM, then SIGKILL if necessary to the container. RM (2) Immediately send SIGKILL to the container. This is the default behavior if deferParam is set to None. .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- The name/ID of the container. .IP \(bu 2 \fBentrypoint\fP (\fI\%str\fP) \-\- Prepends commands sent to the container. See: \fI\%https://docker\-py.readthedocs.io/en/stable/containers.html\fP .IP \(bu 2 \fBdetach\fP (\fI\%bool\fP) \-\- Run the container in detached mode. (equivalent to \(aq\-d\(aq) .IP \(bu 2 \fBstdout\fP (\fI\%bool\fP) \-\- Return logs from STDOUT when detach=False (default: True). Block and capture stdout to a file when detach=True (default: False). Output capture defaults to output.log, and can be specified with the \(dqstreamfile\(dq kwarg. .IP \(bu 2 \fBstderr\fP (\fI\%bool\fP) \-\- Return logs from STDERR when detach=False (default: False). Block and capture stderr to a file when detach=True (default: False). Output capture defaults to output.log, and can be specified with the \(dqstreamfile\(dq kwarg. .IP \(bu 2 \fBstream\fP (\fI\%bool\fP) \-\- If True and detach=False, return a log generator instead of a string. Ignored if detach=True. (default: False). .IP \(bu 2 \fBdemux\fP (\fI\%bool\fP) \-\- Similar to \fIdemux\fP in container.exec_run(). If True and detach=False, returns a tuple of (stdout, stderr). If stream=True, returns a log generator with tuples of (stdout, stderr). Ignored if detach=True. (default: False). .IP \(bu 2 \fBstreamfile\fP (\fI\%str\fP) \-\- Collect container output to this file if detach=True and stderr and/or stdout are True. Defaults to \(dqoutput.log\(dq. .IP \(bu 2 \fBlog_config\fP (\fI\%dict\fP) \-\- Specify the logs to return from the container. See: \fI\%https://docker\-py.readthedocs.io/en/stable/containers.html\fP .IP \(bu 2 \fBremove\fP (\fI\%bool\fP) \-\- Remove the container on exit or not. .IP \(bu 2 \fBuser\fP (\fI\%str\fP) \-\- The container will be run with the privileges of the user specified. Can be an actual name, such as \(aqroot\(aq or \(aqlifeisaboutfishtacos\(aq, or it can be the uid or gid of the user (\(aq0\(aq is root; \(aq1000\(aq is an example of a less privileged uid or gid), or a complement of the uid:gid (RECOMMENDED), such as \(aq0:0\(aq (root user : root group) or \(aq1000:1000\(aq (some other user : some other user group). .IP \(bu 2 \fBenvironment\fP \-\- Allows one to set environment variables inside of the container, such as: .IP \(bu 2 \fBtimeout\fP (\fI\%int\fP) \-\- Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely). .IP \(bu 2 \fBaccelerators\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP\fI]\fP) \-\- Toil accelerator numbers (usually GPUs) to forward to the container. These are interpreted in the current Python process\(aqs environment. See toil.lib.accelerators.get_individual_local_accelerators() for the menu of available accelerators. .IP \(bu 2 \fBkwargs\fP \-\- Additional keyword arguments supplied to the docker API\(aqs run command. The list is 75 keywords total, for examples and full documentation see: \fI\%https://docker\-py.readthedocs.io/en/stable/containers.html\fP .UNINDENT .TP .B Returns Returns the standard output/standard error text, as requested, when detach=False. Returns the underlying docker.models.containers.Container object from the Docker API when detach=True. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.dockerKill(container_name, gentleKill=False, remove=False, timeout=365 * 24 * 60 * 60) Immediately kills a container. Equivalent to \(dqdocker kill\(dq: \fI\%https://docs.docker.com/engine/reference/commandline/kill/\fP .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcontainer_name\fP (\fI\%str\fP) \-\- Name of the container being killed. .IP \(bu 2 \fBgentleKill\fP (\fI\%bool\fP) \-\- If True, trigger a graceful shutdown. .IP \(bu 2 \fBremove\fP (\fI\%bool\fP) \-\- If True, remove the container after it exits. .IP \(bu 2 \fBtimeout\fP (\fI\%int\fP) \-\- Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely). .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.dockerStop(container_name, remove=False) Gracefully kills a container. Equivalent to \(dqdocker stop\(dq: \fI\%https://docs.docker.com/engine/reference/commandline/stop/\fP .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcontainer_name\fP (\fI\%str\fP) \-\- Name of the container being stopped. .IP \(bu 2 \fBremove\fP (\fI\%bool\fP) \-\- If True, remove the container after it exits. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.containerIsRunning(container_name, timeout=365 * 24 * 60 * 60) Checks whether the container is running or not. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcontainer_name\fP (\fI\%str\fP) \-\- Name of the container being checked. .IP \(bu 2 \fBtimeout\fP (\fI\%int\fP) \-\- Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely). .UNINDENT .TP .B Returns True if status is \(aqrunning\(aq, False if status is anything else, and None if the container does not exist. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.docker.getContainerName(job) Create a random string including the job name, and return it. Name will match \fB[a\-zA\-Z0\-9][a\-zA\-Z0\-9_.\-]\fP\&. .UNINDENT .SS \fI\%toil.lib.ec2\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%not_found\fP(e) T} T{ T} _ T{ \fI\%inconsistencies_detected\fP(e) T} T{ T} _ T{ \fI\%retry_ec2\fP([t, retry_for, retry_while]) T} T{ T} _ T{ \fI\%wait_transition\fP(resource, from_states, to_state[, ...]) T} T{ Wait until the specified EC2 resource (instance, image, volume, ...) transitions from any T} _ T{ \fI\%wait_instances_running\fP(ec2, instances) T} T{ Wait until no instance in the given iterable is \(aqpending\(aq. Yield every instance that T} _ T{ \fI\%wait_spot_requests_active\fP(ec2, requests[, timeout, ...]) T} T{ Wait until no spot request in the given iterator is in the \(aqopen\(aq state or, optionally, T} _ T{ \fI\%create_spot_instances\fP(ec2, price, image_id, spec[, ...]) T} T{ Create instances on the spot market. T} _ T{ \fI\%create_ondemand_instances\fP(ec2, image_id, spec[, ...]) T} T{ Requests the RunInstances EC2 API call but accounts for the race between recently created T} _ T{ \fI\%prune\fP(bushy) T} T{ Prune entries in the given dict with false\-y values. T} _ T{ \fI\%wait_until_instance_profile_arn_exists\fP(...) T} T{ T} _ T{ \fI\%create_instances\fP(ec2_resource, image_id, key_name, ...) T} T{ Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts. T} _ T{ \fI\%create_launch_template\fP(ec2_client, template_name, ...) T} T{ Creates a launch template with the given name for launching instances with the given parameters. T} _ T{ \fI\%create_auto_scaling_group\fP(autoscaling_client, ...[, ...]) T} T{ Create a new Auto Scaling Group with the given name (which is also its T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%a_short_time\fP T} T{ T} _ T{ \fI\%a_long_time\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%INCONSISTENCY_ERRORS\fP T} T{ T} _ T{ \fI\%iam_client\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.ec2.a_short_time = 5 .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.a_long_time .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.logger .UNINDENT .INDENT 0.0 .TP .B exception toil.lib.ec2.UserError(message=None, cause=None) Bases: \fI\%RuntimeError\fP .sp Unspecified run\-time error. .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.not_found(e) .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.inconsistencies_detected(e) .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.INCONSISTENCY_ERRORS .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.retry_ec2(t=a_short_time, retry_for=10 * a_short_time, retry_while=not_found) .UNINDENT .INDENT 0.0 .TP .B exception toil.lib.ec2.UnexpectedResourceState(resource, to_state, state) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.wait_transition(resource, from_states, to_state, state_getter=attrgetter(\(aqstate\(aq)) Wait until the specified EC2 resource (instance, image, volume, ...) transitions from any of the given \(aqfrom\(aq states to the specified \(aqto\(aq state. If the instance is found in a state other that the to state or any of the from states, an exception will be thrown. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBresource\fP \-\- the resource to monitor .IP \(bu 2 \fBfrom_states\fP \-\- a set of states that the resource is expected to be in before the transition occurs .IP \(bu 2 \fBto_state\fP \-\- the state of the resource when this method returns .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.wait_instances_running(ec2, instances) Wait until no instance in the given iterable is \(aqpending\(aq. Yield every instance that entered the running state as soon as it does. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBec2\fP (\fIboto.ec2.connection.EC2Connection\fP) \-\- the EC2 connection to use for making requests .IP \(bu 2 \fBinstances\fP (\fIIterable\fP\fI[\fP\fIBoto2Instance\fP\fI]\fP) \-\- the instances to wait on .UNINDENT .TP .B Return type Iterable[Boto2Instance] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.wait_spot_requests_active(ec2, requests, timeout=None, tentative=False) Wait until no spot request in the given iterator is in the \(aqopen\(aq state or, optionally, a timeout occurs. Yield spot requests as soon as they leave the \(aqopen\(aq state. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequests\fP (\fIIterable\fP\fI[\fP\fIboto.ec2.spotinstancerequest.SpotInstanceRequest\fP\fI]\fP) \-\- The requests to wait on. .IP \(bu 2 \fBtimeout\fP (\fI\%float\fP) \-\- Maximum time in seconds to spend waiting or None to wait forever. If a timeout occurs, the remaining open requests will be cancelled. .IP \(bu 2 \fBtentative\fP (\fI\%bool\fP) \-\- if True, give up on a spot request at the earliest indication of it not being fulfilled immediately .UNINDENT .TP .B Return type Iterable[List[boto.ec2.spotinstancerequest.SpotInstanceRequest]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.create_spot_instances(ec2, price, image_id, spec, num_instances=1, timeout=None, tentative=False, tags=None) Create instances on the spot market. .INDENT 7.0 .TP .B Return type Iterable[List[boto.ec2.instance.Instance]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.create_ondemand_instances(ec2, image_id, spec, num_instances=1) Requests the RunInstances EC2 API call but accounts for the race between recently created instance profiles, IAM roles and an instance creation that refers to them. .INDENT 7.0 .TP .B Return type List[Boto2Instance] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.prune(bushy) Prune entries in the given dict with false\-y values. Boto3 may not like None and instead wants no key. .INDENT 7.0 .TP .B Parameters \fBbushy\fP (\fI\%dict\fP) \-\- .TP .B Return type \fI\%dict\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.iam_client .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.wait_until_instance_profile_arn_exists(instance_profile_arn) .INDENT 7.0 .TP .B Parameters \fBinstance_profile_arn\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.create_instances(ec2_resource, image_id, key_name, instance_type, num_instances=1, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None) Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts. .INDENT 7.0 .TP .B See \(dqcreate_instances\(dq (returns a list of ec2.Instance objects): \fI\%https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances\fP .TP .B Not to be confused with \(dqrun_instances\(dq (same input args; returns a dictionary): \fI\%https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances\fP .UNINDENT .sp Tags, if given, are applied to the instances, and all volumes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBec2_resource\fP (\fIboto3.resources.base.ServiceResource\fP) \-\- .IP \(bu 2 \fBimage_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBinstance_type\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBnum_instances\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBsecurity_group_ids\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI]\fP) \-\- .IP \(bu 2 \fBuser_data\fP (\fIOptional\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%bytes\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBblock_device_map\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fIDict\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBinstance_profile_arn\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBplacement_az\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBsubnet_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtags\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type List[\fI\%dict\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.create_launch_template(ec2_client, template_name, image_id, key_name, instance_type, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None) Creates a launch template with the given name for launching instances with the given parameters. .sp We only ever use the default version of any launch template. .sp Internally calls \fI\%https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html?highlight=create_launch_template#EC2.Client.create_launch_template\fP .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtags\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Tags, if given, are applied to the template itself, all instances, and all volumes. .IP \(bu 2 \fBuser_data\fP (\fIOptional\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%bytes\fP\fI]\fP\fI]\fP) \-\- non\-base64\-encoded user data to pass to the instances. .IP \(bu 2 \fBec2_client\fP (\fIbotocore.client.BaseClient\fP) \-\- .IP \(bu 2 \fBtemplate_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBimage_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBinstance_type\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBsecurity_group_ids\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI]\fP) \-\- .IP \(bu 2 \fBblock_device_map\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fIDict\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBinstance_profile_arn\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBplacement_az\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBsubnet_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns the ID of the launch template. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2.create_auto_scaling_group(autoscaling_client, asg_name, launch_template_ids, vpc_subnets, min_size, max_size, instance_types=None, spot_bid=None, spot_cheapest=False, tags=None) Create a new Auto Scaling Group with the given name (which is also its unique identifier). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBautoscaling_client\fP (\fIbotocore.client.BaseClient\fP) \-\- Boto3 client for autoscaling. .IP \(bu 2 \fBasg_name\fP (\fI\%str\fP) \-\- Unique name for the autoscaling group. .IP \(bu 2 \fBlaunch_template_ids\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- ID of the launch template to make instances from, for each instance type. .IP \(bu 2 \fBvpc_subnets\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- One or more subnet IDs to place instances in the group into. Determine the availability zone(s) instances will launch into. .IP \(bu 2 \fBmin_size\fP (\fI\%int\fP) \-\- Minimum number of instances to have in the group at all times. .IP \(bu 2 \fBmax_size\fP (\fI\%int\fP) \-\- Maximum number of instances to allow in the group at any time. .IP \(bu 2 \fBinstance_types\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Use a pool over the given instance types, instead of the type given in the launch template. For on\-demand groups, this is a prioritized list. For spot groups, we let AWS balance according to spot_strategy. Must be 20 types or shorter. .IP \(bu 2 \fBspot_bid\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- If set, the ASG will be a spot market ASG. Bid is in dollars per instance hour. All instance types in the group are bid on equivalently. .IP \(bu 2 \fBspot_cheapest\fP (\fI\%bool\fP) \-\- If true, use the cheapest spot instances available out of instance_types, instead of the spot instances that minimize eviction probability. .IP \(bu 2 \fBtags\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Tags to apply to the ASG only. Tags for the instances should be added to the launch template instead. .UNINDENT .TP .B Return type None .UNINDENT .sp The default version of the launch template is used. .UNINDENT .SS \fI\%toil.lib.ec2nodes\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%InstanceType\fP T} T{ T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%is_number\fP(s) T} T{ Determines if a unicode string (that may include commas) is a number. T} _ T{ \fI\%parse_storage\fP(storage_info) T} T{ Parses EC2 JSON storage param string into a number. T} _ T{ \fI\%parse_memory\fP(mem_info) T} T{ Returns EC2 \(aqmemory\(aq string as a float. T} _ T{ \fI\%download_region_json\fP(filename[, region]) T} T{ Downloads and writes the AWS Billing JSON to a file using the AWS pricing API. T} _ T{ \fI\%reduce_region_json_size\fP(filename) T} T{ Deletes information in the json file that we don\(aqt need, and rewrites it. This makes the file smaller. T} _ T{ \fI\%updateStaticEC2Instances\fP() T} T{ Generates a new python file of fetchable EC2 Instances by region with current prices and specs. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%manager\fP T} T{ T} _ T{ \fI\%dirname\fP T} T{ T} _ T{ \fI\%region_json_dirname\fP T} T{ T} _ T{ \fI\%EC2Regions\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.ec2nodes.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.manager .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.dirname .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.region_json_dirname .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.EC2Regions .UNINDENT .INDENT 0.0 .TP .B class toil.lib.ec2nodes.InstanceType(name, cores, memory, disks, disk_capacity, architecture) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcores\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBdisks\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBdisk_capacity\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __slots__ = (\(aqname\(aq, \(aqcores\(aq, \(aqmemory\(aq, \(aqdisks\(aq, \(aqdisk_capacity\(aq, \(aqarchitecture\(aq) .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __eq__(other) Return self==value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%object\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.is_number(s) Determines if a unicode string (that may include commas) is a number. .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- Any unicode string. .TP .B Returns True if s represents a number, False otherwise. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.parse_storage(storage_info) Parses EC2 JSON storage param string into a number. .INDENT 7.0 .TP .B Examples: \(dq2 x 160 SSD\(dq \(dq3 x 2000 HDD\(dq \(dqEBS only\(dq \(dq1 x 410\(dq \(dq8 x 1.9 NVMe SSD\(dq \(dq900 GB NVMe SSD\(dq .UNINDENT .INDENT 7.0 .TP .B Parameters \fBstorage_info\fP (\fI\%str\fP) \-\- EC2 JSON storage param string. .TP .B Returns Two floats representing: (# of disks), and (disk_capacity in GiB of each disk). .TP .B Return type Union[List[\fI\%int\fP], Tuple[Union[\fI\%int\fP, \fI\%float\fP], \fI\%float\fP]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.parse_memory(mem_info) Returns EC2 \(aqmemory\(aq string as a float. .sp Format should always be \(aq#\(aq GiB (example: \(aq244 GiB\(aq or \(aq1,952 GiB\(aq). Amazon loves to put commas in their numbers, so we have to accommodate that. If the syntax ever changes, this will raise. .INDENT 7.0 .TP .B Parameters \fBmem_info\fP (\fI\%str\fP) \-\- EC2 JSON memory param string. .TP .B Returns A float representing memory in GiB. .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.download_region_json(filename, region=\(aqus\-east\-1\(aq) Downloads and writes the AWS Billing JSON to a file using the AWS pricing API. .sp See: \fI\%https://aws.amazon.com/blogs/aws/new\-aws\-price\-list\-api/\fP .INDENT 7.0 .TP .B Returns A dict of InstanceType objects, where the key is the string: aws instance name (example: \(aqt2.micro\(aq), and the value is an InstanceType object representing that aws instance name. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfilename\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBregion\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.reduce_region_json_size(filename) Deletes information in the json file that we don\(aqt need, and rewrites it. This makes the file smaller. .sp The reason being: we used to download the unified AWS Bulk API JSON, which eventually crept up to 5.6Gb, the loading of which could not be done on a 32Gb RAM machine. Now we download each region JSON individually (with AWS\(aqs new Query API), but even those may eventually one day grow ridiculously large, so we do what we can to keep the file sizes down (and thus also the amount loaded into memory) to keep this script working for longer. .INDENT 7.0 .TP .B Parameters \fBfilename\fP (\fI\%str\fP) \-\- .TP .B Return type List[Dict[\fI\%str\fP, Any]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.ec2nodes.updateStaticEC2Instances() Generates a new python file of fetchable EC2 Instances by region with current prices and specs. .sp Takes a few (~3+) minutes to run (you\(aqll need decent internet). .INDENT 7.0 .TP .B Returns Nothing. Writes a new \(aqgeneratedEC2Lists.py\(aq file. .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.lib.exceptions\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%panic\fP T} T{ The Python idiom for reraising a primary exception fails when the except block raises a T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%raise_\fP(exc_type, exc_value, traceback) T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.lib.exceptions.panic(log=None) The Python idiom for reraising a primary exception fails when the except block raises a secondary exception, e.g. while trying to cleanup. In that case the original exception is lost and the secondary exception is reraised. The solution seems to be to save the primary exception info as returned from sys.exc_info() and then reraise that. .sp This is a contextmanager that should be used like this .INDENT 7.0 .TP .B try: # do something that can fail .TP .B except: .INDENT 7.0 .TP .B with panic( log ): # do cleanup that can also fail .UNINDENT .UNINDENT .sp If a logging logger is passed to panic(), any secondary Exception raised within the with block will be logged. Otherwise those exceptions are swallowed. At the end of the with block the primary exception will be reraised. .INDENT 7.0 .TP .B __enter__() .UNINDENT .INDENT 7.0 .TP .B __exit__(*exc_info) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.exceptions.raise_(exc_type, exc_value, traceback) .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.lib.expando\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Expando\fP T} T{ Pass inital attributes to the constructor: T} _ T{ \fI\%MagicExpando\fP T} T{ Use MagicExpando for chained attribute access. T} _ .TE .INDENT 0.0 .TP .B class toil.lib.expando.Expando(*args, **kwargs) Bases: \fI\%dict\fP .sp Pass inital attributes to the constructor: .sp .EX >>> o = Expando(foo=42) >>> o.foo 42 .EE .sp Dynamically create new attributes: .sp .EX >>> o.bar = \(aqhi\(aq >>> o.bar \(aqhi\(aq .EE .sp Expando is a dictionary: .sp .EX >>> isinstance(o,dict) True >>> o[\(aqfoo\(aq] 42 .EE .sp Works great with JSON: .sp .EX >>> import json >>> s=\(aq{\(dqfoo\(dq:42}\(aq >>> o = json.loads(s,object_hook=Expando) >>> o.foo 42 >>> o.bar = \(aqhi\(aq >>> o.bar \(aqhi\(aq .EE .sp And since Expando is a dict, it serializes back to JSON just fine: .sp .EX >>> json.dumps(o, sort_keys=True) \(aq{\(dqbar\(dq: \(dqhi\(dq, \(dqfoo\(dq: 42}\(aq .EE .sp Attributes can be deleted, too: .sp .EX >>> o = Expando(foo=42) >>> o.foo 42 >>> del o.foo >>> o.foo Traceback (most recent call last): \&... AttributeError: \(aqExpando\(aq object has no attribute \(aqfoo\(aq >>> o[\(aqfoo\(aq] Traceback (most recent call last): \&... KeyError: \(aqfoo\(aq .EE .sp .EX >>> del o.foo Traceback (most recent call last): \&... AttributeError: foo .EE .sp And copied: .sp .EX >>> o = Expando(foo=42) >>> p = o.copy() >>> isinstance(p,Expando) True >>> o == p True >>> o is p False .EE .sp Same with MagicExpando ... .sp .EX >>> o = MagicExpando() >>> o.foo.bar = 42 >>> p = o.copy() >>> isinstance(p,MagicExpando) True >>> o == p True >>> o is p False .EE .sp \&... but the copy is shallow: .sp .EX >>> o.foo is p.foo True .EE .INDENT 7.0 .TP .B copy() D.copy() \-> a shallow copy of D .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.lib.expando.MagicExpando(*args, **kwargs) Bases: \fI\%Expando\fP .sp Use MagicExpando for chained attribute access. .sp The first time a missing attribute is accessed, it will be set to a new child MagicExpando. .sp .EX >>> o=MagicExpando() >>> o.foo = 42 >>> o {\(aqfoo\(aq: 42} >>> o.bar.hello = \(aqhi\(aq >>> o.bar {\(aqhello\(aq: \(aqhi\(aq} .EE .INDENT 7.0 .TP .B __getattribute__(name) Return getattr(self, name). .INDENT 7.0 .TP .B Parameters \fBname\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.lib.generatedEC2Lists\fP .SS Module Contents .INDENT 0.0 .TP .B toil.lib.generatedEC2Lists.E2Instances .UNINDENT .INDENT 0.0 .TP .B toil.lib.generatedEC2Lists.regionDict .UNINDENT .INDENT 0.0 .TP .B toil.lib.generatedEC2Lists.ec2InstancesByRegion .UNINDENT .SS \fI\%toil.lib.humanize\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%bytes2human\fP(n) T} T{ Convert n bytes into a human readable string. T} _ T{ \fI\%human2bytes\fP(s) T} T{ Attempts to guess the string format based on default symbols T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.humanize.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.humanize.bytes2human(n) Convert n bytes into a human readable string. .INDENT 7.0 .TP .B Parameters \fBn\fP (\fISupportsInt\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.humanize.human2bytes(s) Attempts to guess the string format based on default symbols set and return the corresponding bytes as an integer. .sp When unable to recognize the format ValueError is raised. .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .SS \fI\%toil.lib.io\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%WriteWatchingStream\fP T} T{ A stream wrapping class that calls any functions passed to onWrite() with the number of bytes written for every write. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%mkdtemp\fP([suffix, prefix, dir]) T} T{ Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions. T} _ T{ \fI\%robust_rmtree\fP(path) T} T{ Robustly tries to delete paths. T} _ T{ \fI\%atomic_tmp_file\fP(final_path) T} T{ Return a tmp file name to use with atomic_install. This will be in the T} _ T{ \fI\%atomic_install\fP(tmp_path, final_path) T} T{ atomic install of tmp_path as final_path T} _ T{ \fI\%AtomicFileCreate\fP(final_path[, keep]) T} T{ Context manager to create a temporary file. Entering returns path to T} _ T{ \fI\%atomic_copy\fP(src_path, dest_path[, executable]) T} T{ Copy a file using posix atomic creations semantics. T} _ T{ \fI\%atomic_copyobj\fP(src_fh, dest_path[, length, executable]) T} T{ Copy an open file using posix atomic creations semantics. T} _ T{ \fI\%make_public_dir\fP([in_directory]) T} T{ Try to make a random directory name with length 4 that doesn\(aqt exist, with the given prefix. T} _ T{ \fI\%try_path\fP(path[, min_size]) T} T{ Try to use the given path. Return it if it exists or can be made, T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.io.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.mkdtemp(suffix=None, prefix=None, dir=None) Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions. .sp The permissions on the directory will be 711 instead of 700, allowing the group and all other users to traverse the directory. This is necessary if the direcotry is on NFS and the Docker daemon would like to mount it or a file inside it into a container, because on NFS even the Docker daemon appears bound by the file permissions. .sp See <\fI\%https://github.com/DataBiosphere/toil/issues/4644\fP>, and <\fI\%https://stackoverflow.com/a/67928880\fP> which talks about a similar problem but in the context of user namespaces. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBprefix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.robust_rmtree(path) Robustly tries to delete paths. .sp Continues silently if the path to be removed is already gone, or if it goes away while this function is executing. .sp May raise an error if a path changes between file and directory while the function is executing, or if a permission error is encountered. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%bytes\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.atomic_tmp_file(final_path) Return a tmp file name to use with atomic_install. This will be in the same directory as final_path. The temporary file will have the same extension as finalPath. It the final path is in /dev (/dev/null, /dev/stdout), it is returned unchanged and atomic_tmp_install will do nothing. .INDENT 7.0 .TP .B Parameters \fBfinal_path\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.atomic_install(tmp_path, final_path) atomic install of tmp_path as final_path .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.AtomicFileCreate(final_path, keep=False) Context manager to create a temporary file. Entering returns path to the temporary file in the same directory as finalPath. If the code in context succeeds, the file renamed to its actually name. If an error occurs, the file is not installed and is removed unless keep is specified. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfinal_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkeep\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type Iterator[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.atomic_copy(src_path, dest_path, executable=None) Copy a file using posix atomic creations semantics. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdest_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBexecutable\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.atomic_copyobj(src_fh, dest_path, length=16384, executable=False) Copy an open file using posix atomic creations semantics. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc_fh\fP (\fI\%io.BytesIO\fP) \-\- .IP \(bu 2 \fBdest_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBlength\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBexecutable\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.make_public_dir(in_directory=None) Try to make a random directory name with length 4 that doesn\(aqt exist, with the given prefix. Otherwise, try length 5, length 6, etc, up to a max of 32 (len of uuid4 with dashes replaced). This function\(aqs purpose is mostly to avoid having long file names when generating directories. If somehow this fails, which should be incredibly unlikely, default to a normal uuid4, which was our old default. .INDENT 7.0 .TP .B Parameters \fBin_directory\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.io.try_path(path, min_size=100 * 1024 * 1024) Try to use the given path. Return it if it exists or can be made, and we can make things within it, or None otherwise. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmin_size\fP (\fI\%int\fP) \-\- Reject paths on filesystems smaller than this many bytes. .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.lib.io.WriteWatchingStream(backingStream) A stream wrapping class that calls any functions passed to onWrite() with the number of bytes written for every write. .sp Not seekable. .INDENT 7.0 .TP .B Parameters \fBbackingStream\fP (\fIIO\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B onWrite(listener) Call the given listener with the number of bytes written on every write. .INDENT 7.0 .TP .B Parameters \fBlistener\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%int\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write(data) Write the given data to the file. .UNINDENT .INDENT 7.0 .TP .B writelines(datas) Write each string from the given iterable, without newlines. .UNINDENT .INDENT 7.0 .TP .B flush() Flush the backing stream. .UNINDENT .INDENT 7.0 .TP .B close() Close the backing stream. .UNINDENT .UNINDENT .SS \fI\%toil.lib.iterables\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%concat\fP T} T{ A literal iterable to combine sequence literals (lists, set) with generators or list comprehensions. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%flatten\fP(iterables) T} T{ Flatten an iterable, except for string elements. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%IT\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.iterables.IT .UNINDENT .INDENT 0.0 .TP .B toil.lib.iterables.flatten(iterables) Flatten an iterable, except for string elements. .INDENT 7.0 .TP .B Parameters \fBiterables\fP (\fIIterable\fP\fI[\fP\fIIT\fP\fI]\fP) \-\- .TP .B Return type Iterator[IT] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.lib.iterables.concat(*args) A literal iterable to combine sequence literals (lists, set) with generators or list comprehensions. .sp Instead of .sp .EX >>> [ \-1 ] + [ x * 2 for x in range( 3 ) ] + [ \-1 ] [\-1, 0, 2, 4, \-1] .EE .sp you can write .sp .EX >>> list( concat( \-1, ( x * 2 for x in range( 3 ) ), \-1 ) ) [\-1, 0, 2, 4, \-1] .EE .sp This is slightly shorter (not counting the list constructor) and does not involve array construction or concatenation. .sp Note that concat() flattens (or chains) all iterable arguments into a single result iterable: .sp .EX >>> list( concat( 1, range( 2, 4 ), 4 ) ) [1, 2, 3, 4] .EE .sp It only does so one level deep. If you need to recursively flatten a data structure, check out crush(). .sp If you want to prevent that flattening for an iterable argument, wrap it in concat(): .sp .EX >>> list( concat( 1, concat( range( 2, 4 ) ), 4 ) ) [1, range(2, 4), 4] .EE .sp Some more example. .sp .EX >>> list( concat() ) # empty concat [] >>> list( concat( 1 ) ) # non\-iterable [1] >>> list( concat( concat() ) ) # empty iterable [] >>> list( concat( concat( 1 ) ) ) # singleton iterable [1] >>> list( concat( 1, concat( 2 ), 3 ) ) # flattened iterable [1, 2, 3] >>> list( concat( 1, [2], 3 ) ) # flattened iterable [1, 2, 3] >>> list( concat( 1, concat( [2] ), 3 ) ) # protecting an iterable from being flattened [1, [2], 3] >>> list( concat( 1, concat( [2], 3 ), 4 ) ) # protection only works with a single argument [1, 2, 3, 4] >>> list( concat( 1, 2, concat( 3, 4 ), 5, 6 ) ) [1, 2, 3, 4, 5, 6] >>> list( concat( 1, 2, concat( [ 3, 4 ] ), 5, 6 ) ) [1, 2, [3, 4], 5, 6] .EE .sp Note that while strings are technically iterable, concat() does not flatten them. .sp .EX >>> list( concat( \(aqab\(aq ) ) [\(aqab\(aq] >>> list( concat( concat( \(aqab\(aq ) ) ) [\(aqab\(aq] .EE .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fIAny\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __iter__() .INDENT 7.0 .TP .B Return type Iterator[Any] .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.lib.memoize\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%sync_memoize\fP(f) T} T{ Like memoize, but guarantees that decorated function is only called once, even when multiple T} _ T{ \fI\%parse_iso_utc\fP(s) T} T{ Parses an ISO time with a hard\-coded Z for zulu\-time (UTC) at the end. Other timezones are T} _ T{ \fI\%strict_bool\fP(s) T} T{ Variant of bool() that only accepts two possible string values. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%memoize\fP T} T{ Memoize a function result based on its parameters using this decorator. T} _ T{ \fI\%MAT\fP T} T{ T} _ T{ \fI\%MRT\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.memoize.memoize Memoize a function result based on its parameters using this decorator. .sp For example, this can be used in place of lazy initialization. If the decorating function is invoked by multiple threads, the decorated function may be called more than once with the same arguments. .UNINDENT .INDENT 0.0 .TP .B toil.lib.memoize.MAT .UNINDENT .INDENT 0.0 .TP .B toil.lib.memoize.MRT .UNINDENT .INDENT 0.0 .TP .B toil.lib.memoize.sync_memoize(f) Like memoize, but guarantees that decorated function is only called once, even when multiple threads are calling the decorating function with multiple parameters. .INDENT 7.0 .TP .B Parameters \fBf\fP (\fICallable\fP\fI[\fP\fI[\fP\fIMAT\fP\fI]\fP\fI, \fP\fIMRT\fP\fI]\fP) \-\- .TP .B Return type Callable[[MAT], MRT] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.memoize.parse_iso_utc(s) Parses an ISO time with a hard\-coded Z for zulu\-time (UTC) at the end. Other timezones are not supported. Returns a timezone\-naive datetime object. .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- The ISO\-formatted time .TP .B Returns A timezone\-naive datetime object .TP .B Return type \fI\%datetime.datetime\fP .UNINDENT .sp .EX >>> parse_iso_utc(\(aq2016\-04\-27T00:28:04.000Z\(aq) datetime.datetime(2016, 4, 27, 0, 28, 4) >>> parse_iso_utc(\(aq2016\-04\-27T00:28:04Z\(aq) datetime.datetime(2016, 4, 27, 0, 28, 4) >>> parse_iso_utc(\(aq2016\-04\-27T00:28:04X\(aq) Traceback (most recent call last): \&... ValueError: Not a valid ISO datetime in UTC: 2016\-04\-27T00:28:04X .EE .UNINDENT .INDENT 0.0 .TP .B toil.lib.memoize.strict_bool(s) Variant of bool() that only accepts two possible string values. .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .SS \fI\%toil.lib.misc\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%get_public_ip\fP() T} T{ Get the IP that this machine uses to contact the internet. T} _ T{ \fI\%get_user_name\fP() T} T{ Get the current user name, or a suitable substitute string if the user name T} _ T{ \fI\%utc_now\fP() T} T{ Return a datetime in the UTC timezone corresponding to right now. T} _ T{ \fI\%unix_now_ms\fP() T} T{ Return the current time in milliseconds since the Unix epoch. T} _ T{ \fI\%slow_down\fP(seconds) T} T{ Toil jobs that have completed are not allowed to have taken 0 seconds, but T} _ T{ \fI\%printq\fP(msg, quiet) T} T{ T} _ T{ \fI\%truncExpBackoff\fP() T} T{ T} _ T{ \fI\%call_command\fP(cmd, *args[, input, timeout, useCLocale, ...]) T} T{ Simplified calling of external commands. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.misc.logger .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.get_public_ip() Get the IP that this machine uses to contact the internet. .sp If behind a NAT, this will still be this computer\(aqs IP, and not the router\(aqs. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.get_user_name() Get the current user name, or a suitable substitute string if the user name is not available. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.utc_now() Return a datetime in the UTC timezone corresponding to right now. .INDENT 7.0 .TP .B Return type \fI\%datetime.datetime\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.unix_now_ms() Return the current time in milliseconds since the Unix epoch. .INDENT 7.0 .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.slow_down(seconds) Toil jobs that have completed are not allowed to have taken 0 seconds, but Kubernetes timestamps round things to the nearest second. It is possible in some batch systems for a pod to have identical start and end timestamps. .sp This function takes a possibly 0 job length in seconds and enforces a minimum length to satisfy Toil. .INDENT 7.0 .TP .B Parameters \fBseconds\fP (\fI\%float\fP) \-\- Timestamp difference .TP .B Returns seconds, or a small positive number if seconds is 0 .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.printq(msg, quiet) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmsg\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBquiet\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.truncExpBackoff() .INDENT 7.0 .TP .B Return type Iterator[\fI\%float\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.lib.misc.CalledProcessErrorStderr(returncode, cmd, output=None, stderr=None) Bases: \fI\%subprocess.CalledProcessError\fP .sp Version of CalledProcessError that include stderr in the error message if it is set .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.misc.call_command(cmd, *args, input=None, timeout=None, useCLocale=True, env=None, quiet=False) Simplified calling of external commands. .sp If the process fails, CalledProcessErrorStderr is raised. .sp The captured stderr is always printed, regardless of if an exception occurs, so it can be logged. .sp Always logs the command at debug log level. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBquiet\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- If True, do not log the command output. If False (the default), do log the command output at debug log level. .IP \(bu 2 \fBuseCLocale\fP (\fI\%bool\fP) \-\- If True, C locale is forced, to prevent failures that can occur in some batch systems when using UTF\-8 locale. .IP \(bu 2 \fBcmd\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBargs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBinput\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBtimeout\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBenv\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns Command standard output, decoded as utf\-8. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .SS \fI\%toil.lib.objects\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%InnerClass\fP T} T{ Note that this is EXPERIMENTAL code. T} _ .TE .INDENT 0.0 .TP .B class toil.lib.objects.InnerClass(inner_class) Note that this is EXPERIMENTAL code. .sp A nested class (the inner class) decorated with this will have an additional attribute called \(aqouter\(aq referencing the instance of the nesting class (the outer class) that was used to create the inner class. The outer instance does not need to be passed to the inner class\(aqs constructor, it will be set magically. Shamelessly stolen from .sp \fI\%http://stackoverflow.com/questions/2278426/inner\-classes\-how\-can\-i\-get\-the\-outer\-class\-object\-at\-construction\-time#answer\-2278595\fP\&. .sp with names made more descriptive (I hope) and added caching of the BoundInner classes. .sp Caveat: Within the inner class, self.__class__ will not be the inner class but a dynamically created subclass thereof. It\(aqs name will be the same as that of the inner class, but its __module__ will be different. There will be one such dynamic subclass per inner class and instance of outer class, if that outer class instance created any instances of inner the class. .sp .EX >>> class Outer(object): \&... def new_inner(self): \&... # self is an instance of the outer class \&... inner = self.Inner() \&... # the inner instance\(aqs \(aqouter\(aq attribute is set to the outer instance \&... assert inner.outer is self \&... return inner \&... @InnerClass \&... class Inner(object): \&... def get_outer(self): \&... return self.outer \&... @classmethod \&... def new_inner(cls): \&... return cls() >>> o = Outer() >>> i = o.new_inner() >>> i bound to .EE .sp .EX >>> i.get_outer() .EE .sp Now with inheritance for both inner and outer: .sp .EX >>> class DerivedOuter(Outer): \&... def new_inner(self): \&... return self.DerivedInner() \&... @InnerClass \&... class DerivedInner(Outer.Inner): \&... def get_outer(self): \&... assert super( DerivedOuter.DerivedInner, self ).get_outer() == self.outer \&... return self.outer >>> derived_outer = DerivedOuter() >>> derived_inner = derived_outer.new_inner() >>> derived_inner bound to .EE .sp .EX >>> derived_inner.get_outer() .EE .sp Test a static references: >>> Outer.Inner # doctest: +ELLIPSIS >>> DerivedOuter.Inner # doctest: +ELLIPSIS >>> DerivedOuter.DerivedInner #doctest: +ELLIPSIS .sp Can\(aqt decorate top\-level classes. Unfortunately, this is detected when the instance is created, not when the class is defined. >>> @InnerClass \&... class Foo(object): \&... pass >>> Foo() Traceback (most recent call last): \&... RuntimeError: Inner classes must be nested in another class. .sp All inner instances should refer to a single outer instance: >>> o = Outer() >>> o.new_inner().outer == o == o.new_inner().outer True .sp All inner instances should be of the same class ... >>> o.new_inner().__class__ == o.new_inner().__class__ True .sp \&... but that class isn\(aqt the inner class ... >>> o.new_inner().__class__ != Outer.Inner True .sp \&... but a subclass of the inner class. >>> isinstance( o.new_inner(), Outer.Inner ) True .sp Static and class methods, e.g. should work, too .sp .EX >>> o.Inner.new_inner().outer == o True .EE .INDENT 7.0 .TP .B __get__(instance, owner) .UNINDENT .INDENT 7.0 .TP .B __call__(**kwargs) .UNINDENT .UNINDENT .SS \fI\%toil.lib.resources\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%get_total_cpu_time_and_memory_usage\fP() T} T{ Gives the total cpu time of itself and all its children, and the maximum RSS memory usage of T} _ T{ \fI\%get_total_cpu_time\fP() T} T{ Gives the total cpu time, including the children. T} _ T{ \fI\%glob\fP(glob_pattern, directoryname) T} T{ Walks through a directory and its subdirectories looking for files matching T} _ .TE .INDENT 0.0 .TP .B toil.lib.resources.get_total_cpu_time_and_memory_usage() Gives the total cpu time of itself and all its children, and the maximum RSS memory usage of itself and its single largest child (in kibibytes). .INDENT 7.0 .TP .B Return type Tuple[\fI\%float\fP, \fI\%int\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.resources.get_total_cpu_time() Gives the total cpu time, including the children. .INDENT 7.0 .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.resources.glob(glob_pattern, directoryname) Walks through a directory and its subdirectories looking for files matching the glob_pattern and returns a list=[]. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdirectoryname\fP (\fI\%str\fP) \-\- Any accessible folder name on the filesystem. .IP \(bu 2 \fBglob_pattern\fP (\fI\%str\fP) \-\- A string like \fB*.txt\fP, which would find all text files. .UNINDENT .TP .B Returns A list=[] of absolute filepaths matching the glob pattern. .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .SS \fI\%toil.lib.retry\fP .sp This file holds the retry() decorator function and RetryCondition object. .sp retry() can be used to decorate any function based on the list of errors one wishes to retry on. .sp This list of errors can contain normal Exception objects, and/or RetryCondition objects wrapping Exceptions to include additional conditions. .sp For example, retrying on a one Exception (HTTPError): .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[HTTPError]) def update_my_wallpaper(): return get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp Or: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[HTTPError, ValueError]) def update_my_wallpaper(): return get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp The examples above will retry for the default interval on any errors specified the \(dqerrors=\(dq arg list. .sp To retry on specifically 500/502/503/504 errors, you could specify an ErrorCondition object instead, for example: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[ ErrorCondition( error=HTTPError, error_codes=[500, 502, 503, 504] )]) def update_my_wallpaper(): return requests.get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp To retry on specifically errors containing the phrase \(dqNotFound\(dq: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[ ErrorCondition( error=HTTPError, error_message_must_include=\(dqNotFound\(dq )]) def update_my_wallpaper(): return requests.get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp To retry on all HTTPError errors EXCEPT an HTTPError containing the phrase \(dqNotFound\(dq: .INDENT 0.0 .INDENT 3.5 .sp .EX from requests import get from requests.exceptions import HTTPError @retry(errors=[ HTTPError, ErrorCondition( error=HTTPError, error_message_must_include=\(dqNotFound\(dq, retry_on_this_condition=False )]) def update_my_wallpaper(): return requests.get(\(aqhttps://www.deviantart.com/\(aq) .EE .UNINDENT .UNINDENT .sp To retry on boto3\(aqs specific status errors, an example of the implementation is: .INDENT 0.0 .INDENT 3.5 .sp .EX import boto3 from botocore.exceptions import ClientError @retry(errors=[ ErrorCondition( error=ClientError, boto_error_codes=[\(dqBucketNotFound\(dq] )]) def boto_bucket(bucket_name): boto_session = boto3.session.Session() s3_resource = boto_session.resource(\(aqs3\(aq) return s3_resource.Bucket(bucket_name) .EE .UNINDENT .UNINDENT .sp Any combination of these will also work, provided the codes are matched to the correct exceptions. A ValueError will not return a 404, for example. .sp The retry function as a decorator should make retrying functions easier and clearer It also encourages smaller independent functions, as opposed to lumping many different things that may need to be retried on different conditions in the same function. .sp The ErrorCondition object tries to take some of the heavy lifting of writing specific retry conditions and boil it down to an API that covers all common use\-cases without the user having to write any new bespoke functions. .sp Use\-cases covered currently: .INDENT 0.0 .IP 1. 3 Retrying on a normal error, like a KeyError. .IP 2. 3 Retrying on HTTP error codes (use ErrorCondition). .IP 3. 3 Retrying on boto 3\(aqs specific status errors, like \(dqBucketNotFound\(dq (use ErrorCondition). .IP 4. 3 Retrying when an error message contains a certain phrase (use ErrorCondition). .IP 5. 3 Explicitly NOT retrying on a condition (use ErrorCondition). .UNINDENT .sp If new functionality is needed, it\(aqs currently best practice in Toil to add functionality to the ErrorCondition itself rather than making a new custom retry method. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ErrorCondition\fP T} T{ A wrapper describing an error condition. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%retry\fP([intervals, infinite_retries, errors, ...]) T} T{ Retry a function if it fails with any Exception defined in \(dqerrors\(dq. T} _ T{ \fI\%return_status_code\fP(e) T} T{ T} _ T{ \fI\%get_error_code\fP(e) T} T{ Get the error code name from a Boto 2 or 3 error, or compatible types. T} _ T{ \fI\%get_error_message\fP(e) T} T{ Get the error message string from a Boto 2 or 3 error, or compatible types. T} _ T{ \fI\%get_error_status\fP(e) T} T{ Get the HTTP status code from a compatible source. T} _ T{ \fI\%get_error_body\fP(e) T} T{ Get the body from a Boto 2 or 3 error, or compatible types. T} _ T{ \fI\%meets_error_message_condition\fP(e, error_message) T} T{ T} _ T{ \fI\%meets_error_code_condition\fP(e, error_codes) T} T{ These are expected to be normal HTTP error codes, like 404 or 500. T} _ T{ \fI\%meets_boto_error_code_condition\fP(e, boto_error_codes) T} T{ These are expected to be AWS\(aqs custom error aliases, like \(aqBucketNotFound\(aq or \(aqAccessDenied\(aq. T} _ T{ \fI\%error_meets_conditions\fP(e, error_conditions) T} T{ T} _ T{ \fI\%old_retry\fP([delays, timeout, predicate]) T} T{ Deprecated. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%SUPPORTED_HTTP_ERRORS\fP T} T{ T} _ T{ \fI\%kubernetes\fP T} T{ T} _ T{ \fI\%botocore\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%DEFAULT_DELAYS\fP T} T{ T} _ T{ \fI\%DEFAULT_TIMEOUT\fP T} T{ T} _ T{ \fI\%retry_flaky_test\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.retry.SUPPORTED_HTTP_ERRORS .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.kubernetes .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.botocore .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.logger .UNINDENT .INDENT 0.0 .TP .B class toil.lib.retry.ErrorCondition(error=None, error_codes=None, boto_error_codes=None, error_message_must_include=None, retry_on_this_condition=True) A wrapper describing an error condition. .sp ErrorCondition events may be used to define errors in more detail to determine whether to retry. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBerror\fP (\fIOptional\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBerror_codes\fP (\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBboto_error_codes\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBerror_message_must_include\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBretry_on_this_condition\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None) Retry a function if it fails with any Exception defined in \(dqerrors\(dq. .sp Does so every x seconds, where x is defined by a list of numbers (ints or floats) in \(dqintervals\(dq. Also accepts ErrorCondition events for more detailed retry attempts. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBintervals\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI]\fP) \-\- A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back\-off before failing: 1s, 1s, 2s, 4s, 8s, 16s .IP \(bu 2 \fBinfinite_retries\fP (\fI\%bool\fP) \-\- If this is True, reset the intervals when they run out. Defaults to: False. .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fISequence\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%ErrorCondition\fP\fI, \fP\fIType\fP\fI[\fP\fI\%Exception\fP\fI]\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .sp A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: \- Exception (required) \- Error codes that must match to be retried (optional; defaults to not checking) \- A string that must be in the error message to be retried (optional; defaults to not checking) \- A bool that can be set to False to always error on this condition. .sp If not specified, this will default to a generic Exception. .IP \(bu 2 \fBlog_message\fP (\fIOptional\fP\fI[\fP\fITuple\fP\fI[\fP\fICallable\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Optional tuple of (\(dqlog/print function()\(dq, \(dqmessage string\(dq) that will precede each attempt. .IP \(bu 2 \fBprepare\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fICallable\fP\fI]\fP\fI]\fP) \-\- Optional list of functions to call, with the function\(aqs arguments, between retries, to reset state. .UNINDENT .TP .B Returns The result of the wrapped function or raise. .TP .B Return type Callable[[Any], Any] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.return_status_code(e) .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.get_error_code(e) Get the error code name from a Boto 2 or 3 error, or compatible types. .sp Returns empty string for other errors. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.get_error_message(e) Get the error message string from a Boto 2 or 3 error, or compatible types. .sp Note that error message conditions also check more than this; this function does not fall back to the traceback for incompatible types. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.get_error_status(e) Get the HTTP status code from a compatible source. .sp Such as a Boto 2 or 3 error, kubernetes.client.rest.ApiException, http.client.HTTPException, urllib3.exceptions.HTTPError, requests.exceptions.HTTPError, urllib.error.HTTPError, or compatible type .sp Returns 0 from other errors. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.get_error_body(e) Get the body from a Boto 2 or 3 error, or compatible types. .sp Returns the code and message if the error does not have a body. .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.meets_error_message_condition(e, error_message) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBe\fP (\fI\%Exception\fP) \-\- .IP \(bu 2 \fBerror_message\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.meets_error_code_condition(e, error_codes) These are expected to be normal HTTP error codes, like 404 or 500. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBe\fP (\fI\%Exception\fP) \-\- .IP \(bu 2 \fBerror_codes\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%int\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.meets_boto_error_code_condition(e, boto_error_codes) These are expected to be AWS\(aqs custom error aliases, like \(aqBucketNotFound\(aq or \(aqAccessDenied\(aq. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBe\fP (\fI\%Exception\fP) \-\- .IP \(bu 2 \fBboto_error_codes\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.error_meets_conditions(e, error_conditions) .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.DEFAULT_DELAYS = (0, 1, 1, 4, 16, 64) .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.DEFAULT_TIMEOUT = 300 .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.old_retry(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=lambda e: ...) Deprecated. .sp Retry an operation while the failure matches a given predicate and until a given timeout expires, waiting a given amount of time in between attempts. This function is a generator that yields contextmanagers. See doctests below for example usage. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdelays\fP (\fIIterable\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- an interable yielding the time in seconds to wait before each retried attempt, the last element of the iterable will be repeated. .IP \(bu 2 \fBtimeout\fP (\fI\%float\fP) \-\- a overall timeout that should not be exceeded for all attempts together. This is a best\-effort mechanism only and it won\(aqt abort an ongoing attempt, even if the timeout expires during that attempt. .IP \(bu 2 \fBpredicate\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%Exception\fP\fI]\fP\fI,\fP\fI\%bool\fP\fI]\fP) \-\- a unary callable returning True if another attempt should be made to recover from the given exception. The default value for this parameter will prevent any retries! .UNINDENT .TP .B Returns a generator yielding context managers, one per attempt .TP .B Return type Iterator .UNINDENT .sp Retry for a limited amount of time: .sp .EX >>> true = lambda _:True >>> false = lambda _:False >>> i = 0 >>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ): \&... with attempt: \&... i += 1 \&... raise RuntimeError(\(aqfoo\(aq) Traceback (most recent call last): \&... RuntimeError: foo >>> i > 1 True .EE .sp If timeout is 0, do exactly one attempt: .sp .EX >>> i = 0 >>> for attempt in old_retry( timeout=0 ): \&... with attempt: \&... i += 1 \&... raise RuntimeError( \(aqfoo\(aq ) Traceback (most recent call last): \&... RuntimeError: foo >>> i 1 .EE .sp Don\(aqt retry on success: .sp .EX >>> i = 0 >>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ): \&... with attempt: \&... i += 1 >>> i 1 .EE .sp Don\(aqt retry on unless predicate returns True: .sp .EX >>> i = 0 >>> for attempt in old_retry( delays=[0], timeout=.1, predicate=false): \&... with attempt: \&... i += 1 \&... raise RuntimeError( \(aqfoo\(aq ) Traceback (most recent call last): \&... RuntimeError: foo >>> i 1 .EE .UNINDENT .INDENT 0.0 .TP .B toil.lib.retry.retry_flaky_test .UNINDENT .SS \fI\%toil.lib.threading\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ExceptionalThread\fP T} T{ A thread whose join() method re\-raises exceptions raised during run(). While join() is T} _ T{ \fI\%LastProcessStandingArena\fP T} T{ Class that lets a bunch of processes detect and elect a last process T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%cpu_count\fP() T} T{ Get the rounded\-up integer number of whole CPUs available. T} _ T{ \fI\%collect_process_name_garbage\fP() T} T{ Delete all the process names that point to files that don\(aqt exist anymore T} _ T{ \fI\%destroy_all_process_names\fP() T} T{ Delete all our process name files because our process is going away. T} _ T{ \fI\%get_process_name\fP(base_dir) T} T{ Return the name of the current process. Like a PID but visible between T} _ T{ \fI\%process_name_exists\fP(base_dir, name) T} T{ Return true if the process named by the given name (from process_name) exists, and false otherwise. T} _ T{ \fI\%global_mutex\fP(base_dir, mutex) T} T{ Context manager that locks a mutex. The mutex is identified by the given T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%current_process_name_lock\fP T} T{ T} _ T{ \fI\%current_process_name_for\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.lib.threading.logger .UNINDENT .INDENT 0.0 .TP .B class toil.lib.threading.ExceptionalThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None) Bases: \fI\%threading.Thread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B exc_info .UNINDENT .INDENT 7.0 .TP .B run() Method representing the thread\(aqs activity. .sp You may override this method in a subclass. The standard run() method invokes the callable object passed to the object\(aqs constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tryRun() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B join(*args, **kwargs) Wait until the thread terminates. .sp This blocks the calling thread until the thread whose join() method is called terminates \-\- either normally or through an unhandled exception or until the optional timeout occurs. .sp When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened \-\- if the thread is still alive, the join() call timed out. .sp When the timeout argument is not present or None, the operation will block until the thread terminates. .sp A thread can be join()ed many times. .sp join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.cpu_count() Get the rounded\-up integer number of whole CPUs available. .sp Counts hyperthreads as CPUs. .sp Uses the system\(aqs actual CPU count, or the current v1 cgroup\(aqs quota per period, if the quota is set. .sp Ignores the cgroup\(aqs cpu shares value, because it\(aqs extremely difficult to interpret. See \fI\%https://github.com/kubernetes/kubernetes/issues/81021\fP\&. .sp Caches result for efficiency. .INDENT 7.0 .TP .B Returns Integer count of available CPUs, minimum 1. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.current_process_name_lock .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.current_process_name_for: Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.collect_process_name_garbage() Delete all the process names that point to files that don\(aqt exist anymore (because the work directory was temporary and got cleaned up). This is known to happen during the tests, which get their own temp directories. .sp Caller must hold current_process_name_lock. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.destroy_all_process_names() Delete all our process name files because our process is going away. .sp We let all our FDs get closed by the process death. .sp We assume there is nobody else using the system during exit to race with. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.get_process_name(base_dir) Return the name of the current process. Like a PID but visible between containers on what to Toil appears to be a node. .INDENT 7.0 .TP .B Parameters \fBbase_dir\fP (\fI\%str\fP) \-\- Base directory to work in. Defines the shared namespace. .TP .B Returns Process\(aqs assigned name .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.process_name_exists(base_dir, name) Return true if the process named by the given name (from process_name) exists, and false otherwise. .sp Can see across container boundaries using the given node workflow directory. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_dir\fP (\fI\%str\fP) \-\- Base directory to work in. Defines the shared namespace. .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- Process\(aqs name to poll .UNINDENT .TP .B Returns True if the named process is still alive, and False otherwise. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lib.threading.global_mutex(base_dir, mutex) Context manager that locks a mutex. The mutex is identified by the given name, and scoped to the given directory. Works across all containers that have access to the given diectory. Mutexes held by dead processes are automatically released. .sp Only works between processes, NOT between threads. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_dir\fP (\fI\%str\fP) \-\- Base directory to work in. Defines the shared namespace. .IP \(bu 2 \fBmutex\fP (\fI\%str\fP) \-\- Mutex to lock. Must be a permissible path component. .UNINDENT .TP .B Return type Iterator[None] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.lib.threading.LastProcessStandingArena(base_dir, name) Class that lets a bunch of processes detect and elect a last process standing. .sp Process enter and leave (sometimes due to sudden existence failure). We guarantee that the last process to leave, if it leaves properly, will get a chance to do some cleanup. If new processes try to enter during the cleanup, they will be delayed until after the cleanup has happened and the previous \(dqlast\(dq process has finished leaving. .sp The user is responsible for making sure you always leave if you enter! Consider using a try/finally; this class is not a context manager. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B enter() This process is entering the arena. If cleanup is in progress, blocks until it is finished. .sp You may not enter the arena again before leaving it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B leave() This process is leaving the arena. If this process happens to be the last process standing, yields something, with other processes blocked from joining the arena until the loop body completes and the process has finished leaving. Otherwise, does not yield anything. .sp Should be used in a loop: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .TP .B for _ in arena.leave(): # If we get here, we were the last process. Do the cleanup pass .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B Return type Iterator[\fI\%bool\fP] .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.lib.throttle\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%LocalThrottle\fP T} T{ A thread\-safe rate limiter that throttles each thread independently. Can be used as a T} _ T{ \fI\%throttle\fP T} T{ A context manager for ensuring that the execution of its body takes at least a given amount T} _ .TE .INDENT 0.0 .TP .B class toil.lib.throttle.LocalThrottle(min_interval) A thread\-safe rate limiter that throttles each thread independently. Can be used as a function or method decorator or as a simple object, via its .throttle() method. .sp The use as a decorator is deprecated in favor of throttle(). .INDENT 7.0 .TP .B Parameters \fBmin_interval\fP (\fI\%int\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B throttle(wait=True) If the wait parameter is True, this method returns True after suspending the current thread as necessary to ensure that no less than the configured minimum interval has passed since the last invocation of this method in the current thread returned True. .sp If the wait parameter is False, this method immediatly returns True (if at least the configured minimum interval has passed since the last time this method returned True in the current thread) or False otherwise. .INDENT 7.0 .TP .B Parameters \fBwait\fP (\fI\%bool\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __call__(function) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.lib.throttle.throttle(min_interval) A context manager for ensuring that the execution of its body takes at least a given amount of time, sleeping if necessary. It is a simpler version of LocalThrottle if used as a decorator. .sp Ensures that body takes at least the given amount of time. .sp .EX >>> start = time.time() >>> with throttle(1): \&... pass >>> 1 <= time.time() \- start <= 1.1 True .EE .sp Ditto when used as a decorator. .sp .EX >>> @throttle(1) \&... def f(): \&... pass >>> start = time.time() >>> f() >>> 1 <= time.time() \- start <= 1.1 True .EE .sp If the body takes longer by itself, don\(aqt throttle. .sp .EX >>> start = time.time() >>> with throttle(1): \&... time.sleep(2) >>> 2 <= time.time() \- start <= 2.1 True .EE .sp Ditto when used as a decorator. .sp .EX >>> @throttle(1) \&... def f(): \&... time.sleep(2) >>> start = time.time() >>> f() >>> 2 <= time.time() \- start <= 2.1 True .EE .sp If an exception occurs, don\(aqt throttle. .sp .EX >>> start = time.time() >>> try: \&... with throttle(1): \&... raise ValueError(\(aqfoo\(aq) \&... except ValueError: \&... end = time.time() \&... raise Traceback (most recent call last): \&... ValueError: foo >>> 0 <= end \- start <= 0.1 True .EE .sp Ditto when used as a decorator. .sp .EX >>> @throttle(1) \&... def f(): \&... raise ValueError(\(aqfoo\(aq) >>> start = time.time() >>> try: \&... f() \&... except ValueError: \&... end = time.time() \&... raise Traceback (most recent call last): \&... ValueError: foo >>> 0 <= end \- start <= 0.1 True .EE .INDENT 7.0 .TP .B Parameters \fBmin_interval\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __enter__() .UNINDENT .INDENT 7.0 .TP .B __exit__(exc_type, exc_val, exc_tb) .UNINDENT .INDENT 7.0 .TP .B __call__(function) .UNINDENT .UNINDENT .SS \fI\%toil.options\fP .SS Submodules .SS \fI\%toil.options.common\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%parse_set_env\fP(l) T} T{ Parse a list of strings of the form \(dqNAME=VALUE\(dq or just \(dqNAME\(dq into a dictionary. T} _ T{ \fI\%parse_str_list\fP(s) T} T{ T} _ T{ \fI\%parse_int_list\fP(s) T} T{ T} _ T{ \fI\%iC\fP(min_value[, max_value]) T} T{ Returns a function that checks if a given int is in the given half\-open interval. T} _ T{ \fI\%fC\fP(minValue[, maxValue]) T} T{ Returns a function that checks if a given float is in the given half\-open interval. T} _ T{ \fI\%parse_accelerator_list\fP(specs) T} T{ Parse a string description of one or more accelerator requirements. T} _ T{ \fI\%parseBool\fP(val) T} T{ T} _ T{ \fI\%make_open_interval_action\fP(min[, max]) T} T{ Returns an argparse action class to check if the input is within the given half\-open interval. T} _ T{ \fI\%parse_jobstore\fP(jobstore_uri) T} T{ Turn the jobstore string into it\(aqs corresponding URI T} _ T{ \fI\%add_base_toil_options\fP(parser[, jobstore_as_flag, cwl]) T} T{ Add base Toil command line options to the parser. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%defaultTargetTime\fP T} T{ T} _ T{ \fI\%SYS_MAX_SIZE\fP T} T{ T} _ T{ \fI\%JOBSTORE_HELP\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.options.common.logger .UNINDENT .INDENT 0.0 .TP .B toil.options.common.defaultTargetTime = 1800 .UNINDENT .INDENT 0.0 .TP .B toil.options.common.SYS_MAX_SIZE = 9223372036854775807 .UNINDENT .INDENT 0.0 .TP .B toil.options.common.parse_set_env(l) Parse a list of strings of the form \(dqNAME=VALUE\(dq or just \(dqNAME\(dq into a dictionary. .sp Strings of the latter from will result in dictionary entries whose value is None. .sp .EX >>> parse_set_env([]) {} >>> parse_set_env([\(aqa\(aq]) {\(aqa\(aq: None} >>> parse_set_env([\(aqa=\(aq]) {\(aqa\(aq: \(aq\(aq} >>> parse_set_env([\(aqa=b\(aq]) {\(aqa\(aq: \(aqb\(aq} >>> parse_set_env([\(aqa=a\(aq, \(aqa=b\(aq]) {\(aqa\(aq: \(aqb\(aq} >>> parse_set_env([\(aqa=b\(aq, \(aqc=d\(aq]) {\(aqa\(aq: \(aqb\(aq, \(aqc\(aq: \(aqd\(aq} >>> parse_set_env([\(aqa=b=c\(aq]) {\(aqa\(aq: \(aqb=c\(aq} >>> parse_set_env([\(aq\(aq]) Traceback (most recent call last): \&... ValueError: Empty name >>> parse_set_env([\(aq=1\(aq]) Traceback (most recent call last): \&... ValueError: Empty name .EE .INDENT 7.0 .TP .B Parameters \fBl\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, Optional[\fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.parse_str_list(s) .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.parse_int_list(s) .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- .TP .B Return type List[\fI\%int\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.iC(min_value, max_value=None) Returns a function that checks if a given int is in the given half\-open interval. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmin_value\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBmax_value\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Callable[[\fI\%int\fP], \fI\%bool\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.fC(minValue, maxValue=None) Returns a function that checks if a given float is in the given half\-open interval. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBminValue\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBmaxValue\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Callable[[\fI\%float\fP], \fI\%bool\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.parse_accelerator_list(specs) Parse a string description of one or more accelerator requirements. .INDENT 7.0 .TP .B Parameters \fBspecs\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type List[\fI\%toil.job.AcceleratorRequirement\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.parseBool(val) .INDENT 7.0 .TP .B Parameters \fBval\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.make_open_interval_action(min, max=None) Returns an argparse action class to check if the input is within the given half\-open interval. ex: Provided value to argparse must be within the interval [min, max) Types of min and max must be the same (max may be None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmin\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- float/int .IP \(bu 2 \fBmax\fP (\fIOptional\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP\fI]\fP) \-\- optional float/int .UNINDENT .TP .B Returns argparse action class .TP .B Return type Type[\fI\%argparse.Action\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.parse_jobstore(jobstore_uri) Turn the jobstore string into it\(aqs corresponding URI ex: /path/to/jobstore \-> \fI\%file:/path/to/jobstore\fP .sp If the jobstore string already is a URI, return the jobstore: aws:/path/to/jobstore \-> aws:/path/to/jobstore :param jobstore_uri: string of the jobstore :return: URI of the jobstore .INDENT 7.0 .TP .B Parameters \fBjobstore_uri\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.JOBSTORE_HELP = Multiline\-String .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .INDENT 3.5 .sp .EX \(dq\(dq\(dqThe location of the job store for the workflow. A job store holds persistent information about the jobs, stats, and files in a workflow. If the workflow is run with a distributed batch system, the job store must be accessible by all worker nodes. Depending on the desired job store implementation, the location should be formatted according to one of the following schemes: file: where points to a directory on the file systen aws:: where is the name of an AWS region like us\-west\-2 and will be prepended to the names of any top\-level AWS resources in use by job store, e.g. S3 buckets. google:: TODO: explain For backwards compatibility, you may also specify ./foo (equivalent to file:./foo or just file:foo) or /bar (equivalent to file:/bar).\(dq\(dq\(dq .EE .UNINDENT .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.options.common.add_base_toil_options(parser, jobstore_as_flag=False, cwl=False) Add base Toil command line options to the parser. :param parser: Argument parser to add options to :param jobstore_as_flag: make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. :param cwl: whether CWL should be included or not .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- .IP \(bu 2 \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBcwl\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.options.cwl\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%add_cwl_options\fP(parser[, suppress]) T} T{ Add CWL options to the parser. This only adds nonpositional CWL arguments. T} _ .TE .INDENT 0.0 .TP .B toil.options.cwl.add_cwl_options(parser, suppress=True) Add CWL options to the parser. This only adds nonpositional CWL arguments. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- Parser to add options to .IP \(bu 2 \fBsuppress\fP (\fI\%bool\fP) \-\- Suppress help output .UNINDENT .TP .B Returns None .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.options.wdl\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%add_wdl_options\fP(parser[, suppress]) T} T{ Add WDL options to a parser. This only adds nonpositional WDL arguments T} _ .TE .INDENT 0.0 .TP .B toil.options.wdl.add_wdl_options(parser, suppress=True) Add WDL options to a parser. This only adds nonpositional WDL arguments :param parser: Parser to add options to :param suppress: Suppress help output :return: None .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- .IP \(bu 2 \fBsuppress\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.provisioners\fP .SS Subpackages .SS \fI\%toil.provisioners.aws\fP .SS Submodules .SS \fI\%toil.provisioners.aws.awsProvisioner\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AWSProvisioner\fP T} T{ Interface for provisioning worker nodes to use in a Toil cluster. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%awsRetryPredicate\fP(e) T} T{ T} _ T{ \fI\%expectedShutdownErrors\fP(e) T} T{ Matches errors that we expect to occur during shutdown, and which indicate T} _ T{ \fI\%awsRetry\fP(f) T} T{ This decorator retries the wrapped function if aws throws unexpected errors T} _ T{ \fI\%awsFilterImpairedNodes\fP(nodes, ec2) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.aws.awsProvisioner.logger .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.awsProvisioner.awsRetryPredicate(e) .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.awsProvisioner.expectedShutdownErrors(e) Matches errors that we expect to occur during shutdown, and which indicate that we need to wait or try again. .sp Should \fInot\fP match any errors which indicate that an operation is impossible or unnecessary (such as errors resulting from a thing not existing to be deleted). .INDENT 7.0 .TP .B Parameters \fBe\fP (\fI\%Exception\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.awsProvisioner.awsRetry(f) This decorator retries the wrapped function if aws throws unexpected errors errors. It should wrap any function that makes use of boto .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.awsProvisioner.awsFilterImpairedNodes(nodes, ec2) .UNINDENT .INDENT 0.0 .TP .B exception toil.provisioners.aws.awsProvisioner.InvalidClusterStateException Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.aws.awsProvisioner.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey) Bases: \fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP .sp Interface for provisioning worker nodes to use in a Toil cluster. .INDENT 7.0 .TP .B supportedClusterTypes() Get all the cluster types that this provisioner implementation supports. .UNINDENT .INDENT 7.0 .TP .B createClusterSettings() Create a new set of cluster settings for a cluster to be deployed into AWS. .UNINDENT .INDENT 7.0 .TP .B readClusterSettings() Reads the cluster settings from the instance metadata, which assumes the instance is the leader. .UNINDENT .INDENT 7.0 .TP .B launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs) Starts a single leader node and populates this class with the leader\(aqs metadata. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBleaderNodeType\fP (\fI\%str\fP) \-\- An AWS instance type, like \(dqt2.medium\(dq, for example. .IP \(bu 2 \fBleaderStorage\fP (\fI\%int\fP) \-\- An integer number of gigabytes to provide the leader instance with. .IP \(bu 2 \fBowner\fP (\fI\%str\fP) \-\- Resources will be tagged with this owner string. .IP \(bu 2 \fBkeyName\fP (\fI\%str\fP) \-\- The ssh key to use to access the leader node. .IP \(bu 2 \fBbotoPath\fP (\fI\%str\fP) \-\- The path to the boto credentials directory. .IP \(bu 2 \fBuserTags\fP (\fIOptional\fP\fI[\fP\fI\%dict\fP\fI]\fP) \-\- Optionally provided user tags to put on the cluster. .IP \(bu 2 \fBvpcSubnet\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Optionally specify the VPC subnet for the leader. .IP \(bu 2 \fBawsEc2ProfileArn\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Optionally provide the profile ARN. .IP \(bu 2 \fBawsEc2ExtraSecurityGroupIds\fP (\fIOptional\fP\fI[\fP\fI\%list\fP\fI]\fP) \-\- Optionally provide additional security group IDs. .UNINDENT .TP .B Returns None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B toil_service_env_options() Set AWS tags in user docker container .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getKubernetesAutoscalerSetupCommands(values) Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS. .INDENT 7.0 .TP .B Parameters \fBvalues\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getKubernetesCloudProvider() Use the \(dqaws\(dq Kubernetes cloud provider when setting up Kubernetes. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getNodeShape(instance_type, preemptible=False) Get the Shape for the given instance type (e.g. \(aqt2.medium\(aq). .INDENT 7.0 .TP .B Parameters \fBinstance_type\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%toil.provisioners.abstractProvisioner.Shape\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static retryPredicate(e) Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff. .INDENT 7.0 .TP .B Parameters \fBe\fP \-\- exception raised during execution of setNodeCount .TP .B Returns boolean indicating whether the exception e should be retried .UNINDENT .UNINDENT .INDENT 7.0 .TP .B destroyCluster() Terminate instances and delete the profile and security group. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B terminateNodes(nodes) Terminate the nodes represented by given Node objects .INDENT 7.0 .TP .B Parameters \fBnodes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.node.Node\fP\fI]\fP) \-\- list of Node objects .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addNodes(nodeTypes, numNodes, preemptible, spotBid=None) Used to add worker nodes to the cluster .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnumNodes\fP \-\- The number of nodes to add .IP \(bu 2 \fBpreemptible\fP \-\- whether or not the nodes will be preemptible .IP \(bu 2 \fBspotBid\fP \-\- The bid for preemptible nodes if applicable (this can be set in config, also). .IP \(bu 2 \fBnodeTypes\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns number of nodes successfully added .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None) Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load. .sp Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can\(aqt have managed nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBminNodes\fP \-\- The minimum number of nodes to scale to .IP \(bu 2 \fBmaxNodes\fP \-\- The maximum number of nodes to scale to .IP \(bu 2 \fBpreemptible\fP \-\- whether or not the nodes will be preemptible .IP \(bu 2 \fBspotBid\fP \-\- The bid for preemptible nodes if applicable (this can be set in config, also). .IP \(bu 2 \fBnodeTypes\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getProvisionedWorkers(instance_type=None, preemptible=None) Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- Boolean value to restrict to preemptible nodes or non\-preemptible nodes .IP \(bu 2 \fBinstance_type\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns list of Node objects .TP .B Return type List[\fI\%toil.provisioners.node.Node\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLeader(wait=False) Get the leader for the cluster as a Toil Node object. .INDENT 7.0 .TP .B Return type \fI\%toil.provisioners.node.Node\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B full_policy(resource) Produce a dict describing the JSON form of a full\-access\-granting AWS IAM policy for the service with the given name (e.g. \(aqs3\(aq). .INDENT 7.0 .TP .B Parameters \fBresource\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%dict\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B kubernetes_policy() Get the Kubernetes policy grants not provided by the full grants on EC2 and IAM. See <\fI\%https://github.com/DataBiosphere/toil/wiki/Manual\-Autoscaling\-Kubernetes\-Setup#leader\-policy\fP> and <\fI\%https://github.com/DataBiosphere/toil/wiki/Manual\-Autoscaling\-Kubernetes\-Setup#worker\-policy\fP>. .sp These are mostly needed to support Kubernetes\(aq AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler\(aqs AWS integration. .sp Some of these are really only needed on the leader. .INDENT 7.0 .TP .B Return type \fI\%dict\fP .UNINDENT .UNINDENT .UNINDENT .SS Package Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%get_aws_zone_from_boto\fP() T} T{ Get the AWS zone from the Boto config file, if it is configured and the T} _ T{ \fI\%get_aws_zone_from_environment\fP() T} T{ Get the AWS zone from TOIL_AWS_ZONE if set. T} _ T{ \fI\%get_aws_zone_from_environment_region\fP() T} T{ Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set. T} _ T{ \fI\%get_aws_zone_from_metadata\fP() T} T{ Get the AWS zone from instance metadata, if on EC2 and the boto module is T} _ T{ \fI\%get_aws_zone_from_spot_market\fP(spotBid, nodeType, ...) T} T{ If a spot bid, node type, and Boto2 EC2 connection are specified, picks a T} _ T{ \fI\%get_best_aws_zone\fP([spotBid, nodeType, boto2_ec2, ...]) T} T{ Get the right AWS zone to use. T} _ T{ \fI\%choose_spot_zone\fP(zones, bid, spot_history) T} T{ Returns the zone to put the spot request based on, in order of priority: T} _ T{ \fI\%optimize_spot_bid\fP(boto2_ec2, instance_type, spot_bid, ...) T} T{ Check whether the bid is in line with history and makes an effort to place T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%ZoneTuple\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.aws.get_aws_zone_from_boto() Get the AWS zone from the Boto config file, if it is configured and the boto module is available. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.get_aws_zone_from_environment() Get the AWS zone from TOIL_AWS_ZONE if set. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.get_aws_zone_from_environment_region() Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.get_aws_zone_from_metadata() Get the AWS zone from instance metadata, if on EC2 and the boto module is available. Otherwise, gets the AWS zone from ECS task metadata, if on ECS. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.logger .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.ZoneTuple .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.get_aws_zone_from_spot_market(spotBid, nodeType, boto2_ec2, zone_options) If a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all. .sp In this case, zone_options can be used to restrict to a subset of the zones in the region. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBspotBid\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBnodeType\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBboto2_ec2\fP (\fIOptional\fP\fI[\fP\fIboto.connection.AWSAuthConnection\fP\fI]\fP) \-\- .IP \(bu 2 \fBzone_options\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.get_best_aws_zone(spotBid=None, nodeType=None, boto2_ec2=None, zone_options=None) Get the right AWS zone to use. .sp Reports the TOIL_AWS_ZONE environment variable if set. .sp Otherwise, if we are running on EC2 or ECS, reports the zone we are running in. .sp Otherwise, if a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all. .sp In this case, zone_options can be used to restrict to a subset of the zones in the region. .sp Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region. .sp Finally, if a default region is configured in Boto 2, chooses a zone in that region. .sp Returns None if no method can produce a zone to use. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBspotBid\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBnodeType\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBboto2_ec2\fP (\fIOptional\fP\fI[\fP\fIboto.connection.AWSAuthConnection\fP\fI]\fP) \-\- .IP \(bu 2 \fBzone_options\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.choose_spot_zone(zones, bid, spot_history) Returns the zone to put the spot request based on, in order of priority: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP 1. 3 zones with prices currently under the bid .IP 2. 3 zones with the most stable price .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B Returns the name of the selected zone .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBzones\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBbid\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBspot_history\fP (\fIList\fP\fI[\fP\fIboto.ec2.spotpricehistory.SpotPriceHistory\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .sp .EX >>> from collections import namedtuple >>> FauxHistory = namedtuple(\(aqFauxHistory\(aq, [\(aqprice\(aq, \(aqavailability_zone\(aq]) >>> zones = [\(aqus\-west\-2a\(aq, \(aqus\-west\-2b\(aq] >>> spot_history = [FauxHistory(0.1, \(aqus\-west\-2a\(aq), FauxHistory(0.2, \(aqus\-west\-2a\(aq), FauxHistory(0.3, \(aqus\-west\-2b\(aq), FauxHistory(0.6, \(aqus\-west\-2b\(aq)] >>> choose_spot_zone(zones, 0.15, spot_history) \(aqus\-west\-2a\(aq .EE .sp .EX >>> spot_history=[FauxHistory(0.3, \(aqus\-west\-2a\(aq), FauxHistory(0.2, \(aqus\-west\-2a\(aq), FauxHistory(0.1, \(aqus\-west\-2b\(aq), FauxHistory(0.6, \(aqus\-west\-2b\(aq)] >>> choose_spot_zone(zones, 0.15, spot_history) \(aqus\-west\-2b\(aq .EE .sp .EX >>> spot_history=[FauxHistory(0.1, \(aqus\-west\-2a\(aq), FauxHistory(0.7, \(aqus\-west\-2a\(aq), FauxHistory(0.1, \(aqus\-west\-2b\(aq), FauxHistory(0.6, \(aqus\-west\-2b\(aq)] >>> choose_spot_zone(zones, 0.15, spot_history) \(aqus\-west\-2b\(aq .EE .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.aws.optimize_spot_bid(boto2_ec2, instance_type, spot_bid, zone_options) Check whether the bid is in line with history and makes an effort to place the instance in a sensible zone. .INDENT 7.0 .TP .B Parameters \fBzone_options\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The collection of allowed zones to consider, within the region associated with the Boto2 connection. .UNINDENT .UNINDENT .SS Submodules .SS \fI\%toil.provisioners.abstractProvisioner\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Shape\fP T} T{ Represents a job or a node\(aqs \(dqshape\(dq, in terms of the dimensions of memory, cores, disk and T} _ T{ \fI\%AbstractProvisioner\fP T} T{ Interface for provisioning worker nodes to use in a Toil cluster. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%a_short_time\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.abstractProvisioner.a_short_time = 5 .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.abstractProvisioner.logger .UNINDENT .INDENT 0.0 .TP .B exception toil.provisioners.abstractProvisioner.ManagedNodesNotSupportedException Bases: \fI\%RuntimeError\fP .sp Raised when attempting to add managed nodes (which autoscale up and down by themselves, without the provisioner doing the work) to a provisioner that does not support them. .sp Polling with this and try/except is the Right Way to check if managed nodes are available from a provisioner. .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.abstractProvisioner.Shape(wallTime, memory, cores, disk, preemptible) Represents a job or a node\(aqs \(dqshape\(dq, in terms of the dimensions of memory, cores, disk and wall\-time allocation. .sp The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs? .sp The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwallTime\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBmemory\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBcores\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __eq__(other) Return self==value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B greater_than(other) .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __gt__(other) Return self>value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __hash__() Return hash(self). .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.abstractProvisioner.AbstractProvisioner(clusterName=None, clusterType=\(aqmesos\(aq, zone=None, nodeStorage=50, nodeStorageOverrides=None) Bases: \fI\%abc.ABC\fP .sp Interface for provisioning worker nodes to use in a Toil cluster. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBclusterName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBclusterType\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBzone\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBnodeStorage\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBnodeStorageOverrides\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class InstanceConfiguration Allows defining the initial setup for an instance and then turning it into an Ignition configuration for instance user data. .INDENT 7.0 .TP .B addFile(path, filesystem=\(aqroot\(aq, mode=\(aq0755\(aq, contents=\(aq\(aq, append=False) Make a file on the instance with the given filesystem, mode, and contents. .sp See the storage.files section: \fI\%https://github.com/kinvolk/ignition/blob/flatcar\-master/doc/configuration\-v2_2.md\fP .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfilesystem\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBmode\fP (\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBcontents\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBappend\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addUnit(name, enabled=True, contents=\(aq\(aq) Make a systemd unit on the instance with the given name (including \&.service), and content. Units will be enabled by default. .INDENT 7.0 .TP .B Unit logs can be investigated with: systemctl status whatever.service .TP .B or: journalctl \-xe .UNINDENT .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBenabled\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBcontents\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addSSHRSAKey(keyData) Authorize the given bare, encoded RSA key (without \(dqssh\-rsa\(dq). .INDENT 7.0 .TP .B Parameters \fBkeyData\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B toIgnitionConfig() Return an Ignition configuration describing the desired config. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B LEADER_HOME_DIR = \(aq/root/\(aq .UNINDENT .INDENT 7.0 .TP .B cloud: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B abstract supportedClusterTypes() Get all the cluster types that this provisioner implementation supports. .INDENT 7.0 .TP .B Return type Set[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract createClusterSettings() Initialize class for a new cluster, to be deployed, when running outside the cloud. .UNINDENT .INDENT 7.0 .TP .B abstract readClusterSettings() Initialize class from an existing cluster. This method assumes that the instance we are running on is the leader. .sp Implementations must call _setLeaderWorkerAuthentication(). .UNINDENT .INDENT 7.0 .TP .B setAutoscaledNodeTypes(nodeTypes) Set node types, shapes and spot bids for Toil\-managed autoscaling. :param nodeTypes: A list of node types, as parsed with parse_node_types. .INDENT 7.0 .TP .B Parameters \fBnodeTypes\fP (\fIList\fP\fI[\fP\fITuple\fP\fI[\fP\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasAutoscaledNodeTypes() Check if node types have been configured on the provisioner (via setAutoscaledNodeTypes). .INDENT 7.0 .TP .B Returns True if node types are configured for autoscaling, and false otherwise. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getAutoscaledInstanceShapes() Get all the node shapes and their named instance types that the Toil autoscaler should manage. .INDENT 7.0 .TP .B Return type Dict[\fI\%Shape\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static retryPredicate(e) Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff. .INDENT 7.0 .TP .B Parameters \fBe\fP \-\- exception raised during execution of setNodeCount .TP .B Returns boolean indicating whether the exception e should be retried .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract launchCluster(*args, **kwargs) Initialize a cluster and create a leader node. .sp Implementations must call _setLeaderWorkerAuthentication() with the leader so that workers can be launched. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBleaderNodeType\fP \-\- The leader instance. .IP \(bu 2 \fBleaderStorage\fP \-\- The amount of disk to allocate to the leader in gigabytes. .IP \(bu 2 \fBowner\fP \-\- Tag identifying the owner of the instances. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract addNodes(nodeTypes, numNodes, preemptible, spotBid=None) Used to add worker nodes to the cluster .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnumNodes\fP (\fI\%int\fP) \-\- The number of nodes to add .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- whether or not the nodes will be preemptible .IP \(bu 2 \fBspotBid\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- The bid for preemptible nodes if applicable (this can be set in config, also). .IP \(bu 2 \fBnodeTypes\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns number of nodes successfully added .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None) Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load. .sp Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can\(aqt have managed nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBminNodes\fP \-\- The minimum number of nodes to scale to .IP \(bu 2 \fBmaxNodes\fP \-\- The maximum number of nodes to scale to .IP \(bu 2 \fBpreemptible\fP \-\- whether or not the nodes will be preemptible .IP \(bu 2 \fBspotBid\fP \-\- The bid for preemptible nodes if applicable (this can be set in config, also). .IP \(bu 2 \fBnodeTypes\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract terminateNodes(nodes) Terminate the nodes represented by given Node objects .INDENT 7.0 .TP .B Parameters \fBnodes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.node.Node\fP\fI]\fP) \-\- list of Node objects .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getLeader() .INDENT 7.0 .TP .B Returns The leader node. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getProvisionedWorkers(instance_type=None, preemptible=None) Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- Boolean value to restrict to preemptible nodes or non\-preemptible nodes .IP \(bu 2 \fBinstance_type\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns list of Node objects .TP .B Return type List[\fI\%toil.provisioners.node.Node\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getNodeShape(instance_type, preemptible=False) The shape of a preemptible or non\-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals. .INDENT 7.0 .TP .B Parameters \fBinstance_type\fP (\fI\%str\fP) \-\- Instance type name to return the shape of. .TP .B Return type \fI\%Shape\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract destroyCluster() Terminates all nodes in the specified cluster and cleans up all resources associated with the cluster. :param clusterName: identifier of the cluster to terminate. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getBaseInstanceConfiguration() Get the base configuration for both leader and worker instances for all cluster types. .INDENT 7.0 .TP .B Return type \fI\%InstanceConfiguration\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addVolumesService(config) Add a service to prepare and mount local scratch volumes. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%InstanceConfiguration\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addNodeExporterService(config) Add the node exporter service for Prometheus to an instance configuration. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%InstanceConfiguration\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B toil_service_env_options() .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B add_toil_service(config, role, keyPath=None, preemptible=False) Add the Toil leader or worker service to an instance configuration. .sp Will run Mesos master or agent as appropriate in Mesos clusters. For Kubernetes clusters, will just sleep to provide a place to shell into on the leader, and shouldn\(aqt run on the worker. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrole\fP (\fI\%str\fP) \-\- Should be \(aqleader\(aq or \(aqworker\(aq. Will not work for \(aqworker\(aq until leader credentials have been collected. .IP \(bu 2 \fBkeyPath\fP (\fI\%str\fP) \-\- path on the node to a server\-side encryption key that will be added to the node after it starts. The service will wait until the key is present before starting. .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- Whether a worker should identify itself as preemptible or not to the scheduler. .IP \(bu 2 \fBconfig\fP (\fI\%InstanceConfiguration\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getKubernetesValues(architecture=\(aqamd64\(aq) Returns a dict of Kubernetes component versions and paths for formatting into Kubernetes\-related templates. .INDENT 7.0 .TP .B Parameters \fBarchitecture\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addKubernetesServices(config, architecture=\(aqamd64\(aq) Add installing Kubernetes and Kubeadm and setting up the Kubelet to run when configured to an instance configuration. The same process applies to leaders and workers. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%InstanceConfiguration\fP) \-\- .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract getKubernetesAutoscalerSetupCommands(values) Return Bash commands that set up the Kubernetes cluster autoscaler for provisioning from the environment supported by this provisioner. .sp Should only be implemented if Kubernetes clusters are supported. .INDENT 7.0 .TP .B Parameters \fBvalues\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Contains definitions of cluster variables, like AUTOSCALER_VERSION and CLUSTER_NAME. .TP .B Returns Bash snippet .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getKubernetesCloudProvider() Return the Kubernetes cloud provider (for example, \(aqaws\(aq), to pass to the kubelets in a Kubernetes cluster provisioned using this provisioner. .sp Defaults to None if not overridden, in which case no cloud provider integration will be used. .INDENT 7.0 .TP .B Returns Cloud provider name, or None .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addKubernetesLeader(config) Add services to configure as a Kubernetes leader, if Kubernetes is already set to be installed. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%InstanceConfiguration\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addKubernetesWorker(config, authVars, preemptible=False) Add services to configure as a Kubernetes worker, if Kubernetes is already set to be installed. .sp Authenticate back to the leader using the JOIN_TOKEN, JOIN_CERT_HASH, and JOIN_ENDPOINT set in the given authentication data dict. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%InstanceConfiguration\fP) \-\- The configuration to add services to .IP \(bu 2 \fBauthVars\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Dict with authentication info .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- Whether the worker should be labeled as preemptible or not .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.provisioners.clusterScaler\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BinPackedFit\fP T} T{ If jobShapes is a set of tasks with run requirements (mem/disk/cpu), and nodeShapes is a sorted T} _ T{ \fI\%NodeReservation\fP T} T{ The amount of resources that we expect to be available on a given node at each point in time. T} _ T{ \fI\%ClusterScaler\fP T} T{ T} _ T{ \fI\%ScalerThread\fP T} T{ A thread that automatically scales the number of either preemptible or non\-preemptible worker T} _ T{ \fI\%ClusterStats\fP T} T{ T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%adjustEndingReservationForJob\fP(reservation, jobShape, ...) T} T{ Add a job to an ending reservation that ends at wallTime. T} _ T{ \fI\%split\fP(nodeShape, jobShape, wallTime) T} T{ Partition a node allocation into two to fit the job. T} _ T{ \fI\%binPacking\fP(nodeShapes, jobShapes, goalTime) T} T{ Using the given node shape bins, pack the given job shapes into nodes to T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%EVICTION_THRESHOLD\fP T} T{ T} _ T{ \fI\%RESERVE_SMALL_LIMIT\fP T} T{ T} _ T{ \fI\%RESERVE_SMALL_AMOUNT\fP T} T{ T} _ T{ \fI\%RESERVE_BREAKPOINTS\fP T} T{ T} _ T{ \fI\%RESERVE_FRACTIONS\fP T} T{ T} _ T{ \fI\%OS_SIZE\fP T} T{ T} _ T{ \fI\%FailedConstraint\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.logger .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.EVICTION_THRESHOLD .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.RESERVE_SMALL_LIMIT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.RESERVE_SMALL_AMOUNT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.RESERVE_BREAKPOINTS: List[\fI\%int\fP | \fI\%float\fP] .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.RESERVE_FRACTIONS = [0.25, 0.2, 0.1, 0.06, 0.02] .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.OS_SIZE .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.FailedConstraint .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.clusterScaler.BinPackedFit(nodeShapes, targetTime=defaultTargetTime) If jobShapes is a set of tasks with run requirements (mem/disk/cpu), and nodeShapes is a sorted list of available computers to run these jobs on, this function attempts to return a dictionary representing the minimum set of computerNode computers needed to run the tasks in jobShapes. .sp Uses a first fit decreasing (FFD) bin packing like algorithm to calculate an approximate minimum number of nodes that will fit the given list of jobs. BinPackingFit assumes the ordered list, nodeShapes, is ordered for \(dqnode preference\(dq outside of BinPackingFit beforehand. So when virtually \(dqcreating\(dq nodes, the first node within nodeShapes that fits the job is the one that\(aqs added. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodeShapes\fP (\fI\%list\fP) \-\- The properties of an atomic node allocation, in terms of wall\-time, memory, cores, disk, and whether it is preemptible or not. .IP \(bu 2 \fBtargetTime\fP (\fI\%float\fP) \-\- The time before which all jobs should at least be started. .UNINDENT .TP .B Returns The minimum number of minimal node allocations estimated to be required to run all the jobs in jobShapes. .UNINDENT .INDENT 7.0 .TP .B nodeReservations: Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, List[\fI\%NodeReservation\fP]] .UNINDENT .INDENT 7.0 .TP .B binPack(jobShapes) Pack a list of jobShapes into the fewest nodes reasonable. .sp Can be run multiple times. .sp Returns any distinct Shapes that did not fit, mapping to reasons they did not fit. .INDENT 7.0 .TP .B Parameters \fBjobShapes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI]\fP) \-\- .TP .B Return type Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, List[FailedConstraint]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addJobShape(jobShape) Add the job to the first node reservation in which it will fit. (This is the bin\-packing aspect). .sp Returns the job shape again, and a list of failed constraints, if it did not fit. .INDENT 7.0 .TP .B Parameters \fBjobShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .TP .B Return type Optional[Tuple[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, List[FailedConstraint]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRequiredNodes() Return a dict from node shape to number of nodes required to run the packed jobs. .INDENT 7.0 .TP .B Return type Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, \fI\%int\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.clusterScaler.NodeReservation(shape) The amount of resources that we expect to be available on a given node at each point in time. .sp To represent the resources available in a reservation, we represent a reservation as a linked list of NodeReservations, each giving the resources free within a single timeslice. .INDENT 7.0 .TP .B Parameters \fBshape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_failed_constraints(job_shape) Check if a job shape\(aqs resource requirements will fit within this allocation. .sp If the job does \fInot\fP fit, returns the failing constraints: the resources that can\(aqt be accomodated, and the limits that were hit. .sp If the job \fIdoes\fP fit, returns an empty list. .sp Must always agree with fits()! This codepath is slower and used for diagnosis. .INDENT 7.0 .TP .B Parameters \fBjob_shape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .TP .B Return type List[FailedConstraint] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fits(jobShape) Check if a job shape\(aqs resource requirements will fit within this allocation. .INDENT 7.0 .TP .B Parameters \fBjobShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shapes() Get all time\-slice shapes, in order, from this reservation on. .INDENT 7.0 .TP .B Return type List[\fI\%toil.provisioners.abstractProvisioner.Shape\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B subtract(jobShape) Subtract the resources necessary to run a jobShape from the reservation. .INDENT 7.0 .TP .B Parameters \fBjobShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B attemptToAddJob(jobShape, nodeShape, targetTime) Attempt to pack a job into this reservation timeslice and/or the reservations after it. .sp jobShape is the Shape of the job requirements, nodeShape is the Shape of the node this is a reservation for, and targetTime is the maximum time to wait before starting this job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .IP \(bu 2 \fBnodeShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .IP \(bu 2 \fBtargetTime\fP (\fI\%float\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.adjustEndingReservationForJob(reservation, jobShape, wallTime) Add a job to an ending reservation that ends at wallTime. .sp (splitting the reservation if the job doesn\(aqt fill the entire timeslice) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBreservation\fP (\fI\%NodeReservation\fP) \-\- .IP \(bu 2 \fBjobShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .IP \(bu 2 \fBwallTime\fP (\fI\%float\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.split(nodeShape, jobShape, wallTime) Partition a node allocation into two to fit the job. .sp Returning the modified shape of the node and a new node reservation for the extra time that the job didn\(aqt fill. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodeShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .IP \(bu 2 \fBjobShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .IP \(bu 2 \fBwallTime\fP (\fI\%float\fP) \-\- .UNINDENT .TP .B Return type Tuple[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, \fI\%NodeReservation\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.clusterScaler.binPacking(nodeShapes, jobShapes, goalTime) Using the given node shape bins, pack the given job shapes into nodes to get them done in the given amount of time. .sp Returns a dict saying how many of each node will be needed, a dict from job shapes that could not fit to reasons why. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodeShapes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI]\fP) \-\- .IP \(bu 2 \fBjobShapes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI]\fP) \-\- .IP \(bu 2 \fBgoalTime\fP (\fI\%float\fP) \-\- .UNINDENT .TP .B Return type Tuple[Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, \fI\%int\fP], Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, List[FailedConstraint]]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.clusterScaler.ClusterScaler(provisioner, leader, config) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprovisioner\fP (\fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP) \-\- .IP \(bu 2 \fBleader\fP (\fI\%toil.leader.Leader\fP) \-\- .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getAverageRuntime(jobName, service=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBservice\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addCompletedJob(job, wallTime) Adds the shape of a completed job to the queue, allowing the scalar to use the last N completed jobs in factoring how many nodes are required in the cluster. :param toil.job.JobDescription job: The description of the completed job :param int wallTime: The wall\-time taken to complete the job in seconds. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBwallTime\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setStaticNodes(nodes, preemptible) Used to track statically provisioned nodes. This method must be called before any auto\-scaled nodes are provisioned. .sp These nodes are treated differently than auto\-scaled nodes in that they should not be automatically terminated. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.node.Node\fP\fI]\fP) \-\- list of Node objects .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getStaticNodes(preemptible) Returns nodes set in setStaticNodes(). .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP (\fI\%bool\fP) \-\- .TP .B Returns Statically provisioned nodes. .TP .B Return type Dict[\fI\%str\fP, \fI\%toil.provisioners.node.Node\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B smoothEstimate(nodeShape, estimatedNodeCount) Smooth out fluctuations in the estimate for this node compared to previous runs. .sp Returns an integer. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodeShape\fP (\fI\%toil.provisioners.abstractProvisioner.Shape\fP) \-\- .IP \(bu 2 \fBestimatedNodeCount\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getEstimatedNodeCounts(queuedJobShapes, currentNodeCounts) Given the resource requirements of queued jobs and the current size of the cluster. .sp Returns a dict mapping from nodeShape to the number of nodes we want in the cluster right now, and a dict from job shapes that are too big to run on any node to reasons why. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBqueuedJobShapes\fP (\fIList\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI]\fP) \-\- .IP \(bu 2 \fBcurrentNodeCounts\fP (\fIDict\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI, \fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Tuple[Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, \fI\%int\fP], Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, List[FailedConstraint]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B updateClusterSize(estimatedNodeCounts) Given the desired and current size of the cluster, attempts to launch/remove instances to get to the desired size. .sp Also attempts to remove ignored nodes that were marked for graceful removal. .sp Returns the new size of the cluster. .INDENT 7.0 .TP .B Parameters \fBestimatedNodeCounts\fP (\fIDict\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI, \fP\fI\%int\fP\fI]\fP) \-\- .TP .B Return type Dict[\fI\%toil.provisioners.abstractProvisioner.Shape\fP, \fI\%int\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setNodeCount(instance_type, numNodes, preemptible=False, force=False) Attempt to grow or shrink the number of preemptible or non\-preemptible worker nodes in the cluster to the given value, or as close a value as possible, and, after performing the necessary additions or removals of worker nodes, return the resulting number of preemptible or non\-preemptible nodes currently in the cluster. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinstance_type\fP (\fI\%str\fP) \-\- The instance type to add or remove. .IP \(bu 2 \fBnumNodes\fP (\fI\%int\fP) \-\- Desired size of the cluster .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- whether the added nodes will be preemptible, i.e. whether they may be removed spontaneously by the underlying platform at any time. .IP \(bu 2 \fBforce\fP (\fI\%bool\fP) \-\- If False, the provisioner is allowed to deviate from the given number of nodes. For example, when downsizing a cluster, a provisioner might leave nodes running if they have active jobs running on them. .UNINDENT .TP .B Returns the number of worker nodes in the cluster after making the necessary adjustments. This value should be, but is not guaranteed to be, close or equal to the \fInumNodes\fP argument. It represents the closest possible approximation of the actual cluster size at the time this method returns. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B filter_out_static_nodes(nodes, preemptible=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodes\fP (\fIDict\fP\fI[\fP\fI\%toil.provisioners.node.Node\fP\fI, \fP\fI\%toil.batchSystems.abstractBatchSystem.NodeInfo\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type List[Tuple[\fI\%toil.provisioners.node.Node\fP, \fI\%toil.batchSystems.abstractBatchSystem.NodeInfo\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getNodes(preemptible=None) Returns a dictionary mapping node identifiers of preemptible or non\-preemptible nodes to NodeInfo objects, one for each node. .sp This method is the definitive source on nodes in cluster, & is responsible for consolidating cluster state between the provisioner & batch system. .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP (\fI\%bool\fP) \-\- If True (False) only (non\-)preemptible nodes will be returned. If None, all nodes will be returned. .TP .B Return type Dict[\fI\%toil.provisioners.node.Node\fP, \fI\%toil.batchSystems.abstractBatchSystem.NodeInfo\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutDown() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.provisioners.clusterScaler.JobTooBigError(job=None, shape=None, constraints=None) Bases: \fI\%Exception\fP .sp Raised in the scaler thread when a job cannot fit in any available node type and is likely to lock up the workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fIOptional\fP\fI[\fP\fI\%toil.job.JobDescription\fP\fI]\fP) \-\- .IP \(bu 2 \fBshape\fP (\fIOptional\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI]\fP) \-\- .IP \(bu 2 \fBconstraints\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fIFailedConstraint\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Stringify the exception, including the message. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.clusterScaler.ScalerThread(provisioner, leader, config, stop_on_exception=False) Bases: \fI\%toil.lib.threading.ExceptionalThread\fP .sp A thread that automatically scales the number of either preemptible or non\-preemptible worker nodes according to the resource requirements of the queued jobs. .sp The scaling calculation is essentially as follows: start with 0 estimated worker nodes. For each queued job, check if we expect it can be scheduled into a worker node before a certain time (currently one hour). Otherwise, attempt to add a single new node of the smallest type that can fit that job. .sp At each scaling decision point a comparison between the current, C, and newly estimated number of nodes is made. If the absolute difference is less than beta * C then no change is made, else the size of the cluster is adapted. The beta factor is an inertia parameter that prevents continual fluctuations in the number of nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprovisioner\fP (\fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP) \-\- .IP \(bu 2 \fBleader\fP (\fI\%toil.leader.Leader\fP) \-\- .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBstop_on_exception\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Attempt to join any existing scaler threads that may have died or finished. .sp This insures any exceptions raised in the threads are propagated in a timely fashion. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Shutdown the cluster. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addCompletedJob(job, wallTime) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBwallTime\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tryRun() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.clusterScaler.ClusterStats(path, batchSystem, clusterName) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBbatchSystem\fP (\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP) \-\- .IP \(bu 2 \fBclusterName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutDownStats() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B startStats(preemptible) .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B checkStats() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.provisioners.gceProvisioner\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%GCEProvisioner\fP T} T{ Implements a Google Compute Engine Provisioner using libcloud. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.gceProvisioner.logger .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.gceProvisioner.GCEProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey) Bases: \fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP .sp Implements a Google Compute Engine Provisioner using libcloud. .INDENT 7.0 .TP .B NODE_BOTO_PATH = \(aq/root/.boto\(aq .UNINDENT .INDENT 7.0 .TP .B SOURCE_IMAGE = b\(aqprojects/kinvolk\-public/global/images/family/flatcar\-stable\(aq .UNINDENT .INDENT 7.0 .TP .B DEFAULT_TASK_COMPLETION_TIMEOUT = 180 .UNINDENT .INDENT 7.0 .TP .B supportedClusterTypes() Get all the cluster types that this provisioner implementation supports. .UNINDENT .INDENT 7.0 .TP .B createClusterSettings() Initialize class for a new cluster, to be deployed, when running outside the cloud. .UNINDENT .INDENT 7.0 .TP .B readClusterSettings() Read the cluster settings from the instance, which should be the leader. See \fI\%https://cloud.google.com/compute/docs/storing\-retrieving\-metadata\fP for details about reading the metadata. .UNINDENT .INDENT 7.0 .TP .B launchCluster(leaderNodeType, leaderStorage, owner, **kwargs) In addition to the parameters inherited from the abstractProvisioner, the Google launchCluster takes the following parameters: keyName: The key used to communicate with instances botoPath: Boto credentials for reading an AWS jobStore (optional). network: a network (optional) vpcSubnet: A subnet (optional). use_private_ip: even though a public ip exists, ignore it (optional) .UNINDENT .INDENT 7.0 .TP .B getNodeShape(instance_type, preemptible=False) The shape of a preemptible or non\-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals. .INDENT 7.0 .TP .B Parameters \fBinstance_type\fP (\fI\%str\fP) \-\- Instance type name to return the shape of. .TP .B Return type \fI\%toil.provisioners.abstractProvisioner.Shape\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static retryPredicate(e) Not used by GCE .UNINDENT .INDENT 7.0 .TP .B destroyCluster() Try a few times to terminate all of the instances in the group. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B terminateNodes(nodes) Terminate the nodes represented by given Node objects .INDENT 7.0 .TP .B Parameters \fBnodes\fP \-\- list of Node objects .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addNodes(nodeTypes, numNodes, preemptible, spotBid=None) Used to add worker nodes to the cluster .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnumNodes\fP \-\- The number of nodes to add .IP \(bu 2 \fBpreemptible\fP \-\- whether or not the nodes will be preemptible .IP \(bu 2 \fBspotBid\fP \-\- The bid for preemptible nodes if applicable (this can be set in config, also). .IP \(bu 2 \fBnodeTypes\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns number of nodes successfully added .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getProvisionedWorkers(instance_type=None, preemptible=None) Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- Boolean value to restrict to preemptible nodes or non\-preemptible nodes .IP \(bu 2 \fBinstance_type\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns list of Node objects .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLeader() .INDENT 7.0 .TP .B Returns The leader node. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B ex_create_multiple_nodes(base_name, size, image, number, location=None, ex_network=\(aqdefault\(aq, ex_subnetwork=None, ex_tags=None, ex_metadata=None, ignore_errors=True, use_existing_disk=True, poll_interval=2, external_ip=\(aqephemeral\(aq, ex_disk_type=\(aqpd\-standard\(aq, ex_disk_auto_delete=True, ex_service_accounts=None, timeout=DEFAULT_TASK_COMPLETION_TIMEOUT, description=None, ex_can_ip_forward=None, ex_disks_gce_struct=None, ex_nic_gce_struct=None, ex_on_host_maintenance=None, ex_automatic_restart=None, ex_image_family=None, ex_preemptible=None) Monkey patch to gce.py in libcloud to allow disk and images to be specified. Also changed name to a uuid below. The prefix \(aqwp\(aq identifies preemptible nodes and \(aqwn\(aq non\-preemptible nodes. .UNINDENT .UNINDENT .SS \fI\%toil.provisioners.node\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Node\fP T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%a_short_time\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.node.a_short_time = 5 .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.node.logger .UNINDENT .INDENT 0.0 .TP .B class toil.provisioners.node.Node(publicIP, privateIP, name, launchTime, nodeType, preemptible, tags=None, use_private_ip=None) .INDENT 7.0 .TP .B maxWaitTime .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .UNINDENT .INDENT 7.0 .TP .B __hash__() Return hash(self). .UNINDENT .INDENT 7.0 .TP .B remainingBillingInterval() If the node has a launch time, this function returns a floating point value between 0 and 1.0 representing how far we are into the current billing cycle for the given instance. If the return value is .25, we are one quarter into the billing cycle, with three quarters remaining before we will be charged again for that instance. .sp Assumes a billing cycle of one hour. .INDENT 7.0 .TP .B Returns Float from 0 \-> 1.0 representing percentage of pre\-paid time left in cycle. .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B waitForNode(role, keyName=\(aqcore\(aq) .UNINDENT .INDENT 7.0 .TP .B copySshKeys(keyName) Copy authorized_keys file to the core user from the keyName user. .UNINDENT .INDENT 7.0 .TP .B injectFile(fromFile, toFile, role) rysnc a file to the container with the given role .UNINDENT .INDENT 7.0 .TP .B extractFile(fromFile, toFile, role) rysnc a file from the container with the given role .UNINDENT .INDENT 7.0 .TP .B sshAppliance(*args, **kwargs) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP \-\- arguments to execute in the appliance .IP \(bu 2 \fBkwargs\fP \-\- tty=bool tells docker whether or not to create a TTY shell for interactive SSHing. The default value is False. Input=string is passed as input to the Popen call. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B sshInstance(*args, **kwargs) Run a command on the instance. Returns the binary output of the command. .UNINDENT .INDENT 7.0 .TP .B coreSSH(*args, **kwargs) If strict=False, strict host key checking will be temporarily disabled. This is provided as a convenience for internal/automated functions and ought to be set to True whenever feasible, or whenever the user is directly interacting with a resource (e.g. rsync\-cluster or ssh\-cluster). Assumed to be False by default. .sp kwargs: input, tty, appliance, collectStdout, sshOptions, strict .INDENT 7.0 .TP .B Parameters \fBinput\fP (\fI\%bytes\fP) \-\- UTF\-8 encoded input bytes to send to the command .UNINDENT .UNINDENT .INDENT 7.0 .TP .B coreRsync(args, applianceName=\(aqtoil_leader\(aq, **kwargs) .UNINDENT .UNINDENT .SS Package Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%cluster_factory\fP(provisioner[, clusterName, ...]) T} T{ Find and instantiate the appropriate provisioner instance to make clusters in the given cloud. T} _ T{ \fI\%add_provisioner_options\fP(parser) T} T{ T} _ T{ \fI\%parse_node_types\fP(node_type_specs) T} T{ Parse a specification for zero or more node types. T} _ T{ \fI\%check_valid_node_types\fP(provisioner, node_types) T} T{ Raises if an invalid nodeType is specified for aws or gce. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.provisioners.logger .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.cluster_factory(provisioner, clusterName=None, clusterType=\(aqmesos\(aq, zone=None, nodeStorage=50, nodeStorageOverrides=None, sseKey=None) Find and instantiate the appropriate provisioner instance to make clusters in the given cloud. .sp Raises ClusterTypeNotSupportedException if the given provisioner does not implement clusters of the given type. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprovisioner\fP (\fI\%str\fP) \-\- The cloud type of the cluster. .IP \(bu 2 \fBclusterName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The name of the cluster. .IP \(bu 2 \fBclusterType\fP (\fI\%str\fP) \-\- The type of cluster: \(aqmesos\(aq or \(aqkubernetes\(aq. .IP \(bu 2 \fBzone\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The cloud zone .IP \(bu 2 \fBnodeStorage\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBnodeStorageOverrides\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBsseKey\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns A cluster object for the the cloud type. .TP .B Return type Union[\fI\%aws.awsProvisioner.AWSProvisioner\fP, \fI\%gceProvisioner.GCEProvisioner\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.add_provisioner_options(parser) .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.parse_node_types(node_type_specs) Parse a specification for zero or more node types. .sp Takes a comma\-separated list of node types. Each node type is a slash\-separated list of at least one instance type name (like \(aqm5a.large\(aq for AWS), and an optional bid in dollars after a colon. .sp Raises ValueError if a node type cannot be parsed. .sp Inputs should look something like this: .sp .EX >>> parse_node_types(\(aqc5.4xlarge/c5a.4xlarge:0.42,t2.large\(aq) [({\(aqc5.4xlarge\(aq, \(aqc5a.4xlarge\(aq}, 0.42), ({\(aqt2.large\(aq}, None)] .EE .INDENT 7.0 .TP .B Parameters \fBnode_type_specs\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- A string defining node types .TP .B Returns a list of node types, where each type is the set of instance types, and the float bid, or None. .TP .B Return type List[Tuple[Set[\fI\%str\fP], Optional[\fI\%float\fP]]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.provisioners.check_valid_node_types(provisioner, node_types) Raises if an invalid nodeType is specified for aws or gce. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprovisioner\fP (\fI\%str\fP) \-\- \(aqaws\(aq or \(aqgce\(aq to specify which cloud provisioner used. .IP \(bu 2 \fBnode_types\fP (\fIList\fP\fI[\fP\fITuple\fP\fI[\fP\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP\fI]\fP\fI]\fP) \-\- A list of node types. Example: [({\(aqt2.micro\(aq}, None), ({\(aqt2.medium\(aq}, 0.5)] .UNINDENT .TP .B Returns Nothing. Raises if any instance type in the node type isn\(aqt real. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.provisioners.NoSuchClusterException(cluster_name) Bases: \fI\%Exception\fP .sp Indicates that the specified cluster does not exist. .UNINDENT .INDENT 0.0 .TP .B exception toil.provisioners.ClusterTypeNotSupportedException(provisioner_class, cluster_type) Bases: \fI\%Exception\fP .sp Indicates that a provisioner does not support a given cluster type. .UNINDENT .INDENT 0.0 .TP .B exception toil.provisioners.ClusterCombinationNotSupportedException(provisioner_class, cluster_type, architecture, reason=None) Bases: \fI\%Exception\fP .sp Indicates that a provisioner does not support making a given type of cluster with a given architecture. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprovisioner_class\fP (\fIType\fP) \-\- .IP \(bu 2 \fBcluster_type\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBarchitecture\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBreason\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.server\fP .SS Subpackages .SS \fI\%toil.server.api_spec\fP .SS \fI\%toil.server.cli\fP .SS Submodules .SS \fI\%toil.server.cli.wes_cwl_runner\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%WESClientWithWorkflowEngineParameters\fP T} T{ A modified version of the WESClient from the wes\-service package that T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%generate_attachment_path_names\fP(paths) T} T{ Take in a list of path names and return a list of names with the common path T} _ T{ \fI\%get_deps_from_cwltool\fP(cwl_file[, input_file]) T} T{ Return a list of dependencies of the given workflow from cwltool. T} _ T{ \fI\%submit_run\fP(client, cwl_file[, input_file, engine_options]) T} T{ Given a CWL file, its input files, and an optional list of engine options, T} _ T{ \fI\%poll_run\fP(client, run_id) T} T{ Return True if the given workflow run is in a finished state. T} _ T{ \fI\%print_logs_and_exit\fP(client, run_id) T} T{ Fetch the workflow logs from the WES server, print the results, then exit T} _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.logger .UNINDENT .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.generate_attachment_path_names(paths) Take in a list of path names and return a list of names with the common path name stripped out, while preserving the input order. This guarantees that there are no relative paths that traverse up. .sp For example, for the following CWL workflow where \(dqhello.yaml\(dq references a file \(dqmessage.txt\(dq, .INDENT 7.0 .INDENT 3.5 ~/toil/workflows/hello.cwl ~/toil/input_files/hello.yaml ~/toil/input_files/message.txt .UNINDENT .UNINDENT .INDENT 7.0 .TP .B This may be run with the command: toil\-wes\-cwl\-runner hello.cwl ../input_files/hello.yaml .UNINDENT .sp Where \(dqmessage.txt\(dq is resolved to \(dq../input_files/message.txt\(dq. .sp We\(aqd send the workflow file as \(dqworkflows/hello.cwl\(dq, and send the inputs as \(dqinput_files/hello.yaml\(dq and \(dqinput_files/message.txt\(dq. .INDENT 7.0 .TP .B Parameters \fBpaths\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- A list of absolute or relative path names. Relative paths are interpreted as relative to the current working directory. .TP .B Returns The common path name and a list of minimal path names. .TP .B Return type Tuple[\fI\%str\fP, List[\fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.cli.wes_cwl_runner.WESClientWithWorkflowEngineParameters(endpoint, auth=None) Bases: \fBwes_client.util.WESClient\fP .sp A modified version of the WESClient from the wes\-service package that includes workflow_engine_parameters support. .sp TODO: Propose a PR in wes\-service to include workflow_engine_params. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBendpoint\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBauth\fP (\fIOptional\fP\fI[\fP\fITuple\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_version(extension, workflow_file) Determines the version of a .py, .wdl, or .cwl file. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBextension\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBworkflow_file\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B parse_params(workflow_params_file) Parse the CWL input file into a dictionary to be attached to the body of the WES run request. .INDENT 7.0 .TP .B Parameters \fBworkflow_params_file\fP (\fI\%str\fP) \-\- The URL or path to the CWL input file. .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B modify_param_paths(base_dir, workflow_params) Modify the file paths in the input workflow parameters to be relative to base_dir. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_dir\fP (\fI\%str\fP) \-\- The base directory to make the file paths relative to. This should be the common ancestor of all attached files, which will become the root of the execution folder. .IP \(bu 2 \fBworkflow_params\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- A dict containing the workflow parameters. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B build_wes_request(workflow_file, workflow_params_file, attachments, workflow_engine_parameters=None) Build the workflow run request to submit to WES. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_file\fP (\fI\%str\fP) \-\- The path or URL to the CWL workflow document. Only \fI\%file://\fP URL supported at the moment. .IP \(bu 2 \fBworkflow_params_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The path or URL to the CWL input file. .IP \(bu 2 \fBattachments\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- A list of local paths to files that will be uploaded to the server. .IP \(bu 2 \fBworkflow_engine_parameters\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- A list of engine parameters to set along with this workflow run. .UNINDENT .TP .B Returns A dictionary of parameters as the body of the request, and an iterable for the pairs of filename and file contents to upload to the server. .TP .B Return type Tuple[Dict[\fI\%str\fP, \fI\%str\fP], Iterable[Tuple[\fI\%str\fP, Tuple[\fI\%str\fP, \fI\%io.BytesIO\fP]]]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run_with_engine_options(workflow_file, workflow_params_file, attachments, workflow_engine_parameters) Composes and sends a post request that signals the WES server to run a workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_file\fP (\fI\%str\fP) \-\- The path to the CWL workflow document. .IP \(bu 2 \fBworkflow_params_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The path to the CWL input file. .IP \(bu 2 \fBattachments\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- A list of local paths to files that will be uploaded to the server. .IP \(bu 2 \fBworkflow_engine_parameters\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- A list of engine parameters to set along with this workflow run. .UNINDENT .TP .B Returns The body of the post result as a dictionary. .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.get_deps_from_cwltool(cwl_file, input_file=None) Return a list of dependencies of the given workflow from cwltool. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcwl_file\fP (\fI\%str\fP) \-\- The CWL file. .IP \(bu 2 \fBinput_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Omit to get the dependencies from the CWL file. If set, this returns the dependencies from the input file. .UNINDENT .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.submit_run(client, cwl_file, input_file=None, engine_options=None) Given a CWL file, its input files, and an optional list of engine options, submit the CWL workflow to the WES server via the WES client. .sp This function also attempts to find the attachments from the CWL workflow and its input file, and attach them to the WES run request. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBclient\fP (\fI\%WESClientWithWorkflowEngineParameters\fP) \-\- The WES client. .IP \(bu 2 \fBcwl_file\fP (\fI\%str\fP) \-\- The path to the CWL workflow document. .IP \(bu 2 \fBinput_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The path to the CWL input file. .IP \(bu 2 \fBengine_options\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- A list of engine parameters to set along with this workflow run. .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.poll_run(client, run_id) Return True if the given workflow run is in a finished state. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBclient\fP (\fI\%WESClientWithWorkflowEngineParameters\fP) \-\- .IP \(bu 2 \fBrun_id\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.print_logs_and_exit(client, run_id) Fetch the workflow logs from the WES server, print the results, then exit the program with the same exit code as the workflow run. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBclient\fP (\fI\%WESClientWithWorkflowEngineParameters\fP) \-\- The WES client. .IP \(bu 2 \fBrun_id\fP (\fI\%str\fP) \-\- The run_id of the target workflow. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.cli.wes_cwl_runner.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.server.wes\fP .SS Submodules .SS \fI\%toil.server.wes.abstract_backend\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%WESBackend\fP T} T{ A class to represent a GA4GH Workflow Execution Service (WES) API backend. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%handle_errors\fP(func) T} T{ This decorator catches errors from the wrapped function and returns a JSON T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%TaskLog\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.wes.abstract_backend.logger .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.abstract_backend.TaskLog .UNINDENT .INDENT 0.0 .TP .B exception toil.server.wes.abstract_backend.VersionNotImplementedException(wf_type, version=None, supported_versions=None) Bases: \fI\%Exception\fP .sp Raised when the requested workflow version is not implemented. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwf_type\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBversion\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBsupported_versions\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.server.wes.abstract_backend.MalformedRequestException(message) Bases: \fI\%Exception\fP .sp Raised when the request is malformed. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.server.wes.abstract_backend.WorkflowNotFoundException Bases: \fI\%Exception\fP .sp Raised when the requested run ID is not found. .UNINDENT .INDENT 0.0 .TP .B exception toil.server.wes.abstract_backend.WorkflowConflictException(run_id) Bases: \fI\%Exception\fP .sp Raised when the requested workflow is not in the expected state. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.server.wes.abstract_backend.OperationForbidden(message) Bases: \fI\%Exception\fP .sp Raised when the request is forbidden. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.server.wes.abstract_backend.WorkflowExecutionException(message) Bases: \fI\%Exception\fP .sp Raised when an internal error occurred during the execution of the workflow. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.abstract_backend.handle_errors(func) This decorator catches errors from the wrapped function and returns a JSON formatted error message with the appropriate status code defined by the GA4GH WES spec. .INDENT 7.0 .TP .B Parameters \fBfunc\fP (\fICallable\fP\fI[\fP\fIEllipsis\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .TP .B Return type Callable[Ellipsis, Any] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.abstract_backend.WESBackend(options) A class to represent a GA4GH Workflow Execution Service (WES) API backend. Intended to be inherited. Subclasses should implement all abstract methods to handle user requests when they hit different endpoints. .INDENT 7.0 .TP .B Parameters \fBoptions\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B resolve_operation_id(operation_id) Map an operationId defined in the OpenAPI or swagger yaml file to a function. .INDENT 7.0 .TP .B Parameters \fBoperation_id\fP (\fI\%str\fP) \-\- The operation ID defined in the specification. .TP .B Returns A function that should be called when the given endpoint is reached. .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_service_info() Get information about the Workflow Execution Service. .sp GET /service\-info .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract list_runs(page_size=None, page_token=None) List the workflow runs. .sp GET /runs .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpage_size\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBpage_token\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract run_workflow() Run a workflow. This endpoint creates a new workflow run and returns a \fIRunId\fP to monitor its progress. .sp POST /runs .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_run_log(run_id) Get detailed info about a workflow run. .sp GET /runs/{run_id} .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract cancel_run(run_id) Cancel a running workflow. .sp POST /runs/{run_id}/cancel .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract get_run_status(run_id) Get quick status info about a workflow run, returning a simple result with the overall state of the workflow run. .sp GET /runs/{run_id}/status .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static log_for_run(run_id, message) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrun_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBmessage\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static secure_path(path) .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B collect_attachments(run_id, temp_dir) Collect attachments from the current request by staging uploaded files to temp_dir, and return the temp_dir and parsed body of the request. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrun_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The run ID for logging. .IP \(bu 2 \fBtemp_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- The directory where uploaded files should be staged. If None, a temporary directory is created. .UNINDENT .TP .B Return type Tuple[\fI\%str\fP, Dict[\fI\%str\fP, Any]] .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.server.wes.amazon_wes_utils\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%WorkflowPlan\fP T} T{ These functions pass around dicts of a certain type, with \fIdata\fP and \fIfiles\fP keys. T} _ T{ \fI\%DataDict\fP T} T{ Under \fIdata\fP, there can be: T} _ T{ \fI\%FilesDict\fP T} T{ Under \fIfiles\fP, there can be: T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%parse_workflow_zip_file\fP(file, workflow_type) T} T{ Processes a workflow zip bundle T} _ T{ \fI\%parse_workflow_manifest_file\fP(manifest_file) T} T{ Reads a MANIFEST.json file for a workflow zip bundle T} _ T{ \fI\%workflow_manifest_url_to_path\fP(url[, parent_dir]) T} T{ Interpret a possibly\-relative parsed URL, relative to the given parent directory. T} _ T{ \fI\%task_filter\fP(task, job_status) T} T{ AGC requires task names to be annotated with an AWS Batch job ID that they T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%NOTICE\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.wes.amazon_wes_utils.logger .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.amazon_wes_utils.NOTICE = Multiline\-String .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .INDENT 3.5 .sp .EX \(dq\(dq\(dq Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. \(dq\(dq\(dq .EE .UNINDENT .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.amazon_wes_utils.WorkflowPlan Bases: \fBTypedDict\fP .sp These functions pass around dicts of a certain type, with \fIdata\fP and \fIfiles\fP keys. .INDENT 7.0 .TP .B data: \fI\%DataDict\fP .UNINDENT .INDENT 7.0 .TP .B files: \fI\%FilesDict\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.amazon_wes_utils.DataDict Bases: \fBTypedDict\fP .sp Under \fIdata\fP, there can be: * \fIworkflowUrl\fP (required if no \fIworkflowSource\fP): URL to main workflow code. .INDENT 7.0 .TP .B workflowUrl: \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.amazon_wes_utils.FilesDict Bases: \fBTypedDict\fP .sp Under \fIfiles\fP, there can be: * \fIworkflowSource\fP (required if no \fIworkflowUrl\fP): Open binary\-mode file for the main workflow code. * \fIworkflowInputFiles\fP: List of open binary\-mode file for input files. Expected to be JSONs. * \fIworkflowOptions\fP: Open binary\-mode file for a JSON of options sent along with the workflow. * \fIworkflowDependencies\fP: Open binary\-mode file for the zip the workflow came in, if any. .INDENT 7.0 .TP .B workflowSource: IO[\fI\%bytes\fP] .UNINDENT .INDENT 7.0 .TP .B workflowInputFiles: List[IO[\fI\%bytes\fP]] .UNINDENT .INDENT 7.0 .TP .B workflowOptions: IO[\fI\%bytes\fP] .UNINDENT .INDENT 7.0 .TP .B workflowDependencies: IO[\fI\%bytes\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.amazon_wes_utils.parse_workflow_zip_file(file, workflow_type) Processes a workflow zip bundle .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile\fP (\fI\%str\fP) \-\- String or Path\-like path to a workflow.zip file .IP \(bu 2 \fBworkflow_type\fP (\fI\%str\fP) \-\- String, extension of workflow to expect (e.g. \(dqwdl\(dq) .UNINDENT .TP .B Return type dict of \fIdata\fP and \fIfiles\fP .UNINDENT .sp If the zip only contains a single file, that file is set as \fIworkflowSource\fP .sp If the zip contains multiple files with a MANIFEST.json file, the MANIFEST is used to determine appropriate \fIdata\fP and \fIfile\fP arguments. (See: parse_workflow_manifest_file()) .INDENT 7.0 .TP .B If the zip contains multiple files without a MANIFEST.json file: .INDENT 7.0 .IP \(bu 2 a \fImain\fP workflow file with an extension matching the workflow_type is expected and will be set as \fIworkflowSource\fP .IP \(bu 2 optionally, if \fIinputs*.json\fP files are found in the root level of the zip, they will be set as \fIworkflowInputs(_d)*\fP in the order they are found .IP \(bu 2 optionally, if an \fIoptions.json\fP file is found in the root level of the zip, it will be set as \fIworkflowOptions\fP .UNINDENT .UNINDENT .sp If the zip contains multiple files, the original zip is set as \fIworkflowDependencies\fP .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.amazon_wes_utils.parse_workflow_manifest_file(manifest_file) Reads a MANIFEST.json file for a workflow zip bundle .INDENT 7.0 .TP .B Parameters \fBmanifest_file\fP (\fI\%str\fP) \-\- String or Path\-like path to a MANIFEST.json file .TP .B Return type dict of \fIdata\fP and \fIfiles\fP .UNINDENT .sp MANIFEST.json is expected to be formatted like: .INDENT 7.0 .INDENT 3.5 .sp .EX { \(dqmainWorkflowURL\(dq: \(dqrelpath/to/workflow\(dq, \(dqinputFileURLs\(dq: [ \(dqrelpath/to/input\-file\-1\(dq, \(dqrelpath/to/input\-file\-2\(dq, \(dqrelpath/to/input\-file\-3\(dq ], \(dqoptionsFileURL\(dq: \(dqrelpath/to/option\-file\(dq } .EE .UNINDENT .UNINDENT .sp The \fImainWorkflowURL\fP property that provides a relative file path in the zip to a workflow file, which will be set as \fIworkflowSource\fP .sp The inputFileURLs property is optional and provides a list of relative file paths in the zip to input.json files. The list is assumed to be in the order the inputs should be applied \- e.g. higher list index is higher priority. If present, it will be used to set \fIworkflowInputs(_d)\fP arguments. .sp The optionsFileURL property is optional and provides a relative file path in the zip to an options.json file. If present, it will be used to set \fIworkflowOptions\fP\&. .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.amazon_wes_utils.workflow_manifest_url_to_path(url, parent_dir=None) Interpret a possibly\-relative parsed URL, relative to the given parent directory. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBurl\fP (\fI\%urllib.parse.ParseResult\fP) \-\- .IP \(bu 2 \fBparent_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.amazon_wes_utils.task_filter(task, job_status) AGC requires task names to be annotated with an AWS Batch job ID that they were run under. If it encounters an un\-annotated task name, it will crash. See <\fI\%https://github.com/aws/amazon\-genomics\-cli/issues/494\fP>. .sp This encodes the AWSBatchJobID annotation, from the AmazonBatchBatchSystem, into the task name of the given task, and returns the modified task. If no such annotation is available, the task is censored and None is returned. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask\fP (\fItoil.server.wes.abstract_backend.TaskLog\fP) \-\- .IP \(bu 2 \fBjob_status\fP (\fI\%toil.bus.JobStatus\fP) \-\- .UNINDENT .TP .B Return type Optional[toil.server.wes.abstract_backend.TaskLog] .UNINDENT .UNINDENT .SS \fI\%toil.server.wes.tasks\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilWorkflowRunner\fP T} T{ A class to represent a workflow runner to run the requested workflow. T} _ T{ \fI\%TaskRunner\fP T} T{ Abstraction over the Celery API. Runs our run_wes task and allows canceling it. T} _ T{ \fI\%MultiprocessingTaskRunner\fP T} T{ Version of TaskRunner that just runs tasks with Multiprocessing. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%run_wes_task\fP(base_scratch_dir, state_store_url, ...) T} T{ Run a requested workflow. T} _ T{ \fI\%cancel_run\fP(task_id) T} T{ Send a SIGTERM signal to the process that is running task_id. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%WAIT_FOR_DEATH_TIMEOUT\fP T} T{ T} _ T{ \fI\%run_wes\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.wes.tasks.logger .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.tasks.WAIT_FOR_DEATH_TIMEOUT = 20 .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.tasks.ToilWorkflowRunner(base_scratch_dir, state_store_url, workflow_id, request, engine_options) A class to represent a workflow runner to run the requested workflow. .sp Responsible for parsing the user request into a shell command, executing that command, and collecting the outputs of the resulting workflow run. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_scratch_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBstate_store_url\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBrequest\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBengine_options\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_scratch_file(filename, contents) Write a file to the scratch directory. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfilename\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcontents\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_state() .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_workflow(src_url) Fetch the workflow file from its source and write it to a destination file. .INDENT 7.0 .TP .B Parameters \fBsrc_url\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B sort_options(workflow_engine_parameters=None) Sort the command line arguments in the order that can be recognized by the workflow execution engine. .INDENT 7.0 .TP .B Parameters \fBworkflow_engine_parameters\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP\fI]\fP) \-\- User\-specified parameters for this particular workflow. Keys are command\-line options, and values are option arguments, or None for options that are flags. .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B initialize_run() Write workflow and input files and construct a list of shell commands to be executed. Return that list of shell commands that should be executed in order to complete this workflow run. .INDENT 7.0 .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B call_cmd(cmd, cwd) Calls a command with Popen. Writes stdout, stderr, and the command to separate files. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcmd\fP (\fIUnion\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcwd\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%subprocess.Popen\fP[\fI\%bytes\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run() Construct a command to run a the requested workflow with the options, run it, and deposit the outputs in the output directory. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_output_files() Fetch all the files that this workflow generated and output information about them to \fIoutputs.json\fP\&. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.tasks.run_wes_task(base_scratch_dir, state_store_url, workflow_id, request, engine_options) Run a requested workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_scratch_dir\fP (\fI\%str\fP) \-\- Directory where the workflow\(aqs scratch dir will live, under the workflow\(aqs ID. .IP \(bu 2 \fBstate_store_url\fP (\fI\%str\fP) \-\- URL/path at which the server and Celery task communicate about workflow state. .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- ID of the workflow run. .IP \(bu 2 \fBrequest\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBengine_options\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns the state of the workflow run. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.tasks.run_wes .UNINDENT .INDENT 0.0 .TP .B toil.server.wes.tasks.cancel_run(task_id) Send a SIGTERM signal to the process that is running task_id. .INDENT 7.0 .TP .B Parameters \fBtask_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.tasks.TaskRunner Abstraction over the Celery API. Runs our run_wes task and allows canceling it. .sp We can swap this out in the server to allow testing without Celery. .INDENT 7.0 .TP .B static run(args, task_id) Run the given task args with the given ID on Celery. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fITuple\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI, \fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP\fI, \fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBtask_id\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static cancel(task_id) Cancel the task with the given ID on Celery. .INDENT 7.0 .TP .B Parameters \fBtask_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static is_ok(task_id) Make sure that the task running system is working for the given task. If the task system has detected an internal failure, return False. .INDENT 7.0 .TP .B Parameters \fBtask_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.tasks.MultiprocessingTaskRunner Bases: \fI\%TaskRunner\fP .sp Version of TaskRunner that just runs tasks with Multiprocessing. .sp Can\(aqt use threading because there\(aqs no way to send a cancel signal or exception to a Python thread, if loops in the task (i.e. ToilWorkflowRunner) don\(aqt poll for it. .INDENT 7.0 .TP .B static set_up_and_run_task(output_path, args) Set up logging for the process into the given file and then call run_wes_task with the given arguments. .sp If the process finishes successfully, it will clean up the log, but if the process crashes, the caller must clean up the log. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBoutput_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBargs\fP (\fITuple\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI, \fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP\fI, \fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod run(args, task_id) Run the given task args with the given ID. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fITuple\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI, \fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP\fI, \fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBtask_id\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod cancel(task_id) Cancel the task with the given ID. .INDENT 7.0 .TP .B Parameters \fBtask_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod is_ok(task_id) Make sure that the task running system is working for the given task. If the task system has detected an internal failure, return False. .INDENT 7.0 .TP .B Parameters \fBtask_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.server.wes.toil_backend\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilWorkflow\fP T} T{ T} _ T{ \fI\%ToilBackend\fP T} T{ WES backend implemented for Toil to run CWL, WDL, or Toil workflows. This T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.wes.toil_backend.logger .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.toil_backend.ToilWorkflow(base_work_dir, state_store_url, run_id) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbase_work_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBstate_store_url\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBrun_id\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fetch_state(key: \fI\%str\fP, default: \fI\%str\fP) -> \fI\%str\fP .TP .B fetch_state(key: \fI\%str\fP, default: \fI\%None\fP = None) -> \fI\%str\fP | \fI\%None\fP Return the contents of the given key in the workflow\(aqs state store. If the key does not exist, the default value is returned. .UNINDENT .INDENT 7.0 .TP .B fetch_scratch(filename) Get a context manager for either a stream for the given file from the workflow\(aqs scratch directory, or None if it isn\(aqt there. .INDENT 7.0 .TP .B Parameters \fBfilename\fP (\fI\%str\fP) \-\- .TP .B Return type Generator[Optional[TextIO], None, None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B exists() Return True if the workflow run exists. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_state() Return the state of the current run. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_on_run(task_runner) Check to make sure nothing has gone wrong in the task runner for this workflow. If something has, log, and fail the workflow with an error. .INDENT 7.0 .TP .B Parameters \fBtask_runner\fP (\fIType\fP\fI[\fP\fI\%toil.server.wes.tasks.TaskRunner\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_up_run() Set up necessary directories for the run. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clean_up() Clean directory and files related to the run. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B queue_run(task_runner, request, options) This workflow should be ready to run. Hand this to the task system. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask_runner\fP (\fIType\fP\fI[\fP\fI\%toil.server.wes.tasks.TaskRunner\fP\fI]\fP) \-\- .IP \(bu 2 \fBrequest\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_output_files() Return a collection of output files that this workflow generated. .INDENT 7.0 .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_stdout_path() Return the path to the standard output log, relative to the run\(aqs scratch_dir, or None if it doesn\(aqt exist. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_stderr_path() Return the path to the standard output log, relative to the run\(aqs scratch_dir, or None if it doesn\(aqt exist. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_messages_path() Return the path to the bus message log, relative to the run\(aqs scratch_dir, or None if it doesn\(aqt exist. .INDENT 7.0 .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_task_logs(filter_function=None) Return all the task log objects for the individual tasks in the workflow. .sp Task names will be the job_type values from issued/completed/failed messages, with annotations from JobAnnotationMessage messages if available. .INDENT 7.0 .TP .B Parameters \fBfilter_function\fP (\fIOptional\fP\fI[\fP\fICallable\fP\fI[\fP\fI[\fP\fItoil.server.wes.abstract_backend.TaskLog\fP\fI, \fP\fI\%toil.bus.JobStatus\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fItoil.server.wes.abstract_backend.TaskLog\fP\fI]\fP\fI]\fP\fI]\fP) \-\- If set, will be called with each task log and its job annotations. Returns a modified copy of the task log to actually report, or None if the task log should be omitted. .TP .B Return type List[Dict[\fI\%str\fP, Union[\fI\%str\fP, \fI\%int\fP, None]]] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.wes.toil_backend.ToilBackend(work_dir, state_store, options, dest_bucket_base, bypass_celery=False, wes_dialect=\(aqstandard\(aq) Bases: \fI\%toil.server.wes.abstract_backend.WESBackend\fP .sp WES backend implemented for Toil to run CWL, WDL, or Toil workflows. This class is responsible for validating and executing submitted workflows. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwork_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBstate_store\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdest_bucket_base\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBbypass_celery\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBwes_dialect\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_runs() A generator of a list of run ids and their state. .INDENT 7.0 .TP .B Return type Generator[Tuple[\fI\%str\fP, \fI\%str\fP], None, None] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_state(run_id) Return the state of the workflow run with the given run ID. May raise an error if the workflow does not exist. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_service_info() Get information about the Workflow Execution Service. .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B list_runs(page_size=None, page_token=None) List the workflow runs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpage_size\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBpage_token\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run_workflow() Run a workflow. .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_run_log(run_id) Get detailed info about a workflow run. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B cancel_run(run_id) Cancel a running workflow. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_run_status(run_id) Get quick status info about a workflow run, returning a simple result with the overall state of the workflow run. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_stdout(run_id) Get the stdout of a workflow run as a static file. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_stderr(run_id) Get the stderr of a workflow run as a static file. .INDENT 7.0 .TP .B Parameters \fBrun_id\fP (\fI\%str\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_health() Return successfully if the server is healthy. .INDENT 7.0 .TP .B Return type werkzeug.wrappers.response.Response .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_homepage() Provide a sensible result for / other than 404. .INDENT 7.0 .TP .B Return type werkzeug.wrappers.response.Response .UNINDENT .UNINDENT .UNINDENT .SS Submodules .SS \fI\%toil.server.app\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%parser_with_server_options\fP() T} T{ T} _ T{ \fI\%create_app\fP(args) T} T{ Create a \(dqconnexion.FlaskApp\(dq instance with Toil server configurations. T} _ T{ \fI\%start_server\fP(args) T} T{ Start a Toil server. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.app.logger .UNINDENT .INDENT 0.0 .TP .B toil.server.app.parser_with_server_options() .INDENT 7.0 .TP .B Return type \fI\%argparse.ArgumentParser\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.app.create_app(args) Create a \(dqconnexion.FlaskApp\(dq instance with Toil server configurations. .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fI\%argparse.Namespace\fP) \-\- .TP .B Return type connexion.FlaskApp .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.app.start_server(args) Start a Toil server. .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fI\%argparse.Namespace\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.server.celery_app\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%create_celery_app\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%celery\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.celery_app.create_celery_app() .INDENT 7.0 .TP .B Return type celery.Celery .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.celery_app.celery .UNINDENT .SS \fI\%toil.server.utils\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%MemoryStateCache\fP T} T{ An in\-memory place to store workflow state. T} _ T{ \fI\%AbstractStateStore\fP T} T{ A place for the WES server to keep its state: the set of workflows that T} _ T{ \fI\%MemoryStateStore\fP T} T{ An in\-memory place to store workflow state, for testing. T} _ T{ \fI\%FileStateStore\fP T} T{ A place to store workflow state that uses a POSIX\-compatible file system. T} _ T{ \fI\%S3StateStore\fP T} T{ A place to store workflow state that uses an S3\-compatible object store. T} _ T{ \fI\%WorkflowStateStore\fP T} T{ Slice of a state store for the state of a particular workflow. T} _ T{ \fI\%WorkflowStateMachine\fP T} T{ Class for managing the WES workflow state machine. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%get_iso_time\fP() T} T{ Return the current time in ISO 8601 format. T} _ T{ \fI\%link_file\fP(src, dest) T} T{ Create a link to a file from src to dest. T} _ T{ \fI\%download_file_from_internet\fP(src, dest[, content_type]) T} T{ Download a file from the Internet and write it to dest. T} _ T{ \fI\%download_file_from_s3\fP(src, dest[, content_type]) T} T{ Download a file from Amazon S3 and write it to dest. T} _ T{ \fI\%get_file_class\fP(path) T} T{ Return the type of the file as a human readable string. T} _ T{ \fI\%safe_read_file\fP(file) T} T{ Safely read a file by acquiring a shared lock to prevent other processes T} _ T{ \fI\%safe_write_file\fP(file, s) T} T{ Safely write to a file by acquiring an exclusive lock to prevent other T} _ T{ \fI\%connect_to_state_store\fP(url) T} T{ Connect to a place to store state for workflows, defined by a URL. T} _ T{ \fI\%connect_to_workflow_state_store\fP(url, workflow_id) T} T{ Connect to a place to store state for the given workflow, in the state T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%HAVE_S3\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%state_store_cache\fP T} T{ T} _ T{ \fI\%TERMINAL_STATES\fP T} T{ T} _ T{ \fI\%MAX_CANCELING_SECONDS\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.server.utils.HAVE_S3 = True .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.logger .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.get_iso_time() Return the current time in ISO 8601 format. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.link_file(src, dest) Create a link to a file from src to dest. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdest\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.download_file_from_internet(src, dest, content_type=None) Download a file from the Internet and write it to dest. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdest\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcontent_type\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.download_file_from_s3(src, dest, content_type=None) Download a file from Amazon S3 and write it to dest. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsrc\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdest\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcontent_type\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.get_file_class(path) Return the type of the file as a human readable string. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.safe_read_file(file) Safely read a file by acquiring a shared lock to prevent other processes from writing to it while reading. .INDENT 7.0 .TP .B Parameters \fBfile\fP (\fI\%str\fP) \-\- .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.safe_write_file(file, s) Safely write to a file by acquiring an exclusive lock to prevent other processes from reading and writing to it while writing. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBs\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.MemoryStateCache An in\-memory place to store workflow state. .INDENT 7.0 .TP .B get(workflow_id, key) Get a key value from memory. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set(workflow_id, key, value) Set or clear a key value in memory. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.AbstractStateStore A place for the WES server to keep its state: the set of workflows that exist and whether they are done or not. .sp This is a key\-value store, with keys namespaced by workflow ID. Concurrent access from multiple threads or processes is safe and globally consistent. .sp Keys and workflow IDs are restricted to \fB[\-a\-zA\-Z0\-9_]\fP, because backends may use them as path or URL components. .sp Key values are either a string, or None if the key is not set. .sp Workflow existence isn\(aqt a thing; nonexistent workflows just have None for all keys. .sp Note that we don\(aqt yet have a cleanup operation: things are stored permanently. Even clearing all the keys may leave data behind. .sp Also handles storage for a local cache, with a separate key namespace (not a read/write\-through cache). .sp TODO: Can we replace this with just using a JobStore eventually, when AWSJobStore no longer needs SimpleDB? .INDENT 7.0 .TP .B abstract get(workflow_id, key) Get the value of the given key for the given workflow, or None if the key is not set for the workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract set(workflow_id, key, value) Set the value of the given key for the given workflow. If the value is None, clear the key. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_cache(workflow_id, key) Read a value from a local cache, without checking the actual backend. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_cache(workflow_id, key, value) Write a value to a local cache, without modifying the actual backend. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.MemoryStateStore Bases: \fI\%MemoryStateCache\fP, \fI\%AbstractStateStore\fP .sp An in\-memory place to store workflow state, for testing. .sp Inherits from MemoryStateCache first to provide implementations for AbstractStateStore. .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.FileStateStore(url) Bases: \fI\%AbstractStateStore\fP .sp A place to store workflow state that uses a POSIX\-compatible file system. .INDENT 7.0 .TP .B Parameters \fBurl\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B get(workflow_id, key) Get a key value from the filesystem. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set(workflow_id, key, value) Set or clear a key value on the filesystem. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.S3StateStore(url) Bases: \fI\%AbstractStateStore\fP .sp A place to store workflow state that uses an S3\-compatible object store. .INDENT 7.0 .TP .B Parameters \fBurl\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B get(workflow_id, key) Get a key value from S3. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set(workflow_id, key, value) Set or clear a key value on S3. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.state_store_cache: Dict[\fI\%str\fP, \fI\%AbstractStateStore\fP] .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.connect_to_state_store(url) Connect to a place to store state for workflows, defined by a URL. .sp URL may be a local file path or URL or an S3 URL. .INDENT 7.0 .TP .B Parameters \fBurl\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%AbstractStateStore\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.WorkflowStateStore(state_store, workflow_id) Slice of a state store for the state of a particular workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstate_store\fP (\fI\%AbstractStateStore\fP) \-\- .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get(key) Get the given item of workflow state. .INDENT 7.0 .TP .B Parameters \fBkey\fP (\fI\%str\fP) \-\- .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set(key, value) Set the given item of workflow state. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B read_cache(key) Read a value from a local cache, without checking the actual backend. .INDENT 7.0 .TP .B Parameters \fBkey\fP (\fI\%str\fP) \-\- .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B write_cache(key, value) Write a value to a local cache, without modifying the actual backend. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBvalue\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.connect_to_workflow_state_store(url, workflow_id) Connect to a place to store state for the given workflow, in the state store defined by the given URL. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBurl\fP (\fI\%str\fP) \-\- A URL that can be used for connect_to_state_store() .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%WorkflowStateStore\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.TERMINAL_STATES .UNINDENT .INDENT 0.0 .TP .B toil.server.utils.MAX_CANCELING_SECONDS = 30 .UNINDENT .INDENT 0.0 .TP .B class toil.server.utils.WorkflowStateMachine(store) Class for managing the WES workflow state machine. .sp This is the authority on the WES \(dqstate\(dq of a workflow. You need one to read or change the state. .sp Guaranteeing that only certain transitions can be observed is possible but not worth it. Instead, we just let updates clobber each other and grab and cache the first terminal state we see forever. If it becomes important that clients never see e.g. CANCELED \-> COMPLETE or COMPLETE \-> SYSTEM_ERROR, we can implement a real distributed state machine here. .sp We do handle making sure that tasks don\(aqt get stuck in CANCELING. .sp State can be: .sp \(dqUNKNOWN\(dq \(dqQUEUED\(dq \(dqINITIALIZING\(dq \(dqRUNNING\(dq \(dqPAUSED\(dq \(dqCOMPLETE\(dq \(dqEXECUTOR_ERROR\(dq \(dqSYSTEM_ERROR\(dq \(dqCANCELED\(dq \(dqCANCELING\(dq .sp Uses the state store\(aqs local cache to prevent needing to read things we\(aqve seen already. .INDENT 7.0 .TP .B Parameters \fBstore\fP (\fI\%WorkflowStateStore\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B send_enqueue() Send an enqueue message that would move from UNKNOWN to QUEUED. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_initialize() Send an initialize message that would move from QUEUED to INITIALIZING. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_run() Send a run message that would move from INITIALIZING to RUNNING. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_cancel() Send a cancel message that would move to CANCELING from any non\-terminal state. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_canceled() Send a canceled message that would move to CANCELED from CANCELLING. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_complete() Send a complete message that would move from RUNNING to COMPLETE. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_executor_error() Send an executor_error message that would move from QUEUED, INITIALIZING, or RUNNING to EXECUTOR_ERROR. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B send_system_error() .INDENT 7.0 .TP .B Send a system_error message that would move from QUEUED, INITIALIZING, or RUNNING to SYSTEM_ERROR. .UNINDENT .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_current_state() Get the current state of the workflow. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.server.wsgi_app\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%GunicornApplication\fP T} T{ An entry point to integrate a Gunicorn WSGI server in Python. To start a T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%run_app\fP(app[, options]) T} T{ Run a Gunicorn WSGI server. T} _ .TE .INDENT 0.0 .TP .B class toil.server.wsgi_app.GunicornApplication(app, options=None) Bases: \fBgunicorn.app.base.BaseApplication\fP .sp An entry point to integrate a Gunicorn WSGI server in Python. To start a WSGI application with callable \fIapp\fP, run the following code: .INDENT 7.0 .INDENT 3.5 .sp .EX WSGIApplication(app, options={ ... }).run() .EE .UNINDENT .UNINDENT .sp For more details, see: \fI\%https://docs.gunicorn.org/en/latest/custom.html\fP .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBapp\fP (\fI\%object\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B init(*args) .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fIAny\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_config() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load() .INDENT 7.0 .TP .B Return type \fI\%object\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.server.wsgi_app.run_app(app, options=None) Run a Gunicorn WSGI server. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBapp\fP (\fI\%object\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.test\fP .sp Base testing class for Toil. .SS Subpackages .SS \fI\%toil.test.batchSystems\fP .SS Submodules .SS \fI\%toil.test.batchSystems.batchSystemTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BatchSystemPluginTest\fP T} T{ Class for testing batch system plugin functionality. T} _ T{ \fI\%hidden\fP T} T{ Hide abstract base class from unittest\(aqs test case loader T} _ T{ \fI\%KubernetesBatchSystemTest\fP T} T{ Tests against the Kubernetes batch system T} _ T{ \fI\%KubernetesBatchSystemBenchTest\fP T} T{ Kubernetes batch system unit tests that don\(aqt need to actually talk to a cluster. T} _ T{ \fI\%AWSBatchBatchSystemTest\fP T} T{ Tests against the AWS Batch batch system T} _ T{ \fI\%MesosBatchSystemTest\fP T} T{ Tests against the Mesos batch system T} _ T{ \fI\%SingleMachineBatchSystemTest\fP T} T{ Tests against the single\-machine batch system T} _ T{ \fI\%MaxCoresSingleMachineBatchSystemTest\fP T} T{ This test ensures that single machine batch system doesn\(aqt exceed the configured number T} _ T{ \fI\%Service\fP T} T{ Abstract class used to define the interface to a service. T} _ T{ \fI\%GridEngineBatchSystemTest\fP T} T{ Tests against the GridEngine batch system T} _ T{ \fI\%SlurmBatchSystemTest\fP T} T{ Tests against the Slurm batch system T} _ T{ \fI\%LSFBatchSystemTest\fP T} T{ Tests against the LSF batch system T} _ T{ \fI\%TorqueBatchSystemTest\fP T} T{ Tests against the Torque batch system T} _ T{ \fI\%HTCondorBatchSystemTest\fP T} T{ Tests against the HTCondor batch system T} _ T{ \fI\%SingleMachineBatchSystemJobTest\fP T} T{ Tests Toil workflow against the SingleMachine batch system T} _ T{ \fI\%MesosBatchSystemJobTest\fP T} T{ Tests Toil workflow against the Mesos batch system T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%write_temp_file\fP(s, temp_dir) T} T{ Dump a string into a temp file and return its path. T} _ T{ \fI\%parentJob\fP(job, cmd) T} T{ T} _ T{ \fI\%childJob\fP(job, cmd) T} T{ T} _ T{ \fI\%grandChildJob\fP(job, cmd) T} T{ T} _ T{ \fI\%greatGrandChild\fP(cmd) T} T{ T} _ T{ \fI\%measureConcurrency\fP(filepath[, sleep_time]) T} T{ Run in parallel to determine the number of concurrent tasks. T} _ T{ \fI\%count\fP(delta, file_path) T} T{ Increments counter file and returns the max number of times the file T} _ T{ \fI\%getCounters\fP(path) T} T{ T} _ T{ \fI\%resetCounters\fP(path) T} T{ T} _ T{ \fI\%get_omp_threads\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%numCores\fP T} T{ T} _ T{ \fI\%preemptible\fP T} T{ T} _ T{ \fI\%defaultRequirements\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.numCores = 2 .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.preemptible = False .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.defaultRequirements .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Class for testing batch system plugin functionality. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B test_add_batch_system_factory() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.hidden Hide abstract base class from unittest\(aqs test case loader .sp \fI\%http://stackoverflow.com/questions/1323455/python\-unit\-test\-with\-base\-and\-sub\-class#answer\-25695512\fP .INDENT 7.0 .TP .B class AbstractBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A base test case with generic tests that every batch system should pass. .sp Cannot assume that the batch system actually executes commands on the local machine/filesystem. .INDENT 7.0 .TP .B abstract createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B supportsWallTime() .UNINDENT .INDENT 7.0 .TP .B classmethod createConfig() Returns a dummy config for the batch system tests. We need a workflowID to be set up since we are running tests without setting up a jobstore. This is the class version to be used when an instance is not available. .INDENT 7.0 .TP .B Return type \fI\%toil.common.Config\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B get_max_startup_seconds() Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_available_cores() .UNINDENT .INDENT 7.0 .TP .B test_run_jobs() .UNINDENT .INDENT 7.0 .TP .B test_set_env() .UNINDENT .INDENT 7.0 .TP .B test_set_job_env() Test the mechanism for setting per\-job environment variables to batch system jobs. .UNINDENT .INDENT 7.0 .TP .B testCheckResourceRequest() .UNINDENT .INDENT 7.0 .TP .B testScalableBatchSystem() .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class AbstractBatchSystemJobTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp An abstract base class for batch system tests that use a full Toil workflow rather than using the batch system directly. .INDENT 7.0 .TP .B cpuCount .UNINDENT .INDENT 7.0 .TP .B allocatedCores .UNINDENT .INDENT 7.0 .TP .B sleepTime = 5 .UNINDENT .INDENT 7.0 .TP .B abstract getBatchSystemName() .INDENT 7.0 .TP .B Return type (\fI\%str\fP, \fI\%AbstractBatchSystem\fP) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getOptions(tempDir) Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testJobConcurrency() Tests that the batch system is allocating core resources properly for concurrent tasks. .UNINDENT .INDENT 7.0 .TP .B test_omp_threads() Test if the OMP_NUM_THREADS env var is set correctly based on jobs.cores. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class AbstractGridEngineBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fBAbstractBatchSystemTest\fP .sp An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch systems .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the Kubernetes batch system .INDENT 7.0 .TP .B supportsWallTime() .UNINDENT .INDENT 7.0 .TP .B createBatchSystem() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Kubernetes batch system unit tests that don\(aqt need to actually talk to a cluster. .INDENT 7.0 .TP .B test_preemptability_constraints() Make sure we generate the right preemptability constraints. .UNINDENT .INDENT 7.0 .TP .B test_label_constraints() Make sure we generate the right preemptability constraints. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the AWS Batch batch system .INDENT 7.0 .TP .B supportsWallTime() .UNINDENT .INDENT 7.0 .TP .B createBatchSystem() .UNINDENT .INDENT 7.0 .TP .B get_max_startup_seconds() Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP, \fI\%toil.batchSystems.mesos.test.MesosTestSupport\fP .sp Tests against the Mesos batch system .INDENT 7.0 .TP .B classmethod createConfig() needs to set mesos_endpoint to localhost for testing since the default is now the private IP address .UNINDENT .INDENT 7.0 .TP .B supportsWallTime() .UNINDENT .INDENT 7.0 .TP .B createBatchSystem() .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testIgnoreNode() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.write_temp_file(s, temp_dir) Dump a string into a temp file and return its path. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtemp_dir\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the single\-machine batch system .INDENT 7.0 .TP .B supportsWallTime() .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B testProcessEscape(hide=False) Test to make sure that child processes and their descendants go away when the Toil workflow stops. .sp If hide is true, will try and hide the child processes to make them hard to stop. .INDENT 7.0 .TP .B Parameters \fBhide\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B testHidingProcessEscape() Test to make sure that child processes and their descendants go away when the Toil workflow stops, even if the job process stops and leaves children. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp This test ensures that single machine batch system doesn\(aqt exceed the configured number cores .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B scriptCommand() .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test() .UNINDENT .INDENT 7.0 .TP .B testServices() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.parentJob(job, cmd) .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.childJob(job, cmd) .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.grandChildJob(job, cmd) .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.greatGrandChild(cmd) .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.Service(cmd) Bases: \fI\%toil.job.Job.Service\fP .sp Abstract class used to define the interface to a service. .sp Should be subclassed by the user to define services. .sp Is not executed as a job; runs within a ServiceHostJob. .INDENT 7.0 .TP .B start(fileStore) Start the service. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Returns An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see \fI\%toil.job.Job.addService()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Checks the service is still running. .INDENT 7.0 .TP .B Raises \fBexceptions.RuntimeError\fP \-\- If the service failed, this will cause the service job to be labeled failed. .TP .B Returns True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stop(fileStore) Stops the service. Function can block until complete. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the GridEngine batch system .INDENT 7.0 .TP .B createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the Slurm batch system .INDENT 7.0 .TP .B createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the LSF batch system .INDENT 7.0 .TP .B createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the Torque batch system .INDENT 7.0 .TP .B createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the HTCondor batch system .INDENT 7.0 .TP .B createBatchSystem() .INDENT 7.0 .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemJobTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests Toil workflow against the SingleMachine batch system .INDENT 7.0 .TP .B getBatchSystemName() .INDENT 7.0 .TP .B Return type (\fI\%str\fP, \fI\%AbstractBatchSystem\fP) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B testConcurrencyWithDisk() Tests that the batch system is allocating disk resources properly .UNINDENT .INDENT 7.0 .TP .B testNestedResourcesDoNotBlock() Resources are requested in the order Memory > Cpu > Disk. Test that unavailability of cpus for one job that is scheduled does not block another job that can run. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.batchSystemTest.MesosBatchSystemJobTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP, \fI\%toil.batchSystems.mesos.test.MesosTestSupport\fP .sp Tests Toil workflow against the Mesos batch system .INDENT 7.0 .TP .B getOptions(tempDir) Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object .UNINDENT .INDENT 7.0 .TP .B getBatchSystemName() .INDENT 7.0 .TP .B Return type (\fI\%str\fP, \fI\%AbstractBatchSystem\fP) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.measureConcurrency(filepath, sleep_time=10) Run in parallel to determine the number of concurrent tasks. This code was copied from toil.batchSystemTestMaxCoresSingleMachineBatchSystemTest :param str filepath: path to counter file :param int sleep_time: number of seconds to sleep before counting down :return int max concurrency value: .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.count(delta, file_path) Increments counter file and returns the max number of times the file has been modified. Counter data must be in the form: concurrent tasks, max concurrent tasks (counter should be initialized to 0,0) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBdelta\fP (\fI\%int\fP) \-\- increment value .IP \(bu 2 \fBfile_path\fP (\fI\%str\fP) \-\- path to shared counter file .UNINDENT .TP .B Return int max concurrent tasks .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.getCounters(path) .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.resetCounters(path) .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.batchSystemTest.get_omp_threads() .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .SS \fI\%toil.test.batchSystems.test_lsf_helper\fP .sp lsfHelper.py shouldn\(aqt need a batch system and so the unit tests here should aim to run on any system. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%LSFHelperTest\fP T} T{ A common base class for Toil tests. T} _ .TE .INDENT 0.0 .TP .B class toil.test.batchSystems.test_lsf_helper.LSFHelperTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test_parse_mem_and_cmd_from_output() .UNINDENT .UNINDENT .SS \fI\%toil.test.batchSystems.test_slurm\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%FakeBatchSystem\fP T} T{ Class that implements a minimal Batch System, needed to create a Worker (see below). T} _ T{ \fI\%SlurmTest\fP T} T{ Class for unit\-testing SlurmBatchSystem T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%call_sacct\fP(args, **_) T} T{ The arguments passed to \fIcall_command\fP when executing \fIsacct\fP are: T} _ T{ \fI\%call_scontrol\fP(args, **_) T} T{ The arguments passed to \fIcall_command\fP when executing \fIscontrol\fP are: T} _ T{ \fI\%call_sacct_raises\fP(*_) T} T{ Fake that the \fIsacct\fP command fails by raising a \fICalledProcessErrorStderr\fP T} _ .TE .INDENT 0.0 .TP .B toil.test.batchSystems.test_slurm.call_sacct(args, **_) The arguments passed to \fIcall_command\fP when executing \fIsacct\fP are: [\(aqsacct\(aq, \(aq\-n\(aq, \(aq\-j\(aq, \(aq\(aq, \(aq\-\-format\(aq, \(aqJobIDRaw,State,ExitCode\(aq, \(aq\-P\(aq, \(aq\-S\(aq, \(aq1970\-01\-01\(aq] The multi\-line output is something like: .INDENT 7.0 .INDENT 3.5 .sp .EX 1234|COMPLETED|0:0 1234.batch|COMPLETED|0:0 1235|PENDING|0:0 1236|FAILED|0:2 1236.extern|COMPLETED|0:0 .EE .UNINDENT .UNINDENT .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.test_slurm.call_scontrol(args, **_) The arguments passed to \fIcall_command\fP when executing \fIscontrol\fP are: \fB[\(aqscontrol\(aq, \(aqshow\(aq, \(aqjob\(aq]\fP or \fB[\(aqscontrol\(aq, \(aqshow\(aq, \(aqjob\(aq, \(aq\(aq]\fP .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.batchSystems.test_slurm.call_sacct_raises(*_) Fake that the \fIsacct\fP command fails by raising a \fICalledProcessErrorStderr\fP .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.test_slurm.FakeBatchSystem Class that implements a minimal Batch System, needed to create a Worker (see below). .INDENT 7.0 .TP .B getWaitDuration() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.batchSystems.test_slurm.SlurmTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Class for unit\-testing SlurmBatchSystem .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromSacct_one_exists() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromSacct_one_not_exists() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromSacct_many_all_exist() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromSacct_many_some_exist() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromSacct_many_none_exist() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromScontrol_one_exists() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromScontrol_one_not_exists() Asking for the job details of a single job that \fIscontrol\fP doesn\(aqt know about should raise an exception. .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromScontrol_many_all_exist() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromScontrol_many_some_exist() .UNINDENT .INDENT 7.0 .TP .B test_getJobDetailsFromScontrol_many_none_exist() .UNINDENT .INDENT 7.0 .TP .B test_getJobExitCode_job_exists() .UNINDENT .INDENT 7.0 .TP .B test_getJobExitCode_job_not_exists() .UNINDENT .INDENT 7.0 .TP .B test_getJobExitCode_sacct_raises_job_exists() This test forces the use of \fIscontrol\fP to get job information, by letting \fIsacct\fP raise an exception. .UNINDENT .INDENT 7.0 .TP .B test_getJobExitCode_sacct_raises_job_not_exists() This test forces the use of \fIscontrol\fP to get job information, by letting \fIsacct\fP raise an exception. Next, \fIscontrol\fP should also raise because it doesn\(aqt know the job. .UNINDENT .INDENT 7.0 .TP .B test_coalesce_job_exit_codes_one_exists() .UNINDENT .INDENT 7.0 .TP .B test_coalesce_job_exit_codes_one_not_exists() .UNINDENT .INDENT 7.0 .TP .B test_coalesce_job_exit_codes_many_all_exist() .UNINDENT .INDENT 7.0 .TP .B test_coalesce_job_exit_codes_some_exists() .UNINDENT .INDENT 7.0 .TP .B test_coalesce_job_exit_codes_sacct_raises_job_exists() This test forces the use of \fIscontrol\fP to get job information, by letting \fIsacct\fP raise an exception. .UNINDENT .INDENT 7.0 .TP .B test_coalesce_job_exit_codes_sacct_raises_job_not_exists() This test forces the use of \fIscontrol\fP to get job information, by letting \fIsacct\fP raise an exception. Next, \fIscontrol\fP should also raise because it doesn\(aqt know the job. .UNINDENT .UNINDENT .SS \fI\%toil.test.cactus\fP .SS Submodules .SS \fI\%toil.test.cactus.test_cactus_integration\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%CactusIntegrationTest\fP T} T{ Run the Cactus Integration test on a Kubernetes AWS cluster T} _ .TE .INDENT 0.0 .TP .B class toil.test.cactus.test_cactus_integration.CactusIntegrationTest(methodName) Bases: \fI\%toil.test.provisioners.clusterTest.AbstractClusterTest\fP .sp Run the Cactus Integration test on a Kubernetes AWS cluster .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .UNINDENT .INDENT 7.0 .TP .B test_cactus_integration() .UNINDENT .UNINDENT .SS \fI\%toil.test.cwl\fP .SS Submodules .SS \fI\%toil.test.cwl.conftest\fP .SS Module Contents .INDENT 0.0 .TP .B toil.test.cwl.conftest.collect_ignore = [\(aqspec\(aq] .UNINDENT .SS \fI\%toil.test.cwl.cwlTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%CWLWorkflowTest\fP T} T{ CWL tests included in Toil that don\(aqt involve the whole CWL conformance T} _ T{ \fI\%CWLv10Test\fP T} T{ Run the CWL 1.0 conformance tests in various environments. T} _ T{ \fI\%CWLv11Test\fP T} T{ Run the CWL 1.1 conformance tests in various environments. T} _ T{ \fI\%CWLv12Test\fP T} T{ Run the CWL 1.2 conformance tests in various environments. T} _ T{ \fI\%CWLOnARMTest\fP T} T{ Run the CWL 1.2 conformance tests on ARM specifically. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%run_conformance_tests\fP(workDir, yml[, runner, caching, ...]) T} T{ Run the CWL conformance tests. T} _ T{ \fI\%test_workflow_echo_string_scatter_stderr_log_dir\fP(tmp_path) T} T{ T} _ T{ \fI\%test_log_dir_echo_no_output\fP(tmp_path) T} T{ T} _ T{ \fI\%test_log_dir_echo_stderr\fP(tmp_path) T} T{ T} _ T{ \fI\%test_filename_conflict_resolution\fP(tmp_path) T} T{ T} _ T{ \fI\%test_filename_conflict_detection\fP(tmp_path) T} T{ Make sure we don\(aqt just stage files over each other when using a container. T} _ T{ \fI\%test_filename_conflict_detection_at_root\fP(tmp_path) T} T{ Make sure we don\(aqt just stage files over each other. T} _ T{ \fI\%test_pick_value_with_one_null_value\fP(caplog) T} T{ Make sure toil\-cwl\-runner does not false log a warning when pickValue is T} _ T{ \fI\%test_workflow_echo_string\fP() T} T{ T} _ T{ \fI\%test_workflow_echo_string_scatter_capture_stdout\fP() T} T{ T} _ T{ \fI\%test_visit_top_cwl_class\fP() T} T{ T} _ T{ \fI\%test_visit_cwl_class_and_reduce\fP() T} T{ T} _ T{ \fI\%test_download_structure\fP(tmp_path) T} T{ Make sure that download_structure makes the right calls to what it thinks is the file store. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%pkg_root\fP T} T{ T} _ T{ \fI\%log\fP T} T{ T} _ T{ \fI\%CONFORMANCE_TEST_TIMEOUT\fP T} T{ T} _ T{ \fI\%TesterFuncType\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.pkg_root .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.log .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.CONFORMANCE_TEST_TIMEOUT = 10000 .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.run_conformance_tests(workDir, yml, runner=None, caching=False, batchSystem=None, selected_tests=None, selected_tags=None, skipped_tests=None, extra_args=None, must_support_all_features=False, junit_file=None) Run the CWL conformance tests. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkDir\fP (\fI\%str\fP) \-\- Directory to run tests in. .IP \(bu 2 \fByml\fP (\fI\%str\fP) \-\- CWL test list YML to run tests from. .IP \(bu 2 \fBrunner\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If set, use this cwl runner instead of the default toil\-cwl\-runner. .IP \(bu 2 \fBcaching\fP (\fI\%bool\fP) \-\- If True, use Toil file store caching. .IP \(bu 2 \fBbatchSystem\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If set, use this batch system instead of the default single_machine. .IP \(bu 2 \fBselected_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- If set, use this description of test numbers to run (comma\-separated numbers or ranges) .IP \(bu 2 \fBselected_tags\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- As an alternative to selected_tests, run tests with the given tags. .IP \(bu 2 \fBskipped_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Comma\-separated string labels of tests to skip. .IP \(bu 2 \fBextra_args\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Provide these extra arguments to runner for each test. .IP \(bu 2 \fBmust_support_all_features\fP (\fI\%bool\fP) \-\- If set, fail if some CWL optional features are unsupported. .IP \(bu 2 \fBjunit_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- JUnit XML file to write test info to. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.TesterFuncType .UNINDENT .INDENT 0.0 .TP .B class toil.test.cwl.cwlTest.CWLWorkflowTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp CWL tests included in Toil that don\(aqt involve the whole CWL conformance test suite. Tests Toil\-specific functions like URL types supported for inputs. .INDENT 7.0 .TP .B setUp() Runs anew before each test to create farm fresh temp dirs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Clean up outputs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_cwl_cmdline_input() Test that running a CWL workflow with inputs specified on the command line passes. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B revsort(cwl_filename, tester_fn) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcwl_filename\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtester_fn\fP (\fITesterFuncType\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B revsort_no_checksum(cwl_filename, tester_fn) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcwl_filename\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtester_fn\fP (\fITesterFuncType\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B download(inputs, tester_fn) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinputs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtester_fn\fP (\fITesterFuncType\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load_contents(inputs, tester_fn) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinputs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtester_fn\fP (\fITesterFuncType\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B download_directory(inputs, tester_fn) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinputs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtester_fn\fP (\fITesterFuncType\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B download_subdirectory(inputs, tester_fn) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinputs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtester_fn\fP (\fITesterFuncType\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_mpi() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_s3_as_secondary_file() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_revsort() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_revsort_nochecksum() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_revsort2() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_revsort_debug_worker() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_colon_output() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_glob_dir_bypass_file_store() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_s3() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_http() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_https() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_https_reference() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_file() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_directory_s3() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_directory_s3_reference() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_directory_file() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_subdirectory_s3() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_download_subdirectory_file() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_load_contents_s3() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_load_contents_http() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_load_contents_https() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_load_contents_file() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_bioconda() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_default_args() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_biocontainers() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_cuda() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_restart() Enable restarts with toil\-cwl\-runner \-\- run failing test, re\-run correct test. Only implemented for single machine. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_streamable(extra_args=None) Test that a file with \(aqstreamable\(aq=True is a named pipe. This is a CWL1.2 feature. .INDENT 7.0 .TP .B Parameters \fBextra_args\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_streamable_reference() Test that a streamable file is a stream even when passed around by URI. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_preemptible() Tests that the \fI\%http://arvados.org/cwl#UsePreemptible\fP extension is supported. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_preemptible_expression() Tests that the \fI\%http://arvados.org/cwl#UsePreemptible\fP extension is validated. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.cwl.cwlTest.CWLv10Test(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Run the CWL 1.0 conformance tests in various environments. .INDENT 7.0 .TP .B setUp() Runs anew before each test to create farm fresh temp dirs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Clean up outputs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance(batchSystem=None, caching=False, selected_tests=None, skipped_tests=None, extra_args=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbatchSystem\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcaching\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBselected_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBskipped_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBextra_args\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_lsf_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_slurm_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_torque_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_gridengine_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_mesos_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_kubernetes_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_lsf_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_slurm_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_torque_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_gridengine_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_mesos_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_kubernetes_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.cwl.cwlTest.CWLv11Test(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Run the CWL 1.1 conformance tests in various environments. .INDENT 7.0 .TP .B rootDir: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B cwlSpec: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B test_yaml: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Runs anew before each test. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Clean up outputs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance(caching=False, batchSystem=None, skipped_tests=None, extra_args=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcaching\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBbatchSystem\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBskipped_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBextra_args\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_kubernetes_cwl_conformance(caching=False) .INDENT 7.0 .TP .B Parameters \fBcaching\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_kubernetes_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.cwl.cwlTest.CWLv12Test(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Run the CWL 1.2 conformance tests in various environments. .INDENT 7.0 .TP .B rootDir: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B cwlSpec: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B test_yaml: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Runs anew before each test. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Clean up outputs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance(runner=None, caching=False, batchSystem=None, selected_tests=None, skipped_tests=None, extra_args=None, must_support_all_features=False, junit_file=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrunner\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcaching\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBbatchSystem\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBselected_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBskipped_tests\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBextra_args\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBmust_support_all_features\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBjunit_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_conformance_with_in_place_update() Make sure that with \-\-bypass\-file\-store we properly support in place update on a single node, and that this doesn\(aqt break any other features. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_kubernetes_cwl_conformance(caching=False, junit_file=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcaching\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBjunit_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_kubernetes_cwl_conformance_with_caching() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_wes_server_cwl_conformance() Run the CWL conformance tests via WES. TOIL_WES_ENDPOINT must be specified. If the WES server requires authentication, set TOIL_WES_USER and TOIL_WES_PASSWORD. .sp To run manually: .sp TOIL_WES_ENDPOINT=http://localhost:8080 TOIL_WES_USER=test TOIL_WES_PASSWORD=password python \-m pytest src/toil/test/cwl/cwlTest.py::CWLv12Test::test_wes_server_cwl_conformance \-vv \-\-log\-level INFO \-\-log\-cli\-level INFO .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.cwl.cwlTest.CWLOnARMTest(methodName) Bases: \fI\%toil.test.provisioners.clusterTest.AbstractClusterTest\fP .sp Run the CWL 1.2 conformance tests on ARM specifically. .INDENT 7.0 .TP .B Parameters \fBmethodName\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_cwl_on_arm() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_workflow_echo_string_scatter_stderr_log_dir(tmp_path) .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_log_dir_echo_no_output(tmp_path) .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_log_dir_echo_stderr(tmp_path) .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_filename_conflict_resolution(tmp_path) .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_filename_conflict_detection(tmp_path) Make sure we don\(aqt just stage files over each other when using a container. .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_filename_conflict_detection_at_root(tmp_path) Make sure we don\(aqt just stage files over each other. .sp Specifically, when using a container and the files are at the root of the work dir. .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_pick_value_with_one_null_value(caplog) Make sure toil\-cwl\-runner does not false log a warning when pickValue is used but outputSource only contains one null value. See: #3991. .INDENT 7.0 .TP .B Parameters \fBcaplog\fP (\fIpytest.LogCaptureFixture\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_workflow_echo_string() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_workflow_echo_string_scatter_capture_stdout() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_visit_top_cwl_class() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_visit_cwl_class_and_reduce() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cwl.cwlTest.test_download_structure(tmp_path) Make sure that download_structure makes the right calls to what it thinks is the file store. .INDENT 7.0 .TP .B Parameters \fBtmp_path\fP (\fI\%pathlib.Path\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.test.docs\fP .SS Submodules .SS \fI\%toil.test.docs.scriptsTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilDocumentationTest\fP T} T{ Tests for scripts in the toil tutorials. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%pkg_root\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.docs.scriptsTest.pkg_root .UNINDENT .INDENT 0.0 .TP .B class toil.test.docs.scriptsTest.ToilDocumentationTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests for scripts in the toil tutorials. .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B checkExitCode(script) .UNINDENT .INDENT 7.0 .TP .B checkExpectedOut(script, expectedOutput) .UNINDENT .INDENT 7.0 .TP .B checkExpectedPattern(script, expectedPattern) .UNINDENT .INDENT 7.0 .TP .B testCwlexample() .UNINDENT .INDENT 7.0 .TP .B testStats() .UNINDENT .INDENT 7.0 .TP .B testDynamic() .UNINDENT .INDENT 7.0 .TP .B testEncapsulation() .UNINDENT .INDENT 7.0 .TP .B testEncapsulation2() .UNINDENT .INDENT 7.0 .TP .B testHelloworld() .UNINDENT .INDENT 7.0 .TP .B testInvokeworkflow() .UNINDENT .INDENT 7.0 .TP .B testInvokeworkflow2() .UNINDENT .INDENT 7.0 .TP .B testJobFunctions() .UNINDENT .INDENT 7.0 .TP .B testManaging() .UNINDENT .INDENT 7.0 .TP .B testManaging2() .UNINDENT .INDENT 7.0 .TP .B testMultiplejobs() .UNINDENT .INDENT 7.0 .TP .B testMultiplejobs2() .UNINDENT .INDENT 7.0 .TP .B testMultiplejobs3() .UNINDENT .INDENT 7.0 .TP .B testPromises2() .UNINDENT .INDENT 7.0 .TP .B testQuickstart() .UNINDENT .INDENT 7.0 .TP .B testRequirements() .UNINDENT .INDENT 7.0 .TP .B testArguments() .UNINDENT .INDENT 7.0 .TP .B testDocker() .UNINDENT .INDENT 7.0 .TP .B testPromises() .UNINDENT .INDENT 7.0 .TP .B testServices() .UNINDENT .INDENT 7.0 .TP .B testStaging() .UNINDENT .UNINDENT .SS \fI\%toil.test.jobStores\fP .SS Submodules .SS \fI\%toil.test.jobStores.jobStoreTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AbstractJobStoreTest\fP T} T{ Hide abstract base class from unittest\(aqs test case loader T} _ T{ \fI\%AbstractEncryptedJobStoreTest\fP T} T{ T} _ T{ \fI\%FileJobStoreTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%GoogleJobStoreTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%AWSJobStoreTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%InvalidAWSJobStoreTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%EncryptedAWSJobStoreTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%StubHttpRequestHandler\fP T} T{ Simple HTTP request handler with GET and HEAD commands. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%google_retry\fP(x) T} T{ T} _ T{ \fI\%tearDownModule\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.jobStores.jobStoreTest.google_retry(x) .UNINDENT .INDENT 0.0 .TP .B toil.test.jobStores.jobStoreTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.jobStores.jobStoreTest.tearDownModule() .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.AbstractJobStoreTest Hide abstract base class from unittest\(aqs test case loader .sp \fI\%http://stackoverflow.com/questions/1323455/python\-unit\-test\-with\-base\-and\-sub\-class#answer\-25695512\fP .INDENT 7.0 .TP .B class Test(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B externalStoreCache .UNINDENT .INDENT 7.0 .TP .B mpTestPartSize .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testInitialState() Ensure proper handling of nonexistant files. .UNINDENT .INDENT 7.0 .TP .B testJobCreation() Test creation of a job. .sp Does the job exist in the jobstore it is supposed to be in? Are its attributes what is expected? .UNINDENT .INDENT 7.0 .TP .B testConfigEquality() Ensure that the command line configurations are successfully loaded and stored. .sp In setUp() self.jobstore1 is created and initialized. In this test, after creating newJobStore, \&.resume() will look for a previously instantiated job store and load its config options. This is expected to be equal but not the same object. .UNINDENT .INDENT 7.0 .TP .B testJobLoadEquality() Tests that a job created via one JobStore instance can be loaded from another. .UNINDENT .INDENT 7.0 .TP .B testChildLoadingEquality() Test that loading a child job operates as expected. .UNINDENT .INDENT 7.0 .TP .B testPersistantFilesToDelete() Make sure that updating a job persists filesToDelete. .sp The following demonstrates the job update pattern, where files to be deleted atomically with a job update are referenced in \(dqfilesToDelete\(dq array, which is persisted to disk first. If things go wrong during the update, this list of files to delete is used to ensure that the updated job and the files are never both visible at the same time. .UNINDENT .INDENT 7.0 .TP .B testUpdateBehavior() Tests the proper behavior during updating jobs. .UNINDENT .INDENT 7.0 .TP .B testJobDeletions() Tests the consequences of deleting jobs. .UNINDENT .INDENT 7.0 .TP .B testSharedFiles() Tests the sharing of files. .UNINDENT .INDENT 7.0 .TP .B testReadWriteSharedFilesTextMode() Checks if text mode is compatible for shared file streams. .UNINDENT .INDENT 7.0 .TP .B testReadWriteFileStreamTextMode() Checks if text mode is compatible for file streams. .UNINDENT .INDENT 7.0 .TP .B testPerJobFiles() Tests the behavior of files on jobs. .UNINDENT .INDENT 7.0 .TP .B testStatsAndLogging() Tests behavior of reading and writting stats and logging. .UNINDENT .INDENT 7.0 .TP .B testWriteLogFiles() Test writing log files. .UNINDENT .INDENT 7.0 .TP .B testBatchCreate() Test creation of many jobs. .UNINDENT .INDENT 7.0 .TP .B testGrowingAndShrinkingJob() Make sure jobs update correctly if they grow/shrink. .UNINDENT .INDENT 7.0 .TP .B classmethod cleanUpExternalStores() .UNINDENT .INDENT 7.0 .TP .B classmethod makeImportExportTests() .UNINDENT .INDENT 7.0 .TP .B testImportHttpFile() Test importing a file over HTTP. .UNINDENT .INDENT 7.0 .TP .B testImportFtpFile() Test importing a file over FTP .UNINDENT .INDENT 7.0 .TP .B testFileDeletion() Intended to cover the batch deletion of items in the AWSJobStore, but it doesn\(aqt hurt running it on the other job stores. .UNINDENT .INDENT 7.0 .TP .B testMultipartUploads() This test is meant to cover multi\-part uploads in the AWSJobStore but it doesn\(aqt hurt running it against the other job stores as well. .UNINDENT .INDENT 7.0 .TP .B testZeroLengthFiles() Test reading and writing of empty files. .UNINDENT .INDENT 7.0 .TP .B testLargeFile() Test the reading and writing of large files. .UNINDENT .INDENT 7.0 .TP .B fetch_url(url) Fetch the given URL. Throw an error if it cannot be fetched in a reasonable number of attempts. .INDENT 7.0 .TP .B Parameters \fBurl\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B assertUrl(url) .UNINDENT .INDENT 7.0 .TP .B testCleanCache() .UNINDENT .INDENT 7.0 .TP .B testPartialReadFromStream() Test whether readFileStream will deadlock on a partial read. .UNINDENT .INDENT 7.0 .TP .B testDestructionOfCorruptedJobStore() .UNINDENT .INDENT 7.0 .TP .B testDestructionIdempotence() .UNINDENT .INDENT 7.0 .TP .B testEmptyFileStoreIDIsReadable() Simply creates an empty fileStoreID and attempts to read from it. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.AbstractEncryptedJobStoreTest .INDENT 7.0 .TP .B class Test(methodName=\(aqrunTest\(aq) Bases: \fI\%AbstractJobStoreTest\fP .sp A test of job stores that use encryption .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testEncrypted() Create an encrypted file. Read it in encrypted mode then try with encryption off to ensure that it fails. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.FileJobStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%AbstractJobStoreTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testPreserveFileName() Check that the fileID ends with the given file name. .UNINDENT .INDENT 7.0 .TP .B test_jobstore_init_preserves_symlink_path() Test that if we provide a fileJobStore with a symlink to a directory, it doesn\(aqt de\-reference it. .UNINDENT .INDENT 7.0 .TP .B test_jobstore_does_not_leak_symlinks() Test that if we link imports into the FileJobStore, we can\(aqt get hardlinks to symlinks. .UNINDENT .INDENT 7.0 .TP .B test_file_link_imports() Test that imported files are symlinked when when expected .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.GoogleJobStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%AbstractJobStoreTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B projectID .UNINDENT .INDENT 7.0 .TP .B headers .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.AWSJobStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%AbstractJobStoreTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testSDBDomainsDeletedOnFailedJobstoreBucketCreation() This test ensures that SDB domains bound to a jobstore are deleted if the jobstore bucket failed to be created. We simulate a failed jobstore bucket creation by using a bucket in a different region with the same name. .UNINDENT .INDENT 7.0 .TP .B testInlinedFiles() .UNINDENT .INDENT 7.0 .TP .B testOverlargeJob() .UNINDENT .INDENT 7.0 .TP .B testMultiThreadImportFile() Tests that importFile is thread\-safe. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.InvalidAWSJobStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testInvalidJobStoreName() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.EncryptedAWSJobStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%AWSJobStoreTest\fP, \fI\%AbstractEncryptedJobStoreTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .UNINDENT .INDENT 0.0 .TP .B class toil.test.jobStores.jobStoreTest.StubHttpRequestHandler(*args, directory=None, **kwargs) Bases: \fI\%http.server.SimpleHTTPRequestHandler\fP .sp Simple HTTP request handler with GET and HEAD commands. .sp This serves files from the current directory and any of its subdirectories. The MIME type for files is determined by calling the .guess_type() method. .sp The GET and HEAD requests are identical except that the HEAD request omits the actual contents of the file. .INDENT 7.0 .TP .B fileContents = \(aqA good programmer looks both ways before crossing a one\-way street\(aq .UNINDENT .INDENT 7.0 .TP .B do_GET() Serve a GET request. .UNINDENT .UNINDENT .SS \fI\%toil.test.lib\fP .SS Subpackages .SS \fI\%toil.test.lib.aws\fP .SS Submodules .SS \fI\%toil.test.lib.aws.test_iam\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%IAMTest\fP T} T{ Check that given permissions and associated functions perform correctly T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.aws.test_iam.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.aws.test_iam.IAMTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Check that given permissions and associated functions perform correctly .INDENT 7.0 .TP .B test_permissions_iam() .UNINDENT .INDENT 7.0 .TP .B test_negative_permissions_iam() .UNINDENT .INDENT 7.0 .TP .B test_wildcard_handling() .UNINDENT .INDENT 7.0 .TP .B test_get_policy_permissions() .UNINDENT .UNINDENT .SS \fI\%toil.test.lib.aws.test_s3\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%S3Test\fP T} T{ Confirm the workarounds for us\-east\-1. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.aws.test_s3.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.aws.test_s3.S3Test(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Confirm the workarounds for us\-east\-1. .INDENT 7.0 .TP .B s3_resource: mypy_boto3_s3.S3ServiceResource | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B bucket: mypy_boto3_s3.service_resource.Bucket | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_create_bucket() Test bucket creation for us\-east\-1. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_get_bucket_location_public_bucket() Test getting buket location for a bucket we don\(aqt own. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod tearDownClass() Hook method for deconstructing the class fixture after running all tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.lib.aws.test_utils\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%TagGenerationTest\fP T} T{ Test for tag generation from environment variables T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.aws.test_utils.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.aws.test_utils.TagGenerationTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test for tag generation from environment variables .INDENT 7.0 .TP .B test_build_tag() .UNINDENT .INDENT 7.0 .TP .B test_empty_aws_tags() .UNINDENT .INDENT 7.0 .TP .B test_incorrect_json_object() .UNINDENT .INDENT 7.0 .TP .B test_incorrect_json_emoji() .UNINDENT .INDENT 7.0 .TP .B test_build_tag_with_tags() .UNINDENT .UNINDENT .SS Submodules .SS \fI\%toil.test.lib.dockerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%DockerTest\fP T} T{ Tests dockerCall and ensures no containers are left around. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.dockerTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.dockerTest.DockerTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests dockerCall and ensures no containers are left around. When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testDockerClean(caching=False, detached=True, rm=True, deferParam=None) Run the test container that creates a file in the work dir, and sleeps for 5 minutes. Ensure that the calling job gets SIGKILLed after a minute, leaving behind the spooky/ghost/zombie container. Ensure that the container is killed on batch system shutdown (through the deferParam mechanism). .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CRx_FORGO() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CRx_STOP() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CRx_RM() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CRx_None() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CxD_FORGO() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CxD_STOP() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CxD_RM() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_CxD_None() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_Cxx_FORGO() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_Cxx_STOP() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_Cxx_RM() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_Cxx_None() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xRx_FORGO() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xRx_STOP() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xRx_RM() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xRx_None() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxD_FORGO() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxD_STOP() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxD_RM() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxD_None() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxx_FORGO() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxx_STOP() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxx_RM() .UNINDENT .INDENT 7.0 .TP .B testDockerClean_xxx_None() .UNINDENT .INDENT 7.0 .TP .B testDockerPipeChain(caching=False) Test for piping API for dockerCall(). Using this API (activated when list of argument lists is given as parameters), commands a piped together into a chain. ex: \fBparameters=[ [\(aqprintf\(aq, \(aqx\en y\en\(aq], [\(aqwc\(aq, \(aq\-l\(aq] ]\fP should execute: \fBprintf \(aqx\en y\en\(aq | wc \-l\fP .UNINDENT .INDENT 7.0 .TP .B testDockerPipeChainErrorDetection(caching=False) By default, executing cmd1 | cmd2 | ... | cmdN, will only return an error if cmdN fails. This can lead to all manor of errors being silently missed. This tests to make sure that the piping API for dockerCall() throws an exception if non\-last commands in the chain fail. .UNINDENT .INDENT 7.0 .TP .B testNonCachingDockerChain() .UNINDENT .INDENT 7.0 .TP .B testNonCachingDockerChainErrorDetection() .UNINDENT .INDENT 7.0 .TP .B testDockerLogs(stream=False, demux=False) Test for the different log outputs when deatch=False. .UNINDENT .INDENT 7.0 .TP .B testDockerLogs_Stream() .UNINDENT .INDENT 7.0 .TP .B testDockerLogs_Demux() .UNINDENT .INDENT 7.0 .TP .B testDockerLogs_Demux_Stream() .UNINDENT .UNINDENT .SS \fI\%toil.test.lib.test_conversions\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ConversionTest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.test_conversions.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.test_conversions.ConversionTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test_convert() .UNINDENT .INDENT 7.0 .TP .B test_human2bytes() .UNINDENT .INDENT 7.0 .TP .B test_hms_duration_to_seconds() .UNINDENT .UNINDENT .SS \fI\%toil.test.lib.test_ec2\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%FlatcarFeedTest\fP T} T{ Test accessing the Flatcar AMI release feed, independent of the AWS API T} _ T{ \fI\%AMITest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.test_ec2.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.test_ec2.FlatcarFeedTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test accessing the Flatcar AMI release feed, independent of the AWS API .INDENT 7.0 .TP .B test_parse_archive_feed() Make sure we can get a Flatcar release from the Internet Archive. .UNINDENT .INDENT 7.0 .TP .B test_parse_beta_feed() Make sure we can get a Flatcar release from the beta channel. .UNINDENT .INDENT 7.0 .TP .B test_parse_stable_feed() Make sure we can get a Flatcar release from the stable channel. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.test_ec2.AMITest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .UNINDENT .INDENT 7.0 .TP .B test_fetch_flatcar() .UNINDENT .INDENT 7.0 .TP .B test_fetch_arm_flatcar() Test flatcar AMI finder architecture parameter. .UNINDENT .UNINDENT .SS \fI\%toil.test.lib.test_misc\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%UserNameAvailableTest\fP T} T{ Make sure we can get user names when they are available. T} _ T{ \fI\%UserNameUnvailableTest\fP T} T{ Make sure we can get something for a user name when user names are not T} _ T{ \fI\%UserNameVeryBrokenTest\fP T} T{ Make sure we can get something for a user name when user name fetching is T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.lib.test_misc.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.test_misc.UserNameAvailableTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Make sure we can get user names when they are available. .INDENT 7.0 .TP .B test_get_user_name() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.test_misc.UserNameUnvailableTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Make sure we can get something for a user name when user names are not available. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B test_get_user_name() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.lib.test_misc.UserNameVeryBrokenTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Make sure we can get something for a user name when user name fetching is broken in ways we did not expect. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B test_get_user_name() .UNINDENT .UNINDENT .SS \fI\%toil.test.mesos\fP .SS Submodules .SS \fI\%toil.test.mesos.MesosDataStructuresTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%DataStructuresTest\fP T} T{ A common base class for Toil tests. T} _ .TE .INDENT 0.0 .TP .B class toil.test.mesos.MesosDataStructuresTest.DataStructuresTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testJobQueue(testJobs=1000) The mesos JobQueue sorts MesosShape objects by requirement and this test ensures that that sorting is what is expected: non\-preemptible jobs groups first, with priority given to large jobs. .UNINDENT .UNINDENT .SS \fI\%toil.test.mesos.helloWorld\fP .sp A simple user script for Toil .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%hello_world\fP(job) T} T{ T} _ T{ \fI\%hello_world_child\fP(job, hw) T} T{ T} _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%childMessage\fP T} T{ T} _ T{ \fI\%parentMessage\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.mesos.helloWorld.childMessage = \(aqThe child job is now running!\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.mesos.helloWorld.parentMessage = \(aqThe parent job is now running!\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.mesos.helloWorld.hello_world(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.mesos.helloWorld.hello_world_child(job, hw) .UNINDENT .INDENT 0.0 .TP .B toil.test.mesos.helloWorld.main() .UNINDENT .SS \fI\%toil.test.mesos.stress\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%LongTestJob\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%LongTestFollowOn\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%HelloWorldJob\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%HelloWorldFollowOn\fP T} T{ Class represents a unit of work in toil. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%touchFile\fP(fileStore) T} T{ T} _ T{ \fI\%main\fP(numJobs) T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.mesos.stress.touchFile(fileStore) .UNINDENT .INDENT 0.0 .TP .B class toil.test.mesos.stress.LongTestJob(numJobs) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.mesos.stress.LongTestFollowOn Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.mesos.stress.HelloWorldJob(i) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.mesos.stress.HelloWorldFollowOn(i) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.mesos.stress.main(numJobs) .UNINDENT .SS \fI\%toil.test.options\fP .SS Submodules .SS \fI\%toil.test.options.options\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%OptionsTest\fP T} T{ Class to test functionality of all Toil options T} _ .TE .INDENT 0.0 .TP .B class toil.test.options.options.OptionsTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Class to test functionality of all Toil options .INDENT 7.0 .TP .B test_default_caching_slurm() Test to ensure that caching will be set to false when running on Slurm :return: .UNINDENT .INDENT 7.0 .TP .B test_caching_option_priority() Test to ensure that the \-\-caching option takes priority over the default_caching() return value :return: .UNINDENT .UNINDENT .SS \fI\%toil.test.provisioners\fP .SS Subpackages .SS \fI\%toil.test.provisioners.aws\fP .SS Submodules .SS \fI\%toil.test.provisioners.aws.awsProvisionerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AWSProvisionerBenchTest\fP T} T{ Tests for the AWS provisioner that don\(aqt actually provision anything. T} _ T{ \fI\%AbstractAWSAutoscaleTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%AWSAutoscaleTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%AWSStaticAutoscaleTest\fP T} T{ Runs the tests on a statically provisioned cluster with autoscaling enabled. T} _ T{ \fI\%AWSManagedAutoscaleTest\fP T} T{ Runs the tests on a self\-scaling Kubernetes cluster. T} _ T{ \fI\%AWSAutoscaleTestMultipleNodeTypes\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%AWSRestartTest\fP T} T{ This test insures autoscaling works on a restarted Toil run. T} _ T{ \fI\%PreemptibleDeficitCompensationTest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.provisioners.aws.awsProvisionerTest.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AWSProvisionerBenchTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests for the AWS provisioner that don\(aqt actually provision anything. .INDENT 7.0 .TP .B test_AMI_finding() .UNINDENT .INDENT 7.0 .TP .B test_read_write_global_files() Make sure the \fI_write_file_to_cloud()\fP and \fI_read_file_from_cloud()\fP functions of the AWS provisioner work as intended. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AbstractAWSAutoscaleTest(methodName) Bases: \fI\%toil.test.provisioners.clusterTest.AbstractClusterTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B script() Return the full path to the user script on the leader. .UNINDENT .INDENT 7.0 .TP .B data(filename) Return the full path to the data file with the given name on the leader. .UNINDENT .INDENT 7.0 .TP .B rsyncUtil(src, dest) .UNINDENT .INDENT 7.0 .TP .B getRootVolID() .UNINDENT .INDENT 7.0 .TP .B putScript(content) Helper method for _getScript to inject a script file at the configured script path, from text. .INDENT 7.0 .TP .B Parameters \fBcontent\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTest(name) Bases: \fI\%AbstractAWSAutoscaleTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .UNINDENT .INDENT 7.0 .TP .B launchCluster() .UNINDENT .INDENT 7.0 .TP .B getRootVolID() Adds in test to check that EBS volume is build with adequate size. Otherwise is functionally equivalent to parent. :return: volumeID .UNINDENT .INDENT 7.0 .TP .B testAutoScale() .UNINDENT .INDENT 7.0 .TP .B testSpotAutoScale() .UNINDENT .INDENT 7.0 .TP .B testSpotAutoScaleBalancingTypes() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AWSStaticAutoscaleTest(name) Bases: \fI\%AWSAutoscaleTest\fP .sp Runs the tests on a statically provisioned cluster with autoscaling enabled. .INDENT 7.0 .TP .B launchCluster() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AWSManagedAutoscaleTest(name) Bases: \fI\%AWSAutoscaleTest\fP .sp Runs the tests on a self\-scaling Kubernetes cluster. .INDENT 7.0 .TP .B launchCluster() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTestMultipleNodeTypes(name) Bases: \fI\%AbstractAWSAutoscaleTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .UNINDENT .INDENT 7.0 .TP .B testAutoScale() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.AWSRestartTest(name) Bases: \fI\%AbstractAWSAutoscaleTest\fP .sp This test insures autoscaling works on a restarted Toil run. .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .UNINDENT .INDENT 7.0 .TP .B testAutoScaledCluster() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.aws.awsProvisionerTest.PreemptibleDeficitCompensationTest(name) Bases: \fI\%AbstractAWSAutoscaleTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .UNINDENT .INDENT 7.0 .TP .B test() .UNINDENT .UNINDENT .SS Submodules .SS \fI\%toil.test.provisioners.clusterScalerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BinPackingTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%ClusterScalerTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%ScalerThreadTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%MockBatchSystemAndProvisioner\fP T} T{ Mimics a leader, job batcher, provisioner and scalable batch system. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%c4_8xlarge_preemptible\fP T} T{ T} _ T{ \fI\%c4_8xlarge\fP T} T{ T} _ T{ \fI\%r3_8xlarge\fP T} T{ T} _ T{ \fI\%r5_2xlarge\fP T} T{ T} _ T{ \fI\%r5_4xlarge\fP T} T{ T} _ T{ \fI\%t2_micro\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.c4_8xlarge_preemptible .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.c4_8xlarge .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.r3_8xlarge .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.r5_2xlarge .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.r5_4xlarge .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.clusterScalerTest.t2_micro .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.clusterScalerTest.BinPackingTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testPackingOneShape() Pack one shape and check that the resulting reservations look sane. .UNINDENT .INDENT 7.0 .TP .B testSorting() Test that sorting is correct: preemptible, then memory, then cores, then disk, then wallTime. .UNINDENT .INDENT 7.0 .TP .B testAddingInitialNode() Pack one shape when no nodes are available and confirm that we fit one node properly. .UNINDENT .INDENT 7.0 .TP .B testLowTargetTime() Test that a low targetTime (0) parallelizes jobs aggressively (1000 queued jobs require 1000 nodes). .sp Ideally, low targetTime means: Start quickly and maximize parallelization after the cpu/disk/mem have been packed. .sp Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can only run one job at a time with its resources. .sp Each job is parametrized to take 300 seconds, so (the minimum of) 1 of them should fit into each node\(aqs 0 second window, so we expect 1000 nodes. .UNINDENT .INDENT 7.0 .TP .B testHighTargetTime() Test that a high targetTime (3600 seconds) maximizes packing within the targetTime. .sp Ideally, high targetTime means: Maximize packing within the targetTime after the cpu/disk/mem have been packed. .sp Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can only run one job at a time with its resources. .sp Each job is parametrized to take 300 seconds, so 12 of them should fit into each node\(aqs 3600 second window. 1000/12 = 83.33, so we expect 84 nodes. .UNINDENT .INDENT 7.0 .TP .B testZeroResourceJobs() Test that jobs requiring zero cpu/disk/mem pack first, regardless of targetTime. .sp Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can run a seemingly infinite number of jobs with its resources. .sp Since all jobs should pack cpu/disk/mem\-wise on a t2.micro, we expect only one t2.micro to be provisioned. If we raise this, as in testLowTargetTime, it will launch 1000 t2.micros. .UNINDENT .INDENT 7.0 .TP .B testLongRunningJobs() Test that jobs with long run times (especially service jobs) are aggressively parallelized. .sp This is important, because services are one case where the degree of parallelization really, really matters. If you have multiple services, they may all need to be running simultaneously before any real work can be done. .sp Despite setting globalTargetTime=3600, this should launch 1000 t2.micros because each job\(aqs estimated runtime (30000 seconds) extends well beyond 3600 seconds. .UNINDENT .INDENT 7.0 .TP .B run1000JobsOnMicros(jobCores, jobMem, jobDisk, jobTime, globalTargetTime) Test packing 1000 jobs on t2.micros. Depending on the targetTime and resources, these should pack differently. .UNINDENT .INDENT 7.0 .TP .B testPathologicalCase() Test a pathological case where only one node can be requested to fit months\(aq worth of jobs. .sp If the reservation is extended to fit a long job, and the bin\-packer naively searches through all the reservation slices to find the first slice that fits, it will happily assign the first slot that fits the job, even if that slot occurs days in the future. .UNINDENT .INDENT 7.0 .TP .B testJobTooLargeForAllNodes() If a job is too large for all node types, the scaler should print a warning, but definitely not crash. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.clusterScalerTest.ClusterScalerTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testRounding() Test to make sure the ClusterScaler\(aqs rounding rounds properly. .UNINDENT .INDENT 7.0 .TP .B testMaxNodes() Set the scaler to be very aggressive, give it a ton of jobs, and make sure it doesn\(aqt go over maxNodes. .UNINDENT .INDENT 7.0 .TP .B testMinNodes() Without any jobs queued, the scaler should still estimate \(dqminNodes\(dq nodes. .UNINDENT .INDENT 7.0 .TP .B testPreemptibleDeficitResponse() When a preemptible deficit was detected by a previous run of the loop, the scaler should add non\-preemptible nodes to compensate in proportion to preemptibleCompensation. .UNINDENT .INDENT 7.0 .TP .B testPreemptibleDeficitIsSet() Make sure that updateClusterSize sets the preemptible deficit if it can\(aqt launch preemptible nodes properly. That way, the deficit can be communicated to the next run of estimateNodeCount. .UNINDENT .INDENT 7.0 .TP .B testNoLaunchingIfDeltaAlreadyMet() Check that the scaler doesn\(aqt try to launch \(dq0\(dq more instances if the delta was able to be met by unignoring nodes. .UNINDENT .INDENT 7.0 .TP .B testBetaInertia() .UNINDENT .INDENT 7.0 .TP .B test_overhead_accounting_large() If a node has a certain raw memory or disk capacity, that won\(aqt all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.). .sp Make sure this overhead is accounted for for large nodes. .UNINDENT .INDENT 7.0 .TP .B test_overhead_accounting_small() If a node has a certain raw memory or disk capacity, that won\(aqt all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.). .sp Make sure this overhead is accounted for for small nodes. .UNINDENT .INDENT 7.0 .TP .B test_overhead_accounting_observed() If a node has a certain raw memory or disk capacity, that won\(aqt all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.). .sp Make sure this overhead is accounted for so that real\-world observed failures cannot happen again. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.clusterScalerTest.ScalerThreadTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testClusterScaling() Test scaling for a batch of non\-preemptible jobs and no preemptible jobs (makes debugging easier). .UNINDENT .INDENT 7.0 .TP .B testClusterScalingMultipleNodeTypes() .UNINDENT .INDENT 7.0 .TP .B testClusterScalingWithPreemptibleJobs() Test scaling simultaneously for a batch of preemptible and non\-preemptible jobs. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.clusterScalerTest.MockBatchSystemAndProvisioner(config, secondsPerJob) Bases: \fI\%toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem\fP, \fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP .sp Mimics a leader, job batcher, provisioner and scalable batch system. .INDENT 7.0 .TP .B start() .UNINDENT .INDENT 7.0 .TP .B shutDown() .UNINDENT .INDENT 7.0 .TP .B nodeInUse(nodeIP) Can be used to determine if a worker node is running any tasks. If the node is doesn\(aqt exist, this function should simply return False. .INDENT 7.0 .TP .B Parameters \fBnodeIP\fP \-\- The worker nodes private IP address .TP .B Returns True if the worker node has been issued any tasks, else False .UNINDENT .UNINDENT .INDENT 7.0 .TP .B ignoreNode(nodeAddress) Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished. .INDENT 7.0 .TP .B Parameters \fBnodeAddress\fP \-\- IP address of node to ignore. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B unignoreNode(nodeAddress) Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one. .UNINDENT .INDENT 7.0 .TP .B supportedClusterTypes() Get all the cluster types that this provisioner implementation supports. .UNINDENT .INDENT 7.0 .TP .B createClusterSettings() Initialize class for a new cluster, to be deployed, when running outside the cloud. .UNINDENT .INDENT 7.0 .TP .B readClusterSettings() Initialize class from an existing cluster. This method assumes that the instance we are running on is the leader. .sp Implementations must call _setLeaderWorkerAuthentication(). .UNINDENT .INDENT 7.0 .TP .B setAutoscaledNodeTypes(node_types) Set node types, shapes and spot bids for Toil\-managed autoscaling. :param nodeTypes: A list of node types, as parsed with parse_node_types. .INDENT 7.0 .TP .B Parameters \fBnode_types\fP (\fIList\fP\fI[\fP\fITuple\fP\fI[\fP\fISet\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.Shape\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getProvisionedWorkers(instance_type=None, preemptible=None) Returns a list of Node objects, each representing a worker node in the cluster .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP \-\- If True only return preemptible nodes else return non\-preemptible nodes .TP .B Returns list of Node .UNINDENT .UNINDENT .INDENT 7.0 .TP .B terminateNodes(nodes) Terminate the nodes represented by given Node objects .INDENT 7.0 .TP .B Parameters \fBnodes\fP \-\- list of Node objects .UNINDENT .UNINDENT .INDENT 7.0 .TP .B remainingBillingInterval(node) .UNINDENT .INDENT 7.0 .TP .B addJob(jobShape, preemptible=False) Add a job to the job queue .UNINDENT .INDENT 7.0 .TP .B getNumberOfJobsIssued(preemptible=None) .UNINDENT .INDENT 7.0 .TP .B getJobs() .UNINDENT .INDENT 7.0 .TP .B getNodes(preemptible=False, timeout=600) Returns a dictionary mapping node identifiers of preemptible or non\-preemptible nodes to NodeInfo objects, one for each node. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- If True (False) only (non\-)preemptible nodes will be returned. If None, all nodes will be returned. .IP \(bu 2 \fBtimeout\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addNodes(nodeTypes, numNodes, preemptible) Used to add worker nodes to the cluster .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnumNodes\fP \-\- The number of nodes to add .IP \(bu 2 \fBpreemptible\fP \-\- whether or not the nodes will be preemptible .IP \(bu 2 \fBspotBid\fP \-\- The bid for preemptible nodes if applicable (this can be set in config, also). .IP \(bu 2 \fBnodeTypes\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Returns number of nodes successfully added .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getNodeShape(nodeType, preemptible=False) The shape of a preemptible or non\-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals. .INDENT 7.0 .TP .B Parameters \fBinstance_type\fP (\fI\%str\fP) \-\- Instance type name to return the shape of. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getWorkersInCluster(nodeShape) .UNINDENT .INDENT 7.0 .TP .B launchCluster(leaderNodeType, keyName, userTags=None, vpcSubnet=None, leaderStorage=50, nodeStorage=50, botoPath=None, **kwargs) Initialize a cluster and create a leader node. .sp Implementations must call _setLeaderWorkerAuthentication() with the leader so that workers can be launched. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBleaderNodeType\fP \-\- The leader instance. .IP \(bu 2 \fBleaderStorage\fP \-\- The amount of disk to allocate to the leader in gigabytes. .IP \(bu 2 \fBowner\fP \-\- Tag identifying the owner of the instances. .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B destroyCluster() Terminates all nodes in the specified cluster and cleans up all resources associated with the cluster. :param clusterName: identifier of the cluster to terminate. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLeader() .INDENT 7.0 .TP .B Returns The leader node. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getNumberOfNodes(nodeType=None, preemptible=None) .UNINDENT .UNINDENT .SS \fI\%toil.test.provisioners.clusterTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AbstractClusterTest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.provisioners.clusterTest.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.clusterTest.AbstractClusterTest(methodName) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B Parameters \fBmethodName\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B python() Return the full path to the venv Python on the leader. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pip() Return the full path to the venv pip on the leader. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B destroyCluster() Destroy the cluster we built, if it exists. .sp Succeeds if the cluster does not currently exist. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B sshUtil(command) Run the given command on the cluster. Raise subprocess.CalledProcessError if it fails. .INDENT 7.0 .TP .B Parameters \fBcommand\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rsync_util(from_file, to_file) Transfer a file to/from the cluster. .sp The cluster\-side path should have a \(aq:\(aq in front of it. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfrom_file\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBto_file\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B createClusterUtil(args=None) .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B launchCluster() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.provisioners.gceProvisionerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AbstractGCEAutoscaleTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%GCEAutoscaleTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%GCEStaticAutoscaleTest\fP T} T{ Runs the tests on a statically provisioned cluster with autoscaling enabled. T} _ T{ \fI\%GCEAutoscaleTestMultipleNodeTypes\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%GCERestartTest\fP T} T{ This test insures autoscaling works on a restarted Toil run T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.provisioners.gceProvisionerTest.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.gceProvisionerTest.AbstractGCEAutoscaleTest(methodName) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B projectID .UNINDENT .INDENT 7.0 .TP .B sshUtil(command) .UNINDENT .INDENT 7.0 .TP .B rsyncUtil(src, dest) .UNINDENT .INDENT 7.0 .TP .B destroyClusterUtil() .UNINDENT .INDENT 7.0 .TP .B createClusterUtil(args=None) .UNINDENT .INDENT 7.0 .TP .B cleanJobStoreUtil() .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B launchCluster() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.gceProvisionerTest.GCEAutoscaleTest(name) Bases: \fI\%AbstractGCEAutoscaleTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B launchCluster() .UNINDENT .INDENT 7.0 .TP .B testAutoScale() .UNINDENT .INDENT 7.0 .TP .B testSpotAutoScale() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.gceProvisionerTest.GCEStaticAutoscaleTest(name) Bases: \fI\%GCEAutoscaleTest\fP .sp Runs the tests on a statically provisioned cluster with autoscaling enabled. .INDENT 7.0 .TP .B launchCluster() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.gceProvisionerTest.GCEAutoscaleTestMultipleNodeTypes(name) Bases: \fI\%AbstractGCEAutoscaleTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testAutoScale() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.gceProvisionerTest.GCERestartTest(name) Bases: \fI\%AbstractGCEAutoscaleTest\fP .sp This test insures autoscaling works on a restarted Toil run .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testAutoScaledCluster() .UNINDENT .UNINDENT .SS \fI\%toil.test.provisioners.provisionerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ProvisionerTest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.provisioners.provisionerTest.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.provisioners.provisionerTest.ProvisionerTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test_node_type_parsing() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.provisioners.restartScript\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%f0\fP(job) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%parser\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.provisioners.restartScript.f0(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.provisioners.restartScript.parser .UNINDENT .SS \fI\%toil.test.server\fP .SS Submodules .SS \fI\%toil.test.server.serverTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilServerUtilsTest\fP T} T{ Tests for the utility functions used by the Toil server. T} _ T{ \fI\%hidden\fP T} T{ T} _ T{ \fI\%FileStateStoreTest\fP T} T{ Test file\-based state storage. T} _ T{ \fI\%FileStateStoreURLTest\fP T} T{ Test file\-based state storage using URLs instead of local paths. T} _ T{ \fI\%BucketUsingTest\fP T} T{ Base class for tests that need a bucket. T} _ T{ \fI\%AWSStateStoreTest\fP T} T{ Test AWS\-based state storage. T} _ T{ \fI\%AbstractToilWESServerTest\fP T} T{ Class for server tests that provides a self.app in testing mode. T} _ T{ \fI\%ToilWESServerBenchTest\fP T} T{ Tests for Toil\(aqs Workflow Execution Service API that don\(aqt run workflows. T} _ T{ \fI\%ToilWESServerWorkflowTest\fP T} T{ Tests of the WES server running workflows. T} _ T{ \fI\%ToilWESServerCeleryWorkflowTest\fP T} T{ End\-to\-end workflow\-running tests against Celery. T} _ T{ \fI\%ToilWESServerCeleryS3StateWorkflowTest\fP T} T{ Test the server with Celery and state stored in S3. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.server.serverTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.ToilServerUtilsTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests for the utility functions used by the Toil server. .INDENT 7.0 .TP .B test_workflow_canceling_recovery() Make sure that a workflow in CANCELING state will be recovered to a terminal state eventually even if the workflow runner Celery task goes away without flipping the state. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.hidden .INDENT 7.0 .TP .B class AbstractStateStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Basic tests for state stores. .INDENT 7.0 .TP .B abstract get_state_store() Make a state store to test, on a single fixed URL. .INDENT 7.0 .TP .B Return type \fI\%AbstractStateStore\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_state_store() Make sure that the state store under test can store and load keys. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.FileStateStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Test file\-based state storage. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_state_store() Make a state store to test, on a single fixed local path. .INDENT 7.0 .TP .B Return type \fI\%AbstractStateStore\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.FileStateStoreURLTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Test file\-based state storage using URLs instead of local paths. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_state_store() Make a state store to test, on a single fixed URL. .INDENT 7.0 .TP .B Return type \fI\%AbstractStateStore\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.BucketUsingTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Base class for tests that need a bucket. .INDENT 7.0 .TP .B region: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B s3_resource: mypy_boto3_s3.S3ServiceResource | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B bucket: mypy_boto3_s3.service_resource.Bucket | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B bucket_name: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Set up the class with a single pre\-existing AWS bucket for all tests. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod tearDownClass() Hook method for deconstructing the class fixture after running all tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.AWSStateStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP, \fI\%BucketUsingTest\fP .sp Test AWS\-based state storage. .INDENT 7.0 .TP .B bucket_path = \(aqprefix/of/keys\(aq .UNINDENT .INDENT 7.0 .TP .B get_state_store() Make a state store to test, on a single fixed URL. .INDENT 7.0 .TP .B Return type \fI\%AbstractStateStore\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_state_store_paths() Make sure that the S3 state store puts things in the right places. .sp We don\(aqt \fIreally\fP care about the exact internal structure, but we do care about actually being under the path we are supposed to use. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.AbstractToilWESServerTest(*args, **kwargs) Bases: \fI\%toil.test.ToilTest\fP .sp Class for server tests that provides a self.app in testing mode. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.ToilWESServerBenchTest(*args, **kwargs) Bases: \fI\%AbstractToilWESServerTest\fP .sp Tests for Toil\(aqs Workflow Execution Service API that don\(aqt run workflows. .INDENT 7.0 .TP .B test_home() Test the homepage endpoint. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_health() Test the health check endpoint. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_get_service_info() Test the GET /service\-info endpoint. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.ToilWESServerWorkflowTest(*args, **kwargs) Bases: \fI\%AbstractToilWESServerTest\fP .sp Tests of the WES server running workflows. .INDENT 7.0 .TP .B run_zip_workflow(zip_path, include_message=True, include_params=True) We have several zip file tests; this submits a zip file and makes sure it ran OK. .sp If include_message is set to False, don\(aqt send a \(dqmessage\(dq argument in workflow_params. If include_params is also set to False, don\(aqt send workflow_params at all. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBzip_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBinclude_message\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBinclude_params\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_relative_url_no_attachments_fails() Test run example CWL workflow from relative workflow URL but with no attachments. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_relative_url() Test run example CWL workflow from relative workflow URL. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_https_url() Test run example CWL workflow from the Internet. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_single_file_zip() Test run example CWL workflow from single\-file ZIP. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_multi_file_zip() Test run example CWL workflow from multi\-file ZIP. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_manifest_zip() Test run example CWL workflow from ZIP with manifest. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_inputs_zip() Test run example CWL workflow from ZIP without manifest but with inputs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_manifest_and_inputs_zip() Test run example CWL workflow from ZIP with manifest and inputs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_workflow_no_params_zip() Test run example CWL workflow from ZIP without workflow_params. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_run_and_cancel_workflows() Run two workflows, cancel one of them, and make sure they all exist. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.ToilWESServerCeleryWorkflowTest(*args, **kwargs) Bases: \fI\%ToilWESServerWorkflowTest\fP .sp End\-to\-end workflow\-running tests against Celery. .UNINDENT .INDENT 0.0 .TP .B class toil.test.server.serverTest.ToilWESServerCeleryS3StateWorkflowTest(*args, **kwargs) Bases: \fI\%ToilWESServerWorkflowTest\fP, \fI\%BucketUsingTest\fP .sp Test the server with Celery and state stored in S3. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.sort\fP .SS Submodules .SS \fI\%toil.test.sort.restart_sort\fP .sp A demonstration of toil. Sorts the lines of a file into ascending order by doing a parallel merge sort. This is an intentionally buggy version that doesn\(aqt include restart() for testing purposes. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%setup\fP(job, inputFile, N, downCheckpoints, options) T} T{ Sets up the sort. T} _ T{ \fI\%down\fP(job, inputFileStoreID, N, path, downCheckpoints, ...) T} T{ Input is a file, a subdivision size N, and a path in the hierarchy of jobs. T} _ T{ \fI\%up\fP(job, inputFileID1, inputFileID2, path, options[, ...]) T} T{ Merges the two files and places them in the output. T} _ T{ \fI\%sort\fP(file) T} T{ Sorts the given file. T} _ T{ \fI\%merge\fP(fileHandle1, fileHandle2, outputFileHandle) T} T{ Merges together two files maintaining sorted order. T} _ T{ \fI\%copySubRangeOfFile\fP(inputFile, fileStart, fileEnd) T} T{ Copies the range (in bytes) between fileStart and fileEnd to the given T} _ T{ \fI\%getMidPoint\fP(file, fileStart, fileEnd) T} T{ Finds the point in the file to split. T} _ T{ \fI\%makeFileToSort\fP(fileName[, lines, lineLen]) T} T{ T} _ T{ \fI\%main\fP([options]) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%defaultLines\fP T} T{ T} _ T{ \fI\%defaultLineLen\fP T} T{ T} _ T{ \fI\%sortMemory\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.sort.restart_sort.defaultLines = 1000 .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.defaultLineLen = 50 .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.sortMemory = \(aq600M\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.setup(job, inputFile, N, downCheckpoints, options) Sets up the sort. Returns the FileID of the sorted file .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory) Input is a file, a subdivision size N, and a path in the hierarchy of jobs. If the range is larger than a threshold N the range is divided recursively and a follow on job is then created which merges back the results else the file is sorted and placed in the output. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory) Merges the two files and places them in the output. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.sort(file) Sorts the given file. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.merge(fileHandle1, fileHandle2, outputFileHandle) Merges together two files maintaining sorted order. .sp All handles must be text\-mode streams. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.copySubRangeOfFile(inputFile, fileStart, fileEnd) Copies the range (in bytes) between fileStart and fileEnd to the given output file handle. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.getMidPoint(file, fileStart, fileEnd) Finds the point in the file to split. Returns an int i such that fileStart <= i < fileEnd .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.makeFileToSort(fileName, lines=defaultLines, lineLen=defaultLineLen) .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.restart_sort.main(options=None) .UNINDENT .SS \fI\%toil.test.sort.sort\fP .sp A demonstration of toil. Sorts the lines of a file into ascending order by doing a parallel merge sort. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%setup\fP(job, inputFile, N, downCheckpoints, options) T} T{ Sets up the sort. T} _ T{ \fI\%down\fP(job, inputFileStoreID, N, path, downCheckpoints, ...) T} T{ Input is a file, a subdivision size N, and a path in the hierarchy of jobs. T} _ T{ \fI\%up\fP(job, inputFileID1, inputFileID2, path, options[, ...]) T} T{ Merges the two files and places them in the output. T} _ T{ \fI\%sort\fP(file) T} T{ Sorts the given file. T} _ T{ \fI\%merge\fP(fileHandle1, fileHandle2, outputFileHandle) T} T{ Merges together two files maintaining sorted order. T} _ T{ \fI\%copySubRangeOfFile\fP(inputFile, fileStart, fileEnd) T} T{ Copies the range (in bytes) between fileStart and fileEnd to the given T} _ T{ \fI\%getMidPoint\fP(file, fileStart, fileEnd) T} T{ Finds the point in the file to split. T} _ T{ \fI\%makeFileToSort\fP(fileName[, lines, lineLen]) T} T{ T} _ T{ \fI\%main\fP([options]) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%defaultLines\fP T} T{ T} _ T{ \fI\%defaultLineLen\fP T} T{ T} _ T{ \fI\%sortMemory\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.sort.sort.defaultLines = 1000 .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.defaultLineLen = 50 .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.sortMemory = \(aq600M\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.setup(job, inputFile, N, downCheckpoints, options) Sets up the sort. Returns the FileID of the sorted file .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory) Input is a file, a subdivision size N, and a path in the hierarchy of jobs. If the range is larger than a threshold N the range is divided recursively and a follow on job is then created which merges back the results else the file is sorted and placed in the output. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory) Merges the two files and places them in the output. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.sort(file) Sorts the given file. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.merge(fileHandle1, fileHandle2, outputFileHandle) Merges together two files maintaining sorted order. .sp All handles must be text\-mode streams. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.copySubRangeOfFile(inputFile, fileStart, fileEnd) Copies the range (in bytes) between fileStart and fileEnd to the given output file handle. .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.getMidPoint(file, fileStart, fileEnd) Finds the point in the file to split. Returns an int i such that fileStart <= i < fileEnd .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.makeFileToSort(fileName, lines=defaultLines, lineLen=defaultLineLen) .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sort.main(options=None) .UNINDENT .SS \fI\%toil.test.sort.sortTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%SortTest\fP T} T{ Tests Toil by sorting a file in parallel on various combinations of job stores and batch T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%runMain\fP(options) T} T{ make sure the output file is deleted every time main is run T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%defaultLineLen\fP T} T{ T} _ T{ \fI\%defaultLines\fP T} T{ T} _ T{ \fI\%defaultN\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.sort.sortTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sortTest.defaultLineLen .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sortTest.defaultLines .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sortTest.defaultN .UNINDENT .INDENT 0.0 .TP .B toil.test.sort.sortTest.runMain(options) make sure the output file is deleted every time main is run .UNINDENT .INDENT 0.0 .TP .B class toil.test.sort.sortTest.SortTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP, \fI\%toil.batchSystems.mesos.test.MesosTestSupport\fP .sp Tests Toil by sorting a file in parallel on various combinations of job stores and batch systems. .INDENT 7.0 .TP .B testNo = 5 .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testAwsSingle() .UNINDENT .INDENT 7.0 .TP .B testAwsMesos() .UNINDENT .INDENT 7.0 .TP .B testFileMesos() .UNINDENT .INDENT 7.0 .TP .B testGoogleSingle() .UNINDENT .INDENT 7.0 .TP .B testGoogleMesos() .UNINDENT .INDENT 7.0 .TP .B testFileSingle() .UNINDENT .INDENT 7.0 .TP .B testFileSingleNonCaching() .UNINDENT .INDENT 7.0 .TP .B testFileSingleCheckpoints() .UNINDENT .INDENT 7.0 .TP .B testFileSingle10000() .UNINDENT .INDENT 7.0 .TP .B testFileGridEngine() .UNINDENT .INDENT 7.0 .TP .B testFileTorqueEngine() .UNINDENT .INDENT 7.0 .TP .B testSort() .UNINDENT .INDENT 7.0 .TP .B testMerge() .UNINDENT .INDENT 7.0 .TP .B testCopySubRangeOfFile() .UNINDENT .INDENT 7.0 .TP .B testGetMidPoint() .UNINDENT .UNINDENT .SS \fI\%toil.test.src\fP .SS Submodules .SS \fI\%toil.test.src.autoDeploymentTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%AutoDeploymentTest\fP T} T{ Tests various auto\-deployment scenarios. Using the appliance, i.e. a docker container, T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.autoDeploymentTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.autoDeploymentTest.AutoDeploymentTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ApplianceTestSupport\fP .sp Tests various auto\-deployment scenarios. Using the appliance, i.e. a docker container, for these tests allows for running worker processes on the same node as the leader process while keeping their file systems separate from each other and the leader process. Separate file systems are crucial to prove that auto\-deployment does its job. .INDENT 7.0 .TP .B sitePackages .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testRestart() Test whether auto\-deployment works on restart. .UNINDENT .INDENT 7.0 .TP .B testSplitRootPackages() Test whether auto\-deployment works with a virtualenv in which jobs are defined in completely separate branches of the package hierarchy. Initially, auto\-deployment did deploy the entire virtualenv but jobs could only be defined in one branch of the package hierarchy. We define a branch as the maximum set of fully qualified package paths that share the same first component. IOW, a.b and a.c are in the same branch, while a.b and d.c are not. .UNINDENT .INDENT 7.0 .TP .B testUserTypesInJobFunctionArgs() Test encapsulated, function\-wrapping jobs where the function arguments reference user\-defined types. .sp Mainly written to cover \fI\%https://github.com/BD2KGenomics/toil/issues/1259\fP but then also revealed \fI\%https://github.com/BD2KGenomics/toil/issues/1278\fP\&. .UNINDENT .INDENT 7.0 .TP .B testDeferralWithConcurrentEncapsulation() Ensure that the following DAG succeeds: .INDENT 7.0 .INDENT 3.5 .sp .EX ┌───────────┐ │ Root (W1) │ └───────────┘ │ ┌──────────┴─────────┐ ▼ ▼ ┌────────────────┐ ┌────────────────────┐ │ Deferring (W2) │ │ Encapsulating (W3) │═══════════════╗ └────────────────┘ └────────────────────┘ ║ │ ║ ▼ ▼ ┌───────────────────┐ ┌────────────────┐ │ Encapsulated (W3) │ │ Follow\-on (W6) │ └───────────────────┘ └────────────────┘ │ │ ┌───────┴────────┐ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Dummy 1 (W4) │ │ Dummy 2 (W5) │ │ Last (W6) │ └──────────────┘ └──────────────┘ └──────────────┘ .EE .UNINDENT .UNINDENT .sp The Wn numbers denote the worker processes that a particular job is run in. \fIDeferring\fP adds a deferred function and then runs for a long time. The deferred function will be present in the cache state for the duration of \fIDeferred\fP\&. \fIFollow\-on\fP is the generic Job instance that\(aqs added by encapsulating a job. It runs on the same worker node but in a separate worker process, as the first job in that worker. Because … .sp 1) it is the first job in its worker process (the user script has not been made available on the sys.path by a previous job in that worker) and .INDENT 7.0 .IP 2. 3 it shares the cache state with the \fIDeferring\fP job and .IP 3. 3 it is an instance of Job (and so does not introduce the user script to sys.path itself), .UNINDENT .sp … it might cause problems with deserializing a defered function defined in the user script. .sp \fIEncapsulated\fP has two children to ensure that \fIFollow\-on\fP is run in a separate worker. .UNINDENT .INDENT 7.0 .TP .B testDeferralWithFailureAndEncapsulation() Ensure that the following DAG succeeds: .INDENT 7.0 .INDENT 3.5 .sp .EX ┌───────────┐ │ Root (W1) │ └───────────┘ │ ┌──────────┴─────────┐ ▼ ▼ ┌────────────────┐ ┌────────────────────┐ │ Deferring (W2) │ │ Encapsulating (W3) │═══════════════════════╗ └────────────────┘ └────────────────────┘ ║ │ ║ ▼ ▼ ┌───────────────────┐ ┌────────────────┐ │ Encapsulated (W3) │════════════╗ │ Follow\-on (W7) │ └───────────────────┘ ║ └────────────────┘ │ ║ ┌──────┴──────┐ ║ ▼ ▼ ▼ ┌────────────┐┌────────────┐ ┌──────────────┐ │ Dummy (W4) ││ Dummy (W5) │ │ Trigger (W6) │ └────────────┘└────────────┘ └──────────────┘ .EE .UNINDENT .UNINDENT .sp \fITrigger\fP causes \fIDeferring\fP to crash. \fIFollow\-on\fP runs next, detects \fIDeferring\fP\(aqs left\-overs and runs the deferred function. \fIFollow\-on\fP is an instance of \fIJob\fP and the first job in its worker process. This test ensures that despite these circumstances, the user script is loaded before the deferred functions defined in it are being run. .sp \fIEncapsulated\fP has two children to ensure that \fIFollow\-on\fP is run in a new worker. That\(aqs the only way to guarantee that the user script has not been loaded yet, which would cause the test to succeed coincidentally. We want to test that auto\-deploying and loading of the user script are done properly \fIbefore\fP deferred functions are being run and before any jobs have been executed by that worker. .UNINDENT .UNINDENT .SS \fI\%toil.test.src.busTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%MessageBusTest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%failing_job_fn\fP(job) T} T{ This function is guaranteed to fail. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.busTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.busTest.MessageBusTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test_enum_ints_in_file() Make sure writing bus messages to files works with enums. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_cross_thread_messaging() Make sure message bus works across threads. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_restart_without_bus_path() Test the ability to restart a workflow when the message bus path used by the previous attempt is gone. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.busTest.failing_job_fn(job) This function is guaranteed to fail. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%toil.job.Job\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.test.src.checkpointTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%CheckpointTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%CheckRetryCount\fP T} T{ Fail N times, succeed on the next try. T} _ T{ \fI\%AlwaysFail\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%CheckpointFailsFirstTime\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%FailOnce\fP T} T{ Fail the first time the workflow is run, but succeed thereafter. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.checkpointTest.CheckpointTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testCheckpointNotRetried() A checkpoint job should not be retried if the workflow has a retryCount of 0. .UNINDENT .INDENT 7.0 .TP .B testCheckpointRetriedOnce() A checkpoint job should be retried exactly once if the workflow has a retryCount of 1. .UNINDENT .INDENT 7.0 .TP .B testCheckpointedRestartSucceeds() A checkpointed job should succeed on restart of a failed run if its child job succeeds. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.checkpointTest.CheckRetryCount(numFailuresBeforeSuccess) Bases: \fI\%toil.job.Job\fP .sp Fail N times, succeed on the next try. .INDENT 7.0 .TP .B getNumRetries(fileStore) Mark a retry in the fileStore, and return the number of retries so far. .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.checkpointTest.AlwaysFail(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName=\(aq\(aq, checkpoint=False, displayName=\(aq\(aq, descriptionClass=None, local=None) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmemory\fP (\fIOptional\fP\fI[\fP\fIParseableIndivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBcores\fP (\fIOptional\fP\fI[\fP\fIParseableDivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fIOptional\fP\fI[\fP\fIParseableIndivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBaccelerators\fP (\fIOptional\fP\fI[\fP\fIParseableAcceleratorRequirement\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fIParseableFlag\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptable\fP (\fIOptional\fP\fI[\fP\fIParseableFlag\fP\fI]\fP) \-\- .IP \(bu 2 \fBunitName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcheckpoint\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisplayName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdescriptionClass\fP (\fIOptional\fP\fI[\fP\fI\%type\fP\fI]\fP) \-\- .IP \(bu 2 \fBlocal\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.checkpointTest.CheckpointFailsFirstTime Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.checkpointTest.FailOnce(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName=\(aq\(aq, checkpoint=False, displayName=\(aq\(aq, descriptionClass=None, local=None) Bases: \fI\%toil.job.Job\fP .sp Fail the first time the workflow is run, but succeed thereafter. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmemory\fP (\fIOptional\fP\fI[\fP\fIParseableIndivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBcores\fP (\fIOptional\fP\fI[\fP\fIParseableDivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fIOptional\fP\fI[\fP\fIParseableIndivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBaccelerators\fP (\fIOptional\fP\fI[\fP\fIParseableAcceleratorRequirement\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fIParseableFlag\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptable\fP (\fIOptional\fP\fI[\fP\fIParseableFlag\fP\fI]\fP) \-\- .IP \(bu 2 \fBunitName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcheckpoint\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisplayName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdescriptionClass\fP (\fIOptional\fP\fI[\fP\fI\%type\fP\fI]\fP) \-\- .IP \(bu 2 \fBlocal\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.deferredFunctionTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%DeferredFunctionTest\fP T} T{ Test the deferred function system. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.deferredFunctionTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.deferredFunctionTest.DeferredFunctionTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test the deferred function system. .INDENT 7.0 .TP .B jobStoreType = \(aqfile\(aq .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testDeferredFunctionRunsWithMethod() Refer docstring in _testDeferredFunctionRuns. Test with Method .UNINDENT .INDENT 7.0 .TP .B testDeferredFunctionRunsWithClassMethod() Refer docstring in _testDeferredFunctionRuns. Test with Class Method .UNINDENT .INDENT 7.0 .TP .B testDeferredFunctionRunsWithLambda() Refer docstring in _testDeferredFunctionRuns. Test with Lambda .UNINDENT .INDENT 7.0 .TP .B testDeferredFunctionRunsWithFailures() Create 2 non local filesto use as flags. Create a job that registers a function that deletes one non local file. If that file exists, the job SIGKILLs itself. If it doesn\(aqt exist, the job registers a second deferred function to delete the second non local file and exits normally. .sp Initially the first file exists, so the job should SIGKILL itself and neither deferred function will run (in fact, the second should not even be registered). On the restart, the first deferred function should run and the first file should not exist, but the second one should. We assert the presence of the second, then register the second deferred function and exit normally. At the end of the test, neither file should exist. .sp Incidentally, this also tests for multiple registered deferred functions, and the case where a deferred function fails (since the first file doesn\(aqt exist on the retry). .UNINDENT .INDENT 7.0 .TP .B testNewJobsCanHandleOtherJobDeaths() Create 2 non\-local files and then create 2 jobs. The first job registers a deferred job to delete the second non\-local file, deletes the first non\-local file and then kills itself. The second job waits for the first file to be deleted, then sleeps for a few seconds and then spawns a child. the child of the second does nothing. However starting it should handle the untimely demise of the first job and run the registered deferred function that deletes the first file. We assert the absence of the two files at the end of the run. .UNINDENT .INDENT 7.0 .TP .B testBatchSystemCleanupCanHandleWorkerDeaths() Create some non\-local files. Create a job that registers a deferred function to delete the file and then kills its worker. .sp Assert that the file is missing after the pipeline fails, because we\(aqre using a single\-machine batch system and the leader\(aqs batch system cleanup will find and run the deferred function. .UNINDENT .UNINDENT .SS \fI\%toil.test.src.dockerCheckTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%DockerCheckTest\fP T} T{ Tests checking whether a docker image exists or not. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.dockerCheckTest.DockerCheckTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests checking whether a docker image exists or not. .INDENT 7.0 .TP .B testOfficialUbuntuRepo() Image exists. This should pass. .UNINDENT .INDENT 7.0 .TP .B testBroadDockerRepo() Image exists. This should pass. .UNINDENT .INDENT 7.0 .TP .B testBroadDockerRepoBadTag() Bad tag. This should raise. .UNINDENT .INDENT 7.0 .TP .B testNonexistentRepo() Bad image. This should raise. .UNINDENT .INDENT 7.0 .TP .B testToilQuayRepo() Image exists. Should pass. .UNINDENT .INDENT 7.0 .TP .B testBadQuayRepoNTag() Bad repo and tag. This should raise. .UNINDENT .INDENT 7.0 .TP .B testBadQuayRepo() Bad repo. This should raise. .UNINDENT .INDENT 7.0 .TP .B testBadQuayTag() Bad tag. This should raise. .UNINDENT .INDENT 7.0 .TP .B testGoogleRepo() Image exists. Should pass. .UNINDENT .INDENT 7.0 .TP .B testBadGoogleRepo() Bad repo and tag. This should raise. .UNINDENT .INDENT 7.0 .TP .B testApplianceParser() Test that a specified appliance is parsed correctly. .UNINDENT .UNINDENT .SS \fI\%toil.test.src.fileStoreTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%hidden\fP T} T{ Hiding the abstract test classes from the Unittest loader so it can be inherited in different T} _ T{ \fI\%NonCachingFileStoreTestWithFileJobStore\fP T} T{ Abstract tests for the the various functions in T} _ T{ \fI\%CachingFileStoreTestWithFileJobStore\fP T} T{ Abstract tests for the the various cache\-related functions in T} _ T{ \fI\%NonCachingFileStoreTestWithAwsJobStore\fP T} T{ Abstract tests for the the various functions in T} _ T{ \fI\%CachingFileStoreTestWithAwsJobStore\fP T} T{ Abstract tests for the the various cache\-related functions in T} _ T{ \fI\%NonCachingFileStoreTestWithGoogleJobStore\fP T} T{ Abstract tests for the the various functions in T} _ T{ \fI\%CachingFileStoreTestWithGoogleJobStore\fP T} T{ Abstract tests for the the various cache\-related functions in T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%testingIsAutomatic\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.fileStoreTest.testingIsAutomatic = True .UNINDENT .INDENT 0.0 .TP .B toil.test.src.fileStoreTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.hidden Hiding the abstract test classes from the Unittest loader so it can be inherited in different test suites for the different job stores. .INDENT 7.0 .TP .B class AbstractFileStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp An abstract base class for testing the various general functions described in :class:toil.fileStores.abstractFileStore.AbstractFileStore .INDENT 7.0 .TP .B jobStoreType .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B create_file(content, executable=False) .UNINDENT .INDENT 7.0 .TP .B testToilIsNotBroken() Runs a simple DAG to test if if any features other that caching were broken. .UNINDENT .INDENT 7.0 .TP .B testFileStoreLogging() Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. .UNINDENT .INDENT 7.0 .TP .B testFileStoreOperations() Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. .UNINDENT .INDENT 7.0 .TP .B testWriteReadGlobalFilePermissions() Ensures that uploaded files preserve their file permissions when they are downloaded again. This function checks that a written executable file maintains its executability after being read. .UNINDENT .INDENT 7.0 .TP .B testWriteExportFileCompatibility() Ensures that files created in a job preserve their executable permissions when they are exported from the leader. .UNINDENT .INDENT 7.0 .TP .B testImportReadFileCompatibility() Ensures that files imported to the leader preserve their executable permissions when they are read by the fileStore. .UNINDENT .INDENT 7.0 .TP .B testReadWriteFileStreamTextMode() Checks if text mode is compatible with file streams. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class AbstractNonCachingFileStoreTest(methodName=\(aqrunTest\(aq) Bases: \fBAbstractFileStoreTest\fP .sp Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class AbstractCachingFileStoreTest(methodName=\(aqrunTest\(aq) Bases: \fBAbstractFileStoreTest\fP .sp Abstract tests for the the various cache\-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testExtremeCacheSetup() Try to create the cache with bad worker active and then have 10 child jobs try to run in the chain. This tests whether the cache is created properly even when the job crashes randomly. .UNINDENT .INDENT 7.0 .TP .B testCacheEvictionPartialEvict() Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected. .UNINDENT .INDENT 7.0 .TP .B testCacheEvictionTotalEvict() Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected. .UNINDENT .INDENT 7.0 .TP .B testCacheEvictionFailCase() Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected. .UNINDENT .INDENT 7.0 .TP .B testAsyncWriteWithCaching() Ensure the Async Writing of files happens as expected. The first Job forcefully modifies the cache size to 1GB. The second asks for 1GB of disk and writes a 900MB file into cache then rewrites it to the job store triggering an async write since the two unique jobstore IDs point to the same local file. Also, the second write is not cached since the first was written to cache, and there \(dqisn\(aqt enough space\(dq to cache the second. Imediately assert that the second write isn\(aqt cached, and is being asynchronously written to the job store. .sp Attempting to get the file from the jobstore should not fail. .UNINDENT .INDENT 7.0 .TP .B testWriteNonLocalFileToJobStore() Write a file not in localTempDir to the job store. Such a file should not be cached. Ensure the file is not cached. .UNINDENT .INDENT 7.0 .TP .B testWriteLocalFileToJobStore() Write a file from the localTempDir to the job store. Such a file will be cached by default. Ensure the file is cached. .UNINDENT .INDENT 7.0 .TP .B testReadCacheMissFileFromJobStoreWithoutCachingReadFile() Read a file from the file store that does not have a corresponding cached copy. Do not cache the read file. Ensure the number of links on the file are appropriate. .UNINDENT .INDENT 7.0 .TP .B testReadCacheMissFileFromJobStoreWithCachingReadFile() Read a file from the file store that does not have a corresponding cached copy. Cache the read file. Ensure the number of links on the file are appropriate. .UNINDENT .INDENT 7.0 .TP .B testReadCachHitFileFromJobStore() Read a file from the file store that has a corresponding cached copy. Ensure the number of links on the file are appropriate. .UNINDENT .INDENT 7.0 .TP .B testMultipleJobsReadSameCacheHitGlobalFile() Write a local file to the job store (hence adding a copy to cache), then have 10 jobs read it. Assert cached file size never goes up, assert unused job required disk space is always \fB(a multiple of job reqs) \- (number of current file readers * filesize)\fP\&. At the end, assert the cache shows unused job\-required space = 0. .UNINDENT .INDENT 7.0 .TP .B testMultipleJobsReadSameCacheMissGlobalFile() Write a non\-local file to the job store(hence no cached copy), then have 10 jobs read it. Assert cached file size never goes up, assert unused job required disk space is always \fB(a multiple of job reqs) \- (number of current file readers * filesize)\fP\&. At the end, assert the cache shows unused job\-required space = 0. .UNINDENT .INDENT 7.0 .TP .B testFileStoreExportFile() .UNINDENT .INDENT 7.0 .TP .B testReturnFileSizes() Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. Ensure that after every step that the cache is in a valid state. .UNINDENT .INDENT 7.0 .TP .B testReturnFileSizesWithBadWorker() Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. Ensure that after every step that the cache is in a valid state. .UNINDENT .INDENT 7.0 .TP .B testControlledFailedWorkerRetry() Conduct a couple of job store operations. Then die. Ensure that the restarted job is tracking values in the cache state file appropriately. .UNINDENT .INDENT 7.0 .TP .B testRemoveLocalMutablyReadFile() If a mutably read file is deleted by the user, it is ok. .UNINDENT .INDENT 7.0 .TP .B testRemoveLocalImmutablyReadFile() If an immutably read file is deleted by the user, it is not ok. .UNINDENT .INDENT 7.0 .TP .B testDeleteLocalFile() Test the deletion capabilities of deleteLocalFile .UNINDENT .INDENT 7.0 .TP .B testSimultaneousReadsUncachedStream() Test many simultaneous read attempts on a file created via a stream directly to the job store. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore. .INDENT 7.0 .TP .B jobStoreType = \(aqfile\(aq .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Abstract tests for the the various cache\-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. .INDENT 7.0 .TP .B jobStoreType = \(aqfile\(aq .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithAwsJobStore(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore. .INDENT 7.0 .TP .B jobStoreType = \(aqaws\(aq .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.CachingFileStoreTestWithAwsJobStore(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Abstract tests for the the various cache\-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. .INDENT 7.0 .TP .B jobStoreType = \(aqaws\(aq .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithGoogleJobStore(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore. .INDENT 7.0 .TP .B jobStoreType = \(aqgoogle\(aq .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.fileStoreTest.CachingFileStoreTestWithGoogleJobStore(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Abstract tests for the the various cache\-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore. .INDENT 7.0 .TP .B jobStoreType = \(aqgoogle\(aq .UNINDENT .UNINDENT .SS \fI\%toil.test.src.helloWorldTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%HelloWorldTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%HelloWorld\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%FollowOn\fP T} T{ Class represents a unit of work in toil. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%childFn\fP(job) T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.helloWorldTest.HelloWorldTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testHelloWorld() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.helloWorldTest.HelloWorld Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.helloWorldTest.childFn(job) .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.helloWorldTest.FollowOn(fileId) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.importExportFileTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ImportExportFileTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%RestartingJob\fP T} T{ Class represents a unit of work in toil. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.importExportFileTest.ImportExportFileTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B create_file(content, executable=False) .UNINDENT .INDENT 7.0 .TP .B test_import_export_restart_true() .UNINDENT .INDENT 7.0 .TP .B test_import_export_restart_false() .UNINDENT .INDENT 7.0 .TP .B test_basic_import_export() Ensures that uploaded files preserve their file permissions when they are downloaded again. This function checks that an imported executable file maintains its executability after being exported. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.importExportFileTest.RestartingJob(msg_portion_file_id, trigger_file_id, message_portion_2) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(file_store) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.jobDescriptionTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%JobDescriptionTest\fP T} T{ A common base class for Toil tests. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.jobDescriptionTest.JobDescriptionTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testJobDescription() Tests the public interface of a JobDescription. .UNINDENT .INDENT 7.0 .TP .B testJobDescriptionSequencing() .UNINDENT .UNINDENT .SS \fI\%toil.test.src.jobEncapsulationTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%JobEncapsulationTest\fP T} T{ Tests testing the EncapsulationJob class. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%noOp\fP() T} T{ T} _ T{ \fI\%encapsulatedJobFn\fP(job, string, outFile) T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.jobEncapsulationTest.JobEncapsulationTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests testing the EncapsulationJob class. .INDENT 7.0 .TP .B testEncapsulation() Tests the Job.encapsulation method, which uses the EncapsulationJob class. .UNINDENT .INDENT 7.0 .TP .B testAddChildEncapsulate() Make sure that the encapsulate child does not have two parents with unique roots. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobEncapsulationTest.noOp() .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobEncapsulationTest.encapsulatedJobFn(job, string, outFile) .UNINDENT .SS \fI\%toil.test.src.jobFileStoreTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%JobFileStoreTest\fP T} T{ Tests testing the methods defined in :class:toil.fileStores.abstractFileStore.AbstractFileStore. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%fileTestJob\fP(job, inputFileStoreIDs, testStrings, ...) T} T{ Test job exercises toil.fileStores.abstractFileStore.AbstractFileStore functions T} _ T{ \fI\%simpleFileStoreJob\fP(job) T} T{ T} _ T{ \fI\%fileStoreChild\fP(job, testID1, testID2) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%PREFIX_LENGTH\fP T} T{ T} _ T{ \fI\%fileStoreString\fP T} T{ T} _ T{ \fI\%streamingFileStoreString\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.PREFIX_LENGTH = 200 .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobFileStoreTest.JobFileStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests testing the methods defined in :class:toil.fileStores.abstractFileStore.AbstractFileStore. .INDENT 7.0 .TP .B testCachingFileStore() .UNINDENT .INDENT 7.0 .TP .B testNonCachingFileStore() .UNINDENT .INDENT 7.0 .TP .B testJobFileStore() Tests case that about half the files are cached .UNINDENT .INDENT 7.0 .TP .B testJobFileStoreWithBadWorker() Tests case that about half the files are cached and the worker is randomly failing. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.fileTestJob(job, inputFileStoreIDs, testStrings, chainLength) Test job exercises toil.fileStores.abstractFileStore.AbstractFileStore functions .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.fileStoreString = \(aqTesting writeGlobalFile\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.streamingFileStoreString = \(aqTesting writeGlobalFileStream\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.simpleFileStoreJob(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobFileStoreTest.fileStoreChild(job, testID1, testID2) .UNINDENT .SS \fI\%toil.test.src.jobServiceTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%JobServiceTest\fP T} T{ Tests testing the Job.Service class T} _ T{ \fI\%PerfectServiceTest\fP T} T{ Tests testing the Job.Service class T} _ T{ \fI\%ToyService\fP T} T{ Abstract class used to define the interface to a service. T} _ T{ \fI\%ToySerializableService\fP T} T{ Abstract class used to define the interface to a service. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%serviceTest\fP(job, outFile, messageInt) T} T{ Creates one service and one accessing job, which communicate with two files to establish T} _ T{ \fI\%serviceTestRecursive\fP(job, outFile, messages) T} T{ Creates a chain of services and accessing jobs, each paired together. T} _ T{ \fI\%serviceTestParallelRecursive\fP(job, outFiles, messageBundles) T} T{ Creates multiple chains of services and accessing jobs. T} _ T{ \fI\%serviceAccessor\fP(job, communicationFiles, outFile, randInt) T} T{ Writes a random integer iinto the inJobStoreFileID file, then tries 10 times reading T} _ T{ \fI\%fnTest\fP(strings, outputFile) T} T{ Function concatenates the strings together and writes them to the output file T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.jobServiceTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobServiceTest.JobServiceTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests testing the Job.Service class .INDENT 7.0 .TP .B testServiceSerialization() Tests that a service can receive a promise without producing a serialization error. .UNINDENT .INDENT 7.0 .TP .B testService(checkpoint=False) Tests the creation of a Job.Service with random failures of the worker. .UNINDENT .INDENT 7.0 .TP .B testServiceDeadlock() Creates a job with more services than maxServices, checks that deadlock is detected. .UNINDENT .INDENT 7.0 .TP .B testServiceWithCheckpoints() Tests the creation of a Job.Service with random failures of the worker, making the root job use checkpointing to restart the subtree. .UNINDENT .INDENT 7.0 .TP .B testServiceRecursive(checkpoint=True) Tests the creation of a Job.Service, creating a chain of services and accessing jobs. Randomly fails the worker. .UNINDENT .INDENT 7.0 .TP .B testServiceParallelRecursive(checkpoint=True) Tests the creation of a Job.Service, creating parallel chains of services and accessing jobs. Randomly fails the worker. .UNINDENT .INDENT 7.0 .TP .B runToil(rootJob, retryCount=1, badWorker=0.5, badWorkedFailInterval=0.1, maxServiceJobs=sys.maxsize, deadlockWait=60) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobServiceTest.PerfectServiceTest(methodName=\(aqrunTest\(aq) Bases: \fI\%JobServiceTest\fP .sp Tests testing the Job.Service class .INDENT 7.0 .TP .B runToil(rootJob, retryCount=1, badWorker=0, badWorkedFailInterval=1000, maxServiceJobs=sys.maxsize, deadlockWait=60) Let us run all the tests in the other service test class, but without worker failures. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobServiceTest.serviceTest(job, outFile, messageInt) Creates one service and one accessing job, which communicate with two files to establish that both run concurrently. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobServiceTest.serviceTestRecursive(job, outFile, messages) Creates a chain of services and accessing jobs, each paired together. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobServiceTest.serviceTestParallelRecursive(job, outFiles, messageBundles) Creates multiple chains of services and accessing jobs. .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobServiceTest.ToyService(messageInt, *args, **kwargs) Bases: \fI\%toil.job.Job.Service\fP .sp Abstract class used to define the interface to a service. .sp Should be subclassed by the user to define services. .sp Is not executed as a job; runs within a ServiceHostJob. .INDENT 7.0 .TP .B start(job) Start the service. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Returns An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see \fI\%toil.job.Job.addService()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stop(job) Stops the service. Function can block until complete. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Checks the service is still running. .INDENT 7.0 .TP .B Raises \fBexceptions.RuntimeError\fP \-\- If the service failed, this will cause the service job to be labeled failed. .TP .B Returns True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static serviceWorker(jobStore, terminate, error, inJobStoreID, outJobStoreID, messageInt) .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobServiceTest.serviceAccessor(job, communicationFiles, outFile, randInt) Writes a random integer iinto the inJobStoreFileID file, then tries 10 times reading from outJobStoreFileID to get a pair of integers, the first equal to i the second written into the outputFile. .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobServiceTest.ToySerializableService(messageInt, *args, **kwargs) Bases: \fI\%toil.job.Job.Service\fP .sp Abstract class used to define the interface to a service. .sp Should be subclassed by the user to define services. .sp Is not executed as a job; runs within a ServiceHostJob. .INDENT 7.0 .TP .B start(job) Start the service. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Returns An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see \fI\%toil.job.Job.addService()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stop(job) Stops the service. Function can block until complete. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Checks the service is still running. .INDENT 7.0 .TP .B Raises \fBexceptions.RuntimeError\fP \-\- If the service failed, this will cause the service job to be labeled failed. .TP .B Returns True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobServiceTest.fnTest(strings, outputFile) Function concatenates the strings together and writes them to the output file .UNINDENT .SS \fI\%toil.test.src.jobTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%JobTest\fP T} T{ Tests the job class. T} _ T{ \fI\%TrivialService\fP T} T{ Abstract class used to define the interface to a service. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%simpleJobFn\fP(job, value) T} T{ T} _ T{ \fI\%fn1Test\fP(string, outputFile) T} T{ Function appends the next character after the last character in the given T} _ T{ \fI\%fn2Test\fP(pStrings, s, outputFile) T} T{ Function concatenates the strings in pStrings and s, in that order, and writes the result to T} _ T{ \fI\%trivialParent\fP(job) T} T{ T} _ T{ \fI\%parent\fP(job) T} T{ T} _ T{ \fI\%diamond\fP(job) T} T{ T} _ T{ \fI\%child\fP(job) T} T{ T} _ T{ \fI\%errorChild\fP(job) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.jobTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobTest.JobTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests the job class. .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .UNINDENT .INDENT 7.0 .TP .B testStatic() Create a DAG of jobs non\-dynamically and run it. DAG is: .INDENT 7.0 .INDENT 3.5 .sp .EX A \-> F \e\-\-\-\-\-\-\- B \-> D \e \e \e \-\-\-\-\-\-\- C \-> E .EE .UNINDENT .UNINDENT .sp Follow on is marked by \fB\->\fP .UNINDENT .INDENT 7.0 .TP .B testStatic2() Create a DAG of jobs non\-dynamically and run it. DAG is: .INDENT 7.0 .INDENT 3.5 .sp .EX A \-> F \e\-\-\-\-\-\-\- B \-> D \e \e \e \-\-\-\-\-\-\- C \-> E .EE .UNINDENT .UNINDENT .sp Follow on is marked by \fB\->\fP .UNINDENT .INDENT 7.0 .TP .B testTrivialDAGConsistency() .UNINDENT .INDENT 7.0 .TP .B testDAGConsistency() .UNINDENT .INDENT 7.0 .TP .B testSiblingDAGConsistency() Slightly more complex case. The stranded job\(aqs predecessors are siblings instead of parent/child. .UNINDENT .INDENT 7.0 .TP .B testDeadlockDetection() Randomly generate job graphs with various types of cycle in them and check they cause an exception properly. Also check that multiple roots causes a deadlock exception. .UNINDENT .INDENT 7.0 .TP .B testNewCheckpointIsLeafVertexNonRootCase() Test for issue #1465: Detection of checkpoint jobs that are not leaf vertices identifies leaf vertices incorrectly .sp Test verification of new checkpoint jobs being leaf verticies, starting with the following baseline workflow: .INDENT 7.0 .INDENT 3.5 .sp .EX Parent | Child # Checkpoint=True .EE .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B testNewCheckpointIsLeafVertexRootCase() .INDENT 7.0 .TP .B Test for issue #1466: Detection of checkpoint jobs that are not leaf vertices omits the workflow root job .UNINDENT .sp Test verification of a new checkpoint job being leaf vertex, starting with a baseline workflow of a single, root job: .INDENT 7.0 .INDENT 3.5 .sp .EX Root # Checkpoint=True .EE .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B runNewCheckpointIsLeafVertexTest(createWorkflowFn) Test verification that a checkpoint job is a leaf vertex using both valid and invalid cases. .INDENT 7.0 .TP .B Parameters \fBcreateWorkflowFn\fP \-\- .sp function to create and new workflow and return a tuple of: .INDENT 7.0 .IP 0. 4 the workflow root job .IP 1. 4 a checkpoint job to test within the workflow .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B runCheckpointVertexTest(workflowRootJob, checkpointJob, checkpointJobService=None, checkpointJobChild=None, checkpointJobFollowOn=None, expectedException=None) Modifies the checkpoint job according to the given parameters then runs the workflow, checking for the expected exception, if any. .UNINDENT .INDENT 7.0 .TP .B testEvaluatingRandomDAG() Randomly generate test input then check that the job graph can be run successfully, using the existence of promises to validate the run. .UNINDENT .INDENT 7.0 .TP .B static getRandomEdge(nodeNumber) .UNINDENT .INDENT 7.0 .TP .B static makeRandomDAG(nodeNumber) Makes a random dag with \(dqnodeNumber\(dq nodes in which all nodes are connected. Return value is list of edges, each of form (a, b), where a and b are integers >= 0 < nodeNumber referring to nodes and the edge is from a to b. .UNINDENT .INDENT 7.0 .TP .B static getAdjacencyList(nodeNumber, edges) Make adjacency list representation of edges .UNINDENT .INDENT 7.0 .TP .B reachable(node, adjacencyList, followOnAdjacencyList=None) Find the set of nodes reachable from this node (including the node). Return is a set of integers. .UNINDENT .INDENT 7.0 .TP .B addRandomFollowOnEdges(childAdjacencyList) Adds random follow on edges to the graph, represented as an adjacency list. The follow on edges are returned as a set and their augmented edges are added to the adjacency list. .UNINDENT .INDENT 7.0 .TP .B makeJobGraph(nodeNumber, childEdges, followOnEdges, outPath, addServices=True) Converts a DAG into a job graph. childEdges and followOnEdges are the lists of child and followOn edges. .UNINDENT .INDENT 7.0 .TP .B isAcyclic(adjacencyList) Returns true if there are any cycles in the graph, which is represented as an adjacency list. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.simpleJobFn(job, value) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.fn1Test(string, outputFile) Function appends the next character after the last character in the given string to the string, writes the string to a file, and returns it. For example, if string is \(dqAB\(dq, we will write and return \(dqABC\(dq. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.fn2Test(pStrings, s, outputFile) Function concatenates the strings in pStrings and s, in that order, and writes the result to the output file. Returns s. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.trivialParent(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.parent(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.diamond(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.child(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.jobTest.errorChild(job) .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.jobTest.TrivialService(message, *args, **kwargs) Bases: \fI\%toil.job.Job.Service\fP .sp Abstract class used to define the interface to a service. .sp Should be subclassed by the user to define services. .sp Is not executed as a job; runs within a ServiceHostJob. .INDENT 7.0 .TP .B start(job) Start the service. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Returns An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see \fI\%toil.job.Job.addService()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stop(job) Stops the service. Function can block until complete. .INDENT 7.0 .TP .B Parameters \fBjob\fP \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Checks the service is still running. .INDENT 7.0 .TP .B Raises \fBexceptions.RuntimeError\fP \-\- If the service failed, this will cause the service job to be labeled failed. .TP .B Returns True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.miscTests\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%MiscTests\fP T} T{ This class contains miscellaneous tests that don\(aqt have enough content to be their own test T} _ T{ \fI\%TestPanic\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.miscTests.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.miscTests.MiscTests(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp This class contains miscellaneous tests that don\(aqt have enough content to be their own test file, and that don\(aqt logically fit in with any of the other test suites. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testIDStability() .UNINDENT .INDENT 7.0 .TP .B testGetSizeOfDirectoryWorks() A test to make sure toil.common.getDirSizeRecursively does not underestimate the amount of disk space needed. .sp Disk space allocation varies from system to system. The computed value should always be equal to or slightly greater than the creation value. This test generates a number of random directories and randomly sized files to test this using getDirSizeRecursively. .UNINDENT .INDENT 7.0 .TP .B test_atomic_install() .UNINDENT .INDENT 7.0 .TP .B test_atomic_install_dev() .UNINDENT .INDENT 7.0 .TP .B test_atomic_context_ok() .UNINDENT .INDENT 7.0 .TP .B test_atomic_context_error() .UNINDENT .INDENT 7.0 .TP .B test_call_command_ok() .UNINDENT .INDENT 7.0 .TP .B test_call_command_err() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.miscTests.TestPanic(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test_panic_by_hand() .UNINDENT .INDENT 7.0 .TP .B test_panic() .UNINDENT .INDENT 7.0 .TP .B test_panic_with_secondary() .UNINDENT .INDENT 7.0 .TP .B test_nested_panic() .UNINDENT .INDENT 7.0 .TP .B try_and_panic_by_hand() .UNINDENT .INDENT 7.0 .TP .B try_and_panic() .UNINDENT .INDENT 7.0 .TP .B try_and_panic_with_secondary() .UNINDENT .INDENT 7.0 .TP .B try_and_nested_panic_with_secondary() .UNINDENT .UNINDENT .SS \fI\%toil.test.src.promisedRequirementTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%hidden\fP T} T{ Hide abstract base class from unittest\(aqs test case loader. T} _ T{ \fI\%SingleMachinePromisedRequirementsTest\fP T} T{ Tests against the SingleMachine batch system T} _ T{ \fI\%MesosPromisedRequirementsTest\fP T} T{ Tests against the Mesos batch system T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%maxConcurrency\fP(job, cpuCount, filename, coresPerJob) T} T{ Returns the max number of concurrent tasks when using a PromisedRequirement instance T} _ T{ \fI\%getOne\fP() T} T{ T} _ T{ \fI\%getThirtyTwoMb\fP() T} T{ T} _ T{ \fI\%logDiskUsage\fP(job, funcName[, sleep]) T} T{ Logs the job\(aqs disk usage to master and sleeps for specified amount of time. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.promisedRequirementTest.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.promisedRequirementTest.hidden Hide abstract base class from unittest\(aqs test case loader. .sp \fI\%http://stackoverflow.com/questions/1323455/python\-unit\-test\-with\-base\-and\-sub\-class#answer\-25695512\fP .INDENT 7.0 .TP .B class AbstractPromisedRequirementsTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemJobTest\fP .sp An abstract base class for testing Toil workflows with promised requirements. .INDENT 7.0 .TP .B testConcurrencyDynamic() Asserts that promised core resources are allocated properly using a dynamic Toil workflow .UNINDENT .INDENT 7.0 .TP .B testConcurrencyStatic() Asserts that promised core resources are allocated properly using a static DAG .UNINDENT .INDENT 7.0 .TP .B getOptions(tempDir, caching=True) Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object .UNINDENT .INDENT 7.0 .TP .B getCounterPath(tempDir) Returns path to a counter file :param str tempDir: path to test directory :return: path to counter file .UNINDENT .INDENT 7.0 .TP .B testPromisesWithJobStoreFileObjects(caching=True) Check whether FileID objects are being pickled properly when used as return values of functions. Then ensure that lambdas of promised FileID objects can be used to describe the requirements of a subsequent job. This type of operation will be used commonly in Toil scripts. :return: None .UNINDENT .INDENT 7.0 .TP .B testPromisesWithNonCachingFileStore() .UNINDENT .INDENT 7.0 .TP .B testPromiseRequirementRaceStatic() Checks for a race condition when using promised requirements and child job functions. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisedRequirementTest.maxConcurrency(job, cpuCount, filename, coresPerJob) Returns the max number of concurrent tasks when using a PromisedRequirement instance to allocate the number of cores per job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcpuCount\fP (\fI\%int\fP) \-\- number of available cpus .IP \(bu 2 \fBfilename\fP (\fI\%str\fP) \-\- path to counter file .IP \(bu 2 \fBcoresPerJob\fP (\fI\%int\fP) \-\- number of cores assigned to each job .UNINDENT .TP .B Return int max concurrency value .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisedRequirementTest.getOne() .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisedRequirementTest.getThirtyTwoMb() .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisedRequirementTest.logDiskUsage(job, funcName, sleep=0) Logs the job\(aqs disk usage to master and sleeps for specified amount of time. .INDENT 7.0 .TP .B Returns job function\(aqs disk usage .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.promisedRequirementTest.SingleMachinePromisedRequirementsTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP .sp Tests against the SingleMachine batch system .INDENT 7.0 .TP .B getBatchSystemName() .INDENT 7.0 .TP .B Return type (\fI\%str\fP, \fI\%AbstractBatchSystem\fP) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.promisedRequirementTest.MesosPromisedRequirementsTest(methodName=\(aqrunTest\(aq) Bases: \fI\%hidden\fP, \fI\%toil.batchSystems.mesos.test.MesosTestSupport\fP .sp Tests against the Mesos batch system .INDENT 7.0 .TP .B getOptions(tempDir, caching=True) Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object .UNINDENT .INDENT 7.0 .TP .B getBatchSystemName() .INDENT 7.0 .TP .B Return type (\fI\%str\fP, \fI\%AbstractBatchSystem\fP) .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .UNINDENT .SS \fI\%toil.test.src.promisesTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%CachedUnpicklingJobStoreTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%ChainedIndexedPromisesTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%PathIndexingPromiseTest\fP T} T{ Test support for indexing promises of arbitrarily nested data structures of lists, dicts and T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%parent\fP(job) T} T{ T} _ T{ \fI\%child\fP() T} T{ T} _ T{ \fI\%a\fP(job) T} T{ T} _ T{ \fI\%b\fP(job) T} T{ T} _ T{ \fI\%c\fP() T} T{ T} _ T{ \fI\%d\fP(job) T} T{ T} _ T{ \fI\%e\fP() T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.promisesTest.CachedUnpicklingJobStoreTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test() Runs two identical Toil workflows with different job store paths .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.parent(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.child() .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.promisesTest.ChainedIndexedPromisesTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B test() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.a(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.b(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.c() .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.promisesTest.PathIndexingPromiseTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test support for indexing promises of arbitrarily nested data structures of lists, dicts and tuples, or any other object supporting the __getitem__() protocol. .INDENT 7.0 .TP .B test() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.d(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.promisesTest.e() .UNINDENT .SS \fI\%toil.test.src.realtimeLoggerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%RealtimeLoggerTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%MessageDetector\fP T} T{ Detect the secret message and set a flag. T} _ T{ \fI\%LogTest\fP T} T{ Class represents a unit of work in toil. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.realtimeLoggerTest.RealtimeLoggerTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B testRealtimeLogger() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.realtimeLoggerTest.MessageDetector Bases: \fI\%logging.StreamHandler\fP .sp Detect the secret message and set a flag. .INDENT 7.0 .TP .B emit(record) Emit a record. .sp If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an \(aqencoding\(aq attribute, it is used to determine how to do the output to the stream. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.realtimeLoggerTest.LogTest Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.regularLogTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%RegularLogTest\fP T} T{ A common base class for Toil tests. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.regularLogTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.regularLogTest.RegularLogTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B testLogToMaster() .UNINDENT .INDENT 7.0 .TP .B testWriteLogs() .UNINDENT .INDENT 7.0 .TP .B testWriteGzipLogs() .UNINDENT .INDENT 7.0 .TP .B testMultipleLogToMaster() .UNINDENT .INDENT 7.0 .TP .B testRegularLog() .UNINDENT .UNINDENT .SS \fI\%toil.test.src.resourceTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ResourceTest\fP T} T{ Test module descriptors and resources derived from them. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%tempFileContaining\fP(content[, suffix]) T} T{ Write a file with the given contents, and keep it on disk as long as the context is active. T} _ .TE .INDENT 0.0 .TP .B toil.test.src.resourceTest.tempFileContaining(content, suffix=\(aq\(aq) Write a file with the given contents, and keep it on disk as long as the context is active. :param str content: The contents of the file. :param str suffix: The extension to use for the temporary file. .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.resourceTest.ResourceTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test module descriptors and resources derived from them. .INDENT 7.0 .TP .B testStandAlone() .UNINDENT .INDENT 7.0 .TP .B testPackage() .UNINDENT .INDENT 7.0 .TP .B testVirtualEnv() .UNINDENT .INDENT 7.0 .TP .B testStandAloneInPackage() .UNINDENT .INDENT 7.0 .TP .B testBuiltIn() .UNINDENT .INDENT 7.0 .TP .B testNonPyStandAlone() Asserts that Toil enforces the user script to have a .py or .pyc extension because that\(aqs the only way auto\-deployment can re\-import the module on a worker. See .sp \fI\%https://github.com/BD2KGenomics/toil/issues/631\fP and \fI\%https://github.com/BD2KGenomics/toil/issues/858\fP .UNINDENT .UNINDENT .SS \fI\%toil.test.src.restartDAGTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%RestartDAGTest\fP T} T{ Tests that restarted job DAGs don\(aqt run children of jobs that failed in the first run till the T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%passingFn\fP(job[, fileName]) T} T{ This function is guaranteed to pass as it does nothing out of the ordinary. If fileName is T} _ T{ \fI\%failingFn\fP(job, failType, fileName) T} T{ This function is guaranteed to fail via a raised assertion, or an os.kill T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.restartDAGTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.restartDAGTest.RestartDAGTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests that restarted job DAGs don\(aqt run children of jobs that failed in the first run till the parent completes successfully in the restart. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testRestartedWorkflowSchedulesCorrectJobsOnFailedParent() .UNINDENT .INDENT 7.0 .TP .B testRestartedWorkflowSchedulesCorrectJobsOnKilledParent() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.restartDAGTest.passingFn(job, fileName=None) This function is guaranteed to pass as it does nothing out of the ordinary. If fileName is provided, it will be created. .INDENT 7.0 .TP .B Parameters \fBfileName\fP (\fI\%str\fP) \-\- The name of a file that must be created if provided. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.restartDAGTest.failingFn(job, failType, fileName) This function is guaranteed to fail via a raised assertion, or an os.kill .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP \-\- Job .IP \(bu 2 \fBfailType\fP (\fI\%str\fP) \-\- \(aqraise\(aq or \(aqkill .IP \(bu 2 \fBfileName\fP (\fI\%str\fP) \-\- The name of a file that must be created. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.resumabilityTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ResumabilityTest\fP T} T{ \fI\%https://github.com/BD2KGenomics/toil/issues/808\fP T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%parent\fP(job) T} T{ Set up a bunch of dummy child jobs, and a bad job that needs to be T} _ T{ \fI\%chaining_parent\fP(job) T} T{ Set up a failing job to chain to. T} _ T{ \fI\%goodChild\fP(job) T} T{ Does nothing. T} _ T{ \fI\%badChild\fP(job) T} T{ Fails the first time it\(aqs run, succeeds the second time. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.resumabilityTest.ResumabilityTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp \fI\%https://github.com/BD2KGenomics/toil/issues/808\fP .INDENT 7.0 .TP .B test() Tests that a toil workflow that fails once can be resumed without a NoSuchJobException. .UNINDENT .INDENT 7.0 .TP .B test_chaining() Tests that a job which is chained to and fails can resume and succeed. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.resumabilityTest.parent(job) Set up a bunch of dummy child jobs, and a bad job that needs to be restarted as the follow on. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.resumabilityTest.chaining_parent(job) Set up a failing job to chain to. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.resumabilityTest.goodChild(job) Does nothing. .UNINDENT .INDENT 0.0 .TP .B toil.test.src.resumabilityTest.badChild(job) Fails the first time it\(aqs run, succeeds the second time. .UNINDENT .SS \fI\%toil.test.src.retainTempDirTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%CleanWorkDirTest\fP T} T{ Tests testing :class:toil.fileStores.abstractFileStore.AbstractFileStore T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%tempFileTestJob\fP(job) T} T{ T} _ T{ \fI\%tempFileTestErrorJob\fP(job) T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.retainTempDirTest.CleanWorkDirTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests testing :class:toil.fileStores.abstractFileStore.AbstractFileStore .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testNever() .UNINDENT .INDENT 7.0 .TP .B testAlways() .UNINDENT .INDENT 7.0 .TP .B testOnErrorWithError() .UNINDENT .INDENT 7.0 .TP .B testOnErrorWithNoError() .UNINDENT .INDENT 7.0 .TP .B testOnSuccessWithError() .UNINDENT .INDENT 7.0 .TP .B testOnSuccessWithSuccess() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.retainTempDirTest.tempFileTestJob(job) .UNINDENT .INDENT 0.0 .TP .B toil.test.src.retainTempDirTest.tempFileTestErrorJob(job) .UNINDENT .SS \fI\%toil.test.src.systemTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%SystemTest\fP T} T{ Test various assumptions about the operating system\(aqs behavior. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.systemTest.SystemTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test various assumptions about the operating system\(aqs behavior. .INDENT 7.0 .TP .B testAtomicityOfNonEmptyDirectoryRenames() .UNINDENT .UNINDENT .SS \fI\%toil.test.src.threadingTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ThreadingTest\fP T} T{ Test Toil threading/synchronization tools. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%log\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.src.threadingTest.log .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.threadingTest.ThreadingTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test Toil threading/synchronization tools. .INDENT 7.0 .TP .B testGlobalMutexOrdering() .UNINDENT .INDENT 7.0 .TP .B testLastProcessStanding() .UNINDENT .UNINDENT .SS \fI\%toil.test.src.toilContextManagerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilContextManagerTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%HelloWorld\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%FollowOn\fP T} T{ Class represents a unit of work in toil. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%childFn\fP(job) T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.toilContextManagerTest.ToilContextManagerTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B testContextManger() .UNINDENT .INDENT 7.0 .TP .B testNoContextManger() .UNINDENT .INDENT 7.0 .TP .B testExportAfterFailedExport() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.toilContextManagerTest.HelloWorld Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.toilContextManagerTest.childFn(job) .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.toilContextManagerTest.FollowOn(fileId) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.src.userDefinedJobArgTypeTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%UserDefinedJobArgTypeTest\fP T} T{ Test for issue #423 (Toil can\(aqt unpickle classes defined in user scripts) and variants T} _ T{ \fI\%JobClass\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%Foo\fP T} T{ T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%jobFunction\fP(job, level, foo) T} T{ T} _ T{ \fI\%main\fP() T} T{ T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test for issue #423 (Toil can\(aqt unpickle classes defined in user scripts) and variants thereof. .sp \fI\%https://github.com/BD2KGenomics/toil/issues/423\fP .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testJobFunction() Test with first job being a function .UNINDENT .INDENT 7.0 .TP .B testJobClass() Test with first job being an instance of a class .UNINDENT .INDENT 7.0 .TP .B testJobFunctionFromMain() Test with first job being a function defined in __main__ .UNINDENT .INDENT 7.0 .TP .B testJobClassFromMain() Test with first job being an instance of a class defined in __main__ .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.userDefinedJobArgTypeTest.JobClass(level, foo) Bases: \fI\%toil.job.Job\fP .sp Class represents a unit of work in toil. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.userDefinedJobArgTypeTest.jobFunction(job, level, foo) .UNINDENT .INDENT 0.0 .TP .B class toil.test.src.userDefinedJobArgTypeTest.Foo .INDENT 7.0 .TP .B assertIsCopy() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.src.userDefinedJobArgTypeTest.main() .UNINDENT .SS \fI\%toil.test.src.workerTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%WorkerTests\fP T} T{ Test miscellaneous units of the worker. T} _ .TE .INDENT 0.0 .TP .B class toil.test.src.workerTest.WorkerTests(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test miscellaneous units of the worker. .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B testNextChainable() Make sure chainable/non\-chainable jobs are identified correctly. .UNINDENT .UNINDENT .SS \fI\%toil.test.utils\fP .SS Submodules .SS \fI\%toil.test.utils.toilDebugTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%DebugJobTest\fP T} T{ Test the toil debug\-job command. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%workflow_debug_jobstore\fP() T} T{ T} _ T{ \fI\%testJobStoreContents\fP() T} T{ Test toilDebugFile.printContentsOfJobStore(). T} _ T{ \fI\%fetchFiles\fP(symLink, jobStoreDir, outputDir) T} T{ Fn for testFetchJobStoreFiles() and testFetchJobStoreFilesWSymlinks(). T} _ T{ \fI\%testFetchJobStoreFiles\fP() T} T{ Test toilDebugFile.fetchJobStoreFiles() symlinks. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.utils.toilDebugTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.toilDebugTest.workflow_debug_jobstore() .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.toilDebugTest.testJobStoreContents() Test toilDebugFile.printContentsOfJobStore(). .sp Runs a workflow that imports \(aqB.txt\(aq and \(aqmkFile.py\(aq into the jobStore. \(aqA.txt\(aq, \(aqC.txt\(aq, \(aqABC.txt\(aq are then created. This checks to make sure these contents are found in the jobStore and printed. .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.toilDebugTest.fetchFiles(symLink, jobStoreDir, outputDir) Fn for testFetchJobStoreFiles() and testFetchJobStoreFilesWSymlinks(). .sp Runs a workflow that imports \(aqB.txt\(aq and \(aqmkFile.py\(aq into the jobStore. \(aqA.txt\(aq, \(aqC.txt\(aq, \(aqABC.txt\(aq are then created. This test then attempts to get a list of these files and copy them over into our output diectory from the jobStore, confirm that they are present, and then delete them. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsymLink\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBjobStoreDir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBoutputDir\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.toilDebugTest.testFetchJobStoreFiles() Test toilDebugFile.fetchJobStoreFiles() symlinks. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.utils.toilDebugTest.DebugJobTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Test the toil debug\-job command. .INDENT 7.0 .TP .B test_run_job() Make sure that we can use toil debug\-job to try and run a job in\-process. .UNINDENT .INDENT 7.0 .TP .B test_print_job_info() Make sure that we can use \-\-printJobInfo to get information on a job from a job store. .UNINDENT .UNINDENT .SS \fI\%toil.test.utils.toilKillTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilKillTest\fP T} T{ A set of test cases for \(dqtoil kill\(dq. T} _ T{ \fI\%ToilKillTestWithAWSJobStore\fP T} T{ A set of test cases for \(dqtoil kill\(dq using the AWS job store. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%pkg_root\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.utils.toilKillTest.logger .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.toilKillTest.pkg_root .UNINDENT .INDENT 0.0 .TP .B class toil.test.utils.toilKillTest.ToilKillTest(*args, **kwargs) Bases: \fI\%toil.test.ToilTest\fP .sp A set of test cases for \(dqtoil kill\(dq. .INDENT 7.0 .TP .B setUp() Shared test variables. .UNINDENT .INDENT 7.0 .TP .B tearDown() Default tearDown for unittest. .UNINDENT .INDENT 7.0 .TP .B test_cwl_toil_kill() Test \(dqtoil kill\(dq on a CWL workflow with a 100 second sleep. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.utils.toilKillTest.ToilKillTestWithAWSJobStore(*args, **kwargs) Bases: \fI\%ToilKillTest\fP .sp A set of test cases for \(dqtoil kill\(dq using the AWS job store. .UNINDENT .SS \fI\%toil.test.utils.utilsTest\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%UtilsTest\fP T} T{ Tests the utilities that toil ships with, e.g. stats and status, in conjunction with restart T} _ T{ \fI\%RunTwoJobsPerWorker\fP T} T{ Runs child job with same resources as self in an attempt to chain the jobs on the same worker T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%printUnicodeCharacter\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%pkg_root\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.test.utils.utilsTest.pkg_root .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.utilsTest.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.utils.utilsTest.UtilsTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Tests the utilities that toil ships with, e.g. stats and status, in conjunction with restart functionality. .INDENT 7.0 .TP .B property toilMain .UNINDENT .INDENT 7.0 .TP .B property cleanCommand .UNINDENT .INDENT 7.0 .TP .B property statsCommand .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .UNINDENT .INDENT 7.0 .TP .B statusCommand(failIfNotComplete=False) .UNINDENT .INDENT 7.0 .TP .B test_config_functionality() Ensure that creating and reading back the config file works .UNINDENT .INDENT 7.0 .TP .B testAWSProvisionerUtils() Runs a number of the cluster utilities in sequence. .sp Launches a cluster with custom tags. Verifies the tags exist. ssh\(aqs into the cluster. Does some weird string comparisons. Makes certain that TOIL_WORKDIR is set as expected in the ssh\(aqed cluster. Rsyncs a file and verifies it exists on the leader. Destroys the cluster. .INDENT 7.0 .TP .B Returns .UNINDENT .UNINDENT .INDENT 7.0 .TP .B testUtilsSort() Tests the status and stats commands of the toil command line utility using the sort example with the \-\-restart flag. .UNINDENT .INDENT 7.0 .TP .B testUtilsStatsSort() Tests the stats commands on a complete run of the stats test. .UNINDENT .INDENT 7.0 .TP .B testUnicodeSupport() .UNINDENT .INDENT 7.0 .TP .B testMultipleJobsPerWorkerStats() Tests case where multiple jobs are run on 1 worker to ensure that all jobs report back their data .UNINDENT .INDENT 7.0 .TP .B check_status(status, status_fn, seconds=20) .UNINDENT .INDENT 7.0 .TP .B testGetPIDStatus() Test that ToilStatus.getPIDStatus() behaves as expected. .UNINDENT .INDENT 7.0 .TP .B testGetStatusFailedToilWF() Test that ToilStatus.getStatus() behaves as expected with a failing Toil workflow. While this workflow could be called by importing and evoking its main function, doing so would remove the opportunity to test the \(aqRUNNING\(aq functionality of getStatus(). .UNINDENT .INDENT 7.0 .TP .B testGetStatusFailedCWLWF() Test that ToilStatus.getStatus() behaves as expected with a failing CWL workflow. .UNINDENT .INDENT 7.0 .TP .B testGetStatusSuccessfulCWLWF() Test that ToilStatus.getStatus() behaves as expected with a successful CWL workflow. .UNINDENT .INDENT 7.0 .TP .B testPrintJobLog(mock_print) Test that ToilStatus.printJobLog() reads the log from a failed command without error. .UNINDENT .INDENT 7.0 .TP .B testRestartAttribute() Test that the job store is only destroyed when we observe a succcessful workflow run. The following simulates a failing workflow that attempts to resume without restart(). In this case, the job store should not be destroyed until restart() is called. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.utils.utilsTest.printUnicodeCharacter() .UNINDENT .INDENT 0.0 .TP .B class toil.test.utils.utilsTest.RunTwoJobsPerWorker Bases: \fI\%toil.job.Job\fP .sp Runs child job with same resources as self in an attempt to chain the jobs on the same worker .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.test.wdl\fP .SS Submodules .SS \fI\%toil.test.wdl.wdltoil_test\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%BaseWDLTest\fP T} T{ Base test class for WDL tests. T} _ T{ \fI\%WDLConformanceTests\fP T} T{ WDL conformance tests for Toil. T} _ T{ \fI\%WDLTests\fP T} T{ Tests for Toil\(aqs MiniWDL\-based implementation. T} _ T{ \fI\%WDLKubernetesClusterTest\fP T} T{ Ensure WDL works on the Kubernetes batchsystem. T} _ .TE .INDENT 0.0 .TP .B class toil.test.wdl.wdltoil_test.BaseWDLTest(methodName=\(aqrunTest\(aq) Bases: \fI\%toil.test.ToilTest\fP .sp Base test class for WDL tests. .INDENT 7.0 .TP .B setUp() Runs anew before each test to create farm fresh temp dirs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.wdl.wdltoil_test.WDLConformanceTests(methodName=\(aqrunTest\(aq) Bases: \fI\%BaseWDLTest\fP .sp WDL conformance tests for Toil. .INDENT 7.0 .TP .B wdl_dir = \(aqwdl\-conformance\-tests\(aq .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_conformance_tests_v10() .UNINDENT .INDENT 7.0 .TP .B test_conformance_tests_v11() .UNINDENT .INDENT 7.0 .TP .B classmethod tearDownClass() Hook method for deconstructing the class fixture after running all tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.wdl.wdltoil_test.WDLTests(methodName=\(aqrunTest\(aq) Bases: \fI\%BaseWDLTest\fP .sp Tests for Toil\(aqs MiniWDL\-based implementation. .INDENT 7.0 .TP .B classmethod setUpClass() Runs once for all tests. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_MD5sum() Test if Toil produces the same outputs as known good outputs for WDL\(aqs GATK tutorial #1. .UNINDENT .INDENT 7.0 .TP .B test_miniwdl_self_test(extra_args=None) Test if the MiniWDL self test runs and produces the expected output. .INDENT 7.0 .TP .B Parameters \fBextra_args\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_miniwdl_self_test_by_reference() Test if the MiniWDL self test works when passing input files by URL reference. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_giraffe_deepvariant() Test if Giraffe and GPU DeepVariant run. This could take 25 minutes. .UNINDENT .INDENT 7.0 .TP .B test_giraffe() Test if Giraffe runs. This could take 12 minutes. Also we scale it down but it still demands lots of memory. .UNINDENT .INDENT 7.0 .TP .B test_gs_uri() Test if Toil can access Google Storage URIs. .UNINDENT .INDENT 7.0 .TP .B test_coalesce() Test if WDLSectionJob can coalesce WDL decls. .sp White box test; will need to be changed or removed if the WDL interpreter changes. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.wdl.wdltoil_test.WDLKubernetesClusterTest(name) Bases: \fI\%toil.test.provisioners.clusterTest.AbstractClusterTest\fP .sp Ensure WDL works on the Kubernetes batchsystem. .INDENT 7.0 .TP .B setUp() Set up for the test. Must be overridden to call this method and set self.jobStore. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B launchCluster() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B test_wdl_kubernetes_cluster() Test that a wdl workflow works on a kubernetes cluster. Launches a cluster with 1 worker. This runs a wdl workflow that performs an image pull on the worker. :return: .UNINDENT .UNINDENT .SS Package Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%concat\fP T} T{ A literal iterable to combine sequence literals (lists, set) with generators or list comprehensions. T} _ T{ \fI\%ExceptionalThread\fP T} T{ A thread whose join() method re\-raises exceptions raised during run(). While join() is T} _ T{ \fI\%ToilTest\fP T} T{ A common base class for Toil tests. T} _ T{ \fI\%ApplianceTestSupport\fP T} T{ A Toil test that runs a user script on a minimal cluster of appliance containers. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%applianceSelf\fP([forceDockerAppliance]) T} T{ Return the fully qualified name of the Docker image to start Toil appliance containers from. T} _ T{ \fI\%toilPackageDirPath\fP() T} T{ Return the absolute path of the directory that corresponds to the top\-level toil package. T} _ T{ \fI\%have_working_nvidia_docker_runtime\fP() T} T{ Return True if Docker exists and can handle an \(dqnvidia\(dq runtime and the \(dq\-\-gpus\(dq option. T} _ T{ \fI\%have_working_nvidia_smi\fP() T} T{ Return True if the nvidia\-smi binary, from nvidia\(aqs CUDA userspace T} _ T{ \fI\%running_on_ec2\fP() T} T{ Return True if we are currently running on EC2, and false otherwise. T} _ T{ \fI\%mkdtemp\fP([suffix, prefix, dir]) T} T{ Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions. T} _ T{ \fI\%cpu_count\fP() T} T{ Get the rounded\-up integer number of whole CPUs available. T} _ T{ \fI\%get_temp_file\fP([suffix, rootDir]) T} T{ Return a string representing a temporary file, that must be manually deleted. T} _ T{ \fI\%needs_env_var\fP(var_name[, comment]) T} T{ Use as a decorator before test classes or methods to run only if the given T} _ T{ \fI\%needs_rsync3\fP(test_item) T} T{ Decorate classes or methods that depend on any features from rsync version 3.0.0+. T} _ T{ \fI\%needs_online\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if we are meant to talk to the Internet. T} _ T{ \fI\%needs_aws_s3\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if AWS S3 is usable. T} _ T{ \fI\%needs_aws_ec2\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if AWS EC2 is usable. T} _ T{ \fI\%needs_aws_batch\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if AWS Batch T} _ T{ \fI\%needs_google_storage\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if Google T} _ T{ \fI\%needs_google_project\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if we have a Google Cloud project set. T} _ T{ \fI\%needs_gridengine\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if GridEngine is installed. T} _ T{ \fI\%needs_torque\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if PBS/Torque is installed. T} _ T{ \fI\%needs_kubernetes_installed\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if Kubernetes is installed. T} _ T{ \fI\%needs_kubernetes\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if Kubernetes is installed and configured. T} _ T{ \fI\%needs_mesos\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if Mesos is installed. T} _ T{ \fI\%needs_slurm\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if Slurm is installed. T} _ T{ \fI\%needs_htcondor\fP(test_item) T} T{ Use a decorator before test classes or methods to run only if the HTCondor is installed. T} _ T{ \fI\%needs_lsf\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if LSF is installed. T} _ T{ \fI\%needs_java\fP(test_item) T} T{ Use as a test decorator to run only if java is installed. T} _ T{ \fI\%needs_docker\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%needs_singularity\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%needs_singularity_or_docker\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%needs_local_cuda\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%needs_docker_cuda\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%needs_encryption\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if PyNaCl is installed T} _ T{ \fI\%needs_cwl\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if CWLTool is installed T} _ T{ \fI\%needs_server\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if Connexion is installed. T} _ T{ \fI\%needs_celery_broker\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if RabbitMQ is set up to take Celery jobs. T} _ T{ \fI\%needs_wes_server\fP(test_item) T} T{ Use as a decorator before test classes or methods to run only if a WES T} _ T{ \fI\%needs_local_appliance\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%needs_fetchable_appliance\fP(test_item) T} T{ Use as a decorator before test classes or methods to only run them if T} _ T{ \fI\%integrative\fP(test_item) T} T{ Use this to decorate integration tests so as to skip them during regular builds. T} _ T{ \fI\%slow\fP(test_item) T} T{ Use this decorator to identify tests that are slow and not critical. T} _ T{ \fI\%timeLimit\fP(seconds) T} T{ Use to limit the execution time of a function. T} _ T{ \fI\%make_tests\fP(generalMethod, targetClass, **kwargs) T} T{ This method dynamically generates test methods using the generalMethod as a template. Each T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%memoize\fP T} T{ Memoize a function result based on its parameters using this decorator. T} _ T{ \fI\%distVersion\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%MT\fP T} T{ T} _ T{ \fI\%methodNamePartRegex\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B exception toil.test.ApplianceImageNotFound(origAppliance, url, statusCode) Bases: \fBdocker.errors.ImageNotFound\fP .sp Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBorigAppliance\fP (\fI\%str\fP) \-\- The full url of the docker image originally specified by the user (or the default). e.g. \(dqquay.io/ucsc_cgl/toil:latest\(dq .IP \(bu 2 \fBurl\fP (\fI\%str\fP) \-\- The URL at which the image\(aqs manifest is supposed to appear .IP \(bu 2 \fBstatusCode\fP (\fI\%int\fP) \-\- the failing HTTP status code returned by the URL .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.applianceSelf(forceDockerAppliance=False) Return the fully qualified name of the Docker image to start Toil appliance containers from. .sp The result is determined by the current version of Toil and three environment variables: \fBTOIL_DOCKER_REGISTRY\fP, \fBTOIL_DOCKER_NAME\fP and \fBTOIL_APPLIANCE_SELF\fP\&. .sp \fBTOIL_DOCKER_REGISTRY\fP specifies an account on a publicly hosted docker registry like Quay or Docker Hub. The default is UCSC\(aqs CGL account on Quay.io where the Toil team publishes the official appliance images. \fBTOIL_DOCKER_NAME\fP specifies the base name of the image. The default of \fItoil\fP will be adequate in most cases. \fBTOIL_APPLIANCE_SELF\fP fully qualifies the appliance image, complete with registry, image name and version tag, overriding both \fBTOIL_DOCKER_NAME\fP and \fITOIL_DOCKER_REGISTRY\(ga\fP as well as the version tag of the image. Setting TOIL_APPLIANCE_SELF will not be necessary in most cases. .INDENT 7.0 .TP .B Parameters \fBforceDockerAppliance\fP (\fI\%bool\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.toilPackageDirPath() Return the absolute path of the directory that corresponds to the top\-level toil package. .sp The return value is guaranteed to end in \(aq/toil\(aq. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.have_working_nvidia_docker_runtime() Return True if Docker exists and can handle an \(dqnvidia\(dq runtime and the \(dq\-\-gpus\(dq option. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.have_working_nvidia_smi() Return True if the nvidia\-smi binary, from nvidia\(aqs CUDA userspace utilities, is installed and can be run successfully. .sp TODO: This isn\(aqt quite the same as the check that cwltool uses to decide if it can fulfill a CUDARequirement. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.running_on_ec2() Return True if we are currently running on EC2, and false otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.mkdtemp(suffix=None, prefix=None, dir=None) Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions. .sp The permissions on the directory will be 711 instead of 700, allowing the group and all other users to traverse the directory. This is necessary if the direcotry is on NFS and the Docker daemon would like to mount it or a file inside it into a container, because on NFS even the Docker daemon appears bound by the file permissions. .sp See <\fI\%https://github.com/DataBiosphere/toil/issues/4644\fP>, and <\fI\%https://stackoverflow.com/a/67928880\fP> which talks about a similar problem but in the context of user namespaces. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBprefix\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.test.concat(*args) A literal iterable to combine sequence literals (lists, set) with generators or list comprehensions. .sp Instead of .sp .EX >>> [ \-1 ] + [ x * 2 for x in range( 3 ) ] + [ \-1 ] [\-1, 0, 2, 4, \-1] .EE .sp you can write .sp .EX >>> list( concat( \-1, ( x * 2 for x in range( 3 ) ), \-1 ) ) [\-1, 0, 2, 4, \-1] .EE .sp This is slightly shorter (not counting the list constructor) and does not involve array construction or concatenation. .sp Note that concat() flattens (or chains) all iterable arguments into a single result iterable: .sp .EX >>> list( concat( 1, range( 2, 4 ), 4 ) ) [1, 2, 3, 4] .EE .sp It only does so one level deep. If you need to recursively flatten a data structure, check out crush(). .sp If you want to prevent that flattening for an iterable argument, wrap it in concat(): .sp .EX >>> list( concat( 1, concat( range( 2, 4 ) ), 4 ) ) [1, range(2, 4), 4] .EE .sp Some more example. .sp .EX >>> list( concat() ) # empty concat [] >>> list( concat( 1 ) ) # non\-iterable [1] >>> list( concat( concat() ) ) # empty iterable [] >>> list( concat( concat( 1 ) ) ) # singleton iterable [1] >>> list( concat( 1, concat( 2 ), 3 ) ) # flattened iterable [1, 2, 3] >>> list( concat( 1, [2], 3 ) ) # flattened iterable [1, 2, 3] >>> list( concat( 1, concat( [2] ), 3 ) ) # protecting an iterable from being flattened [1, [2], 3] >>> list( concat( 1, concat( [2], 3 ), 4 ) ) # protection only works with a single argument [1, 2, 3, 4] >>> list( concat( 1, 2, concat( 3, 4 ), 5, 6 ) ) [1, 2, 3, 4, 5, 6] >>> list( concat( 1, 2, concat( [ 3, 4 ] ), 5, 6 ) ) [1, 2, [3, 4], 5, 6] .EE .sp Note that while strings are technically iterable, concat() does not flatten them. .sp .EX >>> list( concat( \(aqab\(aq ) ) [\(aqab\(aq] >>> list( concat( concat( \(aqab\(aq ) ) ) [\(aqab\(aq] .EE .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fIAny\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B __iter__() .INDENT 7.0 .TP .B Return type Iterator[Any] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.memoize Memoize a function result based on its parameters using this decorator. .sp For example, this can be used in place of lazy initialization. If the decorating function is invoked by multiple threads, the decorated function may be called more than once with the same arguments. .UNINDENT .INDENT 0.0 .TP .B class toil.test.ExceptionalThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None) Bases: \fI\%threading.Thread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B exc_info .UNINDENT .INDENT 7.0 .TP .B run() Method representing the thread\(aqs activity. .sp You may override this method in a subclass. The standard run() method invokes the callable object passed to the object\(aqs constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tryRun() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B join(*args, **kwargs) Wait until the thread terminates. .sp This blocks the calling thread until the thread whose join() method is called terminates \-\- either normally or through an unhandled exception or until the optional timeout occurs. .sp When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened \-\- if the thread is still alive, the join() call timed out. .sp When the timeout argument is not present or None, the operation will block until the thread terminates. .sp A thread can be join()ed many times. .sp join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.cpu_count() Get the rounded\-up integer number of whole CPUs available. .sp Counts hyperthreads as CPUs. .sp Uses the system\(aqs actual CPU count, or the current v1 cgroup\(aqs quota per period, if the quota is set. .sp Ignores the cgroup\(aqs cpu shares value, because it\(aqs extremely difficult to interpret. See \fI\%https://github.com/kubernetes/kubernetes/issues/81021\fP\&. .sp Caches result for efficiency. .INDENT 7.0 .TP .B Returns Integer count of available CPUs, minimum 1. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.distVersion = \(aq6.1.0\(aq .UNINDENT .INDENT 0.0 .TP .B toil.test.logger .UNINDENT .INDENT 0.0 .TP .B class toil.test.ToilTest(methodName=\(aqrunTest\(aq) Bases: \fI\%unittest.TestCase\fP .sp A common base class for Toil tests. .sp Please have every test case directly or indirectly inherit this one. .sp When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn\(aqt exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system\(aqs default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left\-over files will not be removed. .INDENT 7.0 .TP .B setup_method(method) .INDENT 7.0 .TP .B Parameters \fBmethod\fP (\fIAny\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod setUpClass() Hook method for setting up class fixture before running tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod tearDownClass() Hook method for deconstructing the class fixture after running all tests in the class. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setUp() Hook method for setting up the test fixture before exercising it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tearDown() Hook method for deconstructing the test fixture after testing it. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod awsRegion() Pick an appropriate AWS region. .sp Use us\-west\-2 unless running on EC2, in which case use the region in which the instance is located .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.MT .UNINDENT .INDENT 0.0 .TP .B toil.test.get_temp_file(suffix=\(aq\(aq, rootDir=None) Return a string representing a temporary file, that must be manually deleted. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBsuffix\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBrootDir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_env_var(var_name, comment=None) Use as a decorator before test classes or methods to run only if the given environment variable is set. Can include a comment saying what the variable should be set to. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBvar_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcomment\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Callable[[MT], MT] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_rsync3(test_item) Decorate classes or methods that depend on any features from rsync version 3.0.0+. .sp Necessary because \fButilsTest.testAWSProvisionerUtils()\fP uses option \fI\-\-protect\-args\fP which is only available in rsync 3 .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_online(test_item) Use as a decorator before test classes or methods to run only if we are meant to talk to the Internet. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_aws_s3(test_item) Use as a decorator before test classes or methods to run only if AWS S3 is usable. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_aws_ec2(test_item) Use as a decorator before test classes or methods to run only if AWS EC2 is usable. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_aws_batch(test_item) Use as a decorator before test classes or methods to run only if AWS Batch is usable. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_google_storage(test_item) Use as a decorator before test classes or methods to run only if Google Cloud is installed and we ought to be able to access public Google Storage URIs. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_google_project(test_item) Use as a decorator before test classes or methods to run only if we have a Google Cloud project set. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_gridengine(test_item) Use as a decorator before test classes or methods to run only if GridEngine is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_torque(test_item) Use as a decorator before test classes or methods to run only if PBS/Torque is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_kubernetes_installed(test_item) Use as a decorator before test classes or methods to run only if Kubernetes is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_kubernetes(test_item) Use as a decorator before test classes or methods to run only if Kubernetes is installed and configured. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_mesos(test_item) Use as a decorator before test classes or methods to run only if Mesos is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_slurm(test_item) Use as a decorator before test classes or methods to run only if Slurm is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_htcondor(test_item) Use a decorator before test classes or methods to run only if the HTCondor is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_lsf(test_item) Use as a decorator before test classes or methods to only run them if LSF is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_java(test_item) Use as a test decorator to run only if java is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_docker(test_item) Use as a decorator before test classes or methods to only run them if docker is installed and docker\-based tests are enabled. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_singularity(test_item) Use as a decorator before test classes or methods to only run them if singularity is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_singularity_or_docker(test_item) Use as a decorator before test classes or methods to only run them if docker is installed and docker\-based tests are enabled, or if Singularity is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_local_cuda(test_item) Use as a decorator before test classes or methods to only run them if a CUDA setup legible to cwltool (i.e. providing userspace nvidia\-smi) is present. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_docker_cuda(test_item) Use as a decorator before test classes or methods to only run them if a CUDA setup is available through Docker. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_encryption(test_item) Use as a decorator before test classes or methods to only run them if PyNaCl is installed and configured. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_cwl(test_item) Use as a decorator before test classes or methods to only run them if CWLTool is installed and configured. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_server(test_item) Use as a decorator before test classes or methods to only run them if Connexion is installed. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_celery_broker(test_item) Use as a decorator before test classes or methods to run only if RabbitMQ is set up to take Celery jobs. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_wes_server(test_item) Use as a decorator before test classes or methods to run only if a WES server is available to run against. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_local_appliance(test_item) Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is downloaded. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.needs_fetchable_appliance(test_item) Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is able to be downloaded from the Internet. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.integrative(test_item) Use this to decorate integration tests so as to skip them during regular builds. .sp We define integration tests as A) involving other, non\-Toil software components that we develop and/or B) having a higher cost (time or money). .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.slow(test_item) Use this decorator to identify tests that are slow and not critical. Skip if TOIL_TEST_QUICK is true. .INDENT 7.0 .TP .B Parameters \fBtest_item\fP (\fIMT\fP) \-\- .TP .B Return type MT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.test.methodNamePartRegex .UNINDENT .INDENT 0.0 .TP .B toil.test.timeLimit(seconds) Use to limit the execution time of a function. .sp Raises an exception if the execution of the function takes more than the specified amount of time. See <\fI\%http://stackoverflow.com/a/601168\fP>. .INDENT 7.0 .TP .B Parameters \fBseconds\fP (\fI\%int\fP) \-\- maximum allowable time, in seconds .TP .B Return type Generator[None, None, None] .UNINDENT .sp .EX >>> import time >>> with timeLimit(2): \&... time.sleep(1) >>> import time >>> with timeLimit(1): \&... time.sleep(2) Traceback (most recent call last): ... RuntimeError: Timed out .EE .UNINDENT .INDENT 0.0 .TP .B toil.test.make_tests(generalMethod, targetClass, **kwargs) This method dynamically generates test methods using the generalMethod as a template. Each generated function is the result of a unique combination of parameters applied to the generalMethod. Each of the parameters has a corresponding string that will be used to name the method. These generated functions are named in the scheme: test_[generalMethodName]___[ firstParamaterName]_[someValueName]__[secondParamaterName]_... .sp The arguments following the generalMethodName should be a series of one or more dictionaries of the form {str : type, ...} where the key represents the name of the value. The names will be used to represent the permutation of values passed for each parameter in the generalMethod. .sp The generated method names will list the parameters in lexicographic order by parameter name. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBgeneralMethod\fP \-\- A method that will be parameterized with values passed as kwargs. Note that the generalMethod must be a regular method. .IP \(bu 2 \fBtargetClass\fP \-\- This represents the class to which the generated test methods will be bound. If no targetClass is specified the class of the generalMethod is assumed the target. .IP \(bu 2 \fBkwargs\fP \-\- a series of dictionaries defining values, and their respective names where each keyword is the name of a parameter in generalMethod. .UNINDENT .UNINDENT .sp .EX >>> class Foo: \&... def has(self, num, letter): \&... return num, letter \&... \&... def hasOne(self, num): \&... return num .EE .sp .EX >>> class Bar(Foo): \&... pass .EE .sp .EX >>> make_tests(Foo.has, Bar, num={\(aqone\(aq:1, \(aqtwo\(aq:2}, letter={\(aqa\(aq:\(aqa\(aq, \(aqb\(aq:\(aqb\(aq}) .EE .sp .EX >>> b = Bar() .EE .sp Note that num comes lexicographically before letter and so appears first in the generated method names. .sp .EX >>> assert b.test_has__letter_a__num_one() == b.has(1, \(aqa\(aq) .EE .sp .EX >>> assert b.test_has__letter_b__num_one() == b.has(1, \(aqb\(aq) .EE .sp .EX >>> assert b.test_has__letter_a__num_two() == b.has(2, \(aqa\(aq) .EE .sp .EX >>> assert b.test_has__letter_b__num_two() == b.has(2, \(aqb\(aq) .EE .sp .EX >>> f = Foo() .EE .sp .EX >>> hasattr(f, \(aqtest_has__num_one__letter_a\(aq) # should be false because Foo has no test methods False .EE .UNINDENT .INDENT 0.0 .TP .B class toil.test.ApplianceTestSupport(methodName=\(aqrunTest\(aq) Bases: \fI\%ToilTest\fP .sp A Toil test that runs a user script on a minimal cluster of appliance containers. .sp i.e. one leader container and one worker container. .INDENT 7.0 .TP .B class Appliance(outer, mounts, cleanMounts=False) Bases: \fI\%toil.lib.threading.ExceptionalThread\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBouter\fP (\fI\%ApplianceTestSupport\fP) \-\- .IP \(bu 2 \fBmounts\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanMounts\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B lock .UNINDENT .INDENT 7.0 .TP .B __enter__() .INDENT 7.0 .TP .B Return type \fI\%Appliance\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __exit__(exc_type, exc_val, exc_tb) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexc_type\fP (\fIType\fP\fI[\fP\fI\%BaseException\fP\fI]\fP) \-\- .IP \(bu 2 \fBexc_val\fP (\fI\%Exception\fP) \-\- .IP \(bu 2 \fBexc_tb\fP (\fIAny\fP) \-\- .UNINDENT .TP .B Return type Literal[False] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B tryRun() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B runOnAppliance(*args, **kwargs) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBargs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B writeToAppliance(path, contents) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcontents\fP (\fIAny\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B deployScript(path, packagePath, script) Deploy a Python module on the appliance. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fI\%str\fP) \-\- the path (absolute or relative to the WORDIR of the appliance container) to the root of the package hierarchy where the given module should be placed. The given directory should be on the Python path. .IP \(bu 2 \fBpackagePath\fP (\fI\%str\fP) \-\- the desired fully qualified module name (dotted form) of the module .IP \(bu 2 \fBscript\fP (\fI\%str\fP\fI|\fP\fIcallable\fP) \-\- the contents of the Python module. If a callable is given, its source code will be extracted. This is a convenience that lets you embed user scripts into test code as nested function. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class LeaderThread(outer, mounts, cleanMounts=False) Bases: \fBAppliance\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBouter\fP (\fI\%ApplianceTestSupport\fP) \-\- .IP \(bu 2 \fBmounts\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcleanMounts\fP (\fI\%bool\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class WorkerThread(outer, mounts, numCores) Bases: \fBAppliance\fP .sp A thread whose join() method re\-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run(). .sp When subclassing this thread, override tryRun() instead of run(). .sp .EX >>> def f(): \&... assert 0 >>> t = ExceptionalThread(target=f) >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .sp .EX >>> class MyThread(ExceptionalThread): \&... def tryRun( self ): \&... assert 0 >>> t = MyThread() >>> t.start() >>> t.join() Traceback (most recent call last): \&... AssertionError .EE .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBouter\fP (\fI\%ApplianceTestSupport\fP) \-\- .IP \(bu 2 \fBmounts\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBnumCores\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.utils\fP .SS Submodules .SS \fI\%toil.utils.toilClean\fP .sp Delete a job store used by a previous Toil workflow invocation. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilClean.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilClean.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilConfig\fP .sp Create a config file with all default Toil options. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilConfig.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilConfig.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilDebugFile\fP .sp Debug tool for copying files contained in a toil jobStore. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%fetchJobStoreFiles\fP(jobStore, options) T} T{ Takes a list of file names as glob patterns, searches for these within a T} _ T{ \fI\%printContentsOfJobStore\fP(job_store[, job_id]) T} T{ Fetch a list of all files contained in the job store if nameOfJob is not T} _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilDebugFile.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilDebugFile.fetchJobStoreFiles(jobStore, options) Takes a list of file names as glob patterns, searches for these within a given directory, and attempts to take all of the files found and copy them into options.localFilePath. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.fileJobStore.FileJobStore\fP) \-\- A fileJobStore object. .IP \(bu 2 \fBoptions.fetch\fP \-\- List of file glob patterns to search for in the jobStore and copy into options.localFilePath. .IP \(bu 2 \fBoptions.localFilePath\fP \-\- Local directory to copy files into. .IP \(bu 2 \fBoptions.jobStore\fP \-\- The path to the jobStore directory. .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilDebugFile.printContentsOfJobStore(job_store, job_id=None) Fetch a list of all files contained in the job store if nameOfJob is not declared, otherwise it only prints out the names of files for that specific job for which it can find a match. Also creates a log file of these file names in the current directory. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_store\fP (\fI\%toil.jobStores.fileJobStore.FileJobStore\fP) \-\- Job store to ask for files from. .IP \(bu 2 \fBjob_id\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Default is None, which prints out all files in the jobStore. If specified, it will print all jobStore files that have been written to the jobStore by that job. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilDebugFile.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilDebugJob\fP .sp Debug tool for running a toil job locally. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilDebugJob.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilDebugJob.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilDestroyCluster\fP .sp Terminates the specified cluster and associated resources. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilDestroyCluster.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilDestroyCluster.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilKill\fP .sp Kills rogue toil processes. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilKill.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilKill.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilLaunchCluster\fP .sp Launches a toil leader instance with the specified provisioner. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%create_tags_dict\fP(tags) T} T{ T} _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilLaunchCluster.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilLaunchCluster.create_tags_dict(tags) .INDENT 7.0 .TP .B Parameters \fBtags\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilLaunchCluster.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilMain\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ T{ \fI\%get_or_die\fP(module, name) T} T{ Get an object from a module or complain that it is missing. T} _ T{ \fI\%loadModules\fP() T} T{ T} _ T{ \fI\%printHelp\fP(modules) T} T{ T} _ T{ \fI\%printVersion\fP() T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilMain.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilMain.get_or_die(module, name) Get an object from a module or complain that it is missing. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmodule\fP (\fI\%types.ModuleType\fP) \-\- .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilMain.loadModules() .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, \fI\%types.ModuleType\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilMain.printHelp(modules) .INDENT 7.0 .TP .B Parameters \fBmodules\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%types.ModuleType\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilMain.printVersion() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilRsyncCluster\fP .sp Rsyncs into the toil appliance container running on the leader of the cluster. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilRsyncCluster.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilRsyncCluster.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilServer\fP .sp CLI entry for the Toil servers. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilServer.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilServer.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilSshCluster\fP .sp SSH into the toil appliance container running on the leader of the cluster. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilSshCluster.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilSshCluster.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilStats\fP .sp Reports statistical data about a given Toil workflow. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ColumnWidths\fP T} T{ Convenience object that stores the width of columns for printing. Helps make things pretty. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%pad_str\fP(s[, field]) T} T{ Pad the beginning of a string with spaces, if necessary. T} _ T{ \fI\%pretty_space\fP(k[, field, alone]) T} T{ Given input k as kibibytes, return a nicely formatted string. T} _ T{ \fI\%pretty_time\fP(t[, field, unit, alone]) T} T{ Given input t as seconds, return a nicely formatted string. T} _ T{ \fI\%report_unit\fP(unit) T} T{ Format a unit name for display. T} _ T{ \fI\%report_time\fP(t, options[, field, unit, alone]) T} T{ Given t seconds, report back the correct format as string. T} _ T{ \fI\%report_space\fP(k, options[, field, unit, alone]) T} T{ Given k kibibytes, report back the correct format as string. T} _ T{ \fI\%report_number\fP(n[, field, nan_value]) T} T{ Given a number, report back the correct format as string. T} _ T{ \fI\%report\fP(v, category, options[, field, alone]) T} T{ Report a value of the given category formatted as a string. T} _ T{ \fI\%sprint_tag\fP(key, tag, options[, columnWidths]) T} T{ Generate a pretty\-print ready string from a JTTag(). T} _ T{ \fI\%decorate_title\fP(category, title, options) T} T{ Add extra parts to the category titles. T} _ T{ \fI\%decorate_subheader\fP(category, columnWidths, options) T} T{ Add a marker to the correct field if the TITLE is sorted on. T} _ T{ \fI\%get\fP(tree, name) T} T{ Return a float value attribute NAME from TREE. T} _ T{ \fI\%sort_jobs\fP(jobTypes, options) T} T{ Return a jobTypes all sorted. T} _ T{ \fI\%report_pretty_data\fP(root, worker, job, job_types, options) T} T{ Print the important bits out. T} _ T{ \fI\%compute_column_widths\fP(job_types, worker, job, options) T} T{ Return a ColumnWidths() object with the correct max widths. T} _ T{ \fI\%update_column_widths\fP(tag, cw, options) T} T{ Update the column width attributes for this tag\(aqs fields. T} _ T{ \fI\%build_element\fP(element, items, item_name, defaults) T} T{ Create an element for output. T} _ T{ \fI\%create_summary\fP(element, containingItems, ...) T} T{ Figure out how many jobs (or contained items) ran on each worker (or containing item). T} _ T{ \fI\%get_stats\fP(jobStore) T} T{ Sum together all the stats information in the job store. T} _ T{ \fI\%process_data\fP(config, stats) T} T{ Collate the stats and report T} _ T{ \fI\%report_data\fP(tree, options) T} T{ T} _ T{ \fI\%add_stats_options\fP(parser) T} T{ T} _ T{ \fI\%main\fP() T} T{ Reports stats on the workflow, use with \-\-stats option to toil. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%CATEGORIES\fP T} T{ T} _ T{ \fI\%CATEGORY_UNITS\fP T} T{ T} _ T{ \fI\%TITLES\fP T} T{ T} _ T{ \fI\%TIME_CATEGORIES\fP T} T{ T} _ T{ \fI\%SPACE_CATEGORIES\fP T} T{ T} _ T{ \fI\%COMPUTED_CATEGORIES\fP T} T{ T} _ T{ \fI\%LONG_FORMS\fP T} T{ T} _ T{ \fI\%sort_category_choices\fP T} T{ T} _ T{ \fI\%sort_field_choices\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilStats.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.CATEGORIES = [\(aqtime\(aq, \(aqclock\(aq, \(aqwait\(aq, \(aqmemory\(aq, \(aqdisk\(aq] .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.CATEGORY_UNITS .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.TITLES .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.TIME_CATEGORIES .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.SPACE_CATEGORIES .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.COMPUTED_CATEGORIES .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.LONG_FORMS .UNINDENT .INDENT 0.0 .TP .B class toil.utils.toilStats.ColumnWidths Convenience object that stores the width of columns for printing. Helps make things pretty. .INDENT 7.0 .TP .B title(category) Return the total printed length of this category item. .INDENT 7.0 .TP .B Parameters \fBcategory\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_width(category, field) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcategory\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfield\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B set_width(category, field, width) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcategory\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfield\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwidth\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B report() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.pad_str(s, field=None) Pad the beginning of a string with spaces, if necessary. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBs\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.pretty_space(k, field=None, alone=False) Given input k as kibibytes, return a nicely formatted string. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBk\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBalone\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.pretty_time(t, field=None, unit=\(aqs\(aq, alone=False) Given input t as seconds, return a nicely formatted string. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBt\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBunit\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBalone\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report_unit(unit) Format a unit name for display. .INDENT 7.0 .TP .B Parameters \fBunit\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report_time(t, options, field=None, unit=\(aqs\(aq, alone=False) Given t seconds, report back the correct format as string. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBt\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBunit\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBalone\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report_space(k, options, field=None, unit=\(aqKiB\(aq, alone=False) Given k kibibytes, report back the correct format as string. .sp If unit is set to B, convert to KiB first. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBk\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBunit\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBalone\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report_number(n, field=None, nan_value=\(aqNaN\(aq) Given a number, report back the correct format as string. .sp If it is a NaN or None, use nan_value to represent it instead. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBn\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%float\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .IP \(bu 2 \fBnan_value\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report(v, category, options, field=None, alone=False) Report a value of the given category formatted as a string. .sp Uses the given field width if set. .sp If alone is set, the field is being formatted outside a table and might need a unit. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBv\fP (\fI\%float\fP) \-\- .IP \(bu 2 \fBcategory\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .IP \(bu 2 \fBfield\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.sprint_tag(key, tag, options, columnWidths=None) Generate a pretty\-print ready string from a JTTag(). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBkey\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtag\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .IP \(bu 2 \fBcolumnWidths\fP (\fIOptional\fP\fI[\fP\fI\%ColumnWidths\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.decorate_title(category, title, options) Add extra parts to the category titles. .sp Add units to title if they won\(aqt appear in the formatted values. Add a marker to TITLE if the TITLE is sorted on. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcategory\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtitle\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.decorate_subheader(category, columnWidths, options) Add a marker to the correct field if the TITLE is sorted on. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcategory\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcolumnWidths\fP (\fI\%ColumnWidths\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.get(tree, name) Return a float value attribute NAME from TREE. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtree\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%float\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.sort_jobs(jobTypes, options) Return a jobTypes all sorted. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobTypes\fP (\fIList\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type List[Any] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report_pretty_data(root, worker, job, job_types, options) Print the important bits out. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBroot\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBworker\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBjob\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBjob_types\fP (\fIList\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.compute_column_widths(job_types, worker, job, options) Return a ColumnWidths() object with the correct max widths. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_types\fP (\fIList\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .IP \(bu 2 \fBworker\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBjob\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type \fI\%ColumnWidths\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.update_column_widths(tag, cw, options) Update the column width attributes for this tag\(aqs fields. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtag\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBcw\fP (\fI\%ColumnWidths\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.build_element(element, items, item_name, defaults) Create an element for output. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBelement\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBitems\fP (\fIList\fP\fI[\fP\fI\%toil.job.Job\fP\fI]\fP) \-\- .IP \(bu 2 \fBitem_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdefaults\fP (\fI\%dict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%float\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%toil.lib.expando.Expando\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.create_summary(element, containingItems, containingItemName, count_contained) Figure out how many jobs (or contained items) ran on each worker (or containing item). .sp Stick a bunch of xxx_number_per_xxx stats into element to describe this. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcount_contained\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%toil.lib.expando.Expando\fP\fI]\fP\fI, \fP\fI\%int\fP\fI]\fP) \-\- function that maps from containing item to number of contained items. .IP \(bu 2 \fBelement\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBcontainingItems\fP (\fIList\fP\fI[\fP\fI\%toil.lib.expando.Expando\fP\fI]\fP) \-\- .IP \(bu 2 \fBcontainingItemName\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.get_stats(jobStore) Sum together all the stats information in the job store. .sp Produces one object containing lists of the values from all the summed objects. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type \fI\%toil.lib.expando.Expando\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.process_data(config, stats) Collate the stats and report .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBstats\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .UNINDENT .TP .B Return type \fI\%toil.lib.expando.Expando\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.report_data(tree, options) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtree\fP (\fI\%toil.lib.expando.Expando\fP) \-\- .IP \(bu 2 \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.sort_category_choices .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.sort_field_choices = [\(aqmin\(aq, \(aqmed\(aq, \(aqave\(aq, \(aqmax\(aq, \(aqtotal\(aq] .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.add_stats_options(parser) .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStats.main() Reports stats on the workflow, use with \-\-stats option to toil. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilStatus\fP .sp Tool for reporting on job status. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilStatus\fP T} T{ Tool for reporting on job status. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%main\fP() T} T{ Reports the state of a Toil workflow. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilStatus.logger .UNINDENT .INDENT 0.0 .TP .B class toil.utils.toilStatus.ToilStatus(jobStoreName, specifiedJobs=None) Tool for reporting on job status. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBspecifiedJobs\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B print_dot_chart() Print a dot output graph representing the workflow. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B printJobLog() Takes a list of jobs, finds their log files, and prints them to the terminal. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B printJobChildren() Takes a list of jobs, and prints their successors. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B printAggregateJobStats(properties, childNumber) Prints a job\(aqs ID, log file, remaining tries, and other properties. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBproperties\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBchildNumber\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B report_on_jobs() Gathers information about jobs such as its child jobs and status. .INDENT 7.0 .TP .B Returns jobStats Pairings of a useful category and a list of jobs which fall into it. .TP .B Rtype dict .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static getPIDStatus(jobStoreName) Determine the status of a process with a particular local pid. .sp Checks to see if a process exists or not. .INDENT 7.0 .TP .B Returns A string indicating the status of the PID of the workflow as stored in the jobstore. .TP .B Return type \fI\%str\fP .TP .B Parameters \fBjobStoreName\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static getStatus(jobStoreName) Determine the status of a workflow. .sp If the jobstore does not exist, this returns \(aqQUEUED\(aq, assuming it has not been created yet. .sp Checks for the existence of files created in the toil.Leader.run(). In toil.Leader.run(), if a workflow completes with failed jobs, \(aqfailed.log\(aq is created, otherwise \(aqsucceeded.log\(aq is written. If neither of these exist, the leader is still running jobs. .INDENT 7.0 .TP .B Returns A string indicating the status of the workflow. [\(aqCOMPLETED\(aq, \(aqRUNNING\(aq, \(aqERROR\(aq, \(aqQUEUED\(aq] .TP .B Return type \fI\%str\fP .TP .B Parameters \fBjobStoreName\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B print_bus_messages() Goes through bus messages, returns a list of tuples which have correspondence between PID on assigned batch system and .sp Prints a list of the currently running jobs .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fetchRootJob() Fetches the root job from the jobStore that provides context for all other jobs. .sp Exactly the same as the jobStore.loadRootJob() function, but with a different exit message if the root job is not found (indicating the workflow ran successfully to completion and certain stats cannot be gathered from it meaningfully such as which jobs are left to run). .INDENT 7.0 .TP .B Raises \fI\%JobException\fP \-\- if the root job does not exist. .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B fetchUserJobs(jobs) Takes a user input array of jobs, verifies that they are in the jobStore and returns the array of jobsToReport. .INDENT 7.0 .TP .B Parameters \fBjobs\fP (\fI\%list\fP) \-\- A list of jobs to be verified. .TP .B Returns jobsToReport A list of jobs which are verified to be in the jobStore. .TP .B Return type List[\fI\%toil.job.JobDescription\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B traverseJobGraph(rootJob, jobsToReport=None, foundJobStoreIDs=None) Find all current jobs in the jobStore and return them as an Array. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrootJob\fP (\fI\%toil.job.JobDescription\fP) \-\- The root job of the workflow. .IP \(bu 2 \fBjobsToReport\fP (\fI\%list\fP) \-\- A list of jobNodes to be added to and returned. .IP \(bu 2 \fBfoundJobStoreIDs\fP (\fI\%set\fP) \-\- A set of jobStoreIDs used to keep track of jobStoreIDs encountered in traversal. .UNINDENT .TP .B Returns jobsToReport The list of jobs currently in the job graph. .TP .B Return type List[\fI\%toil.job.JobDescription\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilStatus.main() Reports the state of a Toil workflow. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.utils.toilUpdateEC2Instances\fP .sp Updates Toil\(aqs internal list of EC2 instance types. .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%internet_connection\fP() T} T{ Returns True if there is an internet connection present, and False otherwise. T} _ T{ \fI\%main\fP() T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.utils.toilUpdateEC2Instances.logger .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilUpdateEC2Instances.internet_connection() Returns True if there is an internet connection present, and False otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.utils.toilUpdateEC2Instances.main() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.wdl\fP .SS Submodules .SS \fI\%toil.wdl.utils\fP .SS Module Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%get_version\fP(iterable) T} T{ Get the version of the WDL document. T} _ .TE .INDENT 0.0 .TP .B toil.wdl.utils.get_version(iterable) Get the version of the WDL document. .INDENT 7.0 .TP .B Parameters \fBiterable\fP (\fIIterable\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- An iterable that contains the lines of a WDL document. .TP .B Returns The WDL version used in the workflow. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .SS \fI\%toil.wdl.wdltoil\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%NonDownloadingSize\fP T} T{ WDL size() implementation that avoids downloading files. T} _ T{ \fI\%ToilWDLStdLibBase\fP T} T{ Standard library implementation for WDL as run on Toil. T} _ T{ \fI\%ToilWDLStdLibTaskCommand\fP T} T{ Standard library implementation to use inside a WDL task command evaluation. T} _ T{ \fI\%ToilWDLStdLibTaskOutputs\fP T} T{ Standard library implementation for WDL as run on Toil, with additional T} _ T{ \fI\%WDLBaseJob\fP T} T{ Base job class for all WDL\-related jobs. T} _ T{ \fI\%WDLTaskWrapperJob\fP T} T{ Job that determines the resources needed to run a WDL job. T} _ T{ \fI\%WDLTaskJob\fP T} T{ Job that runs a WDL task. T} _ T{ \fI\%WDLWorkflowNodeJob\fP T} T{ Job that evaluates a WDL workflow node. T} _ T{ \fI\%WDLWorkflowNodeListJob\fP T} T{ Job that evaluates a list of WDL workflow nodes, which are in the same T} _ T{ \fI\%WDLCombineBindingsJob\fP T} T{ Job that collects the results from WDL workflow nodes and combines their T} _ T{ \fI\%WDLWorkflowGraph\fP T} T{ Represents a graph of WDL WorkflowNodes. T} _ T{ \fI\%WDLSectionJob\fP T} T{ Job that can create more graph for a section of the wrokflow. T} _ T{ \fI\%WDLScatterJob\fP T} T{ Job that evaluates a scatter in a WDL workflow. Runs the body for each T} _ T{ \fI\%WDLArrayBindingsJob\fP T} T{ Job that takes all new bindings created in an array of input environments, T} _ T{ \fI\%WDLConditionalJob\fP T} T{ Job that evaluates a conditional in a WDL workflow. T} _ T{ \fI\%WDLWorkflowJob\fP T} T{ Job that evaluates an entire WDL workflow. T} _ T{ \fI\%WDLOutputsJob\fP T} T{ Job which evaluates an outputs section (such as for a workflow). T} _ T{ \fI\%WDLRootJob\fP T} T{ Job that evaluates an entire WDL workflow, and returns the workflow outputs T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%wdl_error_reporter\fP(task[, exit, log]) T} T{ Run code in a context where WDL errors will be reported with pretty formatting. T} _ T{ \fI\%report_wdl_errors\fP(task[, exit, log]) T} T{ Create a decorator to report WDL errors with the given task message. T} _ T{ \fI\%potential_absolute_uris\fP(uri, path[, importer]) T} T{ Get potential absolute URIs to check for an imported file. T} _ T{ \fI\%toil_read_source\fP(uri, path, importer) T} T{ Implementation of a MiniWDL read_source function that can use any T} _ T{ \fI\%combine_bindings\fP(all_bindings) T} T{ Combine variable bindings from multiple predecessor tasks into one set for T} _ T{ \fI\%log_bindings\fP(log_function, message, all_bindings) T} T{ Log bindings to the console, even if some are still promises. T} _ T{ \fI\%get_supertype\fP(types) T} T{ Get the supertype that can hold values of all the given types. T} _ T{ \fI\%for_each_node\fP(root) T} T{ Iterate over all WDL workflow nodes in the given node, including inputs, T} _ T{ \fI\%recursive_dependencies\fP(root) T} T{ Get the combined workflow_node_dependencies of root and everything under T} _ T{ \fI\%pack_toil_uri\fP(file_id, dir_id, file_basename) T} T{ Encode a Toil file ID and its source path in a URI that starts with the scheme in TOIL_URI_SCHEME. T} _ T{ \fI\%unpack_toil_uri\fP(toil_uri) T} T{ Unpack a URI made by make_toil_uri to retrieve the FileID and the basename T} _ T{ \fI\%evaluate_output_decls\fP(output_decls, all_bindings, ...) T} T{ Evaluate output decls with a given bindings environment and standard library. T} _ T{ \fI\%is_url\fP(filename[, schemes]) T} T{ Decide if a filename is a known kind of URL T} _ T{ \fI\%evaluate_named_expression\fP(context, name, ...) T} T{ Evaluate an expression when we know the name of it. T} _ T{ \fI\%evaluate_decl\fP(node, environment, stdlib) T} T{ Evaluate the expression of a declaration node, or raise an error. T} _ T{ \fI\%evaluate_call_inputs\fP(context, expressions, ...[, ...]) T} T{ Evaluate a bunch of expressions with names, and make them into a fresh set of bindings. \fIinputs_dict\fP is a mapping of T} _ T{ \fI\%evaluate_defaultable_decl\fP(node, environment, stdlib) T} T{ If the name of the declaration is already defined in the environment, return its value. Otherwise, return the evaluated expression. T} _ T{ \fI\%devirtualize_files\fP(environment, stdlib) T} T{ Make sure all the File values embedded in the given bindings point to files T} _ T{ \fI\%virtualize_files\fP(environment, stdlib) T} T{ Make sure all the File values embedded in the given bindings point to files T} _ T{ \fI\%add_paths\fP(task_container, host_paths) T} T{ Based off of WDL.runtime.task_container.add_paths from miniwdl T} _ T{ \fI\%import_files\fP(environment, toil[, path, skip_remote]) T} T{ Make sure all File values embedded in the given bindings are imported, T} _ T{ \fI\%drop_missing_files\fP(environment[, ...]) T} T{ Make sure all the File values embedded in the given bindings point to files T} _ T{ \fI\%get_file_paths_in_bindings\fP(environment) T} T{ Get the paths of all files in the bindings. Doesn\(aqt guarantee that T} _ T{ \fI\%map_over_typed_files_in_bindings\fP(environment, transform) T} T{ Run all File values embedded in the given bindings through the given T} _ T{ \fI\%map_over_files_in_bindings\fP(bindings, transform) T} T{ Run all File values\(aq types and values embedded in the given bindings T} _ T{ \fI\%map_over_typed_files_in_binding\fP(binding, transform) T} T{ Run all File values\(aq types and values embedded in the given binding\(aqs value through the given T} _ T{ \fI\%map_over_typed_files_in_value\fP(value, transform) T} T{ Run all File values embedded in the given value through the given T} _ T{ \fI\%monkeypatch_coerce\fP(standard_library) T} T{ Monkeypatch miniwdl\(aqs WDL.Value.Base.coerce() function to virtualize files when they are represented as Strings. T} _ T{ \fI\%main\fP() T} T{ A Toil workflow to interpret WDL input files. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%F\fP T} T{ T} _ T{ \fI\%WDLBindings\fP T} T{ T} _ T{ \fI\%TOIL_URI_SCHEME\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.wdl.wdltoil.logger .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.wdl_error_reporter(task, exit=False, log=logger.critical) Run code in a context where WDL errors will be reported with pretty formatting. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBexit\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBlog\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Generator[None, None, None] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.F .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.report_wdl_errors(task, exit=False, log=logger.critical) Create a decorator to report WDL errors with the given task message. .sp Decorator can then be applied to a function, and if a WDL error happens it will say that it could not {task}. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBexit\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBlog\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Callable[[F], F] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.potential_absolute_uris(uri, path, importer=None) Get potential absolute URIs to check for an imported file. .sp Given a URI or bare path, yield in turn all the URIs, with schemes, where we should actually try to find it, given that we want to search under/against the given paths or URIs, the current directory, and the given importing WDL document if any. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBuri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBpath\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBimporter\fP (\fIOptional\fP\fI[\fP\fIWDL.Tree.Document\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Iterator[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B async toil.wdl.wdltoil.toil_read_source(uri, path, importer) Implementation of a MiniWDL read_source function that can use any filename or URL supported by Toil. .sp Needs to be async because MiniWDL will await its result. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBuri\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBpath\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBimporter\fP (\fIOptional\fP\fI[\fP\fIWDL.Tree.Document\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDL.ReadSourceResult .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.WDLBindings .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.combine_bindings(all_bindings) Combine variable bindings from multiple predecessor tasks into one set for the current task. .INDENT 7.0 .TP .B Parameters \fBall_bindings\fP (\fISequence\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.log_bindings(log_function, message, all_bindings) Log bindings to the console, even if some are still promises. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlog_function\fP (\fICallable\fP\fI[\fP\fIEllipsis\fP\fI, \fP\fINone\fP\fI]\fP) \-\- Function (like logger.info) to call to log data .IP \(bu 2 \fBmessage\fP (\fI\%str\fP) \-\- Message to log before the bindings .IP \(bu 2 \fBall_bindings\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- A list of bindings or promises for bindings, to log .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.get_supertype(types) Get the supertype that can hold values of all the given types. .INDENT 7.0 .TP .B Parameters \fBtypes\fP (\fISequence\fP\fI[\fP\fIOptional\fP\fI[\fP\fIWDL.Type.Base\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type WDL.Type.Base .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.for_each_node(root) Iterate over all WDL workflow nodes in the given node, including inputs, internal nodes of conditionals and scatters, and gather nodes. .INDENT 7.0 .TP .B Parameters \fBroot\fP (\fIWDL.Tree.WorkflowNode\fP) \-\- .TP .B Return type Iterator[WDL.Tree.WorkflowNode] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.recursive_dependencies(root) Get the combined workflow_node_dependencies of root and everything under it, which are not on anything in that subtree. .sp Useful because section nodes can have internal nodes with dependencies not reflected in those of the section node itself. .INDENT 7.0 .TP .B Parameters \fBroot\fP (\fIWDL.Tree.WorkflowNode\fP) \-\- .TP .B Return type Set[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.TOIL_URI_SCHEME = \(aqtoilfile:\(aq .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.pack_toil_uri(file_id, dir_id, file_basename) Encode a Toil file ID and its source path in a URI that starts with the scheme in TOIL_URI_SCHEME. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdir_id\fP (\fI\%uuid.UUID\fP) \-\- .IP \(bu 2 \fBfile_basename\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.unpack_toil_uri(toil_uri) Unpack a URI made by make_toil_uri to retrieve the FileID and the basename (no path prefix) that the file is supposed to have. .INDENT 7.0 .TP .B Parameters \fBtoil_uri\fP (\fI\%str\fP) \-\- .TP .B Return type Tuple[\fI\%toil.fileStores.FileID\fP, \fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.evaluate_output_decls(output_decls, all_bindings, standard_library) Evaluate output decls with a given bindings environment and standard library. Creates a new bindings object that only contains the bindings from the given decls. Guarantees that each decl in \fIoutput_decls\fP can access the variables defined by the previous ones. :param all_bindings: Environment to use when evaluating decls :param output_decls: Decls to evaluate :param standard_library: Standard library :return: New bindings object with only the output_decls .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBoutput_decls\fP (\fIList\fP\fI[\fP\fIWDL.Tree.Decl\fP\fI]\fP) \-\- .IP \(bu 2 \fBall_bindings\fP (\fIWDL.Env.Bindings\fP\fI[\fP\fIWDL.Value.Base\fP\fI]\fP) \-\- .IP \(bu 2 \fBstandard_library\fP (\fIWDL.StdLib.Base\fP) \-\- .UNINDENT .TP .B Return type WDL.Env.Bindings[WDL.Value.Base] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.NonDownloadingSize Bases: \fBWDL.StdLib._Size\fP .sp WDL size() implementation that avoids downloading files. .sp MiniWDL\(aqs default size() implementation downloads the whole file to get its size. We want to be able to get file sizes from code running on the leader, where there may not be space to download the whole file. So we override the fancy class that implements it so that we can handle sizes for FileIDs using the FileID\(aqs stored size info. .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.is_url(filename, schemes=[\(aqhttp:\(aq, \(aqhttps:\(aq, \(aqs3:\(aq, \(aqgs:\(aq, TOIL_URI_SCHEME]) Decide if a filename is a known kind of URL .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfilename\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBschemes\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.ToilWDLStdLibBase(file_store, execution_dir=None) Bases: \fBWDL.StdLib.Base\fP .sp Standard library implementation for WDL as run on Toil. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .IP \(bu 2 \fBexecution_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static devirtualze_to(filename, dest_dir, file_source, execution_dir) Download or export a WDL virtualized filename/URL to the given directory. .sp Makes sure sibling files stay siblings and files with the same name don\(aqt clobber each other. Called from within this class for tasks, and statically at the end of the workflow for outputs. .sp Returns the local path to the file. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfilename\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBdest_dir\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfile_source\fP (\fIUnion\fP\fI[\fP\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP\fI, \fP\fI\%toil.common.Toil\fP\fI]\fP) \-\- .IP \(bu 2 \fBexecution_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.ToilWDLStdLibTaskCommand(file_store, container) Bases: \fI\%ToilWDLStdLibBase\fP .sp Standard library implementation to use inside a WDL task command evaluation. .sp Expects all the filenames in variable bindings to be container\-side paths; these are the \(dqvirtualized\(dq filenames, while the \(dqdevirtualized\(dq filenames are host\-side paths. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .IP \(bu 2 \fBcontainer\fP (\fIWDL.runtime.task_container.TaskContainer\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs(file_store, stdout_path, stderr_path, current_directory_override=None) Bases: \fI\%ToilWDLStdLibBase\fP, \fBWDL.StdLib.TaskOutputs\fP .sp Standard library implementation for WDL as run on Toil, with additional functions only allowed in task output sections. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .IP \(bu 2 \fBstdout_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBstderr_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcurrent_directory_override\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stdout_used() Return True if the standard output was read by the WDL. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B stderr_used() Return True if the standard error was read by the WDL. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.evaluate_named_expression(context, name, expected_type, expression, environment, stdlib) Evaluate an expression when we know the name of it. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcontext\fP (\fIUnion\fP\fI[\fP\fIWDL.Error.SourceNode\fP\fI, \fP\fIWDL.Error.SourcePosition\fP\fI]\fP) \-\- .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBexpected_type\fP (\fIOptional\fP\fI[\fP\fIWDL.Type.Base\fP\fI]\fP) \-\- .IP \(bu 2 \fBexpression\fP (\fIOptional\fP\fI[\fP\fIWDL.Expr.Base\fP\fI]\fP) \-\- .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBstdlib\fP (\fIWDL.StdLib.Base\fP) \-\- .UNINDENT .TP .B Return type WDL.Value.Base .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.evaluate_decl(node, environment, stdlib) Evaluate the expression of a declaration node, or raise an error. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnode\fP (\fIWDL.Tree.Decl\fP) \-\- .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBstdlib\fP (\fIWDL.StdLib.Base\fP) \-\- .UNINDENT .TP .B Return type WDL.Value.Base .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.evaluate_call_inputs(context, expressions, environment, stdlib, inputs_dict=None) Evaluate a bunch of expressions with names, and make them into a fresh set of bindings. \fIinputs_dict\fP is a mapping of variable names to their expected type for the input decls in a task. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcontext\fP (\fIUnion\fP\fI[\fP\fIWDL.Error.SourceNode\fP\fI, \fP\fIWDL.Error.SourcePosition\fP\fI]\fP) \-\- .IP \(bu 2 \fBexpressions\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIWDL.Expr.Base\fP\fI]\fP) \-\- .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBstdlib\fP (\fIWDL.StdLib.Base\fP) \-\- .IP \(bu 2 \fBinputs_dict\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIWDL.Type.Base\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.evaluate_defaultable_decl(node, environment, stdlib) If the name of the declaration is already defined in the environment, return its value. Otherwise, return the evaluated expression. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnode\fP (\fIWDL.Tree.Decl\fP) \-\- .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBstdlib\fP (\fIWDL.StdLib.Base\fP) \-\- .UNINDENT .TP .B Return type WDL.Value.Base .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.devirtualize_files(environment, stdlib) Make sure all the File values embedded in the given bindings point to files that are actually available to command line commands. The same virtual file always maps to the same devirtualized filename even with duplicates .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBstdlib\fP (\fIWDL.StdLib.Base\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.virtualize_files(environment, stdlib) Make sure all the File values embedded in the given bindings point to files that are usable from other machines. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBstdlib\fP (\fIWDL.StdLib.Base\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.add_paths(task_container, host_paths) Based off of WDL.runtime.task_container.add_paths from miniwdl Maps the host path to the container paths .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask_container\fP (\fIWDL.runtime.task_container.TaskContainer\fP) \-\- .IP \(bu 2 \fBhost_paths\fP (\fIIterable\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.import_files(environment, toil, path=None, skip_remote=False) Make sure all File values embedded in the given bindings are imported, using the given Toil object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpath\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- If set, try resolving input location relative to the URLs or directories in this list. .IP \(bu 2 \fBskip_remote\fP (\fI\%bool\fP) \-\- If set, don\(aqt try to import files from remote locations. Leave them as URIs. .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBtoil\fP (\fI\%toil.common.Toil\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.drop_missing_files(environment, current_directory_override=None) Make sure all the File values embedded in the given bindings point to files that exist, or are null. .sp Files must not be virtualized. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBcurrent_directory_override\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.get_file_paths_in_bindings(environment) Get the paths of all files in the bindings. Doesn\(aqt guarantee that duplicates are removed. .sp TODO: Duplicative with WDL.runtime.task._fspaths, except that is internal and supports Direcotry objects. .INDENT 7.0 .TP .B Parameters \fBenvironment\fP (\fIWDLBindings\fP) \-\- .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.map_over_typed_files_in_bindings(environment, transform) Run all File values embedded in the given bindings through the given transformation function. .sp TODO: Replace with WDL.Value.rewrite_env_paths or WDL.Value.rewrite_files .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBtransform\fP (\fICallable\fP\fI[\fP\fI[\fP\fIWDL.Type.Base\fP\fI, \fP\fI\%str\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.map_over_files_in_bindings(bindings, transform) Run all File values\(aq types and values embedded in the given bindings through the given transformation function. .sp TODO: Replace with WDL.Value.rewrite_env_paths or WDL.Value.rewrite_files .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbindings\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBtransform\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.map_over_typed_files_in_binding(binding, transform) Run all File values\(aq types and values embedded in the given binding\(aqs value through the given transformation function. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbinding\fP (\fIWDL.Env.Binding\fP\fI[\fP\fIWDL.Value.Base\fP\fI]\fP) \-\- .IP \(bu 2 \fBtransform\fP (\fICallable\fP\fI[\fP\fI[\fP\fIWDL.Type.Base\fP\fI, \fP\fI\%str\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDL.Env.Binding[WDL.Value.Base] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.map_over_typed_files_in_value(value, transform) Run all File values embedded in the given value through the given transformation function. .sp If the transform returns None, the file value is changed to Null. .sp The transform has access to the type information for the value, so it knows if it may return None, depending on if the value is optional or not. .sp The transform is \fIallowed\fP to return None only if the mapping result won\(aqt actually be used, to allow for scans. So error checking needs to be part of the transform itself. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBvalue\fP (\fIWDL.Value.Base\fP) \-\- .IP \(bu 2 \fBtransform\fP (\fICallable\fP\fI[\fP\fI[\fP\fIWDL.Type.Base\fP\fI, \fP\fI\%str\fP\fI]\fP\fI, \fP\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type WDL.Value.Base .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLBaseJob(wdl_options=None, **kwargs) Bases: \fI\%toil.job.Job\fP .sp Base job class for all WDL\-related jobs. .sp Responsible for post\-processing returned bindings, to do things like add in null values for things not defined in a section. Post\-processing operations can be added onto any job before it is saved, and will be applied as long as the job\(aqs run method calls postprocess(). .sp Also responsible for remembering the Toil WDL configuration keys and values. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Run a WDL\-related job. .sp Remember to decorate non\-trivial overrides with \fI\%report_wdl_errors()\fP\&. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B then_underlay(underlay) Apply an underlay of backup bindings to the result. .INDENT 7.0 .TP .B Parameters \fBunderlay\fP (\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B then_remove(remove) Remove the given bindings from the result. .INDENT 7.0 .TP .B Parameters \fBremove\fP (\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B then_namespace(namespace) Put the result bindings into a namespace. .INDENT 7.0 .TP .B Parameters \fBnamespace\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B then_overlay(overlay) Overlay the given bindings on top of the (possibly namespaced) result. .INDENT 7.0 .TP .B Parameters \fBoverlay\fP (\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B postprocess(bindings) Apply queued changes to bindings. .sp Should be applied by subclasses\(aq run() implementations to their return values. .INDENT 7.0 .TP .B Parameters \fBbindings\fP (\fIWDLBindings\fP) \-\- .TP .B Return type WDLBindings .UNINDENT .UNINDENT .INDENT 7.0 .TP .B defer_postprocessing(other) Give our postprocessing steps to a different job. .sp Use this when you are returning a promise for bindings, on the job that issues the promise. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%WDLBaseJob\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLTaskWrapperJob(task, prev_node_results, task_id, namespace, task_path, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that determines the resources needed to run a WDL job. .sp Responsible for evaluating the input declarations for unspecified inputs, evaluating the runtime section, and scheduling or chaining to the real WDL job. .sp All bindings are in terms of task\-internal names. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask\fP (\fIWDL.Tree.Task\fP) \-\- .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBtask_id\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Evaluate inputs and runtime and schedule the task. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLTaskJob(task, task_internal_bindings, runtime_bindings, task_id, namespace, task_path, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that runs a WDL task. .sp Responsible for re\-evaluating input declarations for unspecified inputs, evaluating the runtime section, re\-scheduling if resources are not available, running any command, and evaluating the outputs. .sp All bindings are in terms of task\-internal names. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBtask\fP (\fIWDL.Tree.Task\fP) \-\- .IP \(bu 2 \fBtask_internal_bindings\fP (\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .IP \(bu 2 \fBruntime_bindings\fP (\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .IP \(bu 2 \fBtask_id\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B can_fake_root() Determine if \-\-fakeroot is likely to work for Singularity. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B can_mount_proc() Determine if \-\-containall will work for Singularity. On Kubernetes, this will result in operation not permitted See: \fI\%https://github.com/apptainer/singularity/issues/5857\fP .sp So if Kubernetes is detected, return False :return: bool .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Actually run the task. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLWorkflowNodeJob(node, prev_node_results, namespace, task_path, wdl_options=None, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that evaluates a WDL workflow node. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnode\fP (\fIWDL.Tree.WorkflowNode\fP) \-\- .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Actually execute the workflow node. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLWorkflowNodeListJob(nodes, prev_node_results, namespace, wdl_options=None, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that evaluates a list of WDL workflow nodes, which are in the same scope and in a topological dependency order, and which do not call out to any other workflows or tasks or sections. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnodes\fP (\fIList\fP\fI[\fP\fIWDL.Tree.WorkflowNode\fP\fI]\fP) \-\- .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Actually execute the workflow nodes. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLCombineBindingsJob(prev_node_results, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that collects the results from WDL workflow nodes and combines their environment changes. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Aggregate incoming results. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type WDLBindings .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLWorkflowGraph(nodes) Represents a graph of WDL WorkflowNodes. .sp Operates at a certain level of instantiation (i.e. sub\-sections are represented by single nodes). .sp Assumes all relevant nodes are provided; dependencies outside the provided nodes are assumed to be satisfied already. .INDENT 7.0 .TP .B Parameters \fBnodes\fP (\fISequence\fP\fI[\fP\fIWDL.Tree.WorkflowNode\fP\fI]\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B real_id(node_id) Map multiple IDs for what we consider the same node to one ID. .sp This elides/resolves gathers. .INDENT 7.0 .TP .B Parameters \fBnode_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_decl(node_id) Return True if a node represents a WDL declaration, and false otherwise. .INDENT 7.0 .TP .B Parameters \fBnode_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get(node_id) Get a node by ID. .INDENT 7.0 .TP .B Parameters \fBnode_id\fP (\fI\%str\fP) \-\- .TP .B Return type WDL.Tree.WorkflowNode .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_dependencies(node_id) Get all the nodes that a node depends on, recursively (into the node if it has a body) but not transitively. .sp Produces dependencies after resolving gathers and internal\-to\-section dependencies, on nodes that are also in this graph. .INDENT 7.0 .TP .B Parameters \fBnode_id\fP (\fI\%str\fP) \-\- .TP .B Return type Set[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_transitive_dependencies(node_id) Get all the nodes that a node depends on, transitively. .INDENT 7.0 .TP .B Parameters \fBnode_id\fP (\fI\%str\fP) \-\- .TP .B Return type Set[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B topological_order() Get a topological order of the nodes, based on their dependencies. .INDENT 7.0 .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B leaves() Get all the workflow node IDs that have no dependents in the graph. .INDENT 7.0 .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLSectionJob(namespace, task_path, wdl_options=None, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that can create more graph for a section of the wrokflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static coalesce_nodes(order, section_graph) Given a topological order of WDL workflow node IDs, produce a list of lists of IDs, still in topological order, where each list of IDs can be run under a single Toil job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBorder\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBsection_graph\fP (\fI\%WDLWorkflowGraph\fP) \-\- .UNINDENT .TP .B Return type List[List[\fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create_subgraph(nodes, gather_nodes, environment, local_environment=None, subscript=None) Make a Toil job to evaluate a subgraph inside a workflow or workflow section. .INDENT 7.0 .TP .B Returns a child Job that will return the aggregated environment after running all the things in the section. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBgather_nodes\fP (\fISequence\fP\fI[\fP\fIWDL.Tree.Gather\fP\fI]\fP) \-\- Names exposed by these will always be defined with something, even if the code that defines them does not actually run. .IP \(bu 2 \fBenvironment\fP (\fIWDLBindings\fP) \-\- Bindings in this environment will be used to evaluate the subgraph and will be passed through. .IP \(bu 2 \fBlocal_environment\fP (\fIOptional\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- Bindings in this environment will be used to evaluate the subgraph but will go out of scope at the end of the section. .IP \(bu 2 \fBsubscript\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- If the subgraph is being evaluated multiple times, this should be a disambiguating integer for logging. .IP \(bu 2 \fBnodes\fP (\fISequence\fP\fI[\fP\fIWDL.Tree.WorkflowNode\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%WDLBaseJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B make_gather_bindings(gathers, undefined) Given a collection of Gathers, create bindings from every identifier gathered, to the given \(dqundefined\(dq placeholder (which would be Null for a single execution of the body, or an empty array for a completely unexecuted scatter). .sp These bindings can be overlaid with bindings from the actual execution, so that references to names defined in unexecuted code get a proper default undefined value, and not a KeyError at runtime. .sp The information to do this comes from MiniWDL\(aqs \(dqgathers\(dq system: <\fI\%https://miniwdl.readthedocs.io/en/latest/WDL.html#WDL.Tree.WorkflowSection.gathers\fP> .sp TODO: This approach will scale O(n^2) when run on n nested conditionals, because generating these bindings for the outer conditional will visit all the bindings from the inner ones. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBgathers\fP (\fISequence\fP\fI[\fP\fIWDL.Tree.Gather\fP\fI]\fP) \-\- .IP \(bu 2 \fBundefined\fP (\fIWDL.Value.Base\fP) \-\- .UNINDENT .TP .B Return type WDLBindings .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLScatterJob(scatter, prev_node_results, namespace, task_path, wdl_options=None, **kwargs) Bases: \fI\%WDLSectionJob\fP .sp Job that evaluates a scatter in a WDL workflow. Runs the body for each value in an array, and makes arrays of the new bindings created in each instance of the body. If an instance of the body doesn\(aqt create a binding, it gets a null value in the corresponding array. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBscatter\fP (\fIWDL.Tree.Scatter\fP) \-\- .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Run the scatter. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLArrayBindingsJob(input_bindings, base_bindings, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job that takes all new bindings created in an array of input environments, relative to a base environment, and produces bindings where each new binding name is bound to an array of the values in all the input environments. .sp Useful for producing the results of a scatter. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBinput_bindings\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBbase_bindings\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Actually produce the array\-ified bindings now that promised values are available. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type WDLBindings .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLConditionalJob(conditional, prev_node_results, namespace, task_path, wdl_options=None, **kwargs) Bases: \fI\%WDLSectionJob\fP .sp Job that evaluates a conditional in a WDL workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconditional\fP (\fIWDL.Tree.Conditional\fP) \-\- .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Run the conditional. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLWorkflowJob(workflow, prev_node_results, workflow_id, namespace, task_path, wdl_options=None, **kwargs) Bases: \fI\%WDLSectionJob\fP .sp Job that evaluates an entire WDL workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow\fP (\fIWDL.Tree.Workflow\fP) \-\- .IP \(bu 2 \fBprev_node_results\fP (\fISequence\fP\fI[\fP\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBworkflow_id\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBnamespace\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBtask_path\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Run the workflow. Return the result of the workflow. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLOutputsJob(workflow, bindings, wdl_options=None, **kwargs) Bases: \fI\%WDLBaseJob\fP .sp Job which evaluates an outputs section (such as for a workflow). .sp Returns an environment with just the outputs bound, in no namespace. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow\fP (\fIWDL.Tree.Workflow\fP) \-\- .IP \(bu 2 \fBbindings\fP (\fItoil.job.Promised\fP\fI[\fP\fIWDLBindings\fP\fI]\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Make bindings for the outputs. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type WDLBindings .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.wdl.wdltoil.WDLRootJob(workflow, inputs, wdl_options=None, **kwargs) Bases: \fI\%WDLSectionJob\fP .sp Job that evaluates an entire WDL workflow, and returns the workflow outputs namespaced with the workflow name. Inputs may or may not be namespaced with the workflow name; both forms are accepted. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow\fP (\fIWDL.Tree.Workflow\fP) \-\- .IP \(bu 2 \fBinputs\fP (\fIWDLBindings\fP) \-\- .IP \(bu 2 \fBwdl_options\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBkwargs\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(file_store) Actually build the subgraph. .INDENT 7.0 .TP .B Parameters \fBfile_store\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- .TP .B Return type toil.job.Promised[WDLBindings] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.monkeypatch_coerce(standard_library) Monkeypatch miniwdl\(aqs WDL.Value.Base.coerce() function to virtualize files when they are represented as Strings. Calls _virtualize_filename from a given standard library object. :param standard_library: a standard library object :return .INDENT 7.0 .TP .B Parameters \fBstandard_library\fP (\fI\%ToilWDLStdLibBase\fP) \-\- .TP .B Return type Generator[None, None, None] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.wdl.wdltoil.main() A Toil workflow to interpret WDL input files. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .SS Submodules .SS \fI\%toil.bus\fP .sp Message types and message bus for leader component coordination. .sp Historically, the Toil Leader has been organized around functions calling other functions to \(dqhandle\(dq different things happening. Over time, it has become very brittle: exactly the right handling functions need to be called in exactly the right order, or it gets confused and does the wrong thing. .sp The MessageBus is meant to let the leader avoid this by more losely coupling its components together, by having them communicate by sending messages instead of by calling functions. .sp When events occur (like a job coming back from the batch system with a failed exit status), this will be translated into a message that will be sent to the bus. Then, all the leader components that need to react to this message in some way (by, say, decrementing the retry count) would listen for the relevant messages on the bus and react to them. If a new component needs to be added, it can be plugged into the message bus and receive and react to messages without interfering with existing components\(aq ability to react to the same messages. .sp Eventually, the different aspects of the Leader could become separate objects. .sp By default, messages stay entirely within the Toil leader process, and are not persisted anywhere, not even in the JobStore. .sp The Message Bus also provides an extension point: its messages can be serialized to a file by the leader (see the \-\-writeMessages option), and they can then be decoded using MessageBus.scan_bus_messages() (as is done in the Toil WES server backend). By replaying the messages and tracking their effects on job state, you can get an up\-to\-date view of the state of the jobs in a workflow. This includes information, such as whether jobs are issued or running, or what jobs have completely finished, which is not persisted in the JobStore. .sp The MessageBus instance for the leader process is owned by the Toil leader, but the BatchSystem has an opportunity to connect to it, and can send (or listen for) messages. Right now the BatchSystem deos not \fIhave\fP to send or receive any messages; the Leader is responsible for polling it via the BatchSystem API and generating the events. But a BatchSystem implementation \fImay\fP send additional events (like JobAnnotationMessage). .sp Currently, the MessageBus is implemented using pypubsub, and so messages are always handled in a single Thread, the Toil leader\(aqs main loop thread. If other components send events, they will be shipped over to that thread inside the MessageBus. Communication between processes is allowed using MessageBus.connect_output_file() and MessageBus.scan_bus_messages(). .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Names\fP T} T{ Stores all the kinds of name a job can have. T} _ T{ \fI\%JobIssuedMessage\fP T} T{ Produced when a job is issued to run on the batch system. T} _ T{ \fI\%JobUpdatedMessage\fP T} T{ Produced when a job is \(dqupdated\(dq and ready to have something happen to it. T} _ T{ \fI\%JobCompletedMessage\fP T} T{ Produced when a job is completed, whether successful or not. T} _ T{ \fI\%JobFailedMessage\fP T} T{ Produced when a job is completely failed, and will not be retried again. T} _ T{ \fI\%JobMissingMessage\fP T} T{ Produced when a job goes missing and should be in the batch system but isn\(aqt. T} _ T{ \fI\%JobAnnotationMessage\fP T} T{ Produced when extra information (such as an AWS Batch job ID from the T} _ T{ \fI\%ExternalBatchIdMessage\fP T} T{ Produced when using a batch system, links toil assigned batch ID to T} _ T{ \fI\%QueueSizeMessage\fP T} T{ Produced to describe the size of the queue of jobs issued but not yet T} _ T{ \fI\%ClusterSizeMessage\fP T} T{ Produced by the Toil\-integrated autoscaler describe the number of T} _ T{ \fI\%ClusterDesiredSizeMessage\fP T} T{ Produced by the Toil\-integrated autoscaler to describe the number of T} _ T{ \fI\%MessageBus\fP T} T{ Holds messages that should cause jobs to change their scheduling states. T} _ T{ \fI\%MessageBusClient\fP T} T{ Base class for clients (inboxes and outboxes) of a message bus. Handles T} _ T{ \fI\%MessageInbox\fP T} T{ A buffered connection to a message bus that lets us receive messages. T} _ T{ \fI\%MessageOutbox\fP T} T{ A connection to a message bus that lets us publish messages. T} _ T{ \fI\%MessageBusConnection\fP T} T{ A two\-way connection to a message bus. Buffers incoming messages until you T} _ T{ \fI\%JobStatus\fP T} T{ Records the status of a job. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%get_job_kind\fP(names) T} T{ Return an identifying string for the job. T} _ T{ \fI\%message_to_bytes\fP(message) T} T{ Convert a plain\-old\-data named tuple into a byte string. T} _ T{ \fI\%bytes_to_message\fP(message_type, data) T} T{ Convert bytes from message_to_bytes back to a message of the given type. T} _ T{ \fI\%replay_message_bus\fP(path) T} T{ Replay all the messages and work out what they mean for jobs. T} _ T{ \fI\%gen_message_bus_path\fP() T} T{ Return a file path in tmp to store the message bus at. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%MessageType\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.bus.logger .UNINDENT .INDENT 0.0 .TP .B class toil.bus.Names Bases: \fBNamedTuple\fP .sp Stores all the kinds of name a job can have. .INDENT 7.0 .TP .B job_name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B unit_name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B display_name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B stats_name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B job_store_id: \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.bus.get_job_kind(names) Return an identifying string for the job. .sp The result may contain spaces. .INDENT 7.0 .TP .B Returns: Either the unit name, job name, or display name, which identifies the kind of job it is to toil. Otherwise \(dqUnknown Job\(dq in case no identifier is available .UNINDENT .INDENT 7.0 .TP .B Parameters \fBnames\fP (\fI\%Names\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobIssuedMessage Bases: \fBNamedTuple\fP .sp Produced when a job is issued to run on the batch system. .INDENT 7.0 .TP .B job_type: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B job_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B toil_batch_id: \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobUpdatedMessage Bases: \fBNamedTuple\fP .sp Produced when a job is \(dqupdated\(dq and ready to have something happen to it. .INDENT 7.0 .TP .B job_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B result_status: \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobCompletedMessage Bases: \fBNamedTuple\fP .sp Produced when a job is completed, whether successful or not. .INDENT 7.0 .TP .B job_type: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B job_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B exit_code: \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobFailedMessage Bases: \fBNamedTuple\fP .sp Produced when a job is completely failed, and will not be retried again. .INDENT 7.0 .TP .B job_type: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B job_id: \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobMissingMessage Bases: \fBNamedTuple\fP .sp Produced when a job goes missing and should be in the batch system but isn\(aqt. .INDENT 7.0 .TP .B job_id: \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobAnnotationMessage Bases: \fBNamedTuple\fP .sp Produced when extra information (such as an AWS Batch job ID from the AWSBatchBatchSystem) is available that goes with a job. .INDENT 7.0 .TP .B job_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B annotation_name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B annotation_value: \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.ExternalBatchIdMessage Bases: \fBNamedTuple\fP .sp Produced when using a batch system, links toil assigned batch ID to Batch system ID (Whatever\(aqs returned by local implementation, PID, batch ID, etc) .INDENT 7.0 .TP .B toil_batch_id: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B external_batch_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B batch_system: \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.QueueSizeMessage Bases: \fBNamedTuple\fP .sp Produced to describe the size of the queue of jobs issued but not yet completed. Theoretically recoverable from other messages. .INDENT 7.0 .TP .B queue_size: \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.ClusterSizeMessage Bases: \fBNamedTuple\fP .sp Produced by the Toil\-integrated autoscaler describe the number of instances of a certain type in a cluster. .INDENT 7.0 .TP .B instance_type: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B current_size: \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.ClusterDesiredSizeMessage Bases: \fBNamedTuple\fP .sp Produced by the Toil\-integrated autoscaler to describe the number of instances of a certain type that it thinks will be needed. .INDENT 7.0 .TP .B instance_type: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B desired_size: \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.bus.message_to_bytes(message) Convert a plain\-old\-data named tuple into a byte string. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fINamedTuple\fP) \-\- .TP .B Return type \fI\%bytes\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.bus.MessageType .UNINDENT .INDENT 0.0 .TP .B toil.bus.bytes_to_message(message_type, data) Convert bytes from message_to_bytes back to a message of the given type. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmessage_type\fP (\fIType\fP\fI[\fP\fIMessageType\fP\fI]\fP) \-\- .IP \(bu 2 \fBdata\fP (\fI\%bytes\fP) \-\- .UNINDENT .TP .B Return type MessageType .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.MessageBus Holds messages that should cause jobs to change their scheduling states. Messages are put in and buffered, and can be taken out and handled as batches when convenient. .sp All messages are NamedTuple objects of various subtypes. .sp Message order is guaranteed to be preserved within a type. .INDENT 7.0 .TP .B MessageType .UNINDENT .INDENT 7.0 .TP .B publish(message) Put a message onto the bus. Can be called from any thread. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fIAny\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() If we are in the owning thread, deliver any messages that are in the queue for us. Must be called every once in a while in the main thread, possibly through inbox objects. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B subscribe(message_type, handler) Register the given callable to be called when messages of the given type are sent. It will be called with messages sent after the subscription is created. Returns a subscription object; when the subscription object is GC\(aqd the subscription will end. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmessage_type\fP (\fIType\fP\fI[\fP\fIMessageType\fP\fI]\fP) \-\- .IP \(bu 2 \fBhandler\fP (\fICallable\fP\fI[\fP\fI[\fP\fIMessageType\fP\fI]\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type pubsub.core.listener.Listener .UNINDENT .UNINDENT .INDENT 7.0 .TP .B connect(wanted_types) Get a connection object that serves as an inbox for messages of the given types. Messages of those types will accumulate in the inbox until it is destroyed. You can check for them at any time. .INDENT 7.0 .TP .B Parameters \fBwanted_types\fP (\fIList\fP\fI[\fP\fI\%type\fP\fI]\fP) \-\- .TP .B Return type \fI\%MessageBusConnection\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B outbox() Get a connection object that only allows sending messages. .INDENT 7.0 .TP .B Return type \fI\%MessageOutbox\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B connect_output_file(file_path) Send copies of all messages to the given output file. .sp Returns connection data which must be kept alive for the connection to persist. That data is opaque: the user is not supposed to look at it or touch it or do anything with it other than store it somewhere or delete it. .INDENT 7.0 .TP .B Parameters \fBfile_path\fP (\fI\%str\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod scan_bus_messages(stream, message_types) Get an iterator over all messages in the given log stream of the given types, in order. Discard any trailing partial messages. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstream\fP (\fIIO\fP\fI[\fP\fI\%bytes\fP\fI]\fP) \-\- .IP \(bu 2 \fBmessage_types\fP (\fIList\fP\fI[\fP\fIType\fP\fI[\fP\fINamedTuple\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Iterator[Any] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.MessageBusClient Base class for clients (inboxes and outboxes) of a message bus. Handles keeping a reference to the message bus. .UNINDENT .INDENT 0.0 .TP .B class toil.bus.MessageInbox Bases: \fI\%MessageBusClient\fP .sp A buffered connection to a message bus that lets us receive messages. Buffers incoming messages until you are ready for them. Does not preserve ordering between messages of different types. .INDENT 7.0 .TP .B MessageType .UNINDENT .INDENT 7.0 .TP .B count(message_type) Get the number of pending messages of the given type. .INDENT 7.0 .TP .B Parameters \fBmessage_type\fP (\fI\%type\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B empty() Return True if no messages are pending, and false otherwise. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B for_each(message_type) Loop over all messages currently pending of the given type. Each that is handled without raising an exception will be removed. .sp Messages sent while this function is running will not be yielded by the current call. .INDENT 7.0 .TP .B Parameters \fBmessage_type\fP (\fIType\fP\fI[\fP\fIMessageType\fP\fI]\fP) \-\- .TP .B Return type Iterator[MessageType] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.MessageOutbox Bases: \fI\%MessageBusClient\fP .sp A connection to a message bus that lets us publish messages. .INDENT 7.0 .TP .B publish(message) Publish the given message to the connected message bus. .sp We have this so you don\(aqt need to store both the bus and your connection. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fIAny\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.bus.MessageBusConnection Bases: \fI\%MessageInbox\fP, \fI\%MessageOutbox\fP .sp A two\-way connection to a message bus. Buffers incoming messages until you are ready for them, and lets you send messages. .UNINDENT .INDENT 0.0 .TP .B class toil.bus.JobStatus Records the status of a job. .INDENT 7.0 .TP .B job_store_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B exit_code: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B annotations: Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .INDENT 7.0 .TP .B toil_batch_id: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B external_batch_id: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B batch_system: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.bus.replay_message_bus(path) Replay all the messages and work out what they mean for jobs. .sp We track the state and name of jobs here, by ID. We would use a list of two items but MyPy can\(aqt understand a list of items of multiple types, so we need to define a new class. .sp Returns a dictionary from the job_id to a dataclass, JobStatus. A JobStatus contains information about a job which we have gathered from the message bus, including the job store id, name of the job the exit code, any associated annotations, the toil batch id the external batch id, and the batch system on which the job is running. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI\%str\fP) \-\- .TP .B Return type Dict[\fI\%str\fP, \fI\%JobStatus\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.bus.gen_message_bus_path() Return a file path in tmp to store the message bus at. Calling function is responsible for cleaning the generated file. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .SS \fI\%toil.common\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Config\fP T} T{ Class to represent configuration operations for a toil workflow run. T} _ T{ \fI\%Toil\fP T} T{ A context manager that represents a Toil workflow. T} _ T{ \fI\%ToilMetrics\fP T} T{ T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%check_and_create_toil_home_dir\fP() T} T{ Ensure that TOIL_HOME_DIR exists. T} _ T{ \fI\%check_and_create_default_config_file\fP() T} T{ If the default config file does not exist, create it in the Toil home directory. Create the Toil home directory T} _ T{ \fI\%check_and_create_config_file\fP(filepath) T} T{ If the config file at the filepath does not exist, try creating it. T} _ T{ \fI\%generate_config\fP(filepath) T} T{ Write a Toil config file to the given path. T} _ T{ \fI\%parser_with_common_options\fP([provisioner_options, ...]) T} T{ T} _ T{ \fI\%addOptions\fP(parser[, jobstore_as_flag, cwl, wdl]) T} T{ Add all Toil command line options to a parser. T} _ T{ \fI\%getNodeID\fP() T} T{ Return unique ID of the current node (host). The resulting string will be convertable to a uuid.UUID. T} _ T{ \fI\%cacheDirName\fP(workflowID) T} T{ .INDENT 0.0 .TP .B return Name of the cache directory. .UNINDENT T} _ T{ \fI\%getDirSizeRecursively\fP(dirPath) T} T{ This method will return the cumulative number of bytes occupied by the files T} _ T{ \fI\%getFileSystemSize\fP(dirPath) T} T{ Return the free space, and total size of the file system hosting \fIdirPath\fP\&. T} _ T{ \fI\%safeUnpickleFromStream\fP(stream) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%UUID_LENGTH\fP T} T{ T} _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%TOIL_HOME_DIR\fP T} T{ T} _ T{ \fI\%DEFAULT_CONFIG_FILE\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.common.UUID_LENGTH = 32 .UNINDENT .INDENT 0.0 .TP .B toil.common.logger .UNINDENT .INDENT 0.0 .TP .B toil.common.TOIL_HOME_DIR: \fI\%str\fP .UNINDENT .INDENT 0.0 .TP .B toil.common.DEFAULT_CONFIG_FILE: \fI\%str\fP .UNINDENT .INDENT 0.0 .TP .B class toil.common.Config Class to represent configuration operations for a toil workflow run. .INDENT 7.0 .TP .B logFile: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B logRotating: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B cleanWorkDir: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B max_jobs: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B max_local_jobs: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B manualMemArgs: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B run_local_jobs_on_workers: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B coalesceStatusCalls: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B mesos_endpoint: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B mesos_framework_id: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B mesos_role: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B mesos_name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_host_path: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_owner: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_service_account: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B kubernetes_pod_timeout: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B tes_endpoint: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B tes_user: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B tes_password: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B tes_bearer_token: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B aws_batch_region: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B aws_batch_queue: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B aws_batch_job_role_arn: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B scale: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B batchSystem: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B batch_logs_dir: \fI\%str\fP | \fI\%None\fP The backing scheduler will be instructed, if possible, to save logs to this directory, where the leader can read them. .UNINDENT .INDENT 7.0 .TP .B statePollingWait: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B disableAutoDeployment: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B workflowID: \fI\%str\fP | \fI\%None\fP This attribute uniquely identifies the job store and therefore the workflow. It is necessary in order to distinguish between two consecutive workflows for which self.jobStore is the same, e.g. when a job store name is reused after a previous run has finished successfully and its job store has been clean up. .UNINDENT .INDENT 7.0 .TP .B workflowAttemptNumber: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B jobStore: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B logLevel: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B workDir: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B coordination_dir: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B noStdOutErr: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B stats: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B clean: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B clusterStats: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B restart: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B caching: \fI\%bool\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B symlinkImports: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B moveOutputs: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B provisioner: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B nodeTypes: List[Tuple[Set[\fI\%str\fP], \fI\%float\fP | \fI\%None\fP]] .UNINDENT .INDENT 7.0 .TP .B minNodes: List[\fI\%int\fP] .UNINDENT .INDENT 7.0 .TP .B maxNodes: List[\fI\%int\fP] .UNINDENT .INDENT 7.0 .TP .B targetTime: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B betaInertia: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B scaleInterval: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B preemptibleCompensation: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B nodeStorage: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B nodeStorageOverrides: List[\fI\%str\fP] .UNINDENT .INDENT 7.0 .TP .B metrics: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B assume_zero_overhead: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B maxPreemptibleServiceJobs: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B maxServiceJobs: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B deadlockWait: \fI\%float\fP | \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B deadlockCheckInterval: \fI\%float\fP | \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B defaultMemory: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B defaultCores: \fI\%float\fP | \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B defaultDisk: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B defaultPreemptible: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B defaultAccelerators: List[\fI\%toil.job.AcceleratorRequirement\fP] .UNINDENT .INDENT 7.0 .TP .B maxCores: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B maxMemory: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B maxDisk: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B retryCount: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B enableUnlimitedPreemptibleRetries: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B doubleMem: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B maxJobDuration: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B rescueJobsFrequency: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B maxLogFileSize: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B writeLogs: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B writeLogsGzip: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B writeLogsFromAllJobs: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B write_messages: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B realTimeLogging: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B environment: Dict[\fI\%str\fP, \fI\%str\fP] .UNINDENT .INDENT 7.0 .TP .B disableChaining: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B disableJobStoreChecksumVerification: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B sseKey: \fI\%str\fP | \fI\%None\fP .UNINDENT .INDENT 7.0 .TP .B servicePollingInterval: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B useAsync: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B forceDockerAppliance: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B statusWait: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B disableProgress: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B readGlobalFileMutableByDefault: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B debugWorker: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B disableWorkerOutputCapture: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B badWorker: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B badWorkerFailInterval: \fI\%float\fP .UNINDENT .INDENT 7.0 .TP .B kill_polling_interval: \fI\%int\fP .UNINDENT .INDENT 7.0 .TP .B cwl: \fI\%bool\fP .UNINDENT .INDENT 7.0 .TP .B set_from_default_config() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepare_start() After options are set, prepare for initial start of workflow. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepare_restart() Before restart options are set, prepare for a restart of a workflow. Set up any execution\-specific parameters and clear out any stale ones. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setOptions(options) Creates a config object from the options object. .INDENT 7.0 .TP .B Parameters \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_configuration_consistency() Old checks that cannot be fit into an action class for argparse .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __eq__(other) Return self==value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%object\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __hash__() Return hash(self). .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.check_and_create_toil_home_dir() Ensure that TOIL_HOME_DIR exists. .sp Raises an error if it does not exist and cannot be created. Safe to run simultaneously in multiple processes. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.check_and_create_default_config_file() If the default config file does not exist, create it in the Toil home directory. Create the Toil home directory if needed .sp Raises an error if the default config file cannot be created. Safe to run simultaneously in multiple processes. If this process runs this function, it will always see the default config file existing with parseable contents, even if other processes are racing to create it. .sp No process will see an empty or partially\-written default config file. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.check_and_create_config_file(filepath) If the config file at the filepath does not exist, try creating it. The parent directory should be created prior to calling this :param filepath: path to config file :return: None .INDENT 7.0 .TP .B Parameters \fBfilepath\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.generate_config(filepath) Write a Toil config file to the given path. .sp Safe to run simultaneously in multiple processes. No process will see an empty or partially\-written file at the given path. .sp Set include to \(dqcwl\(dq or \(dqwdl\(dq to include cwl options and wdl options respectfully .INDENT 7.0 .TP .B Parameters \fBfilepath\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.parser_with_common_options(provisioner_options=False, jobstore_option=True, prog=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBprovisioner_options\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBjobstore_option\fP (\fI\%bool\fP) \-\- .IP \(bu 2 \fBprog\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type configargparse.ArgParser .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.addOptions(parser, jobstore_as_flag=False, cwl=False, wdl=False) Add all Toil command line options to a parser. .sp Support for config files if using configargparse. This will also check and set up the default config file. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .IP \(bu 2 \fBcwl\fP (\fI\%bool\fP) \-\- Whether CWL options are expected. If so, CWL options won\(aqt be suppressed. .IP \(bu 2 \fBwdl\fP (\fI\%bool\fP) \-\- Whether WDL options are expected. If so, WDL options won\(aqt be suppressed. .IP \(bu 2 \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.getNodeID() Return unique ID of the current node (host). The resulting string will be convertable to a uuid.UUID. .sp Tries several methods until success. The returned ID should be identical across calls from different processes on the same node at least until the next OS reboot. .sp The last resort method is uuid.getnode() that in some rare OS configurations may return a random ID each time it is called. However, this method should never be reached on a Linux system, because reading from /proc/sys/kernel/random/boot_id will be tried prior to that. If uuid.getnode() is reached, it will be called twice, and exception raised if the values are not identical. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.common.Toil(options) Bases: \fBContextManager\fP[\fI\%Toil\fP] .sp A context manager that represents a Toil workflow. .sp Specifically the batch system, job store, and its configuration. .INDENT 7.0 .TP .B Parameters \fBoptions\fP (\fI\%argparse.Namespace\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B config: \fI\%Config\fP .UNINDENT .INDENT 7.0 .TP .B __enter__() Derive configuration from the command line options. .sp Then load the job store and, on restart, consolidate the derived configuration with the one from the previous invocation of the workflow. .INDENT 7.0 .TP .B Return type \fI\%Toil\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __exit__(exc_type, exc_val, exc_tb) Clean up after a workflow invocation. .sp Depending on the configuration, delete the job store. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexc_type\fP (\fIOptional\fP\fI[\fP\fIType\fP\fI[\fP\fI\%BaseException\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBexc_val\fP (\fIOptional\fP\fI[\fP\fI\%BaseException\fP\fI]\fP) \-\- .IP \(bu 2 \fBexc_tb\fP (\fIOptional\fP\fI[\fP\fI\%types.TracebackType\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type Literal[False] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B start(rootJob) Invoke a Toil workflow with the given job as the root for an initial run. .sp This method must be called in the body of a \fBwith Toil(...) as toil:\fP statement. This method should not be called more than once for a workflow that has not finished. .INDENT 7.0 .TP .B Parameters \fBrootJob\fP (\fI\%toil.job.Job\fP) \-\- The root job of the workflow .TP .B Returns The root job\(aqs return value .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B restart() Restarts a workflow that has been interrupted. .INDENT 7.0 .TP .B Returns The root job\(aqs return value .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod getJobStore(locator) Create an instance of the concrete job store implementation that matches the given locator. .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- The location of the job store to be represent by the instance .TP .B Returns an instance of a concrete subclass of AbstractJobStore .TP .B Return type \fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static parseLocator(locator) .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .TP .B Return type Tuple[\fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static buildLocator(name, rest) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBrest\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod resumeJobStore(locator) .INDENT 7.0 .TP .B Parameters \fBlocator\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static createBatchSystem(config) Create an instance of the batch system specified in the given config. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%Config\fP) \-\- the current configuration .TP .B Returns an instance of a concrete subclass of AbstractBatchSystem .TP .B Return type \fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B importFile(srcUrl: \fI\%str\fP, sharedFileName: \fI\%str\fP, symlink: \fI\%bool\fP = True) -> \fI\%None\fP .TP .B importFile(srcUrl: \fI\%str\fP, sharedFileName: \fI\%None\fP = None, symlink: \fI\%bool\fP = True) -> \fI\%toil.fileStores.FileID\fP .UNINDENT .INDENT 7.0 .TP .B import_file(src_uri: \fI\%str\fP, shared_file_name: \fI\%str\fP, symlink: \fI\%bool\fP = True, check_existence: \fI\%bool\fP = True) -> \fI\%None\fP .TP .B import_file(src_uri: \fI\%str\fP, shared_file_name: \fI\%None\fP = None, symlink: \fI\%bool\fP = True, check_existence: \fI\%bool\fP = True) -> \fI\%toil.fileStores.FileID\fP Import the file at the given URL into the job store. .sp By default, returns None if the file does not exist. .INDENT 7.0 .TP .B Parameters \fBcheck_existence\fP \-\- If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist. .UNINDENT .sp See \fI\%toil.jobStores.abstractJobStore.AbstractJobStore.importFile()\fP for a full description .UNINDENT .INDENT 7.0 .TP .B exportFile(jobStoreFileID, dstUrl) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStoreFileID\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdstUrl\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B export_file(file_id, dst_uri) Export file to destination pointed at by the destination URL. .sp See \fI\%toil.jobStores.abstractJobStore.AbstractJobStore.exportFile()\fP for a full description .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfile_id\fP (\fI\%toil.fileStores.FileID\fP) \-\- .IP \(bu 2 \fBdst_uri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static normalize_uri(uri, check_existence=False) Given a URI, if it has no scheme, prepend \(dqfile:\(dq. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcheck_existence\fP (\fI\%bool\fP) \-\- If set, raise FileNotFoundError if a URI points to a local file that does not exist. .IP \(bu 2 \fBuri\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static getToilWorkDir(configWorkDir=None) Return a path to a writable directory under which per\-workflow directories exist. .sp This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR. .INDENT 7.0 .TP .B Parameters \fBconfigWorkDir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Value passed to the program using the \-\-workDir flag .TP .B Returns Path to the Toil work directory, constant across all machines .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir) Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig_work_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Value passed to the program using the \-\-workDir flag .IP \(bu 2 \fBconfig_coordination_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Value passed to the program using the \-\-coordinationDir flag .UNINDENT .TP .B Returns Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static get_workflow_path_component(workflow_id) Get a safe filesystem path component for a workflow. .sp Will be consistent for all processes on a given machine, and different for all processes on different machines. .INDENT 7.0 .TP .B Parameters \fBworkflow_id\fP (\fI\%str\fP) \-\- The ID of the current Toil workflow. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod getLocalWorkflowDir(workflowID, configWorkDir=None) Return the directory where worker directories and the cache will be located for this workflow on this machine. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfigWorkDir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Value passed to the program using the \-\-workDir flag .IP \(bu 2 \fBworkflowID\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Returns Path to the local workflow directory on this machine .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir) Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine. .sp If an in\-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBworkflow_id\fP (\fI\%str\fP) \-\- Unique ID of the current workflow. .IP \(bu 2 \fBconfig_work_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Value used for the work directory in the current Toil Config. .IP \(bu 2 \fBconfig_coordination_dir\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Value used for the coordination directory in the current Toil Config. .UNINDENT .TP .B Returns Path to the local workflow coordination directory on this machine. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.common.ToilRestartException(message) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.common.ToilContextManagerException Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .UNINDENT .INDENT 0.0 .TP .B class toil.common.ToilMetrics(bus, provisioner=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbus\fP (\fI\%toil.bus.MessageBus\fP) \-\- .IP \(bu 2 \fBprovisioner\fP (\fIOptional\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B startDashboard(clusterName, zone) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBclusterName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBzone\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B add_prometheus_data_source() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log(message) .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logClusterSize(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.ClusterSizeMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logClusterDesiredSize(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.ClusterDesiredSizeMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logQueueSize(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.QueueSizeMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logMissingJob(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.JobMissingMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logIssuedJob(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.JobIssuedMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logFailedJob(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.JobFailedMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B logCompletedJob(m) .INDENT 7.0 .TP .B Parameters \fBm\fP (\fI\%toil.bus.JobCompletedMessage\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.cacheDirName(workflowID) .INDENT 7.0 .TP .B Returns Name of the cache directory. .TP .B Parameters \fBworkflowID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.getDirSizeRecursively(dirPath) This method will return the cumulative number of bytes occupied by the files on disk in the directory and its subdirectories. .sp If the method is unable to access a file or directory (due to insufficient permissions, or due to the file or directory having been removed while this function was attempting to traverse it), the error will be handled internally, and a (possibly 0) lower bound on the size of the directory will be returned. .INDENT 7.0 .TP .B Parameters \fBdirPath\fP (\fI\%str\fP) \-\- A valid path to a directory or file. .TP .B Returns Total size, in bytes, of the file or directory at dirPath. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.getFileSystemSize(dirPath) Return the free space, and total size of the file system hosting \fIdirPath\fP\&. .INDENT 7.0 .TP .B Parameters \fBdirPath\fP (\fI\%str\fP) \-\- A valid path to a directory. .TP .B Returns free space and total size of file system .TP .B Return type Tuple[\fI\%int\fP, \fI\%int\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.common.safeUnpickleFromStream(stream) .INDENT 7.0 .TP .B Parameters \fBstream\fP (\fIIO\fP\fI[\fP\fIAny\fP\fI]\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .SS \fI\%toil.deferred\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%DeferredFunction\fP T} T{ .sp .EX >>> from collections import defaultdict .EE T} _ T{ \fI\%DeferredFunctionManager\fP T} T{ Implements a deferred function system. Each Toil worker will have an T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.deferred.logger .UNINDENT .INDENT 0.0 .TP .B class toil.deferred.DeferredFunction Bases: \fBnamedtuple\fP(\fB\(aqDeferredFunction\(aq\fP, \fB\(aqfunction args kwargs name module\(aq\fP) .sp .EX >>> from collections import defaultdict >>> df = DeferredFunction.create(defaultdict, None, {\(aqx\(aq:1}, y=2) >>> df DeferredFunction(defaultdict, ...) >>> df.invoke() == defaultdict(None, x=1, y=2) True .EE .INDENT 7.0 .TP .B __repr__ .UNINDENT .INDENT 7.0 .TP .B classmethod create(function, *args, **kwargs) Capture the given callable and arguments as an instance of this class. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfunction\fP (\fIcallable\fP) \-\- The deferred action to take in the form of a function .IP \(bu 2 \fBargs\fP (\fI\%tuple\fP) \-\- Non\-keyword arguments to the function .IP \(bu 2 \fBkwargs\fP (\fI\%dict\fP) \-\- Keyword arguments to the function .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B invoke() Invoke the captured function with the captured arguments. .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.deferred.DeferredFunctionManager(stateDirBase) Implements a deferred function system. Each Toil worker will have an instance of this class. When a job is executed, it will happen inside a context manager from this class. If the job registers any \(dqdeferred\(dq functions, they will be executed when the context manager is exited. .sp If the Python process terminates before properly exiting the context manager and running the deferred functions, and some other worker process enters or exits the per\-job context manager of this class at a later time, or when the DeferredFunctionManager is shut down on the worker, the earlier job\(aqs deferred functions will be picked up and run. .sp Note that deferred function cleanup is on a best\-effort basis, and deferred functions may end up getting executed multiple times. .sp Internally, deferred functions are serialized into files in the given directory, which are locked by the owning process. .sp If that process dies, other processes can detect that the files are able to be locked, and will take them over. .INDENT 7.0 .TP .B Parameters \fBstateDirBase\fP (\fI\%str\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B STATE_DIR_STEM = \(aqdeferred\(aq .UNINDENT .INDENT 7.0 .TP .B PREFIX = \(aqfunc\(aq .UNINDENT .INDENT 7.0 .TP .B WIP_SUFFIX = \(aq.tmp\(aq .UNINDENT .INDENT 7.0 .TP .B __del__() Clean up our state on disk. We assume that the deferred functions we manage have all been executed, and none are currently recorded. .UNINDENT .INDENT 7.0 .TP .B open() Yields a single\-argument function that allows for deferred functions of type \fBtoil.DeferredFunction\fP to be registered. We use this design so deferred functions can be registered only inside this context manager. .sp Not thread safe. .UNINDENT .INDENT 7.0 .TP .B classmethod cleanupWorker(stateDirBase) Called by the batch system when it shuts down the node, after all workers are done, if the batch system supports worker cleanup. Checks once more for orphaned deferred functions and runs them. .INDENT 7.0 .TP .B Parameters \fBstateDirBase\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.exceptions\fP .sp Neutral place for exceptions, to break import cycles. .SS Module Contents .INDENT 0.0 .TP .B toil.exceptions.logger .UNINDENT .INDENT 0.0 .TP .B exception toil.exceptions.FailedJobsException(job_store, failed_jobs, exit_code=1) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_store\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBfailed_jobs\fP (\fIList\fP\fI[\fP\fI\%toil.job.JobDescription\fP\fI]\fP) \-\- .IP \(bu 2 \fBexit_code\fP (\fI\%int\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Stringify the exception, including the message. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.job\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%TemporaryID\fP T} T{ Placeholder for a unregistered job ID used by a JobDescription. T} _ T{ \fI\%AcceleratorRequirement\fP T} T{ Requirement for one or more computational accelerators, like a GPU or FPGA. T} _ T{ \fI\%RequirementsDict\fP T} T{ Typed storage for requirements for a job. T} _ T{ \fI\%Requirer\fP T} T{ Base class implementing the storage and presentation of requirements. T} _ T{ \fI\%JobDescription\fP T} T{ Stores all the information that the Toil Leader ever needs to know about a Job. T} _ T{ \fI\%ServiceJobDescription\fP T} T{ A description of a job that hosts a service. T} _ T{ \fI\%CheckpointJobDescription\fP T} T{ A description of a job that is a checkpoint. T} _ T{ \fI\%Job\fP T} T{ Class represents a unit of work in toil. T} _ T{ \fI\%FunctionWrappingJob\fP T} T{ Job used to wrap a function. In its \fIrun\fP method the wrapped function is called. T} _ T{ \fI\%JobFunctionWrappingJob\fP T} T{ A job function is a function whose first argument is a \fI\%Job\fP T} _ T{ \fI\%PromisedRequirementFunctionWrappingJob\fP T} T{ Handles dynamic resource allocation using \fI\%toil.job.Promise\fP instances. T} _ T{ \fI\%PromisedRequirementJobFunctionWrappingJob\fP T} T{ Handles dynamic resource allocation for job functions. T} _ T{ \fI\%EncapsulatedJob\fP T} T{ A convenience Job class used to make a job subgraph appear to be a single job. T} _ T{ \fI\%ServiceHostJob\fP T} T{ Job that runs a service. Used internally by Toil. Users should subclass Service instead of using this. T} _ T{ \fI\%Promise\fP T} T{ References a return value from a method as a \fIpromise\fP before the method itself is run. T} _ T{ \fI\%PromisedRequirement\fP T} T{ Class for dynamically allocating job function resource requirements. T} _ T{ \fI\%UnfulfilledPromiseSentinel\fP T} T{ This should be overwritten by a proper promised value. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%parse_accelerator\fP(spec) T} T{ Parse an AcceleratorRequirement specified by user code. T} _ T{ \fI\%accelerator_satisfies\fP(candidate, requirement[, ignore]) T} T{ Test if candidate partially satisfies the given requirement. T} _ T{ \fI\%accelerators_fully_satisfy\fP(candidates, requirement[, ...]) T} T{ Determine if a set of accelerators satisfy a requirement. T} _ T{ \fI\%unwrap\fP(p) T} T{ Function for ensuring you actually have a promised value, and not just a promise. T} _ T{ \fI\%unwrap_all\fP(p) T} T{ Function for ensuring you actually have a collection of promised values, T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%REQUIREMENT_NAMES\fP T} T{ T} _ T{ \fI\%ParsedRequirement\fP T} T{ T} _ T{ \fI\%ParseableIndivisibleResource\fP T} T{ T} _ T{ \fI\%ParseableDivisibleResource\fP T} T{ T} _ T{ \fI\%ParseableFlag\fP T} T{ T} _ T{ \fI\%ParseableAcceleratorRequirement\fP T} T{ T} _ T{ \fI\%ParseableRequirement\fP T} T{ T} _ T{ \fI\%T\fP T} T{ T} _ T{ \fI\%Promised\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.job.logger .UNINDENT .INDENT 0.0 .TP .B exception toil.job.JobPromiseConstraintError(promisingJob, recipientJob=None) Bases: \fI\%RuntimeError\fP .sp Error for job being asked to promise its return value, but it not available. .sp (Due to the return value not yet been hit in the topological order of the job graph.) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpromisingJob\fP (\fI\%Job\fP) \-\- .IP \(bu 2 \fBrecipientJob\fP (\fIOptional\fP\fI[\fP\fI\%Job\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.job.ConflictingPredecessorError(predecessor, successor) Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpredecessor\fP (\fI\%Job\fP) \-\- .IP \(bu 2 \fBsuccessor\fP (\fI\%Job\fP) \-\- .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.TemporaryID Placeholder for a unregistered job ID used by a JobDescription. .INDENT 7.0 .TP .B Needs to be held: .INDENT 7.0 .IP \(bu 2 By JobDescription objects to record normal relationships. .IP \(bu 2 By Jobs to key their connected\-component registries and to record predecessor relationships to facilitate EncapsulatedJob adding itself as a child. .IP \(bu 2 By Services to tie back to their hosting jobs, so the service tree can be built up from Service objects. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Return str(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __hash__() Return hash(self). .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __eq__(other) Return self==value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __ne__(other) Return self!=value. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.AcceleratorRequirement Bases: \fBTypedDict\fP .sp Requirement for one or more computational accelerators, like a GPU or FPGA. .INDENT 7.0 .TP .B count: \fI\%int\fP How many of the accelerator are needed to run the job. .UNINDENT .INDENT 7.0 .TP .B kind: \fI\%str\fP What kind of accelerator is required. Can be \(dqgpu\(dq. Other kinds defined in the future might be \(dqfpga\(dq, etc. .UNINDENT .INDENT 7.0 .TP .B model: typing_extensions.NotRequired[\fI\%str\fP] What model of accelerator is needed. The exact set of values available depends on what the backing scheduler calls its accelerators; strings like \(dqnvidia\-tesla\-k80\(dq might be expected to work. If a specific model of accelerator is not required, this should be absent. .UNINDENT .INDENT 7.0 .TP .B brand: typing_extensions.NotRequired[\fI\%str\fP] What brand or manufacturer of accelerator is required. The exact set of values available depends on what the backing scheduler calls the brands of its accleerators; strings like \(dqnvidia\(dq or \(dqamd\(dq might be expected to work. If a specific brand of accelerator is not required (for example, because the job can use multiple brands of accelerator that support a given API) this should be absent. .UNINDENT .INDENT 7.0 .TP .B api: typing_extensions.NotRequired[\fI\%str\fP] What API is to be used to communicate with the accelerator. This can be \(dqcuda\(dq. Other APIs supported in the future might be \(dqrocm\(dq, \(dqopencl\(dq, \(dqmetal\(dq, etc. If the job does not need a particular API to talk to the accelerator, this should be absent. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.job.parse_accelerator(spec) Parse an AcceleratorRequirement specified by user code. .sp Supports formats like: .sp .EX >>> parse_accelerator(8) {\(aqcount\(aq: 8, \(aqkind\(aq: \(aqgpu\(aq} .EE .sp .EX >>> parse_accelerator(\(dq1\(dq) {\(aqcount\(aq: 1, \(aqkind\(aq: \(aqgpu\(aq} .EE .sp .EX >>> parse_accelerator(\(dqnvidia\-tesla\-k80\(dq) {\(aqcount\(aq: 1, \(aqkind\(aq: \(aqgpu\(aq, \(aqbrand\(aq: \(aqnvidia\(aq, \(aqmodel\(aq: \(aqnvidia\-tesla\-k80\(aq} .EE .sp .EX >>> parse_accelerator(\(dqnvidia\-tesla\-k80:2\(dq) {\(aqcount\(aq: 2, \(aqkind\(aq: \(aqgpu\(aq, \(aqbrand\(aq: \(aqnvidia\(aq, \(aqmodel\(aq: \(aqnvidia\-tesla\-k80\(aq} .EE .sp .EX >>> parse_accelerator(\(dqgpu\(dq) {\(aqcount\(aq: 1, \(aqkind\(aq: \(aqgpu\(aq} .EE .sp .EX >>> parse_accelerator(\(dqcuda:1\(dq) {\(aqcount\(aq: 1, \(aqkind\(aq: \(aqgpu\(aq, \(aqbrand\(aq: \(aqnvidia\(aq, \(aqapi\(aq: \(aqcuda\(aq} .EE .sp .EX >>> parse_accelerator({\(dqkind\(dq: \(dqgpu\(dq}) {\(aqcount\(aq: 1, \(aqkind\(aq: \(aqgpu\(aq} .EE .sp .EX >>> parse_accelerator({\(dqbrand\(dq: \(dqnvidia\(dq, \(dqcount\(dq: 5}) {\(aqcount\(aq: 5, \(aqkind\(aq: \(aqgpu\(aq, \(aqbrand\(aq: \(aqnvidia\(aq} .EE .sp Assumes that if not specified, we are talking about GPUs, and about one of them. Knows that \(dqgpu\(dq is a kind, and \(dqcuda\(dq is an API, and \(dqnvidia\(dq is a brand. .INDENT 7.0 .TP .B Raises .INDENT 7.0 .IP \(bu 2 \fI\%ValueError\fP \-\- if it gets somethign it can\(aqt parse .IP \(bu 2 \fI\%TypeError\fP \-\- if it gets something it can\(aqt parse because it\(aqs the wrong type. .UNINDENT .TP .B Parameters \fBspec\fP (\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%str\fP\fI, \fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%int\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type \fI\%AcceleratorRequirement\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.job.accelerator_satisfies(candidate, requirement, ignore=[]) Test if candidate partially satisfies the given requirement. .INDENT 7.0 .TP .B Returns True if the given candidate at least partially satisfies the given requirement (i.e. check all fields other than count). .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcandidate\fP (\fI\%AcceleratorRequirement\fP) \-\- .IP \(bu 2 \fBrequirement\fP (\fI\%AcceleratorRequirement\fP) \-\- .IP \(bu 2 \fBignore\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.job.accelerators_fully_satisfy(candidates, requirement, ignore=[]) Determine if a set of accelerators satisfy a requirement. .sp Ignores fields specified in ignore. .INDENT 7.0 .TP .B Returns True if the requirement AcceleratorRequirement is fully satisfied by the ones in the list, taken together (i.e. check all fields including count). .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBcandidates\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%AcceleratorRequirement\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBrequirement\fP (\fI\%AcceleratorRequirement\fP) \-\- .IP \(bu 2 \fBignore\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.RequirementsDict Bases: \fBTypedDict\fP .sp Typed storage for requirements for a job. .sp Where requirement values are of different types depending on the requirement. .INDENT 7.0 .TP .B cores: typing_extensions.NotRequired[\fI\%int\fP | \fI\%float\fP] .UNINDENT .INDENT 7.0 .TP .B memory: typing_extensions.NotRequired[\fI\%int\fP] .UNINDENT .INDENT 7.0 .TP .B disk: typing_extensions.NotRequired[\fI\%int\fP] .UNINDENT .INDENT 7.0 .TP .B accelerators: typing_extensions.NotRequired[List[\fI\%AcceleratorRequirement\fP]] .UNINDENT .INDENT 7.0 .TP .B preemptible: typing_extensions.NotRequired[\fI\%bool\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.job.REQUIREMENT_NAMES = [\(aqdisk\(aq, \(aqmemory\(aq, \(aqcores\(aq, \(aqaccelerators\(aq, \(aqpreemptible\(aq] .UNINDENT .INDENT 0.0 .TP .B toil.job.ParsedRequirement .UNINDENT .INDENT 0.0 .TP .B toil.job.ParseableIndivisibleResource .UNINDENT .INDENT 0.0 .TP .B toil.job.ParseableDivisibleResource .UNINDENT .INDENT 0.0 .TP .B toil.job.ParseableFlag .UNINDENT .INDENT 0.0 .TP .B toil.job.ParseableAcceleratorRequirement .UNINDENT .INDENT 0.0 .TP .B toil.job.ParseableRequirement .UNINDENT .INDENT 0.0 .TP .B class toil.job.Requirer(requirements) Base class implementing the storage and presentation of requirements. .sp Has cores, memory, disk, and preemptability as properties. .INDENT 7.0 .TP .B Parameters \fBrequirements\fP (\fIMapping\fP\fI[\fP\fI\%str\fP\fI, \fP\fIParseableRequirement\fP\fI]\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B property requirements: \fI\%RequirementsDict\fP Get dict containing all non\-None, non\-defaulted requirements. .INDENT 7.0 .TP .B Return type \fI\%RequirementsDict\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property disk: \fI\%int\fP Get the maximum number of bytes of disk required. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property memory: \fI\%int\fP Get the maximum number of bytes of memory required. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property cores: \fI\%int\fP | \fI\%float\fP Get the number of CPU cores required. .INDENT 7.0 .TP .B Return type Union[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property preemptible: \fI\%bool\fP Whether a preemptible node is permitted, or a nonpreemptible one is required. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property accelerators: List[\fI\%AcceleratorRequirement\fP] Any accelerators, such as GPUs, that are needed. .INDENT 7.0 .TP .B Return type List[\fI\%AcceleratorRequirement\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B assignConfig(config) Assign the given config object to be used to provide default values. .sp Must be called exactly once on a loaded JobDescription before any requirements are queried. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- Config object to query .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __getstate__() Return the dict to use as the instance\(aqs __dict__ when pickling. .INDENT 7.0 .TP .B Return type Dict[\fI\%str\fP, Any] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __copy__() Return a semantically\-shallow copy of the object, for \fBcopy.copy()\fP\&. .INDENT 7.0 .TP .B Return type \fI\%Requirer\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __deepcopy__(memo) Return a semantically\-deep copy of the object, for \fBcopy.deepcopy()\fP\&. .INDENT 7.0 .TP .B Parameters \fBmemo\fP (\fIAny\fP) \-\- .TP .B Return type \fI\%Requirer\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B preemptable(val) .INDENT 7.0 .TP .B Parameters \fBval\fP (\fIParseableFlag\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B scale(requirement, factor) Return a copy of this object with the given requirement scaled up or down. .sp Only works on requirements where that makes sense. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirement\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfactor\fP (\fI\%float\fP) \-\- .UNINDENT .TP .B Return type \fI\%Requirer\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B requirements_string() Get a nice human\-readable string of our requirements. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.JobDescription(requirements, jobName, unitName=\(aq\(aq, displayName=\(aq\(aq, command=None, local=None) Bases: \fI\%Requirer\fP .sp Stores all the information that the Toil Leader ever needs to know about a Job. .sp (requirements information, dependency information, commands to issue, etc.) .sp Can be obtained from an actual (i.e. executable) Job object, and can be used to obtain the Job object from the JobStore. .sp Never contains other Jobs or JobDescriptions: all reference is by ID. .sp Subclassed into variants for checkpoint jobs and service jobs that have their specific parameters. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrequirements\fP (\fIMapping\fP\fI[\fP\fI\%str\fP\fI, \fP\fIUnion\fP\fI[\fP\fI\%int\fP\fI, \fP\fI\%str\fP\fI, \fP\fI\%bool\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBunitName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisplayName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcommand\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBlocal\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property services Get a collection of the IDs of service host jobs for this job, in arbitrary order. .sp Will be empty if the job has no unfinished services. .UNINDENT .INDENT 7.0 .TP .B property remainingTryCount Get the number of tries remaining. .sp The try count set on the JobDescription, or the default based on the retry count from the config if none is set. .UNINDENT .INDENT 7.0 .TP .B get_names() Get the names and ID of this job as a named tuple. .INDENT 7.0 .TP .B Return type \fI\%toil.bus.Names\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_chain() Get all the jobs that executed in this job\(aqs chain, in order. .sp For each job, produces a named tuple with its various names and its original job store ID. The jobs in the chain are in execution order. .sp If the job hasn\(aqt run yet or it didn\(aqt chain, produces a one\-item list. .INDENT 7.0 .TP .B Return type List[\fI\%toil.bus.Names\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B serviceHostIDsInBatches() Find all batches of service host job IDs that can be started at the same time. .sp (in the order they need to start in) .INDENT 7.0 .TP .B Return type Iterator[List[\fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B successorsAndServiceHosts() Get an iterator over all child, follow\-on, and service job IDs. .INDENT 7.0 .TP .B Return type Iterator[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B allSuccessors() Get an iterator over all child, follow\-on, and chained, inherited successor job IDs. .sp Follow\-ons will come before children. .INDENT 7.0 .TP .B Return type Iterator[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B successors_by_phase() Get an iterator over all child/follow\-on/chained inherited successor job IDs, along with their phase numbere on the stack. .sp Phases ececute higher numbers to lower numbers. .INDENT 7.0 .TP .B Return type Iterator[Tuple[\fI\%int\fP, \fI\%str\fP]] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B nextSuccessors() Return the collection of job IDs for the successors of this job that are ready to run. .sp If those jobs have multiple predecessor relationships, they may still be blocked on other jobs. .sp Returns None when at the final phase (all successors done), and an empty collection if there are more phases but they can\(aqt be entered yet (e.g. because we are waiting for the job itself to run). .INDENT 7.0 .TP .B Return type Set[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B filterSuccessors(predicate) Keep only successor jobs for which the given predicate function approves. .sp The predicate function is called with the job\(aqs ID. .sp Treats all other successors as complete and forgets them. .INDENT 7.0 .TP .B Parameters \fBpredicate\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fI\%bool\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B filterServiceHosts(predicate) Keep only services for which the given predicate approves. .sp The predicate function is called with the service host job\(aqs ID. .sp Treats all other services as complete and forgets them. .INDENT 7.0 .TP .B Parameters \fBpredicate\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fI\%bool\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clear_nonexistent_dependents(job_store) Remove all references to child, follow\-on, and associated service jobs that do not exist. .sp That is to say, all those that have been completed and removed. .INDENT 7.0 .TP .B Parameters \fBjob_store\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B clear_dependents() Remove all references to successor and service jobs. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_subtree_done() Check if the subtree is done. .INDENT 7.0 .TP .B Returns True if the job appears to be done, and all related child, follow\-on, and service jobs appear to be finished and removed. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B replace(other) Take on the ID of another JobDescription, retaining our own state and type. .sp When updated in the JobStore, we will save over the other JobDescription. .sp Useful for chaining jobs: the chained\-to job can replace the parent job. .sp Merges cleanup state and successors other than this job from the job being replaced into this one. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%JobDescription\fP) \-\- Job description to replace. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check_new_version(other) Make sure a prospective new version of the JobDescription is actually moving forward in time and not backward. .INDENT 7.0 .TP .B Parameters \fBother\fP (\fI\%JobDescription\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChild(childID) Make the job with the given ID a child of the described job. .INDENT 7.0 .TP .B Parameters \fBchildID\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOnID) Make the job with the given ID a follow\-on of the described job. .INDENT 7.0 .TP .B Parameters \fBfollowOnID\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addServiceHostJob(serviceID, parentServiceID=None) Make the ServiceHostJob with the given ID a service of the described job. .sp If a parent ServiceHostJob ID is given, that parent service will be started first, and must have already been added. .UNINDENT .INDENT 7.0 .TP .B hasChild(childID) Return True if the job with the given ID is a child of the described job. .INDENT 7.0 .TP .B Parameters \fBchildID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasFollowOn(followOnID) Test if the job with the given ID is a follow\-on of the described job. .INDENT 7.0 .TP .B Parameters \fBfollowOnID\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasServiceHostJob(serviceID) Test if the ServiceHostJob is a service of the described job. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B renameReferences(renames) Apply the given dict of ID renames to all references to jobs. .sp Does not modify our own ID or those of finished predecessors. IDs not present in the renames dict are left as\-is. .INDENT 7.0 .TP .B Parameters \fBrenames\fP (\fIDict\fP\fI[\fP\fI\%TemporaryID\fP\fI, \fP\fI\%str\fP\fI]\fP) \-\- Rename operations to apply. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addPredecessor() Notify the JobDescription that a predecessor has been added to its Job. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B onRegistration(jobStore) Perform setup work that requires the JobStore. .sp Called by the Job saving logic when this JobDescription meets the JobStore and has its ID assigned. .sp Overridden to perform setup work (like hooking up flag files for service jobs) that requires the JobStore. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- The job store we are being placed into .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B setupJobAfterFailure(exit_status=None, exit_reason=None) Configure job after a failure. .sp Reduce the remainingTryCount if greater than zero and set the memory to be at least as big as the default memory (in case of exhaustion of memory, which is common). .sp Requires a configuration to have been assigned (see \fI\%toil.job.Requirer.assignConfig()\fP). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexit_status\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- The exit code from the job. .IP \(bu 2 \fBexit_reason\fP (\fIOptional\fP\fI[\fP\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP\fI]\fP) \-\- The reason the job stopped, if available from the batch system. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getLogFileHandle(jobStore) Create a context manager that yields a file handle to the log file. .sp Assumes logJobStoreFileID is set. .UNINDENT .INDENT 7.0 .TP .B clearRemainingTryCount() Clear remainingTryCount and set it back to its default value. .INDENT 7.0 .TP .B Returns True if a modification to the JobDescription was made, and False otherwise. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Produce a useful logging string identifying this job. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __repr__() Return repr(self). .UNINDENT .INDENT 7.0 .TP .B reserve_versions(count) Reserve a job version number for later, for journaling asynchronously. .INDENT 7.0 .TP .B Parameters \fBcount\fP (\fI\%int\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pre_update_hook() Run before pickling and saving a created or updated version of this job. .sp Called by the job store. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.ServiceJobDescription(*args, **kwargs) Bases: \fI\%JobDescription\fP .sp A description of a job that hosts a service. .INDENT 7.0 .TP .B onRegistration(jobStore) Setup flag files. .sp When a ServiceJobDescription first meets the JobStore, it needs to set up its flag files. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.CheckpointJobDescription(*args, **kwargs) Bases: \fI\%JobDescription\fP .sp A description of a job that is a checkpoint. .INDENT 7.0 .TP .B restartCheckpoint(jobStore) Restart a checkpoint after the total failure of jobs in its subtree. .sp Writes the changes to the jobStore immediately. All the checkpoint\(aqs successors will be deleted, but its try count will \fInot\fP be decreased. .sp Returns a list with the IDs of any successors deleted. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type List[\fI\%str\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.Job(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName=\(aq\(aq, checkpoint=False, displayName=\(aq\(aq, descriptionClass=None, local=None) Class represents a unit of work in toil. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBmemory\fP (\fIOptional\fP\fI[\fP\fIParseableIndivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBcores\fP (\fIOptional\fP\fI[\fP\fIParseableDivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisk\fP (\fIOptional\fP\fI[\fP\fIParseableIndivisibleResource\fP\fI]\fP) \-\- .IP \(bu 2 \fBaccelerators\fP (\fIOptional\fP\fI[\fP\fIParseableAcceleratorRequirement\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fIParseableFlag\fP\fI]\fP) \-\- .IP \(bu 2 \fBpreemptable\fP (\fIOptional\fP\fI[\fP\fIParseableFlag\fP\fI]\fP) \-\- .IP \(bu 2 \fBunitName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBcheckpoint\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .IP \(bu 2 \fBdisplayName\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBdescriptionClass\fP (\fIOptional\fP\fI[\fP\fI\%type\fP\fI]\fP) \-\- .IP \(bu 2 \fBlocal\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class Runner Used to setup and run Toil workflow. .INDENT 7.0 .TP .B static getDefaultArgumentParser(jobstore_as_flag=False) Get argument parser with added toil workflow options. .INDENT 7.0 .TP .B Parameters \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .TP .B Returns The argument parser used by a toil workflow with added Toil options. .TP .B Return type \fI\%argparse.ArgumentParser\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static getDefaultOptions(jobStore=None, jobstore_as_flag=False) Get default options for a toil workflow. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- A string describing the jobStore for the workflow. .IP \(bu 2 \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .UNINDENT .TP .B Returns The options used by a toil workflow. .TP .B Return type \fI\%argparse.Namespace\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static addToilOptions(parser, jobstore_as_flag=False) Adds the default toil options to an \fI\%optparse\fP or \fI\%argparse\fP parser object. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBparser\fP (\fIUnion\fP\fI[\fP\fI\%optparse.OptionParser\fP\fI, \fP\fI\%argparse.ArgumentParser\fP\fI]\fP) \-\- Options object to add toil options to. .IP \(bu 2 \fBjobstore_as_flag\fP (\fI\%bool\fP) \-\- make the job store option a \-\-jobStore flag instead of a required jobStore positional argument. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static startToil(job, options) Run the toil workflow using the given options. .sp Deprecated by toil.common.Toil.start. .sp (see Job.Runner.getDefaultOptions and Job.Runner.addToilOptions) starting with this job. :param job: root job of the workflow :raises: toil.exceptions.FailedJobsException if at the end of function there remain failed jobs. :return: The return value of the root job\(aqs run function. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B class Service(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName=None) Bases: \fI\%Requirer\fP .sp Abstract class used to define the interface to a service. .sp Should be subclassed by the user to define services. .sp Is not executed as a job; runs within a ServiceHostJob. .INDENT 7.0 .TP .B abstract start(job) Start the service. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Returns An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see \fI\%toil.job.Job.addService()\fP). .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B abstract stop(job) Stops the service. Function can block until complete. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Checks the service is still running. .INDENT 7.0 .TP .B Raises \fBexceptions.RuntimeError\fP \-\- If the service failed, this will cause the service job to be labeled failed. .TP .B Returns True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False! .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property jobStoreID: \fI\%str\fP | \fI\%TemporaryID\fP Get the ID of this Job. .INDENT 7.0 .TP .B Return type Union[\fI\%str\fP, \fI\%TemporaryID\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property description: \fI\%JobDescription\fP Expose the JobDescription that describes this job. .INDENT 7.0 .TP .B Return type \fI\%JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property disk: \fI\%int\fP The maximum number of bytes of disk the job will require to run. .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property memory The maximum number of bytes of memory the job will require to run. .UNINDENT .INDENT 7.0 .TP .B property cores: \fI\%int\fP | \fI\%float\fP The number of CPU cores required. .INDENT 7.0 .TP .B Return type Union[\fI\%int\fP, \fI\%float\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property accelerators: List[\fI\%AcceleratorRequirement\fP] Any accelerators, such as GPUs, that are needed. .INDENT 7.0 .TP .B Return type List[\fI\%AcceleratorRequirement\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property preemptible: \fI\%bool\fP Whether the job can be run on a preemptible node. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property checkpoint: \fI\%bool\fP Determine if the job is a checkpoint job or not. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property tempDir: \fI\%str\fP Shortcut to calling \fBjob.fileStore.getLocalTempDir()\fP\&. .sp Temp dir is created on first call and will be returned for first and future calls :return: Path to tempDir. See \fIjob.fileStore.getLocalTempDir\fP .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __str__() Produce a useful logging string to identify this Job and distinguish it from its JobDescription. .UNINDENT .INDENT 7.0 .TP .B preemptable() .UNINDENT .INDENT 7.0 .TP .B assignConfig(config) Assign the given config object. .sp It will be used by various actions implemented inside the Job class. .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- Config object to query .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP (\fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP) \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChild(childJob) Add a childJob to be run as child of this job. .sp Child jobs will be run directly after this job\(aqs \fI\%toil.job.Job.run()\fP method has completed. .INDENT 7.0 .TP .B Returns childJob: for call chaining .TP .B Parameters \fBchildJob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%Job\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasChild(childJob) Check if childJob is already a child of this job. .INDENT 7.0 .TP .B Returns True if childJob is a child of the job, else False. .TP .B Parameters \fBchildJob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOnJob) Add a follow\-on job. .sp Follow\-on jobs will be run after the child jobs and their successors have been run. .INDENT 7.0 .TP .B Returns followOnJob for call chaining .TP .B Parameters \fBfollowOnJob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%Job\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasPredecessor(job) Check if a given job is already a predecessor of this job. .INDENT 7.0 .TP .B Parameters \fBjob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasFollowOn(followOnJob) Check if given job is already a follow\-on of this job. .INDENT 7.0 .TP .B Returns True if the followOnJob is a follow\-on of this job, else False. .TP .B Parameters \fBfollowOnJob\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addService(service, parentService=None) Add a service. .sp The \fI\%toil.job.Job.Service.start()\fP method of the service will be called after the run method has completed but before any successors are run. The service\(aqs \fI\%toil.job.Job.Service.stop()\fP method will be called once the successors of the job have been run. .sp Services allow things like databases and servers to be started and accessed by jobs in a workflow. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If service has already been made the child of a job or another service. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice\fP (\fI\%Job\fP) \-\- Service to add. .IP \(bu 2 \fBparentService\fP (\fIOptional\fP\fI[\fP\fI\%Job\fP\fI]\fP) \-\- Service that will be started before \(aqservice\(aq is started. Allows trees of services to be established. parentService must be a service of this job. .UNINDENT .TP .B Returns a promise that will be replaced with the return value from \fI\%toil.job.Job.Service.start()\fP of service in any successor of the job. .TP .B Return type \fI\%Promise\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B hasService(service) Return True if the given Service is a service of this job, and False otherwise. .INDENT 7.0 .TP .B Parameters \fBservice\fP (\fI\%Job\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChildFn(fn, *args, **kwargs) Add a function as a child job. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fICallable\fP) \-\- Function to be run as a child job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Returns The new child job that wraps fn. .TP .B Return type \fI\%FunctionWrappingJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOnFn(fn, *args, **kwargs) Add a function as a follow\-on job. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fICallable\fP) \-\- Function to be run as a follow\-on job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Returns The new follow\-on job that wraps fn. .TP .B Return type \fI\%FunctionWrappingJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addChildJobFn(fn, *args, **kwargs) Add a job function as a child job. .sp See \fI\%toil.job.JobFunctionWrappingJob\fP for a definition of a job function. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fICallable\fP) \-\- Job function to be run as a child job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Returns The new child job that wraps fn. .TP .B Return type \fI\%FunctionWrappingJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOnJobFn(fn, *args, **kwargs) Add a follow\-on job function. .sp See \fI\%toil.job.JobFunctionWrappingJob\fP for a definition of a job function. .INDENT 7.0 .TP .B Parameters \fBfn\fP (\fICallable\fP) \-\- Job function to be run as a follow\-on job with \fB*args\fP and \fB**kwargs\fP as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Returns The new follow\-on job that wraps fn. .TP .B Return type \fI\%FunctionWrappingJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B log(text, level=logging.INFO) Log using \fBfileStore.log_to_leader()\fP\&. .INDENT 7.0 .TP .B Parameters \fBtext\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static wrapFn(fn, *args, **kwargs) Makes a Job out of a function. .sp Convenience function for constructor of \fI\%toil.job.FunctionWrappingJob\fP\&. .INDENT 7.0 .TP .B Parameters \fBfn\fP \-\- Function to be run with \fB*args\fP and \fB**kwargs\fP as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Returns The new function that wraps fn. .TP .B Return type \fI\%FunctionWrappingJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static wrapJobFn(fn, *args, **kwargs) Makes a Job out of a job function. .sp Convenience function for constructor of \fI\%toil.job.JobFunctionWrappingJob\fP\&. .INDENT 7.0 .TP .B Parameters \fBfn\fP \-\- Job function to be run with \fB*args\fP and \fB**kwargs\fP as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements. .TP .B Returns The new job function that wraps fn. .TP .B Return type \fI\%JobFunctionWrappingJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B encapsulate(name=None) Encapsulates the job, see \fI\%toil.job.EncapsulatedJob\fP\&. Convenience function for constructor of \fI\%toil.job.EncapsulatedJob\fP\&. .INDENT 7.0 .TP .B Parameters \fBname\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- Human\-readable name for the encapsulated job. .TP .B Returns an encapsulated version of this job. .TP .B Return type \fI\%EncapsulatedJob\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rv(*path) Create a \fIpromise\fP (\fI\%toil.job.Promise\fP). .sp The \(dqpromise\(dq representing a return value of the job\(aqs run method, or, in case of a function\-wrapping job, the wrapped function\(aqs return value. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI(\fP\fIAny\fP\fI)\fP) \-\- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the \fI__getitem__()\fP magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is \fI[6,{\(aqa\(aq:42}]\fP, \fI\&.rv(0)\fP would select \fI6\fP , \fIrv(1)\fP would select \fI{\(aqa\(aq:3}\fP while \fIrv(1,\(aqa\(aq)\fP would select \fI3\fP\&. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a \fIslice\fP object. For example, assuming that the return value is \fI[6, 7, 8, 9]\fP then \fI\&.rv(slice(1, 3))\fP would select \fI[7, 8]\fP\&. Note that slicing really only makes sense at the end of path. .TP .B Returns A promise representing the return value of this jobs \fI\%toil.job.Job.run()\fP method. .TP .B Return type \fI\%Promise\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B registerPromise(path) .UNINDENT .INDENT 7.0 .TP .B prepareForPromiseRegistration(jobStore) Set up to allow this job\(aqs promises to register themselves. .sp Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized. .sp The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B checkJobGraphForDeadlocks() Ensures that a graph of Jobs (that hasn\(aqt yet been saved to the JobStore) doesn\(aqt contain any pathological relationships between jobs that would result in deadlocks if we tried to run the jobs. .sp See \fI\%toil.job.Job.checkJobGraphConnected()\fP, \fBtoil.job.Job.checkJobGraphAcyclic()\fP and \fI\%toil.job.Job.checkNewCheckpointsAreLeafVertices()\fP for more info. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if the job graph is cyclic, contains multiple roots or contains checkpoint jobs that are not leaf vertices when defined (see \fBtoil.job.Job.checkNewCheckpointsAreLeaves()\fP). .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getRootJobs() Return the set of root job objects that contain this job. .sp A root job is a job with no predecessors (i.e. which are not children, follow\-ons, or services). .sp Only deals with jobs created here, rather than loaded from the job store. .INDENT 7.0 .TP .B Return type Set[\fI\%Job\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B checkJobGraphConnected() .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if \fI\%toil.job.Job.getRootJobs()\fP does not contain exactly one root job. .UNINDENT .sp As execution always starts from one root job, having multiple root jobs will cause a deadlock to occur. .sp Only deals with jobs created here, rather than loaded from the job store. .UNINDENT .INDENT 7.0 .TP .B checkJobGraphAcylic() .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if the connected component of jobs containing this job contains any cycles of child/followOn dependencies in the \fIaugmented job graph\fP (see below). Such cycles are not allowed in valid job graphs. .UNINDENT .sp A follow\-on edge (A, B) between two jobs A and B is equivalent to adding a child edge to B from (1) A, (2) from each child of A, and (3) from the successors of each child of A. We call each such edge an edge an \(dqimplied\(dq edge. The augmented job graph is a job graph including all the implied edges. .sp For a job graph G = (V, E) the algorithm is \fBO(|V|^2)\fP\&. It is \fBO(|V| + |E|)\fP for a graph with no follow\-ons. The former follow\-on case could be improved! .sp Only deals with jobs created here, rather than loaded from the job store. .UNINDENT .INDENT 7.0 .TP .B checkNewCheckpointsAreLeafVertices() A checkpoint job is a job that is restarted if either it fails, or if any of its successors completely fails, exhausting their retries. .sp A job is a leaf it is has no successors. .sp A checkpoint job must be a leaf when initially added to the job graph. When its run method is invoked it can then create direct successors. This restriction is made to simplify implementation. .sp Only works on connected components of jobs not yet added to the JobStore. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobGraphDeadlockException\fP \-\- if there exists a job being added to the graph for which checkpoint=True and which is not a leaf. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B defer(function, *args, **kwargs) Register a deferred function, i.e. a callable that will be invoked after the current attempt at running this job concludes. A job attempt is said to conclude when the job function (or the \fI\%toil.job.Job.run()\fP method for class\-based jobs) returns, raises an exception or after the process running it terminates abnormally. A deferred function will be called on the node that attempted to run the job, even if a subsequent attempt is made on another node. A deferred function should be idempotent because it may be called multiple times on the same node or even in the same process. More than one deferred function may be registered per job attempt by calling this method repeatedly with different arguments. If the same function is registered twice with the same or different arguments, it will be called twice per job attempt. .sp Examples for deferred functions are ones that handle cleanup of resources external to Toil, like Docker containers, files outside the work directory, etc. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfunction\fP (\fIcallable\fP) \-\- The function to be called after this job concludes. .IP \(bu 2 \fBargs\fP (\fI\%list\fP) \-\- The arguments to the function .IP \(bu 2 \fBkwargs\fP (\fI\%dict\fP) \-\- The keyword arguments to the function .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUserScript() .INDENT 7.0 .TP .B Return type \fI\%toil.resource.ModuleDescriptor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getTopologicalOrderingOfJobs() .INDENT 7.0 .TP .B Returns a list of jobs such that for all pairs of indices i, j for which i < j, the job at index i can be run before the job at index j. .TP .B Return type List[\fI\%Job\fP] .UNINDENT .sp Only considers jobs in this job\(aqs subgraph that are newly added, not loaded from the job store. .sp Ignores service jobs. .UNINDENT .INDENT 7.0 .TP .B saveBody(jobStore) Save the execution data for just this job to the JobStore, and fill in the JobDescription with the information needed to retrieve it. .sp The Job\(aqs JobDescription must have already had a real jobStoreID assigned to it. .sp Does not save the JobDescription. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- The job store to save the job body into. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B saveAsRootJob(jobStore) Save this job to the given jobStore as the root job of the workflow. .INDENT 7.0 .TP .B Returns the JobDescription describing this job. .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type \fI\%JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod loadJob(jobStore, jobDescription) Retrieves a \fI\%toil.job.Job\fP instance from a JobStore .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- The job store. .IP \(bu 2 \fBjobDescription\fP (\fI\%JobDescription\fP) \-\- the JobDescription of the job to retrieve. .UNINDENT .TP .B Returns The job referenced by the JobDescription. .TP .B Return type \fI\%Job\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.job.JobException(message) Bases: \fI\%Exception\fP .sp General job exception. .INDENT 7.0 .TP .B Parameters \fBmessage\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.job.JobGraphDeadlockException(string) Bases: \fI\%JobException\fP .sp An exception raised in the event that a workflow contains an unresolvable dependency, such as a cycle. See \fI\%toil.job.Job.checkJobGraphForDeadlocks()\fP\&. .UNINDENT .INDENT 0.0 .TP .B class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs) Bases: \fI\%Job\fP .sp Job used to wrap a function. In its \fIrun\fP method the wrapped function is called. .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUserScript() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs) Bases: \fI\%FunctionWrappingJob\fP .sp A job function is a function whose first argument is a \fI\%Job\fP instance that is the wrapping job for the function. This can be used to add successor jobs for the function and perform all the functions the \fI\%Job\fP class provides. .sp To enable the job function to get access to the \fI\%toil.fileStores.abstractFileStore.AbstractFileStore\fP instance (see \fI\%toil.job.Job.run()\fP), it is made a variable of the wrapping job called fileStore. .sp To specify a job\(aqs resource requirements the following default keyword arguments can be specified: .INDENT 7.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 memory .IP \(bu 2 disk .IP \(bu 2 cores .IP \(bu 2 accelerators .IP \(bu 2 preemptible .UNINDENT .UNINDENT .UNINDENT .sp For example to wrap a function into a job we would call: .INDENT 7.0 .INDENT 3.5 .sp .EX Job.wrapJobFn(myJob, memory=\(aq100k\(aq, disk=\(aq1M\(aq, cores=0.1) .EE .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property fileStore .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.PromisedRequirementFunctionWrappingJob(userFunction, *args, **kwargs) Bases: \fI\%FunctionWrappingJob\fP .sp Handles dynamic resource allocation using \fI\%toil.job.Promise\fP instances. Spawns child function using parent function parameters and fulfilled promised resource requirements. .INDENT 7.0 .TP .B classmethod create(userFunction, *args, **kwargs) Creates an encapsulated Toil job function with unfulfilled promised resource requirements. After the promises are fulfilled, a child job function is created using updated resource values. The subgraph is encapsulated to ensure that this child job function is run before other children in the workflow. Otherwise, a different child may try to use an unresolved promise return value from the parent. .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B evaluatePromisedRequirements() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.PromisedRequirementJobFunctionWrappingJob(userFunction, *args, **kwargs) Bases: \fI\%PromisedRequirementFunctionWrappingJob\fP .sp Handles dynamic resource allocation for job functions. See \fI\%toil.job.JobFunctionWrappingJob\fP .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.EncapsulatedJob(job, unitName=None) Bases: \fI\%Job\fP .sp A convenience Job class used to make a job subgraph appear to be a single job. .sp Let A be the root job of a job subgraph and B be another job we\(aqd like to run after A and all its successors have completed, for this use encapsulate: .INDENT 7.0 .INDENT 3.5 .sp .EX # Job A and subgraph, Job B A, B = A(), B() Aprime = A.encapsulate() Aprime.addChild(B) # B will run after A and all its successors have completed, A and its subgraph of # successors in effect appear to be just one job. .EE .UNINDENT .UNINDENT .sp If the job being encapsulated has predecessors (e.g. is not the root job), then the encapsulated job will inherit these predecessors. If predecessors are added to the job being encapsulated after the encapsulated job is created then the encapsulating job will NOT inherit these predecessors automatically. Care should be exercised to ensure the encapsulated job has the proper set of predecessors. .sp The return value of an encapsulated job (as accessed by the \fI\%toil.job.Job.rv()\fP function) is the return value of the root job, e.g. A().encapsulate().rv() and A().rv() will resolve to the same value after A or A.encapsulate() has been run. .INDENT 7.0 .TP .B addChild(childJob) Add a childJob to be run as child of this job. .sp Child jobs will be run directly after this job\(aqs \fI\%toil.job.Job.run()\fP method has completed. .INDENT 7.0 .TP .B Returns childJob: for call chaining .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addService(service, parentService=None) Add a service. .sp The \fI\%toil.job.Job.Service.start()\fP method of the service will be called after the run method has completed but before any successors are run. The service\(aqs \fI\%toil.job.Job.Service.stop()\fP method will be called once the successors of the job have been run. .sp Services allow things like databases and servers to be started and accessed by jobs in a workflow. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If service has already been made the child of a job or another service. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice\fP \-\- Service to add. .IP \(bu 2 \fBparentService\fP \-\- Service that will be started before \(aqservice\(aq is started. Allows trees of services to be established. parentService must be a service of this job. .UNINDENT .TP .B Returns a promise that will be replaced with the return value from \fI\%toil.job.Job.Service.start()\fP of service in any successor of the job. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOnJob) Add a follow\-on job. .sp Follow\-on jobs will be run after the child jobs and their successors have been run. .INDENT 7.0 .TP .B Returns followOnJob for call chaining .UNINDENT .UNINDENT .INDENT 7.0 .TP .B rv(*path) Create a \fIpromise\fP (\fI\%toil.job.Promise\fP). .sp The \(dqpromise\(dq representing a return value of the job\(aqs run method, or, in case of a function\-wrapping job, the wrapped function\(aqs return value. .INDENT 7.0 .TP .B Parameters \fBpath\fP (\fI(\fP\fIAny\fP\fI)\fP) \-\- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the \fI__getitem__()\fP magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is \fI[6,{\(aqa\(aq:42}]\fP, \fI\&.rv(0)\fP would select \fI6\fP , \fIrv(1)\fP would select \fI{\(aqa\(aq:3}\fP while \fIrv(1,\(aqa\(aq)\fP would select \fI3\fP\&. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a \fIslice\fP object. For example, assuming that the return value is \fI[6, 7, 8, 9]\fP then \fI\&.rv(slice(1, 3))\fP would select \fI[7, 8]\fP\&. Note that slicing really only makes sense at the end of path. .TP .B Returns A promise representing the return value of this jobs \fI\%toil.job.Job.run()\fP method. .TP .B Return type \fI\%Promise\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B prepareForPromiseRegistration(jobStore) Set up to allow this job\(aqs promises to register themselves. .sp Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized. .sp The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor. .UNINDENT .INDENT 7.0 .TP .B __reduce__() Called during pickling to define the pickled representation of the job. .sp We don\(aqt want to pickle our internal references to the job we encapsulate, so we elide them here. When actually run, we\(aqre just a no\-op job that can maybe chain. .UNINDENT .INDENT 7.0 .TP .B getUserScript() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.ServiceHostJob(service) Bases: \fI\%Job\fP .sp Job that runs a service. Used internally by Toil. Users should subclass Service instead of using this. .INDENT 7.0 .TP .B property fileStore Return the file store, which the Service may need. .UNINDENT .INDENT 7.0 .TP .B addChild(child) Add a childJob to be run as child of this job. .sp Child jobs will be run directly after this job\(aqs \fI\%toil.job.Job.run()\fP method has completed. .INDENT 7.0 .TP .B Returns childJob: for call chaining .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addFollowOn(followOn) Add a follow\-on job. .sp Follow\-on jobs will be run after the child jobs and their successors have been run. .INDENT 7.0 .TP .B Returns followOnJob for call chaining .UNINDENT .UNINDENT .INDENT 7.0 .TP .B addService(service, parentService=None) Add a service. .sp The \fI\%toil.job.Job.Service.start()\fP method of the service will be called after the run method has completed but before any successors are run. The service\(aqs \fI\%toil.job.Job.Service.stop()\fP method will be called once the successors of the job have been run. .sp Services allow things like databases and servers to be started and accessed by jobs in a workflow. .INDENT 7.0 .TP .B Raises \fI\%toil.job.JobException\fP \-\- If service has already been made the child of a job or another service. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice\fP \-\- Service to add. .IP \(bu 2 \fBparentService\fP \-\- Service that will be started before \(aqservice\(aq is started. Allows trees of services to be established. parentService must be a service of this job. .UNINDENT .TP .B Returns a promise that will be replaced with the return value from \fI\%toil.job.Job.Service.start()\fP of service in any successor of the job. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B saveBody(jobStore) Serialize the service itself before saving the host job\(aqs body. .UNINDENT .INDENT 7.0 .TP .B run(fileStore) Override this function to perform work and dynamically create successor jobs. .INDENT 7.0 .TP .B Parameters \fBfileStore\fP \-\- Used to create local and globally sharable temporary files and to send log messages to the leader process. .TP .B Returns The return value of the function can be passed to other jobs by means of \fI\%toil.job.Job.rv()\fP\&. .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getUserScript() .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.Promise(job, path) References a return value from a method as a \fIpromise\fP before the method itself is run. .sp References a return value from a \fI\%toil.job.Job.run()\fP or \fI\%toil.job.Job.Service.start()\fP method as a \fIpromise\fP before the method itself is run. .sp Let T be a job. Instances of \fI\%Promise\fP (termed a \fIpromise\fP) are returned by T.rv(), which is used to reference the return value of T\(aqs run function. When the promise is passed to the constructor (or as an argument to a wrapped function) of a different, successor job the promise will be replaced by the actual referenced return value. This mechanism allows a return values from one job\(aqs run method to be input argument to job before the former job\(aqs run function has been executed. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob\fP (\fI\%Job\fP) \-\- .IP \(bu 2 \fBpath\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B filesToDelete A set of IDs of files containing promised values when we know we won\(aqt need them anymore .UNINDENT .INDENT 7.0 .TP .B __reduce__() Return the Promise class and construction arguments. .sp Called during pickling when a promise (an instance of this class) is about to be be pickled. Returns the Promise class and construction arguments that will be evaluated during unpickling, namely the job store coordinates of a file that will hold the promised return value. By the time the promise is about to be unpickled, that file should be populated. .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.job.T .UNINDENT .INDENT 0.0 .TP .B toil.job.Promised .UNINDENT .INDENT 0.0 .TP .B toil.job.unwrap(p) Function for ensuring you actually have a promised value, and not just a promise. Mostly useful for satisfying type\-checking. .sp The \(dqunwrap\(dq terminology is borrowed from Rust. .INDENT 7.0 .TP .B Parameters \fBp\fP (\fIPromised\fP\fI[\fP\fIT\fP\fI]\fP) \-\- .TP .B Return type T .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.job.unwrap_all(p) Function for ensuring you actually have a collection of promised values, and not any remaining promises. Mostly useful for satisfying type\-checking. .sp The \(dqunwrap\(dq terminology is borrowed from Rust. .INDENT 7.0 .TP .B Parameters \fBp\fP (\fISequence\fP\fI[\fP\fIPromised\fP\fI[\fP\fIT\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type Sequence[T] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.PromisedRequirement(valueOrCallable, *args) Class for dynamically allocating job function resource requirements. .sp (involving \fI\%toil.job.Promise\fP instances.) .sp Use when resource requirements depend on the return value of a parent function. PromisedRequirements can be modified by passing a function that takes the \fI\%Promise\fP as input. .sp For example, let f, g, and h be functions. Then a Toil workflow can be defined as follows:: A = Job.wrapFn(f) B = A.addChildFn(g, cores=PromisedRequirement(A.rv()) C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv())) .INDENT 7.0 .TP .B getValue() Return PromisedRequirement value. .UNINDENT .INDENT 7.0 .TP .B static convertPromises(kwargs) Return True if reserved resource keyword is a Promise or PromisedRequirement instance. .sp Converts Promise instance to PromisedRequirement. .INDENT 7.0 .TP .B Parameters \fBkwargs\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- function keyword arguments .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.job.UnfulfilledPromiseSentinel(fulfillingJobName, file_id, unpickled) This should be overwritten by a proper promised value. .sp Throws an exception when unpickled. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfulfillingJobName\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBfile_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBunpickled\fP (\fIAny\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B static __setstate__(stateDict) Only called when unpickling. .sp This won\(aqt be unpickled unless the promise wasn\(aqt resolved, so we throw an exception. .INDENT 7.0 .TP .B Parameters \fBstateDict\fP (\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fIAny\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.leader\fP .sp The leader script (of the leader/worker pair) for running jobs. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Leader\fP T} T{ Represents the Toil leader. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.leader.logger .UNINDENT .INDENT 0.0 .TP .B class toil.leader.Leader(config, batchSystem, provisioner, jobStore, rootJob, jobCache=None) Represents the Toil leader. .sp Responsible for determining what jobs are ready to be scheduled, by consulting the job store, and issuing them in the batch system. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBbatchSystem\fP (\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP) \-\- .IP \(bu 2 \fBprovisioner\fP (\fIOptional\fP\fI[\fP\fI\%toil.provisioners.abstractProvisioner.AbstractProvisioner\fP\fI]\fP) \-\- .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBrootJob\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBjobCache\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%toil.job.TemporaryID\fP\fI]\fP\fI, \fP\fI\%toil.job.JobDescription\fP\fI]\fP\fI]\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B run() Run the leader process to issue and manage jobs. .INDENT 7.0 .TP .B Raises toil.exceptions.FailedJobsException if failed jobs remain after running. .TP .B Returns The return value of the root job\(aqs run function. .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 7.0 .TP .B create_status_sentinel_file(fail) Create a file in the jobstore indicating failure or success. .INDENT 7.0 .TP .B Parameters \fBfail\fP (\fI\%bool\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B innerLoop() Process jobs. .sp This is the leader\(aqs main loop. .UNINDENT .INDENT 7.0 .TP .B checkForDeadlocks() Check if the system is deadlocked running service jobs. .UNINDENT .INDENT 7.0 .TP .B feed_deadlock_watchdog() Note that progress has been made and any pending deadlock checks should be reset. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueJob(jobNode) Add a job to the queue of jobs currently trying to run. .INDENT 7.0 .TP .B Parameters \fBjobNode\fP (\fI\%toil.job.JobDescription\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueJobs(jobs) Add a list of jobs, each represented as a jobNode object. .UNINDENT .INDENT 7.0 .TP .B issueServiceJob(service_id) Issue a service job. .sp Put it on a queue if the maximum number of service jobs to be scheduled has been reached. .INDENT 7.0 .TP .B Parameters \fBservice_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B issueQueingServiceJobs() Issues any queuing service jobs up to the limit of the maximum allowed. .UNINDENT .INDENT 7.0 .TP .B getNumberOfJobsIssued(preemptible=None) Get number of jobs that have been added by issueJob(s) and not removed by removeJob. .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- If none, return all types of jobs. If true, return just the number of preemptible jobs. If false, return just the number of non\-preemptible jobs. .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B removeJob(jobBatchSystemID) Remove a job from the system by batch system ID. .INDENT 7.0 .TP .B Returns Job description as it was issued. .TP .B Parameters \fBjobBatchSystemID\fP (\fI\%int\fP) \-\- .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getJobs(preemptible=None) Get all issued jobs. .INDENT 7.0 .TP .B Parameters \fBpreemptible\fP (\fIOptional\fP\fI[\fP\fI\%bool\fP\fI]\fP) \-\- If specified, select only preemptible or only non\-preemptible jobs. .TP .B Return type List[\fI\%toil.job.JobDescription\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B killJobs(jobsToKill) Kills the given set of jobs and then sends them for processing. .sp Returns the jobs that, upon processing, were reissued. .UNINDENT .INDENT 7.0 .TP .B reissueOverLongJobs() Check each issued job. .sp If a job is running for longer than desirable issue a kill instruction. Wait for the job to die then we pass the job to process_finished_job. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B reissueMissingJobs(killAfterNTimesMissing=3) Check all the current job ids are in the list of currently issued batch system jobs. .sp If a job is missing, we mark it as so, if it is missing for a number of runs of this function (say 10).. then we try deleting the job (though its probably lost), we wait then we pass the job to process_finished_job. .UNINDENT .INDENT 7.0 .TP .B processRemovedJob(issuedJob, result_status) .UNINDENT .INDENT 7.0 .TP .B process_finished_job(batch_system_id, result_status, wall_time=None, exit_reason=None) Process finished jobs. .sp Called when an attempt to run a job finishes, either successfully or otherwise. .sp Takes the job out of the issued state, and then works out what to do about the fact that it succeeded or failed. .INDENT 7.0 .TP .B Returns True if the job is going to run again, and False if the job is fully done or completely failed. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B process_finished_job_description(finished_job, result_status, wall_time=None, exit_reason=None, batch_system_id=None) Process a finished JobDescription based upon its succees or failure. .sp If wall\-clock time is available, informs the cluster scaler about the job finishing. .sp If the job failed and a batch system ID is available, checks for and reports batch system logs. .sp Checks if it succeeded and was removed, or if it failed and needs to be set up after failure, and dispatches to the appropriate function. .INDENT 7.0 .TP .B Returns True if the job is going to run again, and False if the job is fully done or completely failed. .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBfinished_job\fP (\fI\%toil.job.JobDescription\fP) \-\- .IP \(bu 2 \fBresult_status\fP (\fI\%int\fP) \-\- .IP \(bu 2 \fBwall_time\fP (\fIOptional\fP\fI[\fP\fI\%float\fP\fI]\fP) \-\- .IP \(bu 2 \fBexit_reason\fP (\fIOptional\fP\fI[\fP\fI\%toil.batchSystems.abstractBatchSystem.BatchJobExitReason\fP\fI]\fP) \-\- .IP \(bu 2 \fBbatch_system_id\fP (\fIOptional\fP\fI[\fP\fI\%int\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B getSuccessors(job_id, alreadySeenSuccessors) Get successors of the given job by walking the job graph recursively. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBalreadySeenSuccessors\fP (\fISet\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- any successor seen here is ignored and not traversed. .IP \(bu 2 \fBjob_id\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Returns The set of found successors. This set is added to alreadySeenSuccessors. .TP .B Return type Set[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B processTotallyFailedJob(job_id) Process a totally failed job. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.realtimeLogger\fP .sp Implements a real\-time UDP\-based logging system that user scripts can use for debugging. .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%LoggingDatagramHandler\fP T} T{ Receive logging messages from the jobs and display them on the leader. T} _ T{ \fI\%JSONDatagramHandler\fP T} T{ Send logging records over UDP serialized as JSON. T} _ T{ \fI\%RealtimeLoggerMetaclass\fP T} T{ Metaclass for RealtimeLogger that lets add logging methods. T} _ T{ \fI\%RealtimeLogger\fP T} T{ Provide a logger that logs over UDP to the leader. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.realtimeLogger.logger .UNINDENT .INDENT 0.0 .TP .B class toil.realtimeLogger.LoggingDatagramHandler(request, client_address, server) Bases: \fI\%socketserver.BaseRequestHandler\fP .sp Receive logging messages from the jobs and display them on the leader. .sp Uses bare JSON message encoding. .INDENT 7.0 .TP .B handle() Handle a single message. SocketServer takes care of splitting out the messages. .sp Messages are JSON\-encoded logging module records. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.realtimeLogger.JSONDatagramHandler(host, port) Bases: \fI\%logging.handlers.DatagramHandler\fP .sp Send logging records over UDP serialized as JSON. .sp They have to fit in a single UDP datagram, so don\(aqt try to log more than 64kb at once. .INDENT 7.0 .TP .B makePickle(record) Actually, encode the record as bare JSON instead. .INDENT 7.0 .TP .B Parameters \fBrecord\fP (\fI\%logging.LogRecord\fP) \-\- .TP .B Return type \fI\%bytes\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.realtimeLogger.RealtimeLoggerMetaclass Bases: \fI\%type\fP .sp Metaclass for RealtimeLogger that lets add logging methods. .sp Like RealtimeLogger.warning(), RealtimeLogger.info(), etc. .INDENT 7.0 .TP .B __getattr__(name) Fallback to attributes on the logger. .INDENT 7.0 .TP .B Parameters \fBname\fP (\fI\%str\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.realtimeLogger.RealtimeLogger(batchSystem, level=defaultLevel) Provide a logger that logs over UDP to the leader. .sp To use in a Toil job, do: .sp .EX >>> from toil.realtimeLogger import RealtimeLogger >>> RealtimeLogger.info(\(dqThis logging message goes straight to the leader\(dq) .EE .sp That\(aqs all a user of Toil would need to do. On the leader, Job.Runner.startToil() automatically starts the UDP server by using an instance of this class as a context manager. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBbatchSystem\fP (\fI\%toil.batchSystems.abstractBatchSystem.AbstractBatchSystem\fP) \-\- .IP \(bu 2 \fBlevel\fP (\fI\%str\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B envPrefix = \(aqTOIL_RT_LOGGING_\(aq .UNINDENT .INDENT 7.0 .TP .B defaultLevel = \(aqINFO\(aq .UNINDENT .INDENT 7.0 .TP .B lock .UNINDENT .INDENT 7.0 .TP .B loggingServer .UNINDENT .INDENT 7.0 .TP .B serverThread .UNINDENT .INDENT 7.0 .TP .B initialized = 0 .UNINDENT .INDENT 7.0 .TP .B logger .UNINDENT .INDENT 7.0 .TP .B classmethod getLogger() Get the logger that logs real\-time to the leader. .sp Note that if the returned logger is used on the leader, you will see the message twice, since it still goes to the normal log handlers, too. .INDENT 7.0 .TP .B Return type \fI\%logging.Logger\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __enter__() .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B __exit__(exc_type, exc_val, exc_tb) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBexc_type\fP (\fIOptional\fP\fI[\fP\fIType\fP\fI[\fP\fI\%BaseException\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBexc_val\fP (\fIOptional\fP\fI[\fP\fI\%BaseException\fP\fI]\fP) \-\- .IP \(bu 2 \fBexc_tb\fP (\fIOptional\fP\fI[\fP\fI\%types.TracebackType\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.resource\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%Resource\fP T} T{ Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked. T} _ T{ \fI\%FileResource\fP T} T{ A resource read from a file on the leader. T} _ T{ \fI\%DirectoryResource\fP T} T{ A resource read from a directory on the leader. T} _ T{ \fI\%VirtualEnvResource\fP T} T{ A resource read from a virtualenv on the leader. T} _ T{ \fI\%ModuleDescriptor\fP T} T{ A path to a Python module decomposed into a namedtuple of three elements T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.resource.logger .UNINDENT .INDENT 0.0 .TP .B class toil.resource.Resource Bases: \fBnamedtuple\fP(\fB\(aqResource\(aq\fP, (\fB\(aqname\(aq\fP, \fB\(aqpathHash\(aq\fP, \fB\(aqurl\(aq\fP, \fB\(aqcontentHash\(aq\fP)) .sp Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked. .sp Each instance is a namedtuple with the following elements: .sp The pathHash element contains the MD5 (in hexdigest form) of the path to the resource on the leader node. The path, and therefore its hash is unique within a job store. .sp The url element is a \(dqfile:\(dq or \(dqhttp:\(dq URL at which the resource can be obtained. .sp The contentHash element is an MD5 checksum of the resource, allowing for validation and caching of resources. .sp If the resource is a regular file, the type attribute will be \(aqfile\(aq. .sp If the resource is a directory, the type attribute will be \(aqdir\(aq and the URL will point at a ZIP archive of that directory. .INDENT 7.0 .TP .B abstract property localPath: \fI\%str\fP Get the path to resource on the worker. .sp The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B property localDirPath: \fI\%str\fP The path to the directory containing the resource on the worker. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B resourceEnvNamePrefix = \(aqJTRES_\(aq .UNINDENT .INDENT 7.0 .TP .B rootDirPathEnvName .UNINDENT .INDENT 7.0 .TP .B classmethod create(jobStore, leaderPath) Saves the content of the file or directory at the given path to the given job store and returns a resource object representing that content for the purpose of obtaining it again at a generic, public URL. This method should be invoked on the leader node. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBleaderPath\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%Resource\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B refresh(jobStore) .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type \fI\%Resource\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod prepareSystem() Prepares this system for the downloading and lookup of resources. This method should only be invoked on a worker node. It is idempotent but not thread\-safe. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod cleanSystem() Remove all downloaded, localized resources. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B register() Register this resource for later retrieval via lookup(), possibly in a child process. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod lookup(leaderPath) Return a resource object representing a resource created from a file or directory at the given path on the leader. .sp This method should be invoked on the worker. The given path does not need to refer to an existing file or directory on the worker, it only identifies the resource within an instance of toil. This method returns None if no resource for the given path exists. .INDENT 7.0 .TP .B Parameters \fBleaderPath\fP (\fI\%str\fP) \-\- .TP .B Return type Optional[\fI\%Resource\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B download(callback=None) Download this resource from its URL to a file on the local system. .sp This method should only be invoked on a worker node after the node was setup for accessing resources via prepareSystem(). .INDENT 7.0 .TP .B Parameters \fBcallback\fP (\fIOptional\fP\fI[\fP\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B pickle() .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod unpickle(s) .INDENT 7.0 .TP .B Parameters \fBs\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%Resource\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.resource.FileResource Bases: \fI\%Resource\fP .sp A resource read from a file on the leader. .INDENT 7.0 .TP .B property localPath: \fI\%str\fP Get the path to resource on the worker. .sp The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.resource.DirectoryResource Bases: \fI\%Resource\fP .sp A resource read from a directory on the leader. .sp The URL will point to a ZIP archive of the directory. All files in that directory (and any subdirectories) will be included. The directory may be a package but it does not need to be. .INDENT 7.0 .TP .B property localPath: \fI\%str\fP Get the path to resource on the worker. .sp The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B class toil.resource.VirtualEnvResource Bases: \fI\%DirectoryResource\fP .sp A resource read from a virtualenv on the leader. .sp All modules and packages found in the virtualenv\(aqs site\-packages directory will be included. .UNINDENT .INDENT 0.0 .TP .B class toil.resource.ModuleDescriptor Bases: \fBnamedtuple\fP(\fB\(aqModuleDescriptor\(aq\fP, (\fB\(aqdirPath\(aq\fP, \fB\(aqname\(aq\fP, \fB\(aqfromVirtualEnv\(aq\fP)) .sp A path to a Python module decomposed into a namedtuple of three elements .INDENT 7.0 .IP \(bu 2 dirPath, the path to the directory that should be added to sys.path before importing the module, .IP \(bu 2 moduleName, the fully qualified name of the module with leading package names separated by dot and .UNINDENT .sp .EX >>> import toil.resource >>> ModuleDescriptor.forModule(\(aqtoil.resource\(aq) ModuleDescriptor(dirPath=\(aq/.../src\(aq, name=\(aqtoil.resource\(aq, fromVirtualEnv=False) .EE .sp .EX >>> import subprocess, tempfile, os >>> dirPath = tempfile.mkdtemp() >>> path = os.path.join( dirPath, \(aqfoo.py\(aq ) >>> with open(path,\(aqw\(aq) as f: \&... _ = f.write(\(aqfrom toil.resource import ModuleDescriptor\en\(aq \&... \(aqprint(ModuleDescriptor.forModule(__name__))\(aq) >>> subprocess.check_output([ sys.executable, path ]) b\(dqModuleDescriptor(dirPath=\(aq...\(aq, name=\(aqfoo\(aq, fromVirtualEnv=False)\en\(dq .EE .sp .EX >>> from shutil import rmtree >>> rmtree( dirPath ) .EE .sp Now test a collision. \(aqcollections\(aq is part of the standard library in Python 2 and 3. >>> dirPath = tempfile.mkdtemp() >>> path = os.path.join( dirPath, \(aqcollections.py\(aq ) >>> with open(path,\(aqw\(aq) as f: \&... _ = f.write(\(aqfrom toil.resource import ModuleDescriptorn\(aq \&... \(aqModuleDescriptor.forModule(__name__)\(aq) .sp This should fail and return exit status 1 due to the collision with the built\-in module: >>> subprocess.call([ sys.executable, path ]) 1 .sp Clean up >>> rmtree( dirPath ) .INDENT 7.0 .TP .B property belongsToToil: \fI\%bool\fP True if this module is part of the Toil distribution .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B dirPath: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B name: \fI\%str\fP .UNINDENT .INDENT 7.0 .TP .B classmethod forModule(name) Return an instance of this class representing the module of the given name. .sp If the given module name is \(dq__main__\(dq, it will be translated to the actual file name of the top\-level script without the .py or .pyc extension. This method assumes that the module with the specified name has already been loaded. .INDENT 7.0 .TP .B Parameters \fBname\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%ModuleDescriptor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B saveAsResourceTo(jobStore) Store the file containing this module\-\-or even the Python package directory hierarchy containing that file\-\-as a resource to the given job store and return the corresponding resource object. Should only be called on a leader node. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .TP .B Return type \fI\%Resource\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B localize() Check if this module was saved as a resource. .sp If it was, return a new module descriptor that points to a local copy of that resource. Should only be called on a worker node. On the leader, this method returns this resource, i.e. self. .INDENT 7.0 .TP .B Return type \fI\%ModuleDescriptor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B globalize() Reverse the effect of localize(). .INDENT 7.0 .TP .B Return type \fI\%ModuleDescriptor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B toCommand() .INDENT 7.0 .TP .B Return type Sequence[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod fromCommand(command) .INDENT 7.0 .TP .B Parameters \fBcommand\fP (\fISequence\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type \fI\%ModuleDescriptor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B makeLoadable() .INDENT 7.0 .TP .B Return type \fI\%ModuleDescriptor\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B load() .INDENT 7.0 .TP .B Return type Optional[\fI\%types.ModuleType\fP] .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B exception toil.resource.ResourceException Bases: \fI\%Exception\fP .sp Common base class for all non\-exit exceptions. .UNINDENT .SS \fI\%toil.serviceManager\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ServiceManager\fP T} T{ Manages the scheduling of services. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.serviceManager.logger .UNINDENT .INDENT 0.0 .TP .B class toil.serviceManager.ServiceManager(job_store, toil_state) Manages the scheduling of services. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjob_store\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBtoil_state\fP (\fI\%toil.toilState.ToilState\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B services_are_starting(job_id) Check if services are being started. .INDENT 7.0 .TP .B Returns True if the services for the given job are currently being started, and False otherwise. .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_job_count() Get the total number of jobs we are working on. .sp (services and their parent non\-service jobs) .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B start() Start the service scheduling thread. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B put_client(client_id) Schedule the services of a job asynchronously. .sp When the job\(aqs services are running the ID for the job will be returned by toil.leader.ServiceManager.get_ready_client. .INDENT 7.0 .TP .B Parameters \fBclient_id\fP (\fI\%str\fP) \-\- ID of job with services to schedule. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_ready_client(maxWait) Fetch a ready client, waiting as needed. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%float\fP) \-\- Time in seconds to wait to get a JobDescription before returning .TP .B Returns the ID of a client whose services are running, or None if no such job is available. .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_unservable_client(maxWait) Fetch a client whos services failed to start. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%float\fP) \-\- Time in seconds to wait to get a JobDescription before returning .TP .B Returns the ID of a client whose services failed to start, or None if no such job is available. .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_startable_service(maxWait) Fetch a service job that is ready to start. .INDENT 7.0 .TP .B Parameters \fBmaxWait\fP (\fI\%float\fP) \-\- Time in seconds to wait to get a job before returning. .TP .B Returns the ID of a service job that the leader can start, or None if no such job exists. .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 7.0 .TP .B kill_services(service_ids, error=False) Stop all the given service jobs. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservices\fP \-\- Service jobStoreIDs to kill .IP \(bu 2 \fBerror\fP (\fI\%bool\fP) \-\- Whether to signal that the service failed with an error when stopping it. .IP \(bu 2 \fBservice_ids\fP (\fIIterable\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_active(service_id) Return true if the service job has not been told to terminate. .INDENT 7.0 .TP .B Parameters \fBservice_id\fP (\fI\%str\fP) \-\- Service to check on .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B is_running(service_id) Return true if the service job has started and is active. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBservice\fP \-\- Service to check on .IP \(bu 2 \fBservice_id\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Check on the service manager thread. .INDENT 7.0 .TP .B Raises \fI\%RuntimeError\fP \-\- If the underlying thread has quit. .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Terminate worker threads cleanly; starting and killing all service threads. .sp Will block until all services are started and blocked. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.statsAndLogging\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%StatsAndLogging\fP T} T{ A thread to aggregate statistics and logging. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%set_log_level\fP(level[, set_logger]) T} T{ Sets the root logger level to a given string level (like \(dqINFO\(dq). T} _ T{ \fI\%add_logging_options\fP(parser) T} T{ Add logging options to set the global log level. T} _ T{ \fI\%configure_root_logger\fP() T} T{ Set up the root logger with handlers and formatting. T} _ T{ \fI\%log_to_file\fP(log_file, log_rotation) T} T{ T} _ T{ \fI\%set_logging_from_options\fP(options) T} T{ T} _ T{ \fI\%suppress_exotic_logging\fP(local_logger) T} T{ Attempts to suppress the loggers of all non\-Toil packages by setting them to CRITICAL. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ T{ \fI\%root_logger\fP T} T{ T} _ T{ \fI\%toil_logger\fP T} T{ T} _ T{ \fI\%DEFAULT_LOGLEVEL\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.statsAndLogging.logger .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.root_logger .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.toil_logger .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.DEFAULT_LOGLEVEL .UNINDENT .INDENT 0.0 .TP .B class toil.statsAndLogging.StatsAndLogging(jobStore, config) A thread to aggregate statistics and logging. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .UNINDENT .UNINDENT .INDENT 7.0 .TP .B start() Start the stats and logging thread. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod formatLogStream(stream, stream_name) Given a stream of text or bytes, and the job name, job itself, or some other optional stringifyable identity info for the job, return a big text string with the formatted job log, suitable for printing for the user. .sp We don\(aqt want to prefix every line of the job\(aqs log with our own logging info, or we get prefixes wider than any reasonable terminal and longer than the messages. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstream\fP (\fIUnion\fP\fI[\fP\fIIO\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fIIO\fP\fI[\fP\fI\%bytes\fP\fI]\fP\fI]\fP) \-\- The stream of text or bytes to print for the user. .IP \(bu 2 \fBstream_name\fP (\fI\%str\fP) \-\- .UNINDENT .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod logWithFormatting(stream_name, jobLogs, method=logger.debug, message=None) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBstream_name\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBjobLogs\fP (\fIUnion\fP\fI[\fP\fIIO\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fIIO\fP\fI[\fP\fI\%bytes\fP\fI]\fP\fI]\fP) \-\- .IP \(bu 2 \fBmethod\fP (\fICallable\fP\fI[\fP\fI[\fP\fI\%str\fP\fI]\fP\fI, \fP\fINone\fP\fI]\fP) \-\- .IP \(bu 2 \fBmessage\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod writeLogFiles(jobNames, jobLogList, config, failed=False) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobNames\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBjobLogList\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .IP \(bu 2 \fBfailed\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B classmethod statsAndLoggingAggregator(jobStore, stop, config) The following function is used for collating stats/reporting log messages from the workers. Works inside of a thread, collates as long as the stop flag is not True. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .IP \(bu 2 \fBstop\fP (\fI\%threading.Event\fP) \-\- .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B check() Check on the stats and logging aggregator. :raise RuntimeError: If the underlying thread has quit. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B shutdown() Finish up the stats/logging aggregation thread. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.set_log_level(level, set_logger=None) Sets the root logger level to a given string level (like \(dqINFO\(dq). .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlevel\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBset_logger\fP (\fIOptional\fP\fI[\fP\fI\%logging.Logger\fP\fI]\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.add_logging_options(parser) Add logging options to set the global log level. .INDENT 7.0 .TP .B Parameters \fBparser\fP (\fI\%argparse.ArgumentParser\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.configure_root_logger() Set up the root logger with handlers and formatting. .sp Should be called before any entry point tries to log anything, to ensure consistent formatting. .INDENT 7.0 .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.log_to_file(log_file, log_rotation) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBlog_file\fP (\fIOptional\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .IP \(bu 2 \fBlog_rotation\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.set_logging_from_options(options) .INDENT 7.0 .TP .B Parameters \fBoptions\fP (\fIUnion\fP\fI[\fP\fI\%toil.common.Config\fP\fI, \fP\fI\%argparse.Namespace\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.statsAndLogging.suppress_exotic_logging(local_logger) Attempts to suppress the loggers of all non\-Toil packages by setting them to CRITICAL. .sp For example: \(aqrequests_oauthlib\(aq, \(aqgoogle\(aq, \(aqboto\(aq, \(aqwebsocket\(aq, \(aqoauthlib\(aq, etc. .sp This will only suppress loggers that have already been instantiated and can be seen in the environment, except for the list declared in \(dqalways_suppress\(dq. .sp This is important because some packages, particularly boto3, are not always instantiated yet in the environment when this is run, and so we create the logger and set the level preemptively. .INDENT 7.0 .TP .B Parameters \fBlocal_logger\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .SS \fI\%toil.toilState\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%ToilState\fP T} T{ Holds the leader\(aqs scheduling information. T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.toilState.logger .UNINDENT .INDENT 0.0 .TP .B class toil.toilState.ToilState(jobStore) Holds the leader\(aqs scheduling information. .sp But onlt that which does not need to be persisted back to the JobStore (such as information on completed and outstanding predecessors) .sp Holds the true single copies of all JobDescription objects that the Leader and ServiceManager will use. The leader and service manager shouldn\(aqt do their own load() and update() calls on the JobStore; they should go through this class. .sp Everything in the leader should reference JobDescriptions by ID. .sp Only holds JobDescription objects, not Job objects, and those JobDescription objects only exist in single copies. .INDENT 7.0 .TP .B Parameters \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- .UNINDENT .INDENT 7.0 .TP .B load_workflow(rootJob, jobCache=None) Load the workflow rooted at the given job. .sp If jobs are loaded that have updated and need to be dealt with by the leader, JobUpdatedMessage messages will be sent to the message bus. .sp The jobCache is a map from jobStoreID to JobDescription or None. Is used to speed up the building of the state when loading initially from the JobStore, and is not preserved. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBrootJob\fP (\fI\%toil.job.JobDescription\fP) \-\- The description for the root job of the workflow being run. .IP \(bu 2 \fBjobCache\fP (\fIOptional\fP\fI[\fP\fIDict\fP\fI[\fP\fI\%str\fP\fI, \fP\fI\%toil.job.JobDescription\fP\fI]\fP\fI]\fP) \-\- A dict to cache downloaded job descriptions in, keyed by ID. .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B job_exists(job_id) Test if the givin job exists now. .sp Returns True if the given job exists right now, and false if it hasn\(aqt been created or it has been deleted elsewhere. .sp Doesn\(aqt guarantee that the job will or will not be gettable, if racing another process, or if it is still cached. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B get_job(job_id) Get the one true copy of the JobDescription with the given ID. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%toil.job.JobDescription\fP .UNINDENT .UNINDENT .INDENT 7.0 .TP .B commit_job(job_id) Save back any modifications made to a JobDescription. .sp (one retrieved from get_job()) .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B delete_job(job_id) Destroy a JobDescription. .sp May raise an exception if the job could not be cleaned up (i.e. files belonging to it failed to delete). .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B reset_job(job_id) Discard any local modifications to a JobDescription. .sp Will make modifications from other hosts visible. .INDENT 7.0 .TP .B Parameters \fBjob_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B successors_pending(predecessor_id, count) Remember that the given job has the given number more pending successors. .sp (that have not yet succeeded or failed.) .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpredecessor_id\fP (\fI\%str\fP) \-\- .IP \(bu 2 \fBcount\fP (\fI\%int\fP) \-\- .UNINDENT .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B successor_returned(predecessor_id) Remember that the given job has one fewer pending successors. .sp (because one has succeeded or failed.) .INDENT 7.0 .TP .B Parameters \fBpredecessor_id\fP (\fI\%str\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 7.0 .TP .B count_pending_successors(predecessor_id) Count number of pending successors of the given job. .sp Pending successors are those which have not yet succeeded or failed. .INDENT 7.0 .TP .B Parameters \fBpredecessor_id\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .UNINDENT .SS \fI\%toil.version\fP .SS Module Contents .INDENT 0.0 .TP .B toil.version.baseVersion = \(aq6.1.0\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.cgcloudVersion = \(aq1.6.0a1.dev393\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.version = \(aq6.1.0\-3f9cba3766e52866ea80d0934498f8c8f3129c3f\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.cacheTag = \(aqcache\-local\-py3.11\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.mainCacheTag = \(aqcache\-master\-py3.11\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.distVersion = \(aq6.1.0\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.exactPython = \(aqpython3.11\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.python = \(aqpython3.11\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.dockerTag = \(aq6.1.0\-3f9cba3766e52866ea80d0934498f8c8f3129c3f\-py3.11\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.currentCommit = \(aq3f9cba3766e52866ea80d0934498f8c8f3129c3f\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.dockerRegistry = \(aqquay.io/ucsc_cgl\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.dockerName = \(aqtoil\(aq .UNINDENT .INDENT 0.0 .TP .B toil.version.dirty = False .UNINDENT .INDENT 0.0 .TP .B toil.version.cwltool_version = \(aq3.1.20240112164112\(aq .UNINDENT .SS \fI\%toil.worker\fP .SS Module Contents .SS Classes .TS center; |l|l|. _ T{ \fI\%StatsDict\fP T} T{ Subclass of MagicExpando for type\-checking purposes. T} _ .TE .SS Functions .TS center; |l|l|. _ T{ \fI\%nextChainable\fP(predecessor, jobStore, config) T} T{ Returns the next chainable job\(aqs JobDescription after the given predecessor T} _ T{ \fI\%workerScript\fP(jobStore, config, jobName, jobStoreID[, ...]) T} T{ Worker process script, runs a job. T} _ T{ \fI\%parse_args\fP(args) T} T{ Parse command\-line arguments to the worker. T} _ T{ \fI\%in_contexts\fP(contexts) T} T{ Unpickle and enter all the pickled, base64\-encoded context managers in the T} _ T{ \fI\%main\fP([argv]) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%logger\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.worker.logger .UNINDENT .INDENT 0.0 .TP .B class toil.worker.StatsDict(*args, **kwargs) Bases: \fI\%toil.lib.expando.MagicExpando\fP .sp Subclass of MagicExpando for type\-checking purposes. .INDENT 7.0 .TP .B jobs: List[\fI\%toil.lib.expando.MagicExpando\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.worker.nextChainable(predecessor, jobStore, config) Returns the next chainable job\(aqs JobDescription after the given predecessor JobDescription, if one exists, or None if the chain must terminate. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBpredecessor\fP (\fI\%toil.job.JobDescription\fP) \-\- The job to chain from .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- The JobStore to fetch JobDescriptions from. .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- The configuration for the current run. .UNINDENT .TP .B Return type Optional[\fI\%toil.job.JobDescription\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.worker.workerScript(jobStore, config, jobName, jobStoreID, redirectOutputToLogFile=True) Worker process script, runs a job. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBjobStore\fP (\fI\%toil.jobStores.abstractJobStore.AbstractJobStore\fP) \-\- The JobStore to fetch JobDescriptions from. .IP \(bu 2 \fBconfig\fP (\fI\%toil.common.Config\fP) \-\- The configuration for the current run. .IP \(bu 2 \fBjobName\fP (\fI\%str\fP) \-\- The \(dqjob name\(dq (a user friendly name) of the job to be run .IP \(bu 2 \fBjobStoreID\fP (\fI\%str\fP) \-\- The job store ID of the job to be run .IP \(bu 2 \fBredirectOutputToLogFile\fP (\fI\%bool\fP) \-\- .UNINDENT .TP .B Return int 1 if a job failed, or 0 if all jobs succeeded .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.worker.parse_args(args) Parse command\-line arguments to the worker. .INDENT 7.0 .TP .B Parameters \fBargs\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type Any .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.worker.in_contexts(contexts) Unpickle and enter all the pickled, base64\-encoded context managers in the given list. Then do the body, then leave them all. .INDENT 7.0 .TP .B Parameters \fBcontexts\fP (\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP) \-\- .TP .B Return type Iterator[None] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.worker.main(argv=None) .INDENT 7.0 .TP .B Parameters \fBargv\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .SS Package Contents .SS Functions .TS center; |l|l|. _ T{ \fI\%retry\fP([intervals, infinite_retries, errors, ...]) T} T{ Retry a function if it fails with any Exception defined in \(dqerrors\(dq. T} _ T{ \fI\%which\fP(cmd[, mode, path]) T} T{ Return the path with conforms to the given mode on the Path. T} _ T{ \fI\%toilPackageDirPath\fP() T} T{ Return the absolute path of the directory that corresponds to the top\-level toil package. T} _ T{ \fI\%inVirtualEnv\fP() T} T{ Test if we are inside a virtualenv or Conda virtual environment. T} _ T{ \fI\%resolveEntryPoint\fP(entryPoint) T} T{ Find the path to the given entry point that \fIshould\fP work on a worker. T} _ T{ \fI\%physicalMemory\fP() T} T{ Calculate the total amount of physical memory, in bytes. T} _ T{ \fI\%physicalDisk\fP(directory) T} T{ T} _ T{ \fI\%applianceSelf\fP([forceDockerAppliance]) T} T{ Return the fully qualified name of the Docker image to start Toil appliance containers from. T} _ T{ \fI\%customDockerInitCmd\fP() T} T{ Return the custom command set by the \fBTOIL_CUSTOM_DOCKER_INIT_COMMAND\fP environment variable. T} _ T{ \fI\%customInitCmd\fP() T} T{ Return the custom command set by the \fBTOIL_CUSTOM_INIT_COMMAND\fP environment variable. T} _ T{ \fI\%lookupEnvVar\fP(name, envName, defaultValue) T} T{ Look up environment variables that control Toil and log the result. T} _ T{ \fI\%checkDockerImageExists\fP(appliance) T} T{ Attempt to check a url registryName for the existence of a docker image with a given tag. T} _ T{ \fI\%parseDockerAppliance\fP(appliance) T} T{ Derive parsed registry, image reference, and tag from a docker image string. T} _ T{ \fI\%checkDockerSchema\fP(appliance) T} T{ T} _ T{ \fI\%requestCheckRegularDocker\fP(origAppliance, registryName, ...) T} T{ Check if an image exists using the requests library. T} _ T{ \fI\%requestCheckDockerIo\fP(origAppliance, imageName, tag) T} T{ Check docker.io to see if an image exists using the requests library. T} _ T{ \fI\%logProcessContext\fP(config) T} T{ T} _ .TE .SS Attributes .TS center; |l|l|. _ T{ \fI\%memoize\fP T} T{ Memoize a function result based on its parameters using this decorator. T} _ T{ \fI\%currentCommit\fP T} T{ T} _ T{ \fI\%log\fP T} T{ T} _ T{ \fI\%KNOWN_EXTANT_IMAGES\fP T} T{ T} _ T{ \fI\%cache_path\fP T} T{ T} _ .TE .INDENT 0.0 .TP .B toil.memoize Memoize a function result based on its parameters using this decorator. .sp For example, this can be used in place of lazy initialization. If the decorating function is invoked by multiple threads, the decorated function may be called more than once with the same arguments. .UNINDENT .INDENT 0.0 .TP .B toil.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None) Retry a function if it fails with any Exception defined in \(dqerrors\(dq. .sp Does so every x seconds, where x is defined by a list of numbers (ints or floats) in \(dqintervals\(dq. Also accepts ErrorCondition events for more detailed retry attempts. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBintervals\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI]\fP) \-\- A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back\-off before failing: 1s, 1s, 2s, 4s, 8s, 16s .IP \(bu 2 \fBinfinite_retries\fP (\fI\%bool\fP) \-\- If this is True, reset the intervals when they run out. Defaults to: False. .IP \(bu 2 \fBerrors\fP (\fIOptional\fP\fI[\fP\fISequence\fP\fI[\fP\fIUnion\fP\fI[\fP\fI\%ErrorCondition\fP\fI, \fP\fIType\fP\fI[\fP\fI\%Exception\fP\fI]\fP\fI]\fP\fI]\fP\fI]\fP) \-\- .sp A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: \- Exception (required) \- Error codes that must match to be retried (optional; defaults to not checking) \- A string that must be in the error message to be retried (optional; defaults to not checking) \- A bool that can be set to False to always error on this condition. .sp If not specified, this will default to a generic Exception. .IP \(bu 2 \fBlog_message\fP (\fIOptional\fP\fI[\fP\fITuple\fP\fI[\fP\fICallable\fP\fI, \fP\fI\%str\fP\fI]\fP\fI]\fP) \-\- Optional tuple of (\(dqlog/print function()\(dq, \(dqmessage string\(dq) that will precede each attempt. .IP \(bu 2 \fBprepare\fP (\fIOptional\fP\fI[\fP\fIList\fP\fI[\fP\fICallable\fP\fI]\fP\fI]\fP) \-\- Optional list of functions to call, with the function\(aqs arguments, between retries, to reset state. .UNINDENT .TP .B Returns The result of the wrapped function or raise. .TP .B Return type Callable[[Any], Any] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.currentCommit = \(aq3f9cba3766e52866ea80d0934498f8c8f3129c3f\(aq .UNINDENT .INDENT 0.0 .TP .B toil.log .UNINDENT .INDENT 0.0 .TP .B toil.which(cmd, mode=os.F_OK | os.X_OK, path=None) Return the path with conforms to the given mode on the Path. .sp [Copy\-pasted in from python3.6\(aqs shutil.which().] .sp \fImode\fP defaults to os.F_OK | os.X_OK. \fIpath\fP defaults to the result of os.environ.get(\(dqPATH\(dq), or can be overridden with a custom search path. .INDENT 7.0 .TP .B Returns The path found, or None. .TP .B Return type Optional[\fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.toilPackageDirPath() Return the absolute path of the directory that corresponds to the top\-level toil package. .sp The return value is guaranteed to end in \(aq/toil\(aq. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.inVirtualEnv() Test if we are inside a virtualenv or Conda virtual environment. .INDENT 7.0 .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.resolveEntryPoint(entryPoint) Find the path to the given entry point that \fIshould\fP work on a worker. .INDENT 7.0 .TP .B Returns The path found, which may be an absolute or a relative path. .TP .B Parameters \fBentryPoint\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.physicalMemory() Calculate the total amount of physical memory, in bytes. .sp .EX >>> n = physicalMemory() >>> n > 0 True >>> n == physicalMemory() True .EE .INDENT 7.0 .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.physicalDisk(directory) .INDENT 7.0 .TP .B Parameters \fBdirectory\fP (\fI\%str\fP) \-\- .TP .B Return type \fI\%int\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.applianceSelf(forceDockerAppliance=False) Return the fully qualified name of the Docker image to start Toil appliance containers from. .sp The result is determined by the current version of Toil and three environment variables: \fBTOIL_DOCKER_REGISTRY\fP, \fBTOIL_DOCKER_NAME\fP and \fBTOIL_APPLIANCE_SELF\fP\&. .sp \fBTOIL_DOCKER_REGISTRY\fP specifies an account on a publicly hosted docker registry like Quay or Docker Hub. The default is UCSC\(aqs CGL account on Quay.io where the Toil team publishes the official appliance images. \fBTOIL_DOCKER_NAME\fP specifies the base name of the image. The default of \fItoil\fP will be adequate in most cases. \fBTOIL_APPLIANCE_SELF\fP fully qualifies the appliance image, complete with registry, image name and version tag, overriding both \fBTOIL_DOCKER_NAME\fP and \fITOIL_DOCKER_REGISTRY\(ga\fP as well as the version tag of the image. Setting TOIL_APPLIANCE_SELF will not be necessary in most cases. .INDENT 7.0 .TP .B Parameters \fBforceDockerAppliance\fP (\fI\%bool\fP) \-\- .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.customDockerInitCmd() Return the custom command set by the \fBTOIL_CUSTOM_DOCKER_INIT_COMMAND\fP environment variable. .sp The custom docker command is run prior to running the workers and/or the primary node\(aqs services. .sp This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab). .INDENT 7.0 .TP .B Returns The custom commmand, or an empty string is returned if the environment variable is not set. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.customInitCmd() Return the custom command set by the \fBTOIL_CUSTOM_INIT_COMMAND\fP environment variable. .sp The custom init command is run prior to running Toil appliance itself in workers and/or the primary node (i.e. this is run one stage before \fBTOIL_CUSTOM_DOCKER_INIT_COMMAND\fP). .sp This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab). .sp returns: the custom command or n empty string is returned if the environment variable is not set. .INDENT 7.0 .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.lookupEnvVar(name, envName, defaultValue) Look up environment variables that control Toil and log the result. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBname\fP (\fI\%str\fP) \-\- the human readable name of the variable .IP \(bu 2 \fBenvName\fP (\fI\%str\fP) \-\- the name of the environment variable to lookup .IP \(bu 2 \fBdefaultValue\fP (\fI\%str\fP) \-\- the fall\-back value .UNINDENT .TP .B Returns the value of the environment variable or the default value the variable is not set .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.checkDockerImageExists(appliance) Attempt to check a url registryName for the existence of a docker image with a given tag. .INDENT 7.0 .TP .B Parameters \fBappliance\fP (\fI\%str\fP) \-\- The url of a docker image\(aqs registry (with a tag) of the form: \(aqquay.io/:\(aq or \(aq:\(aq. Examples: \(aqquay.io/ucsc_cgl/toil:latest\(aq, \(aqubuntu:latest\(aq, or \(aqbroadinstitute/genomes\-in\-the\-cloud:2.0.0\(aq. .TP .B Returns Raises an exception if the docker image cannot be found or is invalid. Otherwise, it will return the appliance string. .TP .B Return type \fI\%str\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.parseDockerAppliance(appliance) Derive parsed registry, image reference, and tag from a docker image string. .sp Example: \(dqquay.io/ucsc_cgl/toil:latest\(dq Should return: \(dqquay.io\(dq, \(dqucsc_cgl/toil\(dq, \(dqlatest\(dq .sp If a registry is not defined, the default is: \(dqdocker.io\(dq If a tag is not defined, the default is: \(dqlatest\(dq .INDENT 7.0 .TP .B Parameters \fBappliance\fP (\fI\%str\fP) \-\- The full url of the docker image originally specified by the user (or the default). e.g. \(dqquay.io/ucsc_cgl/toil:latest\(dq .TP .B Returns registryName, imageName, tag .TP .B Return type Tuple[\fI\%str\fP, \fI\%str\fP, \fI\%str\fP] .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.checkDockerSchema(appliance) .UNINDENT .INDENT 0.0 .TP .B exception toil.ApplianceImageNotFound(origAppliance, url, statusCode) Bases: \fBdocker.errors.ImageNotFound\fP .sp Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBorigAppliance\fP (\fI\%str\fP) \-\- The full url of the docker image originally specified by the user (or the default). e.g. \(dqquay.io/ucsc_cgl/toil:latest\(dq .IP \(bu 2 \fBurl\fP (\fI\%str\fP) \-\- The URL at which the image\(aqs manifest is supposed to appear .IP \(bu 2 \fBstatusCode\fP (\fI\%int\fP) \-\- the failing HTTP status code returned by the URL .UNINDENT .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.KNOWN_EXTANT_IMAGES .UNINDENT .INDENT 0.0 .TP .B toil.requestCheckRegularDocker(origAppliance, registryName, imageName, tag) Check if an image exists using the requests library. .sp URL is based on the \fI\%docker v2 schema\fP\&. .sp This has the following format: \fBhttps://{websitehostname}.io/v2/{repo}/manifests/{tag}\fP .sp Does not work with the official (docker.io) site, because they require an OAuth token, so a separate check is done for docker.io images. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBorigAppliance\fP (\fI\%str\fP) \-\- The full url of the docker image originally specified by the user (or the default). For example, \fBquay.io/ucsc_cgl/toil:latest\fP\&. .IP \(bu 2 \fBregistryName\fP (\fI\%str\fP) \-\- The url of a docker image\(aqs registry. For example, \fBquay.io\fP\&. .IP \(bu 2 \fBimageName\fP (\fI\%str\fP) \-\- The image, including path and excluding the tag. For example, \fBucsc_cgl/toil\fP\&. .IP \(bu 2 \fBtag\fP (\fI\%str\fP) \-\- The tag used at that docker image\(aqs registry. For example, \fBlatest\fP\&. .UNINDENT .TP .B Raises ApplianceImageNotFound if no match is found. .TP .B Returns Return True if match found. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.requestCheckDockerIo(origAppliance, imageName, tag) Check docker.io to see if an image exists using the requests library. .sp URL is based on the docker v2 schema. Requires that an access token be fetched first. .INDENT 7.0 .TP .B Parameters .INDENT 7.0 .IP \(bu 2 \fBorigAppliance\fP (\fI\%str\fP) \-\- The full url of the docker image originally specified by the user (or the default). For example, \fBubuntu:latest\fP\&. .IP \(bu 2 \fBimageName\fP (\fI\%str\fP) \-\- The image, including path and excluding the tag. For example, \fBubuntu\fP\&. .IP \(bu 2 \fBtag\fP (\fI\%str\fP) \-\- The tag used at that docker image\(aqs registry. For example, \fBlatest\fP\&. .UNINDENT .TP .B Raises ApplianceImageNotFound if no match is found. .TP .B Returns Return True if match found. .TP .B Return type \fI\%bool\fP .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.logProcessContext(config) .INDENT 7.0 .TP .B Parameters \fBconfig\fP (\fI\%common.Config\fP) \-\- .TP .B Return type None .UNINDENT .UNINDENT .INDENT 0.0 .TP .B toil.cache_path = \(aq~/.cache/aws/cached_temporary_credentials\(aq .UNINDENT .IP [1] 5 Created with \fI\%sphinx\-autoapi\fP .INDENT 0.0 .IP \(bu 2 \fI\%Index\fP .IP \(bu 2 \fI\%Search Page\fP .UNINDENT .SH AUTHOR UCSC Computational Genomics Lab .SH COPYRIGHT 2024 – 2024 UCSC Computational Genomics Lab .\" Generated by docutils manpage writer. .