I do a lot of performance testing in the course of my research, and so, like a lot of other people I imagine, I wrote a little test harness to manage running jobs. That harness treats each test as a discrete part of a larger test -- and runs through them individually until complete. The benefit of this is that you can use the model to make sure that your conditions are the same for each test, you can see the progress of the overall tests by looking at the job queue, and you can pick up where you left off if something goes wrong. I've spent a lot of time on that script.
But then, I always forget about it and start a long string of jobs using a nested for loop... and then I lose all those benefits. My tests aren't particularly sensitive to conditions, but since I didn't use a job queue of any kind, I have no idea when the overall tests will end, and if I kill the task, I'll lose all my progress. Gar!
