As good tests should be isolated from one another, their order of execution is arbitrary by definition. The tests should produce the same results consistently regardless of where they are located in the execution chain. I thought I might take this to the test, and perhaps find a bug or two along the way.
Working in Python, I now simply shuffle the list of tests prior to execution:
random.shuffle(testList)
executor.exec(testList)
Nice. Now what happens if a certain execution order fails? I would find the issue and fix it, what else?! Right, but how will I reproduce the same exact execution order that failed before, in order to validate the fix?
One naive option is saving the entire execution order as a list of indices, this is not very comfortable. A simpler option is to simply seed your random object with the same seed int value used during the execution to reproduce. Here's the code:
import random
seedValue = options.get('shuffleSeed', random.randint(0, 10000)) # Seed is provided by the user in an attempt to reproduce, otherwise it's set to some random value.
random.seed(seedValue)
random.shuffle(testList)
print 'Shuffled with seed=' + str(seedValue)
executor.exec(testList)
