Let's say I run a job in Spark with speculation = true.
If a task (let's say T1) takes a long time, Spark would launch a copy of task T1, say, T2 on another executor, without killing off T1.
Now, if T2 also takes more time than the median of all successfully completed tasks, would Spark launch another task T3 on another executor?
If yes, is there any limit to this spawning of new tasks? If no, does Spark limit itself to one parallel job, and waits indefinitely for completion of either one?
The spark TaskSetManager is responsible for that logic. It is checking that at most one copy of the original task is running when trying to launch a speculatable task. So in your example it should never launch T3 since there would be 2 copies running.
You can find the relevant part of the code here.
// Speculatable task should only be launched when at most one copy of the original task is running. Doesn't this mean that T3 would be started (since T2 is the copy of the original task), but it would stop at T3, and no T4 would be launched?
copiesRunning(index) would be 1 once T1 is running, so if T2 gets started as well copiesRunning(index) would be 2 (assuming T1 is still running) and then the dequeueTaskFromList function would return nothing (namely None).