There are a few settings you can use to control speculation. The examples are provided only to show how to change values. Mostly, just turning on speculation is good enough:
- Setting spark.speculation (the default is false):
$ spark-shell -conf spark.speculation=true
- Setting spark.speculation.interval (the default is 100 milliseconds) (denotes the rate at which Spark examines tasks to see whether speculation is needed):
$ spark-shell -conf spark.speculation.interval=200
- Setting spark.speculation.multiplier (the default is 1.5) (denotes how many times a task has to be slower than median to be a candidate for speculation):
$ spark-shell -conf spark.speculation.multiplier=1.5
- Setting spark.speculation.quantile (the default is 0.75) (denotes the percentage of tasks to be completed before triggering speculation):
$ spark-shell -conf spark.speculation.quantile=0.95