FOSDEM event "Faster Spark SQL: Adaptive Query Execution in Spark v3"

Nicolas Poggi

Type devroom

Starts on day 1 (2021-02-06) at 11:00 (Brussels time, UTC+1) in room Hpc (duration 00:30)

Matrix room #hpc:fosdem.org

Over the years, there has been extensive efforts to improve Apache Spark SQL performance. This talk will introduce the new Adaptive Query Execution (AQE) framework and how it can automatically improve user query performance. AQE leverages query runtime statistics to dynamically guide Spark's execution as queries run along. The talk will go over the main features in AQE and provide examples on how it can improve on the previous static query plans. Finally, we'll present the significant improvements we have seen on the TPC-DS benchmark with AQE.

=> FOSDEM schedule page