ARTICLE · 1367055
【极速学习spark源码】SortShuffleManager 详解:三种 Writer 的选择逻辑
基于 Spark 4.2,分支
branch-4.2阅读时长约 10 分钟 · 进阶
背景
Spark 4.2 默认 shuffle manager 是 SortShuffleManager。但它内部并不是只有一种写法,而是三种 Writer:
SortShuffleWriterBypassMergeSortShuffleWriterUnsafeShuffleWriter
问题是:什么时候选哪个?
一、入口:registerShuffle
SortShuffleManager.scala:73:
private[spark] class SortShuffleManager(conf: SparkConf, isDriver: Boolean) extends ShuffleManager选择 handle 的入口是 SortShuffleManager.scala:90:
override def registerShuffle[K, V, C](shuffleId: Int,dependency: ShuffleDependency[K, V, C]): ShuffleHandle = {if (SortShuffleWriter.shouldBypassMergeSort(conf, dependency)) {new BypassMergeSortShuffleHandle(shuffleId, dependency)} else if (SortShuffleManager.canUseSerializedShuffle(dependency)) {new SerializedShuffleHandle(shuffleId, dependency)} else {new BaseShuffleHandle(shuffleId, dependency)}}
顺序很重要:
先判断 bypass
再判断 serialized
最后 fallback 到 base
二、handle 到 writer 的映射
getWriter 在 SortShuffleManager.scala:145:
override def getWriter[K, V](handle: ShuffleHandle, ...): ShuffleWriter[K, V] = {handle match {case unsafeShuffleHandle: SerializedShuffleHandle[K @unchecked, V @unchecked] =>new UnsafeShuffleWriter(...)case bypassMergeSortHandle: BypassMergeSortShuffleHandle[K @unchecked, V @unchecked] =>new BypassMergeSortShuffleWriter(...)case other: BaseShuffleHandle[K @unchecked, V @unchecked, _] =>new SortShuffleWriter(...)}}
对应关系:
| ShuffleHandle | Writer | 行号 |
|---|---|---|
SerializedShuffleHandle | UnsafeShuffleWriter | SortShuffleManager.scala:155 |
BypassMergeSortShuffleHandle | BypassMergeSortShuffleWriter | :165 |
BaseShuffleHandle | SortShuffleWriter | :173 |
三、第一优先级:BypassMergeSortShuffleWriter
判断入口:SortShuffleWriter.scala:118
def shouldBypassMergeSort(conf: SparkConf, dep: ShuffleDependency[_, _, _]): Boolean = {if (dep.mapSideCombine) {false} else {val bypassMergeThreshold = conf.get(config.SHUFFLE_SORT_BYPASS_MERGE_THRESHOLD)dep.partitioner.numPartitions <= bypassMergeThreshold}}
条件很简单:
没有 map-side combine
reduce 分区数不超过阈值
阈值配置:
internal/config/package.scala:1750:SHUFFLE_SORT_BYPASS_MERGE_THRESHOLDinternal/config/package.scala:1751:key 是spark.shuffle.sort.bypassMergeThresholdinternal/config/package.scala:1756:默认值 200
Writer 定义:
BypassMergeSortShuffleWriter.java:84:类定义BypassMergeSortShuffleWriter.java:148:writeBypassMergeSortShuffleWriter.java:305:stop
这条路径适合小分区、无聚合场景,避免排序开销。
四、第二优先级:UnsafeShuffleWriter
如果 bypass 不满足,再判断 serialized shuffle。
SortShuffleManager.scala:227:
def canUseSerializedShuffle(dependency: ShuffleDependency[_, _, _]): Boolean = {val shufId = dependency.shuffleIdval numPartitions = dependency.partitioner.numPartitionsif (!dependency.serializer.supportsRelocationOfSerializedObjects) falseelse if (dependency.mapSideCombine) falseelse if (numPartitions > MAX_SHUFFLE_OUTPUT_PARTITIONS_FOR_SERIALIZED_MODE) falseelse true}
serializer 支持 relocation
没有 map-side combine
partition 数不超过上限
Writer 定义:
UnsafeShuffleWriter.java:70:类定义UnsafeShuffleWriter.java:189:writeUnsafeShuffleWriter.java:512:stop
它尽量以序列化数据方式处理,减少对象反序列化和 GC 压力。
五、默认路径:SortShuffleWriter
如果前两条都不满足,走 BaseShuffleHandle + SortShuffleWriter。
Writer 定义:
SortShuffleWriter.scala:29:类定义SortShuffleWriter.scala:65:writeSortShuffleWriter.scala:92:stop
它最通用,支持:
map-side combine
aggregator
ordering
spill
任意较复杂 shuffle 场景
SortShuffleWriter.scala:66-75 根据 dep.mapSideCombine 决定是否把 aggregator 传给 ExternalSorter。
六、ShuffleDependency 决定选择条件
ShuffleDependency.scala 的定义实际在 Dependency.scala:84:
class ShuffleDependency[K, V, C](_rdd: RDD[_ <: Product2[K, V]],val partitioner: Partitioner,val serializer: Serializer,val keyOrdering: Option[Ordering[K]],val aggregator: Option[Aggregator[K, V, C]],val mapSideCombine: Boolean,...)
关键字段:
partitioner:决定 reduce 分区数(Dependency.scala:87)serializer:决定能否 serialized shuffle(:87)aggregator:map-side combine 需要(:89)mapSideCombine:bypass 和 serialized 都会检查(:90)
ShuffleDependency 构造时会注册 shuffle。Dependency.scala:137:
shuffleHandle=shuffleManager.registerShuffle(shuffleId, this)也就是说,writer 选择发生在 ShuffleDependency 创建时。
七、三种 Writer 对比
| Writer | 触发条件 | 优点 | 限制 |
|---|---|---|---|
BypassMergeSortShuffleWriter | 无 map-side combine;分区数 <= 阈值 | 避免排序,适合小分区 | 文件/分区路径更简单,适用窄 |
UnsafeShuffleWriter | serializer 支持 relocation;无 map-side combine;分区数不超限 | 减少对象开销和 GC | 条件严格 |
SortShuffleWriter | 默认 fallback | 最通用,支持 combine/聚合/排序/spill | 开销相对更高 |
选择顺序:
shouldBypassMergeSort ?
yes → BypassMergeSortShuffleWriter
no → canUseSerializedShuffle ?
yes → UnsafeShuffleWriter
no → SortShuffleWriter
八、ExternalSorter 和 ShuffleExternalSorter
SortShuffleWriter 使用 ExternalSorter:
ExternalSorter.scala:95:类定义ExternalSorter.scala:192:insertAllExternalSorter.scala:259:spillExternalSorter.scala:709:writePartitionedMapOutputExternalSorter.scala:774:stop
UnsafeShuffleWriter 走 unsafe sorter:
UnsafeExternalSorter.java:52ShuffleExternalSorter.java:73ShuffleExternalSorter.java:168:writeSortedFileShuffleExternalSorter.java:423:insertRecordShuffleExternalSorter.java:468:closeAndGetSpills
输出抽象由 ShuffleMapOutputWriter 管:
ShuffleMapOutputWriter.java:33:接口定义ShuffleMapOutputWriter.java:51:getPartitionWriterShuffleMapOutputWriter.java:79:commitAllPartitions
本地磁盘实现:
LocalDiskShuffleExecutorComponents.java:67LocalDiskShuffleMapOutputWriter.java:46
九、配置项
最常用配置:
spark.shuffle.sort.bypassMergeThreshold源码位置:
internal/config/package.scala:1750internal/config/package.scala:1751internal/config/package.scala:1756
旧配置:
SparkConf.scala:678:spark.shuffle.spill已 deprecated,标注Not used anymore.
spill 相关当前配置:
spark.shuffle.spill.numElementsForceSpillThreshold:internal/config/package.scala:1615spark.shuffle.spill.maxSizeInBytesForSpillThreshold::1626spark.shuffle.spill.compress::1707spark.shuffle.spill.initialMemoryThreshold::1725
Push-based shuffle 配置:
spark.shuffle.push.enabled:internal/config/package.scala:2629spark.shuffle.push.numPushThreads::2698spark.shuffle.push.maxBlockSizeToPush::2707
十、案例:预测 writer 路径
// 1. 小分区,无 map-side combine,更可能走 bypassval rdd1 = sc.parallelize(1 to 100, 4).map(x => (x % 10, x)).groupByKey(numPartitions = 10)// 2. reduceByKey 有 map-side combine,不能 bypass,也不能 serializedval rdd2 = sc.parallelize(1 to 100, 4).map(x => (x % 10, x)).reduceByKey(_ + _)rdd1.count()rdd2.count()
结果:
groupByKey(numPartitions = 10):无 map-side combine,分区数小于默认阈值 200,可能选择 bypass handlereduceByKey:mapSideCombine = true,不能 bypass,也不能 serialized,走 base sort shuffle
具体 writer 类型需要打日志或断点确认。发布前如果要写成确定结论,应在本地加断点或日志验证。
十一、总结
SortShuffleManager 的选择逻辑:
ShuffleDependency
→ shuffleManager.registerShuffle
→ shouldBypassMergeSort
→ BypassMergeSortShuffleHandle
→ BypassMergeSortShuffleWriter
→ canUseSerializedShuffle
→ SerializedShuffleHandle
→ UnsafeShuffleWriter
→ BaseShuffleHandle
→ SortShuffleWriter
最值得记住的三个观点:
Writer 选择发生在 registerShuffle。 也就是
ShuffleDependency创建时决定 handle。bypass 优先级最高。 只要无 map-side combine 且分区数不超过阈值,就优先走 bypass。
SortShuffleWriter 是兜底通用路径。 map-side combine、aggregator、复杂场景最终都会落到它。















































































