夜雨聆风学习资料网

ARTICLE · 1367055

【极速学习spark源码】SortShuffleManager 详解:三种 Writer 的选择逻辑

【极速学习spark源码】SortShuffleManager 详解:三种 Writer 的选择逻辑

基于 Spark 4.2,分支 branch-4.2

阅读时长约 10 分钟 · 进阶

背景

Spark 4.2 默认 shuffle manager 是 SortShuffleManager。但它内部并不是只有一种写法,而是三种 Writer:

  • SortShuffleWriter

  • BypassMergeSortShuffleWriter

  • UnsafeShuffleWriter

问题是:什么时候选哪个?


一、入口:registerShuffle

SortShuffleManager.scala:73:

private[spark] class SortShuffleManager(conf: SparkConf, isDriver: Boolean) extends ShuffleManager

选择 handle 的入口是 SortShuffleManager.scala:90:

override def registerShuffle[K, V, C](    shuffleId: Int,    dependency: ShuffleDependency[K, V, C]): ShuffleHandle = {  if (SortShuffleWriter.shouldBypassMergeSort(conf, dependency)) {    new BypassMergeSortShuffleHandle(shuffleId, dependency)  } else if (SortShuffleManager.canUseSerializedShuffle(dependency)) {    new SerializedShuffleHandle(shuffleId, dependency)  } else {    new BaseShuffleHandle(shuffleId, dependency)  }}

顺序很重要:

  1. 先判断 bypass

  2. 再判断 serialized

  3. 最后 fallback 到 base


二、handle 到 writer 的映射

getWriter 在 SortShuffleManager.scala:145:

override def getWriter[K, V](handle: ShuffleHandle, ...): ShuffleWriter[K, V] = {  handle match {    case unsafeShuffleHandle: SerializedShuffleHandle[K @unchecked, V @unchecked] =>      new UnsafeShuffleWriter(...)    case bypassMergeSortHandle: BypassMergeSortShuffleHandle[K @unchecked, V @unchecked] =>      new BypassMergeSortShuffleWriter(...)    case other: BaseShuffleHandle[K @unchecked, V @unchecked, _] =>      new SortShuffleWriter(...)  }}

对应关系:

ShuffleHandleWriter行号
SerializedShuffleHandleUnsafeShuffleWriterSortShuffleManager.scala:155
BypassMergeSortShuffleHandleBypassMergeSortShuffleWriter:165
BaseShuffleHandleSortShuffleWriter:173

三、第一优先级:BypassMergeSortShuffleWriter

判断入口:SortShuffleWriter.scala:118

def shouldBypassMergeSort(conf: SparkConf, dep: ShuffleDependency[_, _, _]): Boolean = {  if (dep.mapSideCombine) {    false  } else {    val bypassMergeThreshold = conf.get(config.SHUFFLE_SORT_BYPASS_MERGE_THRESHOLD)    dep.partitioner.numPartitions <= bypassMergeThreshold  }}

条件很简单:

  1. 没有 map-side combine

  2. reduce 分区数不超过阈值

阈值配置:

  • internal/config/package.scala:1750:SHUFFLE_SORT_BYPASS_MERGE_THRESHOLD

  • internal/config/package.scala:1751:key 是 spark.shuffle.sort.bypassMergeThreshold

  • internal/config/package.scala:1756:默认值 200

Writer 定义:

  • BypassMergeSortShuffleWriter.java:84:类定义

  • BypassMergeSortShuffleWriter.java:148:write

  • BypassMergeSortShuffleWriter.java:305:stop

这条路径适合小分区、无聚合场景,避免排序开销。


四、第二优先级:UnsafeShuffleWriter

如果 bypass 不满足,再判断 serialized shuffle。

SortShuffleManager.scala:227:

def canUseSerializedShuffle(dependency: ShuffleDependency[_, _, _]): Boolean = {  val shufId = dependency.shuffleId  val numPartitions = dependency.partitioner.numPartitions  if (!dependency.serializer.supportsRelocationOfSerializedObjects) false  else if (dependency.mapSideCombine) false  else if (numPartitions > MAX_SHUFFLE_OUTPUT_PARTITIONS_FOR_SERIALIZED_MODE) false  else true}
  1. serializer 支持 relocation

  2. 没有 map-side combine

  3. partition 数不超过上限

Writer 定义:

  • UnsafeShuffleWriter.java:70:类定义

  • UnsafeShuffleWriter.java:189:write

  • UnsafeShuffleWriter.java:512:stop

它尽量以序列化数据方式处理,减少对象反序列化和 GC 压力。


五、默认路径:SortShuffleWriter

如果前两条都不满足,走 BaseShuffleHandle + SortShuffleWriter。

Writer 定义:

  • SortShuffleWriter.scala:29:类定义

  • SortShuffleWriter.scala:65:write

  • SortShuffleWriter.scala:92:stop

它最通用,支持:

  • map-side combine

  • aggregator

  • ordering

  • spill

  • 任意较复杂 shuffle 场景

SortShuffleWriter.scala:66-75 根据 dep.mapSideCombine 决定是否把 aggregator 传给 ExternalSorter。


六、ShuffleDependency 决定选择条件

ShuffleDependency.scala 的定义实际在 Dependency.scala:84:

class ShuffleDependency[K, V, C](    _rdd: RDD[_ <: Product2[K, V]],    val partitioner: Partitioner,    val serializer: Serializer,    val keyOrdering: Option[Ordering[K]],    val aggregator: Option[Aggregator[K, V, C]],    val mapSideCombine: Boolean,    ...)

关键字段:

  • partitioner:决定 reduce 分区数(Dependency.scala:87)

  • serializer:决定能否 serialized shuffle(:87)

  • aggregator:map-side combine 需要(:89)

  • mapSideCombine:bypass 和 serialized 都会检查(:90)

ShuffleDependency 构造时会注册 shuffle。Dependency.scala:137:

shuffleHandle=shuffleManager.registerShuffle(shuffleId, this)

也就是说,writer 选择发生在 ShuffleDependency 创建时。


七、三种 Writer 对比

Writer触发条件优点限制
BypassMergeSortShuffleWriter无 map-side combine;分区数 <= 阈值避免排序,适合小分区文件/分区路径更简单,适用窄
UnsafeShuffleWriterserializer 支持 relocation;无 map-side combine;分区数不超限减少对象开销和 GC条件严格
SortShuffleWriter默认 fallback最通用,支持 combine/聚合/排序/spill开销相对更高

选择顺序:

shouldBypassMergeSort ?
  yes → BypassMergeSortShuffleWriter
  no  → canUseSerializedShuffle ?
          yes → UnsafeShuffleWriter
          no  → SortShuffleWriter

八、ExternalSorter 和 ShuffleExternalSorter

SortShuffleWriter 使用 ExternalSorter:

  • ExternalSorter.scala:95:类定义

  • ExternalSorter.scala:192:insertAll

  • ExternalSorter.scala:259:spill

  • ExternalSorter.scala:709:writePartitionedMapOutput

  • ExternalSorter.scala:774:stop

UnsafeShuffleWriter 走 unsafe sorter:

  • UnsafeExternalSorter.java:52

  • ShuffleExternalSorter.java:73

  • ShuffleExternalSorter.java:168:writeSortedFile

  • ShuffleExternalSorter.java:423:insertRecord

  • ShuffleExternalSorter.java:468:closeAndGetSpills

输出抽象由 ShuffleMapOutputWriter 管:

  • ShuffleMapOutputWriter.java:33:接口定义

  • ShuffleMapOutputWriter.java:51:getPartitionWriter

  • ShuffleMapOutputWriter.java:79:commitAllPartitions

本地磁盘实现:

  • LocalDiskShuffleExecutorComponents.java:67

  • LocalDiskShuffleMapOutputWriter.java:46


九、配置项

最常用配置:

spark.shuffle.sort.bypassMergeThreshold

源码位置:

  • internal/config/package.scala:1750

  • internal/config/package.scala:1751

  • internal/config/package.scala:1756

旧配置:

  • SparkConf.scala:678:spark.shuffle.spill 已 deprecated,标注 Not used anymore.

spill 相关当前配置:

  • spark.shuffle.spill.numElementsForceSpillThreshold:internal/config/package.scala:1615

  • spark.shuffle.spill.maxSizeInBytesForSpillThreshold::1626

  • spark.shuffle.spill.compress::1707

  • spark.shuffle.spill.initialMemoryThreshold::1725

Push-based shuffle 配置:

  • spark.shuffle.push.enabled:internal/config/package.scala:2629

  • spark.shuffle.push.numPushThreads::2698

  • spark.shuffle.push.maxBlockSizeToPush::2707


十、案例:预测 writer 路径

// 1. 小分区,无 map-side combine,更可能走 bypassval rdd1 = sc.parallelize(1 to 100, 4)  .map(x => (x % 10, x))  .groupByKey(numPartitions = 10)// 2. reduceByKey 有 map-side combine,不能 bypass,也不能 serializedval rdd2 = sc.parallelize(1 to 100, 4)  .map(x => (x % 10, x))  .reduceByKey(_ + _)rdd1.count()rdd2.count()

结果:

  • groupByKey(numPartitions = 10):无 map-side combine,分区数小于默认阈值 200,可能选择 bypass handle

  • reduceByKey:mapSideCombine = true,不能 bypass,也不能 serialized,走 base sort shuffle

具体 writer 类型需要打日志或断点确认。发布前如果要写成确定结论,应在本地加断点或日志验证。


十一、总结

SortShuffleManager 的选择逻辑:

ShuffleDependency
  → shuffleManager.registerShuffle
    → shouldBypassMergeSort
       → BypassMergeSortShuffleHandle
       → BypassMergeSortShuffleWriter
    → canUseSerializedShuffle
       → SerializedShuffleHandle
       → UnsafeShuffleWriter
    → BaseShuffleHandle
       → SortShuffleWriter

最值得记住的三个观点:

  1. Writer 选择发生在 registerShuffle。 也就是 ShuffleDependency 创建时决定 handle。

  2. bypass 优先级最高。 只要无 map-side combine 且分区数不超过阈值,就优先走 bypass。

  3. SortShuffleWriter 是兜底通用路径。 map-side combine、aggregator、复杂场景最终都会落到它。

回首页,搜索你想要的文档

在首页搜索框输入关键词,快速定位资料

文库首页搜索指引 立即搜索

相关学习资料