黄海峰
2020-02-03
shuffle把相同key发送给同一个reduce，那岂不是还是要传输大量数据？还是实际是把相同key放到相同hdfs文件reduce进程读取？
作者回复: 确实要传输很大量数据。
shuffle不通过HDFS，即使通过HDFS，也解决不了数据传输的问题。
<div class="router-button__Wrapper-sc-1xlmfkg-0 jDKtMA"><div class="frame-button-content__Wrapper-sc-1td9tx7-0 huSWDo"><div class="frame-button-content__BottomLeft-sc-1td9tx7-1 dmMDDL"><label class="icon__Span-sc-5foskt-0 cSUETK"><span type="chat-button-1" width="32px" height="32px" class="base-icon__Container-sc-1ajxhaa-0 ihFnZp"></span></label></div><div class="frame-button-content__ButtonRight-sc-1td9tx7-2 cSpVKy">我们在线，来聊聊吧</div></div></div>
<div class="bubble-arrow__Arrow-sc-3uxf0p-0 bfDInF"></div><div class="router-bubble__Wrapper-k6dy4k-0 itiunK"><div class="frame-bubble-content__Wrapper-lbgrtb-0 diLqpO"></div></div>
<div class="router-invite__Wrapper-sc-1m23l53-0 gORLfI"><div><div class="frame-invite-content__CloseWrapper-sc-1g53vrw-2 eApwIP">✕</div><div class="frame-invite-content__FrameInviteContentBoxStyle-sc-1g53vrw-0 AkiIQ" transform="translate(20px,20px)"><div class="frame-invite-content__FrameInviteContentStyle-sc-1g53vrw-1 blgYRf" width="260px" height="44px"><div class="content">您好，当前有专业客服人员在线，让我们来帮助您吧。</div></div></div></div></div>
<div class="router-call-botton__Wrapper-dkoslj-0 fkxFou"><div class="frame-button-content__Wrapper-sc-1td9tx7-0 fxjwak"><div class="frame-button-content__RoundWrapper-sc-1td9tx7-3 LOaHP"><label class="icon__Span-sc-5foskt-0 cSUETK"><span type="call-callback" width="32px" height="32px" class="base-icon__Container-sc-1ajxhaa-0 cvRvuJ"></span></label></div></div></div>
我们在线，来聊聊吧
2020-02-03shuffle把相同key发送给同一个reduce，那岂不是还是要传输大量数据？还是实际是把相同key放到相同hdfs文件reduce进程读取？