cdh集群mapreduce任务卡住不动

叁歲伎倆 2023-10-11 10:22 111阅读 0赞

# 报错原因: NodeManager内存太小，无法调度mapreduce任务运行的容器 #

#调大nodemanager自身内存   和  最小任务调拨内存
    yarn.nodemanager.resource.memory-mb
    yarn.scheduler.minimum-allocation-mb
    
    #如果yarn cpu核数总大小=1，那么也会导致任务卡住 
    # (yarn的cpu 是对物理cpu的虚拟化，可以为n倍大小) 
    yarn.scheduler.maximum-allocation-vcores =8 
    yarn.nodemanager.resource.cpu-vcores =8

博客参考：  
[yarn任务提交和运行流程：https://blog.csdn.net/shell33168/article/details/87928001][yarn_https_blog.csdn.net_shell33168_article_details_87928001]，  
[yarn任务调度详解：https://www.jianshu.com/p/f50e85bdb9ce][yarn_https_www.jianshu.com_p_f50e85bdb9ce]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]

# 正常提交的yarn任务 #

## 第一阶段：为任务申请资源 ##

1.  NodeManager通过NodeStatusUpdaterImpl向ResourceManager申请资源：Registered with total resource of <memory:1024, vCores:7>
2.  NodeManager通过ContainerManagerImpl通知ContainerManager: Start request for container
3.  nodemanager.containermanager为任务创建一个容器: Creating a new application reference for 任务id
4.  nodemanager.containermanager修改Application状态：transitioned from NEW to INITING
5.  NodeManager通过NMAuditLogger日志检查器记录此经过： USER=hdfs ，IP=192.168.58.152， OPERATION=Start Container Request，RESULT=SUCCESS，APPID=application\_1569158071152\_0001 ，CONTAINERID=container\_1569158071152\_0001\_01\_000003

## 第二阶段：初始化容器-生成.sh脚本 ##

1.  nodemanager.containermanager为Application调度容器：Adding container\_1569158071152\_0001\_01\_000003 to application application\_1569158071152\_0001
2.  nodemanager.containermanager修改Application状态：application\_1569158071152\_0001 transitioned from INITING to RUNNING
3.  nodemanager.containermanager修改Container状态：container\_1569158071152\_0001\_01\_000003 transitioned from NEW to LOCALIZING
4.  nodemanager.containermanager.AuxServices接收event (CONTAINER\_INIT) -->给mapreduce\_shuffle 发送event (APPLICATION\_INIT) --> 为任务创建初始化器（Created localizer ）
5.  nodemanager.containermanager修改Container状态： transitioned from LOCALIZED to RUNNING

## 第三步：运行容器 ##

1.  nodemanager.DefaultContainerExecutor运行容器：launchContainer: \[bash, /path to 容器初始化器创建的执行脚本(default\_container\_executor.sh) \]
2.  nodemanager.containermanager.monitor收集容器内存使用情况
3.  nodemanager.containermanager.launcher.ContainerLaunch: 汇报容器执行脚本的结果 container\_1569158071152\_0001\_01\_000003 succeeded
4.  nodemanager.containermanager修改容器状态： transitioned from RUNNING to EXITED\_WITH\_SUCCESS
5.  .nodemanager.containermanage开始清理容器产生的临时文件

## 提交任务，查看NodeManager日志 ##

此次使用的 yarn 的架构

<table> 
 <thead> 
  <tr> 
   <th>ip</th> 
   <th align="right">角色</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>192.168.56.151 cdh1</td> 
   <td align="right">nodemanager</td> 
  </tr> 
  <tr> 
   <td>192.168.56.152 cdh2</td> 
   <td align="right">resourcemanager，nodemanager</td> 
  </tr> 
 </tbody> 
</table>

*  通过hive执行mapreduce任务，提交到yarn运行

hive> insert into t values('b');
    Query ID = hdfs_20190922211515_45c84336-0b03-48f2-bc00-41d1e7d44d24
    Total jobs = 3
    Launching Job 1 out of 3
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_1569158071152_0001, Tracking URL = http://cdh2:8088/proxy/application_1569158071152_0001/
    Kill Command = /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/bin/hadoop job  -kill job_1569158071152_0001
    Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
    2019-09-22 21:15:21,163 Stage-1 map = 0%,  reduce = 0%
    2019-09-22 21:15:28,605 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.35 sec
    ....

*  查看NodeManager日志  
    tailf /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh1.log.out

#第一阶段：为任务申请资源
    INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as cdh1:8041 with total resource of <memory:1024, vCores:7>
    INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying ContainerManager to unblock new container-requests
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1569158071152_0001_01_000003 by user hdfs
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1569158071152_0001 transitioned from NEW to INITING
    INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=192.168.58.152 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1569158071152_0001 CONTAINERID=container_1569158071152_0001_01_000003
    
    # 第二阶段：初始化容器（生成default_container_executor.sh）
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: rollingMonitorInterval is set as -1. The log rolling monitoring interval is disabled. The logs will be aggregated after this application is finished.
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1569158071152_0001_01_000003 to application application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1569158071152_0001 transitioned from INITING to RUNNING
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1569158071152_0001_01_000003 transitioned from NEW to LOCALIZING
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
    INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1569158071152_0001_01_000003
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /yarn/nm/nmPrivate/container_1569158071152_0001_01_000003.tokens. Credentials list:
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hdfs
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /yarn/nm/nmPrivate/container_1569158071152_0001_01_000003.tokens to /yarn/nm/usercache/hdfs/appcache/application_1569158071152_0001/container_1569158071152_0001_01_000003.tokens
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /yarn/nm/usercache/hdfs/appcache/application_1569158071152_0001 = file:/yarn/nm/usercache/hdfs/appcache/application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1569158071152_0001_01_000003 transitioned from LOCALIZING to LOCALIZED
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1569158071152_0001_01_000003 transitioned from LOCALIZED to RUNNING
    
    ## 第三步：运行容器
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /yarn/nm/usercache/hdfs/appcache/application_1569158071152_0001/container_1569158071152_0001_01_000003/default_container_executor.sh]
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1569158071152_0001_01_000003
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 9194 for container-id container_1569158071152_0001_01_000003: 176.5 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_1569158071152_0001_01_000003 succeeded
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1569158071152_0001_01_000003 transitioned from RUNNING to EXITED_WITH_SUCCESS
    ### 开始清理容器产生的临时文件
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1569158071152_0001_01_000003
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/hdfs/appcache/application_1569158071152_0001/container_1569158071152_0001_01_000003
    INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1569158071152_0001 CONTAINERID=container_1569158071152_0001_01_000003
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1569158071152_0001_01_000003 transitioned from EXITED_WITH_SUCCESS to DONE
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1569158071152_0001_01_000003 from application application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1569158071152_0001_01_000003 for log-aggregation
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1569158071152_0001_01_000003
    INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1569158071152_0001_01_000003]
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1569158071152_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/hdfs/appcache/application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1569158071152_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1569158071152_0001
    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_1569158071152_0001_01_000003. Current good log dirs are /yarn/container-logs
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1569158071152_0001/container_1569158071152_0001_01_000003/stdout
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1569158071152_0001/container_1569158071152_0001_01_000003/syslog
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1569158071152_0001/container_1569158071152_0001_01_000003/stderr
    INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1569158071152_0001

[yarn_https_blog.csdn.net_shell33168_article_details_87928001]: https://blog.csdn.net/shell33168/article/details/87928001
[yarn_https_www.jianshu.com_p_f50e85bdb9ce]: https://www.jianshu.com/p/f50e85bdb9ce
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20190922225827416.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU=,size_16,color_FFFFFF,t_70