Hadoop入门系列(3)-Hadoop三种运行模式及案例

╰半橙微兮° 2023-10-10 11:59 27阅读 0赞

### 前言 ###

可以以三种支持的模式之一启动Hadoop集群：

*  本地（独立）模式
 *  伪分布式模式
 *  全分布式模式

### 本地（独立）模式案例 ###

默认情况下，Hadoop被配置为以非分布式模式作为单个Java进程运行。这对于调试很有用。

下面的示例复制解压缩的conf目录以用作输入，然后查找并显示给定正则表达式的每个匹配项。输出被写入给定的输出目录。  
**案例之grep**

cd /opt/soft/hadoop
    mkdir input
    cp etc/hadoop/*.xml input
    # 运行一个jar，查找
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar grep input output 'dfs[a-z.]+'
    cat output/*

![在这里插入图片描述][20201202094453160.png]  
**案例之WordCount**

cd /opt/soft/hadoop
    mkdir wcinput 
    cd wcinput
    vi wc.input
    # 内容
    hadoop yarn
    hadoop mapreduce yarn yarn yarn count count count
    # :wq保存退出
    cd ../
    # 统计每个单词出现的次数
    hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar wordcount wcinput wcoutput
    cat wcoutput/part-r-00000

![在这里插入图片描述][20201202113852791.png]

### 伪分布式模式配置及案例 ###

伪分布式就是假分布式，一个机器上配置了hadoop的所有节点。

1.  配置集群

vi etc/hadoop/core-site.xml
    # 
    <configuration>
     
     <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
     </property>
     
     <property>
            <name>hadoop.tmp.dir</name>
            <value>/opt/soft/hadoop/data/tmp</value>
    </property>
    </configuration>
    # 
    vi etc/hadoop/hdfs-site.xml
    # 添加内容
    <configuration>
      
      <property>
            <name>dfs.replication</name>
            <value>1</value>
      </property>
    </configuration>
    # 格式化NameNode
    bin/hdfs namenode -format
    # 启动NameNode
    sbin/hadoop-daemon.sh start namenode
    # 启动DataNode
    sbin/hadoop-daemon.sh start datanode
    # 查看
    ps -ef | grep NameNode
    ps -ef | grep DataNode

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70]  
浏览器访问HDFS ：http://192.168.58.241:9870  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 1]

1.  案例测试

# 在HDFS文件系统上创建一个input文件夹
    bin/hdfs dfs -mkdir -p /data/input
    # 将测试文件内容上传到文件系统上,此时可在界面上查看到上传的文件
    bin/hdfs dfs -put etc/hadoop/*.xml /data/input
    # 运行示例程序
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar  grep /data/input /data/output 'dfs[a-z.]+'
    # 查看结果，也可在界面下载查看
    bin/hdfs dfs -cat /data/output/*

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 3]

### 伪分布式模式启动YARN运行MapReduce程序 ###

可以通过设置一些参数并另外运行ResourceManager守护程序和NodeManager守护程序，以伪分布式模式在YARN上运行MapReduce作业

1.  配置参数

# 配置mapred
    vim etc/hadoop/mapred-site.xml
    # 添加
    <configuration>
        
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>
    # 配置yarn
    vim etc/hadoop/yarn-site.xml
    # 添加
    <configuration>
     
     <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
     </property>
     
     <property>
          <name>yarn.resourcemanager.hostname</name>
          <value>192.168.58.241</value>
     </property>
    </configuration>
    # 启动ResourceManager
    sbin/yarn-daemon.sh start resourcemanager
    # 启动NodeManager
    sbin/yarn-daemon.sh start nodemanager

浏览器访问：http://192.168.58.241:8088/cluster  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 4]

1.  案例

# 删除文件系统上的output文件
    bin/hdfs dfs -rm -R /data/output
    # 需要配置hadoop classpath不然会报错错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
    # 查看classpath 
    hadoop classpath 
    # 添加classpath 
    vim etc/hadoop/yarn-site.xml
    # 内容
    <property>
             <name>yarn.application.classpath</name>
             <value>/opt/soft/hadoop/etc/hadoop:/opt/soft/hadoop/share/hadoop/common/lib/*:/opt/soft/hadoop/share/hadoop/common/*:/opt/soft/hadoop/share/hadoop/hdfs:/opt/soft/hadoop/share/hadoop/hdfs/lib/*:/opt/soft/hadoop/share/hadoop/hdfs/*:/opt/soft/hadoop/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop/share/hadoop/mapreduce/*:/opt/soft/hadoop/share/hadoop/yarn:/opt/soft/hadoop/share/hadoop/yarn/lib/*:/opt/soft/hadoop/share/hadoop/yarn/*</value>
    </property>
    # 运行
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar  wordcount  /data/input /data/output

浏览器查看详情  
![在这里插入图片描述][2020120214021644.png]

[20201202094453160.png]: https://img-blog.csdnimg.cn/20201202094453160.png
[20201202113852791.png]: https://img-blog.csdnimg.cn/20201202113852791.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20201202123235186.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20201202123541673.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20201202130712929.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20201202131237684.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20201202133756794.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNDM3ODc0,size_16,color_FFFFFF,t_70
[2020120214021644.png]: https://img-blog.csdnimg.cn/2020120214021644.png