docker下部署spark+python+pyspark+Jupyter

我就是我 2023-06-17 14:57 17阅读 0赞

1、使用singularities/spark2.2 搭建 spark

参考 [https://hub.docker.com/r/singularities/spark][https_hub.docker.com_r_singularities_spark]

singularities/spark:2.2版本中
    Hadoop版本：2.8.2
    Spark版本: 2.2.1
    Scala版本：2.11.8
    Java版本：1.8.0_151

创建docker-compose.yml 文件

version: "2"
    
    services:
      master:
        image: singularities/spark
        command: start-spark master
        hostname: master
        ports:
          - "6066:6066"
          - "7070:7070"
          - "8080:8080"
          - "50070:50070"
      worker:
        image: singularities/spark
        command: start-spark worker master
        environment:
          SPARK_WORKER_CORES: 1
          SPARK_WORKER_MEMORY: 2g
        links:
          - master

2、启动 spark

启动容器
    docker-compose up -d
    查看容器
    docker-compose ps
    停止容器
    docker-compose stop
    删除容器
    docker-compose rm

3、启动成功可以通过浏览器访问 localhost:8080,localhost:50070 查看状态

4、创建Python+pyspark+jupyter环境

新建一个 Dockerfile 文件，写入下面的内容

FROM python:3.7
      
    MAINTAINER Haiyang Lv
    
    USER root
    
    RUN pip install jupyter -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
    
    RUN pip install pyspark -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
    
    CMD jupyter notebook --allow-root --ip=0.0.0.0

5、创建 Python+pyspark+jupyter docker images

在Dockerfile的目录下执行命令
    docker build -t python_jupyter:latest .

6、启动 jupyter 容器

docker run -itd -p 8888:8888 --name jupyter python_jupyter

7、浏览器访问 jupyter 127.0.0.1:8888

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70][]

根据提示进入容器执行 jupyter notebook list 查看 token

进入容器命令 docker exec -it jupyter bash

![20191129161409548.png][]

复制到浏览器中就可以了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 1][]

[https_hub.docker.com_r_singularities_spark]: https://hub.docker.com/r/singularities/spark
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191129161223300.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[20191129161409548.png]: https://img-blog.csdnimg.cn/20191129161409548.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191129161516493.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70