0
点赞
收藏
分享

微信扫一扫

Docker搭建Spark


  1. .docker编排工具docker-compose安装
  2. 使用官网指导方式安装(不推荐)​​https://docs.docker.com/compose/install/​​
  3. 使用国内镜像源安装(推荐)

curl -L https://get.daocloud.io/docker/compose/releases/download/1.22.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

 测试是否安装成功

docker-compose --version

出现

docker-compose version 1.22.0, build f46880fe

就安装成功了

3.安装docker的spark镜像singularities/spark:2.2

docker-spark-2.1.0]# docker pull singularities/spark

查看镜像是否安装成功

docker image ls

创建docker-compose.yml

  1. ​[root@localhost home]# mkdir singularitiesCR​
  2. ​[root@localhost home]# cd singularitiesCR​
  3. ​[root@localhost singularitiesCR]# touch docker-compose.yml​
  4. ​[root@localhost singularitiesCR]# vim docker-compose.yml​

编辑

version: "2"

services:
master:
image: singularities/spark
command: start-spark master
hostname: master
ports:
- "6066:6066"
- "7070:7070"
- "8080:8080"
- "50070:50070"
worker:
image: singularities/spark
command: start-spark worker master
environment:
SPARK_WORKER_CORES: 1
SPARK_WORKER_MEMORY: 2g
links:
- master


在刚刚建立docker-compose.yml文件目录下面使用命令


  1. ​[root@localhost singularitiesCR]# docker-compose up -d​
  2. ​Creating singularitiescr_master_1 ... done​
  3. ​Creating singularitiescr_worker_1 ... done​

查看容器运行 docker ps

CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                                                                                                    NAMES
19070f1091cb singularities/spark "start-spark worker …" 4 minutes ago Up 4 minutes 6066/tcp, 7077/tcp, 8020/tcp, 8080-8081/tcp, 9000/tcp, 10020/tcp, 13562/tcp, 14000/tcp, 19888/tcp, 50010/tcp, 50020/tcp, 50070/tcp, 50075/tcp, 50090/tcp, 50470/tcp, 50475/tcp singularitiescr_worker_1
bbe3797cf9ab singularities/spark "start-spark master" 4 minutes ago Up 4 minutes 0.0.0.0:6066->6066/tcp, 7077/tcp, 0.0.0.0:7070->7070/tcp, 8020/tcp, 8081/tcp, 9000/tcp, 10020/tcp, 13562/tcp, 14000/tcp, 19888/tcp, 50010/tcp, 0.0.0.0:8080->8080/tcp, 50020/tcp, 50075/tcp, 50090/tcp, 50470/tcp, 0.0.0.0:50070->50070/tcp, 50475/tcp singularitiescr_master_1

停止容器

[root@localhost singularitiesCR]# docker-compose stop

删除容器

[root@localhost singularitiesCR]# docker-compose rm

 

登录网页

Docker搭建Spark_docker

Docker搭建Spark_docker_02

举报

相关推荐

0 条评论