大数据开发|spark环境构建

本是古典何须时尚 2023-06-16 03:57 23阅读 0赞

提前说明一下，大数据的搭建环境都是在Linux系统下构建，可能针对一些没有Linux编程基础的同学来说会有一些吃力，请各位客官放心，小店伙计后期会专门有几期来讲解Linux编程基础。绝对保证零基础完成大数据环境的构建。今天大数据环境构建后会暂停其他组件（hue、flume、kafka、oozie等）的构建，后面的文章就是基于该环境讲解大数据的应用。

一

安装zookeeper

参考：大数据开发|Hadoop分布式集群环境构建（1）

二

安装spark

![format_png][]

2.2

安装scala

spark的底层是基于scala语言编写，对于spark的计算程序也可以通过scala语言来编写。

并解压scala：

tar  -zxvf  scala-2.10.4.tar.gz

![format_png 1][]

2.3

spark安装配置

下载spark安装包

并解压：tar  -zxvf  spark-1.5.0-cdh5.5.1.tar.gz

【大数据开发学习资料领取方式】：加入大数据技术学习交流扣裙：数字522数字189数字307，私信管理员即可免费领取开发工具以及入门学习资料

![format_png 2][]

![format_png 3][]

**配置spark-env.sh**

![format_png 4][]

内容如下：

export SPARK\_CLASSPATH=/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/\*:/home/hadoop/hadoop/share/hadoop/common/\*:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/\*:/home/hadoop/hadoop/share/hadoop/hdfs/\*:/home/hadoop/hadoop/share/hadoop/yarn/lib/\*:/home/hadoop/hadoop/share/hadoop/yarn/\*:/home/hadoop/hadoop/share/hadoop/tools/lib/\*:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/\*:/home/hadoop/hadoop/share/hadoop/mapreduce/\*:/contrib/capacity-scheduler/\*.jar

export SPARK\_DAEMON\_JAVA\_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop3:2181,hadoop4:2181,hadoop5:2181 -Dspark.deploy.zookeeper.dir=/spark"

export SCALA\_HOME=/home/hadoop/scala

export JAVA\_HOME=/home/hadoop/jdk

export HADOOP\_HOME=/home/hadoop/hadoop

![format_png 5][]

**配置slaves**

![format_png 6][]

2.4

启动集群

./start-all.sh

![format_png 7][]

结果

![format_png 8][]

![format_png 9][]

因为采用的是HA模式

就需要在备用机子上启动master

![format_png 10][]

![format_png 11][]

至此spark HA集群就搭建成功了

三

测试

./spark-shell

![format_png 12][]

例子：

统计文字的长度

代码：

val lines = sc.textFile("hdfs://mycluster/user/hive/warehouse/lijietest/hive\_test.txt")

val lineLengths = lines.map(s => s.length)

val totalLength = lineLengths.reduce((a, b) => a + b)

![format_png 13][]

运行结果：

![format_png 14][]

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS1iM2U0MzNjNDJhODU5OWJj?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS0yMDIxODZjOGYzMjE1NmI0?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS1iNDRkODI3ZWU1NjdlYjBk?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS05OWNjYzJiMDhkMzA4MDFl?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS1kNDk5ODg1ZDhhNDNhZjBm?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS04ODhiY2Y5NzE5NGU1ZWRj?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS03YmU5Y2UzMzkyNTY5OWM1?x-oss-process=image/format,png
[format_png 7]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS0yNzk5OTY5MDA5Nzc2Mzdm?x-oss-process=image/format,png
[format_png 8]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS0wOTU3N2MyYmFkYWI4NjNj?x-oss-process=image/format,png
[format_png 9]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS03NmIzZDJjZmNkZmFlMzQ4?x-oss-process=image/format,png
[format_png 10]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS03OWFhYjllMWQwNzRhOWQz?x-oss-process=image/format,png
[format_png 11]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS0yMWY3MDYyNGFhMmQyMmM4?x-oss-process=image/format,png
[format_png 12]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS0wNDU2MzEyMTFjNzFkNTBj?x-oss-process=image/format,png
[format_png 13]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS02NDQyNjEyYzk5NjIxMzE2?x-oss-process=image/format,png
[format_png 14]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNDI5MTU0OS0zNzUzNGM3YWRhNjE1Nzc0?x-oss-process=image/format,png