脚本宝典收集整理的这篇文章主要介绍了Spark配置文件，脚本宝典觉得挺不错的，现在分享给大家，也给大家做个参考。
1.上传spark-2.4.0-bin-hadoop2.6.tgz到/opt目录，并解压到/usr/localtar -zxf /opt/spark-2.4.0-bin-hadoop2.6.tgz -C /usr/local/
2.进入/usr/local/spark-2.4.0-bin-hadoop2.6/conf复制slaves.template：cp slaves.template slaves修改slaves，先删除其中的localhost，然后添加：slave1slave2slave3
3.修改spark-defaults.confcp spark-defaults.conf.template spark-defaults.confvi spark-defaults.conf添加：spark.master spark://master:7077spark.eventLOG.enabled truespark.eventLog.dir hDFs://master:8020/spark-logsspark.history.fs.logDirectory hdfs://master:8020/spark-logs
4.修改spark-env.shcp spark-env.sh.template spark-env.shvi spark-env.sh添加：JAVA_HOME=/usr/java/jdk1.8.0_151HADOOP_CONF_DIR=/usr/local/hadoop-2.6.5/etc/hadoopSPARK_MASTER_IP=masterSPARK_MASTER_PORT=7077SPARK_WORKER_MEMORY=512mSPARK_WORKER_CORES=1SPARK_EXECUTOR_MEMORY=512mSPARK_EXECUTOR_CORES=1SPARK_WORKER_INSTANCES=1
5.启动Hadoop集群，在HDFS中新建目录：hdfs dfs -mkdir /spark-logs
6.将Spark安装包分发到其他节点 scp -r /usr/local/spark-2.4.0-bin-hadoop2.6/ slave1:/usr/local/ scp -r /usr/local/spark-2.4.0-bin-hadoop2.6/ slave2:/usr/local/ scp -r /usr/local/spark-2.4.0-bin-hadoop2.6/ slave3:/usr/local/
7.在所有节点配置Spark环境变量vi /etc/PRofile在文件尾加入：export SPARK_HOME=/usr/local/spark-2.4.0-bin-hadoop2.6export PATH=$PATH:$SPARK_HOME/bin
执行source /etc/profile使命令生效
8.启动spark进入/usr/local/spark-2.4.0-bin-hadoop2.6/sbin执行./start-all.sh9.查看客户端http://master:8080
1.进入hive@R_126_2082@bin目录，修改hive文件vi hive将sparkAsSEMblyPath=`ls ${SPARK_HOME}/lib/spark-assembly-*.jar`修改为：sparkAssemblyPath=`ls ${SPARK_HOME}/jars/*.jar`
2.拷贝hive-sITe.XMl到/usr/local/spark-2.4.0-bin-hadoop2.6/confcp /usr/local/apache-hive-1.2.1-bin/conf/hive-site.xML /usr/local/spark-2.4.0-bin-hadoop2.6/conf/scp /usr/local/apache-hive-1.2.1-bin/conf/hive-site.xml slave1:/usr/local/spark-2.4.0-bin-hadoop2.6/conf/scp /usr/local/apache-hive-1.2.1-bin/conf/hive-site.xml slave2:/usr/local/spark-2.4.0-bin-hadoop2.6/conf/scp /usr/local/apache-hive-1.2.1-bin/conf/hive-site.xml slave3:/usr/local/spark-2.4.0-bin-hadoop2.6/conf/
3.拷贝MySQL驱动到/usr/local/spark-2.4.0-bin-hadoop2.6/jarscp /usr/local/apache-hive-1.2.1-bin/lib/mySQL-connector-java-5.1.32-bin.jar /usr/local/spark-2.4.0-bin-hadoop2.6/jars/scp /usr/local/spark-2.4.0-bin-hadoop2.6/jars/mysql-connector-java-5.1.32-bin.jar slave1:/usr/local/spark-2.4.0-bin-hadoop2.6/jars/scp /usr/local/spark-2.4.0-bin-hadoop2.6/jars/mysql-connector-java-5.1.32-bin.jar slave2:/usr/local/spark-2.4.0-bin-hadoop2.6/jars/scp /usr/local/spark-2.4.0-bin-hadoop2.6/jars/mysql-connector-java-5.1.32-bin.jar slave3:/usr/local/spark-2.4.0-bin-hadoop2.6/jars/
4.在所有节点/usr/local/spark-2.4.0-bin-hadoop2.6/conf/spark-env.sh 文件中配置 MySQL 驱动SPARK_CLASSPATH=/usr/local/spark-2.4.0-bin-hadoop2.6/jars/mysql-connector-java-5.1.32-bin.jar
5.启动 MySQL 服务service mysqld start
6.启动 Hive 的 metaStore 服务hive --service metastore &
7.修改日志级别，在各节点：cp /usr/local/spark-2.4.0-bin-hadoop2.6/conf/log4j.proPErties.template /usr/local/spark-2.4.0-bin-hadoop2.6/conf/log4j.properties修改log4j.propertieslog4j.rootCategory=WARN, console
8.启动spark集群9.访问spark-sql
以上是脚本宝典为你收集整理的Spark配置文件全部内容，希望文章能够帮你解决Spark配置文件所遇到的问题。
本图文内容来源于网友网络收集整理提供，作为学习参考使用，版权属于原作者。
如您有任何意见或建议可联系处理。小编QQ：384754419，请注明来意。