首页 > 编程札记 > 编程

使用虚拟机(MobaXterm)从零开始搭建Spark1.3集群

阅读： 评论：0

使用虚拟机(MobaXterm)从零开始搭建Spark1.3集群

使用虚拟机(MobaXterm)从零开始搭建Spark1.3集群

准备工作

1. Hadoop：hadoop-2.4.

2. Spark：下载编译好的基于对应hadoop版本的版本：spark-1.3.

3.JAVA：

4. scala：scala-2.5

 

搭建环境

在MobaXterm界面连接服务器，使用命令sudo virt-manager打开virtual machine manager窗口，使用ubuntu16.04镜像创建一个新的虚拟机，进行如下操作

  • 设置静态ip地址      
    sudo vim /etc/network/interfaces## 更改ip地址，将#iface ens3 inet dhcp改为如下内容
# The primary network interface
auto ens3
iface ens3 inet static
address 192.168.122.54
netmask 255.255.255.0
gateway 192.168.122.1## 重启网络
/etc/init.d/networking restart## 注意：clone机器后仍然需要修改，我设置的静态ip如下
master 192.168.122.54
slave1 192.168.122.55
slave2 192.168.122.56
slave3 192.168.122.57
slave4 192.168.122.58

     

  • 配置hosts文件 
    ## 修改主机名
sudo vim /etc/hostname 
## 改为master，clone之后其他再统一修改## 修改hosts文件，只保留localhost，剩余内容进行追加(否则会出问题，后面会提到！)
sudo vim/etc/hosts192.168.122.54 master
192.168.122.55 slave1
192.168.122.56 slave2 
192.168.122.57 slave3 
192.168.122.58 slave4

     

  • 搭建Java环境 
    ## 解压
sudo tar -zxvf  -C ./software/
sudo mv jdkxxxx jdk## 配置环境变量
sudo vim /etc/profileexport JAVA_HOME=/home/zmx/software/jdk
export JRE_HOME=/home/zmx/software/jdk/jre
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

     

  • 搭建scala环境 
    ## 解压
sudo tar -zxvf scala-2. -C software/
sudo mv scala-2.11.8/ scala## 追加环境变量
sudo vim /etc/profile## 最终效果如图
export JAVA_HOME=/home/zmx/softwarek
export JRE_HOME=/home/zmx/softwarek/jre
export SCALA_HOME=/home/zmx/software/scala
export PATH=$SCALA_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOMEb:$JAVA_HOME/jreb

     

  • 关闭防火墙 
    sudo ufw disable
Firewall stopped and disabled on system startup

     

  • clone虚拟机：在MobaXterm界面输入sudo virt-manager打开vm manager后进行clone

       

  • 搭建集群，测试机器之间能否通信 
    ## clone机器后记得进行ip地址和host的更改,这样我们一共有五台机器：master、slave1-slave4
## master上进行如下操作，其他机器同理
ping slave1
ping slave2
ping slave3
ping slave4

     

  • 配置master-slave ssh 免密登陆 
    ## 在每台机器上生成私钥和公钥
ssh-keygen -t rsa## 将slave上的id_rsa.pub用scp命令发给master
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave1
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave2
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave3
scp ./.ssh/id_rsa.pub zmx@master:~/.ssh/id_rsa.pub.slave4## 在master上，将所有公钥加到用于认证的公钥文件authorized_keys中
zmx@master:~$ cat .ssh/id_rsa.pub* >> ~/.ssh/authorized_keys## 将公钥文件分发给slaves
scp .ssh/authorized_keys zmx@slave1:~/.ssh/
scp .ssh/authorized_keys zmx@slave2:~/.ssh/
scp .ssh/authorized_keys zmx@slave3:~/.ssh/
scp .ssh/authorized_keys zmx@slave4:~/.ssh/## 最后在每台主机上，用SSH命令，检验下是否能免密码登录
ssh slave1
ssh slave2
ssh slave3
ssh slave4

安装Hadoop

  配置环境

  • 配置文件
    • hadoop-env.sh 
      ## 末尾增加
export HADOOP_IDENT_STRING=$USER
export JAVA_HOME=/home/zmx/software/jdk
export HADOOP_PREFIX=/home/zmx/software/hadoop-2.4.1

       

    • yarn-evn.sh 
      ## 末尾增加
export JAVA_HOME=/home/zmx/software/jdk

       

    • slaves: 加入master表示将master也视为slave 
      master
slave1
slave2
slave3
slave4

       

      • <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><property><name>fs.defaultFS</name><value>hdfs://master:9000</value></property><property><name&p.dir</name><value>/home/zmx/software/hadoop-2.4.1/tmp</value></property> </configuration>

       

      <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><property><name>dfs.datanode.ipc.address</name><value>0.0.0.0:50020</value></property><property><name>dfs.datanode.http.address</name><value>0.0.0.0:50075</value></property><property><name&plication</name><value>2</value></property> </configuration>

       

      <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property> </configuration>

       

      <?xml version="1.0"?> <!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file. --> <configuration><!-- Site specific YARN configuration properties --><property><name&demanager.aux-services</name><value>mapreduce_shuffle</value></property><property> <name&sourcemanager.address</name> <value>master:8032</value> </property> <property><name&sourcemanager.scheduler.address</name> <value>master:8030</value> </property><property><name&source-tracker.address</name> <value>master:8031</value> </property> </configuration>

       

  • 分发文件夹给slave: 
    sudo chmod -R 777 ~/software/hadoop-2.4.1
scp -r ~/software/hadoop-2.4.1 zmx@slave1:~/software/
scp -r ~/software/hadoop-2.4.1 zmx@slave2:~/software/
scp -r ~/software/hadoop-2.4.1 zmx@slave3:~/software/
scp -r ~/software/hadoop-2.4.1 zmx@slave4:~/software/

     

  • 启动hadoop 
    ## 进入hadoop目录，格式化hdfs
bin/hdfs namenode -format## 启动hdfs
sbin/start-dfs.sh## 启动yarn
sbin/start-yarn.sh

     

  • 用jps命令查看hadoop进程 
    ## master上
ResourceManager
SecondaryNameNode
NameNode
DataNode(因为把master看成自己的slave所以存在该进程，如果不想这样，将hadoop的配置文件slaves中的master去掉即可)
NodeManager(因为把master看成自己的slave所以存在该进程)## slave上
DataNode 
NodeManager

     

  • 输入yarn node -list查看节点信息 
    zmx@master:~/software/hadoop-2.4.1$ yarn node -list
19/03/18 10:01:32 INFO client.RMProxy: Connecting to ResourceManager at /192.168.122.54:8032
19/03/18 10:01:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for  using builtin-java classes where applicable
Total Nodes:5Node-Id             Node-State Node-Http-Address       Number-of-Running-Containersslave4:36952                RUNNING       slave4:8042                                  0slave2:39254                RUNNING       slave2:8042                                  0master:38718                RUNNING       master:8042                                  0slave1:42168                RUNNING       slave1:8042                                  0slave3:43401                RUNNING       slave3:8042

     

 

安装Spark

  • 更改配置文件
    • spark-env.sh：配置几个基本的，可以根据自己机器的实际情况再进行配置 
      ​## 添加代码到末尾
export SCALA_HOME=/home/zmx/software/scala
export JAVA_HOME=/home/zmx/software/jdk
export HADOOP_HOME=/home/zmx/software/hadoop-2.4.1
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SPARK_MASTER_IP=master
SPARK_LOCAL_DIRS=/home/zmx/software/spark-1.3.0-bin-hadoop2.4
SPARK_DRIVER_MEMORY=512M

       

    • slaves：加上master表示将master也视为worker，可以不加

      master
slave1
slave2
slave3
slave4

       

  • 分发给slave 
    scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave1:~/software/
scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave2:~/software/
scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave3:~/software/
scp -r ~/software/spark-1.3.0-bin-hadoop2.4 zmx@slave4:~/software/

     

  • 启动Spark 
    sbin/start-all.sh##启动成功运行jps可以看到
master节点进程
Masterworker节点进程
Worker

     

使用yarn-cluster模式测试Hadoop和Spark是否安装成功

   参考链接：通过跑较大数据集测试Hadoop 2.4.1是否安装成功

zmx@master:~/software/spark-1.3.0-bin-hadoop2.4$ ./bin/spark-submit 
> --class org.amples.JavaWordCount 
> --master yarn-cluster 
> lib/spark-examples*.jar 
> 
Spark assembly has been built with Hive, including Datanucleus jars on classpath
19/03/18 21:04:05 WARN NativeCodeLoader: Unable to load native-hadoop library for  using builtin-java classes where applicable
19/03/18 21:04:06 INFO RMProxy: Connecting to ResourceManager at /192.168.122.54:8032
19/03/18 21:04:06 INFO Client: Requesting a new application from cluster with 5 NodeManagers
19/03/18 21:04:06 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/03/18 21:04:06 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
19/03/18 21:04:06 INFO Client: Setting up container launch context for our AM
19/03/18 21:04:06 INFO Client: Preparing resources for our AM container
19/03/18 21:04:07 INFO Client: Uploading resource file:/home/zmx/software/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar -> hdfs://master:9000/user/zmx/.sparkStaging/application_1552885048834_0014/spark-assembly-1.3.0-hadoop2.4.0.jar
19/03/18 21:04:10 INFO Client: Uploading resource file:/home/zmx/software/spark-1.3.0-bin-hadoop2.4/lib/spark-examples-1.3.0-hadoop2.4.0.jar -> hdfs://master:9000/user/zmx/.sparkStaging/application_1552885048834_0014/spark-examples-1.3.0-hadoop2.4.0.jar
19/03/18 21:04:12 INFO Client: Setting up the launch environment for our AM container
19/03/18 21:04:12 INFO SecurityManager: Changing view acls to: zmx
19/03/18 21:04:12 INFO SecurityManager: Changing modify acls to: zmx
19/03/18 21:04:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zmx); users with modify permissions: Set(zmx)
19/03/18 21:04:12 INFO Client: Submitting application 14 to ResourceManager
19/03/18 21:04:12 INFO YarnClientImpl: Submitted application application_1552885048834_0014
19/03/18 21:04:13 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:13 INFO Client:client token: N/Adiagnostics: N/AApplicationMaster host: N/AApplicationMaster RPC port: -1queue: defaultstart time: 1552914252846final status: UNDEFINEDtracking URL: master:8088/proxy/application_1552885048834_0014/user: zmx
19/03/18 21:04:14 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:15 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:16 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:17 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:18 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:19 INFO Client: Application report for application_1552885048834_0014 (state: ACCEPTED)
19/03/18 21:04:20 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:20 INFO Client:client token: N/Adiagnostics: N/AApplicationMaster host: slave4ApplicationMaster RPC port: 0queue: defaultstart time: 1552914252846final status: UNDEFINEDtracking URL: master:8088/proxy/application_1552885048834_0014/user: zmx
19/03/18 21:04:21 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:22 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:23 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:24 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:25 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:26 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:27 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:28 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:29 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:30 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:31 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:32 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:33 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:34 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:35 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:37 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:38 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:39 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:40 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:41 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:42 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:43 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:44 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:45 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:46 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:47 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:48 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:49 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:50 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:51 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:52 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:53 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:54 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:55 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:56 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:57 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:58 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:04:59 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:00 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:01 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:02 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:03 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:04 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:05 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:06 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:07 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:08 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:09 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:10 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:11 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:12 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:13 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:14 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:15 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:16 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:17 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:18 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:19 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:20 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:21 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:22 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:23 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:24 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:25 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:26 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:27 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:28 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:29 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:30 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:31 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:32 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:33 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:34 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:35 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:36 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:37 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:38 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:39 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:40 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:41 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:42 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:43 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:44 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:45 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:46 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:47 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:48 INFO Client: Application report for application_1552885048834_0014 (state: RUNNING)
19/03/18 21:05:49 INFO Client: Application report for application_1552885048834_0014 (state: FINISHED)
19/03/18 21:05:49 INFO Client:client token: N/Adiagnostics: N/AApplicationMaster host: slave4ApplicationMaster RPC port: 0queue: defaultstart time: 1552914252846final status: SUCCEEDEDtracking URL: master:8088/proxy/application_1552885048834_0014/Auser: zmx

遇到的问题及解决方案

1. Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS 

错误原因&#l配置错误，修改所有节点上l配置文件，在该文件中配置ResourceManager Master节点所在地址即可解决问题

<property>  <name&sourcemanager.address</name>  <value>master:8032</value>  
</property> 
<property><name&sourcemanager.scheduler.address</name>  <value>master:8030</value>  
</property>
<property><name&source-tracker.address</name>  <value>mster:8031</value>
</property>

2. hadoop启动异常，日志如下

java.lang.IllegalArgumentException: Does not contain a valid host:port authority:  master:8031 (configuration property &#source-tracker.address')at org.apache.ateSocketAddr(NetUtils.java:196)at org.SocketAddr(Configuration.java:1590)at org.apache.hadoop.sourcemanager.ResourceTrackerService.serviceInit(ResourceTrackerService.java:106)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)at org.apache.hadoop.sourcemanager.ResourceManager.serviceInit(ResourceManager.java:288)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.sourcemanager.ResourceManager.main(ResourceManager.java:871)
2014-03-20 21:00:20,545 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state INITED; cause: java.lang.IllegalArgumentException: Does not contain a valid host:port authority:  master:8031 (configuration property &#source-tracker.address')
java.lang.IllegalArgumentException: Does not contain a valid host:port authority:  master:8031 (configuration property &#source-tracker.address')at org.apache.ateSocketAddr(NetUtils.java:196)at org.SocketAddr(Configuration.java:1590)at org.apache.hadoop.sourcemanager.ResourceTrackerService.serviceInit(ResourceTrackerService.java:106)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)at org.apache.hadoop.sourcemanager.ResourceManager.serviceInit(ResourceManager.java:288)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.sourcemanager.ResourceManager.main(ResourceManager.java:871)

 

3. hadoop启动异常，INFO org.apache.hadoop.ipc.Client: Retrying connect to server: maste/192.168.122.54:8031. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

问题：/etc/hosts文件多了条记录

127.0.1.1 master

$ netstat -apn|grep 8031
tcp 0 0 127.0.1.1:8031 0.0.0.0:* LISTEN 4964/java

检查端口8031时发现NameNode进程只监听了127.0.1.1:8031端口，除了自己以外其他主机都不连接，所以导致出错，删除所有节点上hosts文件的该记录：127.0.1.1 xxx(xxx为自己的host)，重启namenode即可

其他端口出现同样问题，解决方法类似。

 

4. hadoop hdfs格式化遇到问题

FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000. Exiting. 
java.io.IOException: Incompatible clusterIDs in /home/lxh/hadoop/hdfs/data: namenode clusterID = CID-a3938a0b-57b5-458d-841c-d096e2b7a71c; datanode clusterID = CID-200e6206-98b5-44b2-9e48-262871884eebat org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:477)at org.apache.hadoop.hdfs.server.verTransitionRead(DataStorage.java:226)at org.apache.hadoop.hdfs.server.verTransitionRead(DataStorage.java:254)at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:974)at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:945)at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:278)at org.apache.hadoop.hdfs.server.tToNNAndHandshake(BPServiceActor.java:220)at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)at java.lang.Thread.run(Thread.java:745)

问题：命令 bin/hdfs namenode -format执行一次就行了，否则就会出现上述问题

解决方法：如果没有多次format namenode仍出现上述问题，参照链接 

如果之前多次执行了format命令，则在记录hadoop配置之后，删除所有节点的hadoop文件夹重新在master配置并分发 

 

5. hadoop启动时出现permission denied问题：sbin/start-dfs.sh

   原因：权限不够，执行chmod命令后再进行hadoop文件夹的分发

sudo chmod -R 777 ~/software/hadoop-2.4.1

  

参考链接

  1. 使用虚拟机从小白开始搭建Spark集群
  2. Hadoop-2.4.1完全分布式环境搭建
  3. hadoop 2.6全分布安装
  4. ubuntu关闭防火墙
  5. Hadoop集群从节点出现错误： Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s);
  6. hadoop下启动异常之一
  7. Hadoop 集群安装及配置实战
    8. <格式化遇到的问题
  8. 重新format namenode后，datanode无法正常启动
  9. Hadoop2.4.1中wordcount示例程序测试过程
  10. Spark On YARN 集群安装部署
  11. SparkDoc：Dynamic Resource Allocation
  12. Spark Dynamic Allocation 分析
  13. spark入门笔记

本文发布于:2024-02-02 17:22:58，感谢您对本站的认可！

本文链接：https://www.4u4v.net/it/170686577845304.html

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系，我们将在24小时内删除。

上一篇：多表操作、外键、视图、多表联合
下一篇：linux操作系统可以对目录进行硬链接,Linux操作系统期末复习题(含答案)
标签：集群   从零开始   虚拟机   MobaXterm
留言与评论（共有 0 条评论）
   
验证码：
推荐文章
排行榜
热门标签

Copyright ©2019-2022 Comsenz Inc.Powered by ©

网站地图1 网站地图2 网站地图3 网站地图4 网站地图5 网站地图6 网站地图7 网站地图8 网站地图9 网站地图10 网站地图11 网站地图12 网站地图13 网站地图14 网站地图15 网站地图16 网站地图17 网站地图18 网站地图19 网站地图20 网站地图21 网站地图22/a> 网站地图23