Hadoop 0.23.9 Wie zu Beginn datanodes

Wie es scheint, ich kann nicht hadoop zu starten, richtig. Ich bin mit hadoop 0.23.9:

[msknapp@localhost sbin]$ hadoop namenode -format
...
[msknapp@localhost sbin]$ ./start-dfs.sh 
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/cloud/hadoop-0.23.9/logs/hadoop-msknapp-namenode-localhost.localdomain.out
localhost: starting datanode, logging to /usr/local/cloud/hadoop-0.23.9/logs/hadoop-msknapp-datanode-localhost.localdomain.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/cloud/hadoop-0.23.9/logs/hadoop-msknapp-secondarynamenode-localhost.localdomain.out
[msknapp@localhost sbin]$ ./start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /usr/local/cloud/hadoop-0.23.9/logs/yarn-msknapp-resourcemanager-localhost.localdomain.out
localhost: starting nodemanager, logging to /usr/local/cloud/hadoop-0.23.9/logs/yarn-msknapp-nodemanager-localhost.localdomain.out
[msknapp@localhost sbin]$ cd /var/local/stock/data
[msknapp@localhost data]$ hadoop fs -ls /
[msknapp@localhost data]$ hadoop fs -mkdir /stock
[msknapp@localhost data]$ ls
companies.csv  raw  slf_series.txt
[msknapp@localhost data]$ hadoop fs -put companies.csv /stock/companies.csv 
13/12/08 11:10:40 WARN hdfs.DFSClient: DataStreamer Exception
java.io.IOException: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1180)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1536)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:414)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:394)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1571)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1567)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1262)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1565)

    at org.apache.hadoop.ipc.Client.call(Client.java:1094)
    at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1130)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1006)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:458)
put: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
13/12/08 11:10:40 ERROR hdfs.DFSClient: Failed to close file /stock/companies.csv._COPYING_
java.io.IOException: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1180)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1536)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:414)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:394)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1571)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1567)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1262)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1565)

    at org.apache.hadoop.ipc.Client.call(Client.java:1094)
    at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1130)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1006)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:458)

Hier ist meine core-site.xml:

<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost/</value>
</property>

und meine hdfs-site.xml:

<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>

und mapred-site.xml:

    <property>
            <name>mapred.job.tracker</name>
            <value>localhost:8021</value>
    </property>

Ich schaute durch all die Unterlagen, die ich habe, ich kann nicht herausfinden, wie start hadoop richtig. Ich finde keine Dokumentation über online-hadoop-0.23.9. Meine Hadoop-Buch ist für 0.22. Die online-Dokumentation ist für 2.1.1, die zufälligerweise konnte ich nicht bekommen, um zu arbeiten.

Kann jemand bitte sagen Sie mir, wie man meine hadoop gestartet, richtig?

Sie können prüfen wollen, die datanode-log-Datei für detaillierte Informationen zu dem Fehler

InformationsquelleAutor msknapp | 2013-12-08

Geben Sie einen port für fs.default.name

wie:

<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:54310</value>
</property>

Danach erstellen Sie ein tmp-Verzeichnis für hdfs:

sudo mkdir -p /app/hadoop/tmp
sudo chown you /app/hadoop/tmp

und hinzufügen core-site.xml:

<property>
   <name>hadoop.tmp.dir</name>
   <value>/app/hadoop/tmp</value>
   <description>A base for other temporary directories.</description>
</property>

Stellen Sie sicher, um starten Sie den cluster.

$HADOOP_HOME/bin/stop-all.sh
$HADOOP_HOME/bin/start-all.sh

InformationsquelleAutor Artem Tsikiridis

1

Löschen Sie alle Daten, die mit hadoop gestoppt:
```
$HADOOP_HOME/bin/hadoop datanode -format
```
oder löschen Sie manuell den Inhalt

/app/hadoop/tmp/dfs/data/

und dann starten hadoop wieder:
```
$HADOOP_HOME/bin/start-all.sh
```
InformationsquelleAutor vefthym

Das zentrale problem in Ihrer Konfiguration ist wie folgt:

java.io.IOException: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

Stellen Sie sicher, dass Ihre HDFS-spezifische Konfiguration hat mindestens mindestens wie folgt:

hdfs-site.xml:
So zeigt in der xml müssen Sie haben /tmp/hdfs23/namenode und /tmp/hdfs23/datanode Ordner bereits existierte. Konfigurieren können Sie einen anderen Ordner für die hdfs-root und dann namenode und datanode Ordner insider es.

<configuration>
        <property>
           <name>dfs.replication</name>
           <value>1</value>
        </property>
        <property>
           <name>dfs.namenode.name.dir</name>
           <value>file:///tmp/hdfs23/namenode</value>
        </property>
        <property>
            <name>fs.checkpoint.dir</name>
            <value>file:///tmp/hdfs23/secnamenode</value>
        </property>
        <property>
             <name>fs.checkpoint.edits.dir</name>
             <value>file:///tmp/hdfs23/secnamenode</value>
        </property>
        <property>
           <name>dfs.datanode.data.dir</name>
           <value>file:///tmp/hdfs23/datanode</value>
        </property>
</configuration>

core-site.xml

<configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
     <property>
         <name>fs.default.name</name>
         <value>hdfs://localhost:9000</value>
     </property>
     <property>
        <name>hadoop.http.staticuser.user</name>
        <value>hdfs</value>
     </property>
</configuration>

Dann müssen Sie formatieren Sie Ihre namenode als Sie schon getan hat:

$ hadoop namenode -format

Danach können Sie beginnen, HDFS als unten:

[Hadoop023_ROOT]/sbin/start-dfs.sh

Wenn es immer noch nicht das problem lösen, versuchen Sie, entfernen Sie alle Dateien und Ordner, die in "real" namenode Ordner und datanode Ordner. Zum Beispiel, entfernen Sie alle Dinge, die in file:///tmp/hdfs23/namenode vor format namenode.

InformationsquelleAutor AvkashChauhan

Schreibe einen Kommentar

Du musst angemeldet sein, um einen Kommentar abzugeben.