10.2 `NameNode`目录结构

运行中的NameNode会有相应的目录文件被创建.

默认情况这些文件和目录存储在: file://${hadoop.tmp.dir}/dfs/name

注意:

${hadoop.tmp.dir}是我们在core-site.xml中进行的配置(/opt/module/hadoop-2.7.2/data/tmp)
也可以在hdfs-site.xml中配置这个目录. 使用属性: dfs.namenode.name.dir

目录结构如下:

/opt/module/hadoop-2.7.2/data/tmp/dfs/name
├── current
│   ├── edits_0000000000000000001-0000000000000000035
│   ├── edits_0000000000000000036-0000000000000000037
│   ├── edits_0000000000000000038-0000000000000000050
│   ├── edits_0000000000000000051-0000000000000000079
│   ├── edits_0000000000000000080-0000000000000000105
│   ├── edits_0000000000000000106-0000000000000000107
│   ├── edits_0000000000000000108-0000000000000000116
│   ├── edits_0000000000000000117-0000000000000000118
│   ├── edits_0000000000000000119-0000000000000000120
│   ├── edits_0000000000000000121-0000000000000000123
│   ├── edits_0000000000000000124-0000000000000000125
│   ├── edits_0000000000000000126-0000000000000000127
│   ├── edits_0000000000000000128-0000000000000000129
│   ├── edits_0000000000000000130-0000000000000000131
│   ├── edits_0000000000000000132-0000000000000000132
│   ├── edits_0000000000000000133-0000000000000000134
│   ├── edits_0000000000000000135-0000000000000000135
│   ├── edits_0000000000000000136-0000000000000000137
│   ├── edits_0000000000000000138-0000000000000000139
│   ├── edits_0000000000000000140-0000000000000000141
│   ├── edits_0000000000000000142-0000000000000000143
│   ├── edits_inprogress_0000000000000000144
│   ├── fsimage_0000000000000000141
│   ├── fsimage_0000000000000000141.md5
│   ├── fsimage_0000000000000000143
│   ├── fsimage_0000000000000000143.md5
│   ├── seen_txid
│   └── VERSION
└── in_use.lock

说明:

in_use.lock文件是一个锁文件, NameNode使用该文件为存储目录加锁. 可以避免其他NameNode实例同时使用同一个存储目录的情况. 保证NameNode的唯一性.
curren目录, 存储当前的NameNode的元数据的目录.
1. VERSION文件, 是一个Java属性文件, 其中包含HDFS的版本信息.该文件一般包含如下内容:
```
#Fri Jul 13 09:08:28 CST 2018
namespaceID=947876838
clusterID=CID-7d3cf35b-0a97-484e-9b0f-5bcabec54352
cTime=0
storageType=NAME_NODE
blockpoolID=BP-241925315-192.168.1.201-1531110926280
layoutVersion=-63
```
  解释:
  - namespaceID: 文件系统命名空间的唯一标识.是在NameNode首次格式化的时候创建的. 这个标识符也要所有的DataNode要和NameNode保持一致
  - clusterID: 是HDFS集群作为一个整体赋予的唯一标识符.
  - cTime: NameNode存储系统的创建时间. 对于刚刚格式化的存储系统,这个属性值为 0. 如果文件系统更新了,则该值为更新到最新的时间戳.
  - storageType: 该存储目录包含的是NameNode的数据结构.
  - blockpoolID: 是数据块池的唯一标识, 数据块池中包含了由一个NameNode管理的命名空间的所有文件.
  - layoutVersion: 是一个负整数. 描述HDFS持久性持久性数据结构(也叫布局)的版本. 此版本号与Hadoop发布包的版本号无关.
2. 以edits_前缀开头的文件, 就是前面所说的"编辑日志". 后缀的一堆长串数字就是事务ID的范围.
3. edits_inprogress_....是唯一的一个处于打开可写状态. 任何时候只能有一个这样的文件.
4. fsimage_.....是命名空间的镜像文件. .md5文件中存储着相应的镜像文件的md5校验码, 用来验证fsimgage...是否有被篡改过.

查看`edits`和`fsimage`文件的内容

edits和fsimage都是二进制文件, 理论上我们是没有办法直接查看文件内容的.

但是Hadoop给我们提供了工具可以把二进制文件转换成文本文件, 然后再查看.

# 转换fsimage文件
hdfs oiv -p XML -i fsimage_0000000000000000153 -o /opt/module/fsimage.xml
# 转换edits文件
hdfs oev -p XML -i edits_0000000000000000001-0000000000000000035 -o /opt/module/edits.xml

`fsimage`内容

<?xml version="1.0" ?>
<fsimage>
    <NameSection>
        <genstampV1>1000</genstampV1>
        <genstampV2>1018</genstampV2>
        <genstampV1Limit>0</genstampV1Limit>
        <lastAllocatedBlockId>1073741839</lastAllocatedBlockId>
        <txid>153</txid>
    </NameSection>
    <INodeSection>
        <lastInodeId>16408</lastInodeId>
        <inode>
            <id>16385</id>
            <type>DIRECTORY</type>
            <name></name>
            <mtime>1531467183668</mtime>
            <permission>atguigu:supergroup:rwxr-xr-x</permission>
            <nsquota>9223372036854775807</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16386</id>
            <type>DIRECTORY</type>
            <name>a</name>
            <mtime>1531111209477</mtime>
            <permission>atguigu:supergroup:rwxr-xr-x</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16389</id>
            <type>FILE</type>
            <name>words.txt</name>
            <replication>3</replication>
            <mtime>1531111377413</mtime>
            <atime>1531111209296</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741827</id>
                    <genstamp>1006</genstamp>
                    <numBytes>118</numBytes>
                </block>
            </blocks>
        </inode>
        <inode>
            <id>16397</id>
            <type>DIRECTORY</type>
            <name>user</name>
            <mtime>1531124467865</mtime>
            <permission>atguigu:supergroup:rwxr-xr-x</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16398</id>
            <type>DIRECTORY</type>
            <name>atguigu</name>
            <mtime>1531124467865</mtime>
            <permission>atguigu:supergroup:rwxr-xr-x</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16399</id>
            <type>DIRECTORY</type>
            <name>java_test</name>
            <mtime>1531124467865</mtime>
            <permission>atguigu:supergroup:rwxr-xr-x</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16401</id>
            <type>DIRECTORY</type>
            <name>java_test_1</name>
            <mtime>1531124558732</mtime>
            <permission>atguigu:supergroup:rwxr-xr-x</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16402</id>
            <type>FILE</type>
            <name>bb.txt</name>
            <replication>3</replication>
            <mtime>1531124558973</mtime>
            <atime>1531124558732</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741834</id>
                    <genstamp>1013</genstamp>
                    <numBytes>14</numBytes>
                </block>
            </blocks>
        </inode>
        <inode>
            <id>16403</id>
            <type>FILE</type>
            <name>words.txt</name>
            <replication>3</replication>
            <mtime>1531125284204</mtime>
            <atime>1531125284042</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741835</id>
                    <genstamp>1014</genstamp>
                    <numBytes>78</numBytes>
                </block>
            </blocks>
        </inode>
        <inode>
            <id>16404</id>
            <type>FILE</type>
            <name>words2.txt</name>
            <replication>3</replication>
            <mtime>1531125289711</mtime>
            <atime>1531125289558</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741836</id>
                    <genstamp>1015</genstamp>
                    <numBytes>13</numBytes>
                </block>
            </blocks>
        </inode>
        <inode>
            <id>16405</id>
            <type>FILE</type>
            <name>words3.txt</name>
            <replication>3</replication>
            <mtime>1531125294811</mtime>
            <atime>1531125294641</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741837</id>
                    <genstamp>1016</genstamp>
                    <numBytes>9</numBytes>
                </block>
            </blocks>
        </inode>
        <inode>
            <id>16407</id>
            <type>FILE</type>
            <name>copy_01.txt</name>
            <replication>3</replication>
            <mtime>1531359605445</mtime>
            <atime>1531371334344</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741838</id>
                    <genstamp>1017</genstamp>
                    <numBytes>14</numBytes>
                </block>
            </blocks>
        </inode>
        <inode>
            <id>16408</id>
            <type>FILE</type>
            <name>wwww.txt</name>
            <replication>3</replication>
            <mtime>1531467183659</mtime>
            <atime>1531467182788</atime>
            <perferredBlockSize>134217728</perferredBlockSize>
            <permission>atguigu:supergroup:rw-r--r--</permission>
            <blocks>
                <block>
                    <id>1073741839</id>
                    <genstamp>1018</genstamp>
                    <numBytes>13</numBytes>
                </block>
            </blocks>
        </inode>
    </INodeSection>
    <INodeReferenceSection></INodeReferenceSection>
    <SnapshotSection>
        <snapshotCounter>0</snapshotCounter>
    </SnapshotSection>
    <INodeDirectorySection>
        <directory>
            <parent>16385</parent>
            <inode>16386</inode>
            <inode>16407</inode>
            <inode>16401</inode>
            <inode>16397</inode>
            <inode>16403</inode>
            <inode>16404</inode>
            <inode>16405</inode>
            <inode>16408</inode>
        </directory>
        <directory>
            <parent>16386</parent>
            <inode>16389</inode>
        </directory>
        <directory>
            <parent>16397</parent>
            <inode>16398</inode>
        </directory>
        <directory>
            <parent>16398</parent>
            <inode>16399</inode>
        </directory>
        <directory>
            <parent>16401</parent>
            <inode>16402</inode>
        </directory>
    </INodeDirectorySection>
    <FileUnderConstructionSection></FileUnderConstructionSection>
    <SnapshotDiffSection>
        <diff>
            <inodeid>16385</inodeid>
        </diff>
    </SnapshotDiffSection>
    <SecretManagerSection>
        <currentId>0</currentId>
        <tokenSequenceNumber>0</tokenSequenceNumber>
    </SecretManagerSection>
    <CacheManagerSection>
        <nextDirectiveId>1</nextDirectiveId>
    </CacheManagerSection>
</fsimage>

`edits`内容

<?xml version="1.0" encoding="UTF-8" ?>
<EDITS>
    <EDITS_VERSION>-63</EDITS_VERSION>
    <RECORD>
        <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
        <DATA>
            <TXID>146</TXID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ADD</OPCODE>
        <DATA>
            <TXID>147</TXID>
            <LENGTH>0</LENGTH>
            <INODEID>16408</INODEID>
            <PATH>/wwww.txt._COPYING_</PATH>
            <REPLICATION>3</REPLICATION>
            <MTIME>1531467182788</MTIME>
            <ATIME>1531467182788</ATIME>
            <BLOCKSIZE>134217728</BLOCKSIZE>
            <CLIENT_NAME>DFSClient_NONMAPREDUCE_1250002793_1</CLIENT_NAME>
            <CLIENT_MACHINE>192.168.1.201</CLIENT_MACHINE>
            <OVERWRITE>true</OVERWRITE>
            <PERMISSION_STATUS>
                <USERNAME>atguigu</USERNAME>
                <GROUPNAME>supergroup</GROUPNAME>
                <MODE>420</MODE>
            </PERMISSION_STATUS>
            <RPC_CLIENTID>645e8d7a-9d15-4e55-8d0c-1eba7a1e32ab</RPC_CLIENTID>
            <RPC_CALLID>3</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE>
        <DATA>
            <TXID>148</TXID>
            <BLOCK_ID>1073741839</BLOCK_ID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_SET_GENSTAMP_V2</OPCODE>
        <DATA>
            <TXID>149</TXID>
            <GENSTAMPV2>1018</GENSTAMPV2>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_ADD_BLOCK</OPCODE>
        <DATA>
            <TXID>150</TXID>
            <PATH>/wwww.txt._COPYING_</PATH>
            <BLOCK>
                <BLOCK_ID>1073741839</BLOCK_ID>
                <NUM_BYTES>0</NUM_BYTES>
                <GENSTAMP>1018</GENSTAMP>
            </BLOCK>
            <RPC_CLIENTID></RPC_CLIENTID>
            <RPC_CALLID>-2</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_CLOSE</OPCODE>
        <DATA>
            <TXID>151</TXID>
            <LENGTH>0</LENGTH>
            <INODEID>0</INODEID>
            <PATH>/wwww.txt._COPYING_</PATH>
            <REPLICATION>3</REPLICATION>
            <MTIME>1531467183659</MTIME>
            <ATIME>1531467182788</ATIME>
            <BLOCKSIZE>134217728</BLOCKSIZE>
            <CLIENT_NAME></CLIENT_NAME>
            <CLIENT_MACHINE></CLIENT_MACHINE>
            <OVERWRITE>false</OVERWRITE>
            <BLOCK>
                <BLOCK_ID>1073741839</BLOCK_ID>
                <NUM_BYTES>13</NUM_BYTES>
                <GENSTAMP>1018</GENSTAMP>
            </BLOCK>
            <PERMISSION_STATUS>
                <USERNAME>atguigu</USERNAME>
                <GROUPNAME>supergroup</GROUPNAME>
                <MODE>420</MODE>
            </PERMISSION_STATUS>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_RENAME_OLD</OPCODE>
        <DATA>
            <TXID>152</TXID>
            <LENGTH>0</LENGTH>
            <SRC>/wwww.txt._COPYING_</SRC>
            <DST>/wwww.txt</DST>
            <TIMESTAMP>1531467183668</TIMESTAMP>
            <RPC_CLIENTID>645e8d7a-9d15-4e55-8d0c-1eba7a1e32ab</RPC_CLIENTID>
            <RPC_CALLID>9</RPC_CALLID>
        </DATA>
    </RECORD>
    <RECORD>
        <OPCODE>OP_END_LOG_SEGMENT</OPCODE>
        <DATA>
            <TXID>153</TXID>
        </DATA>
    </RECORD>
</EDITS>

10.2 NameNode的文件目录结构

10.2 `NameNode`目录结构

注意:

说明:

查看`edits`和`fsimage`文件的内容

`fsimage`内容

`edits`内容

results matching ""

No results matching ""

10.2 NameNode目录结构

注意:

说明:

查看edits和fsimage文件的内容

fsimage内容

edits内容

results matching ""

No results matching ""

10.2 `NameNode`目录结构

查看`edits`和`fsimage`文件的内容

`fsimage`内容

`edits`内容