账号密码登录
微信安全登录
微信扫描二维码登录

登录后绑定QQ、微信即可实现信息互通

手机验证码登录
找回密码返回
邮箱找回 手机找回
注册账号返回
其他登录方式
分享
  • 收藏
    X
    HDFS中的secondaryNameNode如何合并元数据的?
    42
    0
    secondaryNameNode在合并edits和fsimage的时候,是如何合并的?难道这两个记录文件有合并的可能么?一个是记录操作日志的,一个是记录元数据的
    0
    打赏
    收藏
    点击回答
        全部回答
    • 0
    • 君子之交,太淡 普通会员 1楼

      在Hadoop分布式文件系统(HDFS)中,Secondary NameNode(NN)是HDFS的集群中的一个角色,负责处理元数据的合并和更新。以下是如何在Secondary NameNode中合并元数据的步骤:

      1. 获取主NameNode的元数据:首先,Secondary NameNode需要从主NameNode获取并处理元数据,这些元数据通常包含HDFS的结构信息、用户数据、配置信息以及存储数据的磁盘信息等。Secondary NameNode通过NamenodeManager接口(HDFS NameNode API)获取这些元数据,如:

      java NamenodeManager nm = NamenodeManager.getInstance(conf); HdfsAdmin admin = nm.getAdmin(); FsImage image = admin.getFsImage(imageId); FsImageStatus status = image.getStatus(); HdfsMeta meta = status.getMeta();

      1. 处理目录结构信息:Secondary NameNode需要处理HDFS目录结构信息,包括文件的路径、文件的大小、文件的创建时间、文件的权限等。这些信息通常存储在HDFS的HDFS Meta Node(HMN)中。Secondary NameNode通过HMN获取HDFS目录结构信息,并将其存储在自己的HDFS Meta Node中,以便后续合并元数据。

      ```java FsImageMeta meta = nm.getMeta(); HdfsMeta metaCopy = new HdfsMeta(); metaCopy.setPath(meta.getPath()); metaCopy.setSize(meta.getSize()); metaCopy.setLastModified(meta.getLastModified()); metaCopy.setPermissions(meta.getPermissions()); metaCopy.setFsImageId(imageId); metaCopy.setAccessTime(meta.getAccessTime()); metaCopy.setLocalFsImageId(imageId); metaCopy.setGcTime(null); metaCopy.setLruSize(null); metaCopy.setUsedBytes(null); metaCopy.setUsedFiles(null); metaCopy.setWriteTime(null); metaCopy.setTotalSpace(null); metaCopy.setTotalBytes(null); metaCopy.setNumBlocks(null); metaCopy.setFreeSpace(null); metaCopy.setTotalBlocks(null); metaCopy.setFreeSpace(null); metaCopy.setTotalLocations(null); metaCopy.setTotalLocationsSize(null); metaCopy.setNumLocations(null); metaCopy.setTotalLocationsSize(null); metaCopy.setTotalLocationsSize(null); metaCopy.setDataDir(null); metaCopy.setBackupDir(null); metaCopy.setLogDir(null); metaCopy.setVolumeInfo(null); metaCopy.setNumSnapshots(null); metaCopy.setSnapshots(null); metaCopy.setLastCheckpointTime(null); metaCopy.setSafemode(null); metaCopy.setClusterName(null); metaCopy.setDatasetName(null); metaCopy.setNamespace(null); metaCopy.setCurrentFsImage(null); metaCopy.setLastHdfsFsImage(null); metaCopy.setLastSnapshotsInfo(null); metaCopy.setSnapshotTimestamp(null); metaCopy.setSnapshotCount(null); metaCopy.setSnapshotSize(null); metaCopy.setSnapshotsSize(null); metaCopy.setStartTime(null); metaCopy.setStopTime(null); metaCopy.setAllSnapshotsStartTime(null); metaCopy.setAllSnapshotsEndTime(null); metaCopy.setZooKeeperConnectInfo(null); metaCopy.setPathRewriteInfo(null); metaCopy.setConfigulativeEditsInfo(null); metaCopy.setEditsInfo(null); metaCopy.setCommitInfo(null); metaCopy.setEditInfo(null); metaCopy.setMajorEditInfo(null); metaCopy.setMinorEditInfo(null); metaCopy.setMergeInfo(null); metaCopy.setSplitInfo(null); metaCopy.setCreateIndexInfo(null); metaCopy.setDeleteInfo(null); metaCopy.setCorruptionInfo(null); metaCopy.setRollbackInfo(null); metaCopy.setRewindInfo(null); metaCopy.setReplicationInfo(null); metaCopy.setSnapshotOrderInfo(null); metaCopy.setMajorChangeCount(null); metaCopy.setMinorChangeCount(null); metaCopy.setMergeInfoInfo(null); metaCopy.setSplitInfoInfo(null); metaCopy.setUpdateInfo(null); metaCopy.setHdfsFsImageBackupInfo(null); metaCopy.setDeleteInfoBackupInfo(null); metaCopy.setCorruptionInfoBackupInfo(null); metaCopy.setRollbackInfoBackupInfo(null); metaCopy.setRewindInfoBackupInfo(null); metaCopy.setZooKeeperUpdateInfo(null); metaCopy.setRecoveryInfo(null); metaCopy.setAppendInfo(null); metaCopy.setLocalBlockInfo(null); metaCopy.setBlockInfo(null); metaCopy.setHostId(null); metaCopy.setNamespaceId(null); metaCopy.setLastCheckpointVolumeId(null); metaCopy.setSnapshotsCount(null); metaCopy.setSnapshotsSize(null); metaCopy.setStartVersion(null); metaCopy.setStopVersion(null); metaCopy.setSnapshotInfo(null); metaCopy.setSnapshotId(null); metaCopy.setBlockMeta(null); metaCopy.setSnapshotMeta(null); metaCopy.setBlockMetaSnapshot(null); metaCopy.setSnapshotMetaSnapshot(null); metaCopy.setEditMeta(null); metaCopy.setEditMetaSnapshot(null); metaCopy.setMajorEditMeta(null); metaCopy.setMinorEditMeta(null); metaCopy.setMergeMeta(null); metaCopy.setSplitMeta(null); metaCopy.setUpdateMeta(null); metaCopy.setHdfsFsImageBackupMeta(null); metaCopy.setDeleteInfoBackupMeta(null); metaCopy.setCorruptionInfoBackupMeta(null); metaCopy.setRollbackInfoBackupMeta(null); metaCopy.setRewindInfoBackupMeta(null); metaCopy.setZooKeeperUpdateMeta(null); metaCopy.setRecoveryInfoMeta(null); metaCopy.setAppendMetaSnapshot(null); metaCopy.setLocalBlockMetaSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshot(null); metaCopy.setHostIdSnapshot(null); metaCopy.setNamespaceIdSnapshot(null); metaCopy.setLastCheckpointVolumeSnapshotId(null); metaCopy.setSnapshotsCountSnapshot(null); metaCopy.setSnapshotsSizeSnapshot(null); metaCopy.setStartVersionSnapshot(null); metaCopy.setStopVersionSnapshot(null); metaCopy.setSnapshotInfoSnapshot(null); metaCopy.setSnapshotIdSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshot(null); metaCopy.setSnapshotMetaSnapshotSnapshot(null); metaCopy.setEditMetaSnapshotSnapshot(null); metaCopy.setMajorEditMetaSnapshot(null); metaCopy.setMinorEditMetaSnapshot(null); metaCopy.setMergeMetaSnapshot(null); metaCopy.setSplitMetaSnapshot(null); metaCopy.setUpdateMetaSnapshotSnapshot(null); metaCopy.setHdfsFsImageBackupMetaSnapshot(null); metaCopy.setDeleteInfoBackupMetaSnapshot(null); metaCopy.setCorruptionInfoBackupMetaSnapshot(null); metaCopy.setRollbackInfoBackupMetaSnapshot(null); metaCopy.setRewindInfoBackupMetaSnapshot(null); metaCopy.setZooKeeperUpdateMetaSnapshot(null); metaCopy.setRecoveryInfoMetaSnapshot(null); metaCopy.setAppendMetaSnapshotSnapshot(null); metaCopy.setLocalBlockMetaSnapshotSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshotSnapshot(null); metaCopy.setHostIdSnapshotSnapshot(null); metaCopy.setNamespaceIdSnapshotSnapshot(null); metaCopy.setLastCheckpointVolumeSnapshotSnapshotId(null); metaCopy.setSnapshotsCountSnapshotSnapshot(null); metaCopy.setSnapshotsSizeSnapshotSnapshot(null); metaCopy.setStartVersionSnapshotSnapshot(null); metaCopy.setStopVersionSnapshotSnapshot(null); metaCopy.setSnapshotInfoSnapshotSnapshot(null); metaCopy.setSnapshotIdSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshotSnapshot(null); metaCopy.setSnapshotMetaSnapshotSnapshot(null); metaCopy.setEditMetaSnapshotSnapshotSnapshot(null); metaCopy.setMajorEditMetaSnapshot(null); metaCopy.setMinorEditMetaSnapshot(null); metaCopy.setMergeMetaSnapshotSnapshot(null); metaCopy.setSplitMetaSnapshotSnapshot(null); metaCopy.setUpdateMetaSnapshotSnapshot(null); metaCopy.setHdfsFsImageBackupMetaSnapshotSnapshot(null); metaCopy.setDeleteInfoBackupMetaSnapshotSnapshot(null); metaCopy.setCorruptionInfoBackupMetaSnapshotSnapshot(null); metaCopy.setRollbackInfoBackupMetaSnapshotSnapshot(null); metaCopy.setRewindInfoBackupMetaSnapshotSnapshot(null); metaCopy.setZooKeeperUpdateMetaSnapshotSnapshot(null); metaCopy.setRecoveryInfoMetaSnapshotSnapshot(null); metaCopy.setAppendMetaSnapshotSnapshotSnapshot(null); metaCopy.setLocalBlockMetaSnapshotSnapshotSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshotSnapshotSnapshot(null); metaCopy.setHostIdSnapshotSnapshotSnapshot(null); metaCopy.setNamespaceIdSnapshotSnapshot(null); metaCopy.setLastCheckpointVolumeSnapshotSnapshotId(null); metaCopy.setSnapshotsCountSnapshotSnapshot(null); metaCopy.setSnapshotsSizeSnapshotSnapshot(null); metaCopy.setStartVersionSnapshotSnapshot(null); metaCopy.setStopVersionSnapshotSnapshot(null); metaCopy.setSnapshotInfoSnapshotSnapshot(null); metaCopy.setSnapshotIdSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshotSnapshot(null); metaCopy.setSnapshotMetaSnapshotSnapshot(null); metaCopy.setEditMetaSnapshotSnapshotSnapshot(null); metaCopy.setMajorEditMetaSnapshot(null); metaCopy.setMinorEditMetaSnapshot(null); metaCopy.setMergeMetaSnapshotSnapshot(null); metaCopy.setSplitMetaSnapshotSnapshot(null); metaCopy.setUpdateMetaSnapshotSnapshot(null); metaCopy.setHdfsFsImageBackupMetaSnapshotSnapshot(null); metaCopy.setDeleteInfoBackupMetaSnapshotSnapshot(null); metaCopy.setCorruptionInfoBackupMetaSnapshotSnapshot(null); metaCopy.setRollbackInfoBackupMetaSnapshotSnapshot(null); metaCopy.setRewindInfoBackupMetaSnapshotSnapshot(null); metaCopy.setZooKeeperUpdateMetaSnapshotSnapshot(null); metaCopy.setRecoveryInfoMetaSnapshotSnapshot(null); metaCopy.setAppendMetaSnapshotSnapshotSnapshot(null); metaCopy.setLocalBlockMetaSnapshotSnapshotSnapshot(null); metaCopy.setBlockMetaSnapshotSnapshotSnapshotSnapshot(null); metaCopy.setHostIdSnapshotSnapshotSnapshot(null); metaCopy.setNamespaceIdSnapshotSnapshot(null); metaCopy.setLastCheckpointVolumeSnapshotSnapshotId(null); metaCopy.setSnapshotsCountSnapshotSnapshot(null); metaCopy.setSnapshotsSizeSnapshotSnapshot(null); metaCopy.setStartVersionSnapshotSnapshot(null); metaCopy

    更多回答
    扫一扫访问手机版
    • 回到顶部
    • 回到顶部