Skip to main content

HDFS Health Check

HDFS supports the fsck command to check for various inconsistencies. It is designed for reporting problems with various files,
  • Corrupt blocks
  • Missing blocks 
  • Under-replicated blocks
Unlike a traditional fsck utility for native file systems, this command does not correct the errors it detects.
Command: sudo hdfs hdfs fsck / --files --blocks  --locations

HDFS: Corrupted/Missing/Under Replicated Blocks
As per the below screenshot of fsck output, there is 1 corrupt block, 1 missing block and 4 under-replicated blocks and status of HDFS is “CORRUPT”. These indicates HDFS health is bad and these should be addressed ASAP to recover our HDFS into HEALTHY.

Corrupt block: block is called corrupt by HDFS if it has at least one corrupt replica along with at least one live replica. As such, a corrupt block does not indicate unavailable data, but they do indicate an increased chance that data may become unavailable.

Missing block: A missing block is a block with no live replicas. All replicas are either missing or corrupt. This may happen because of corruption or because data nodes are offline or being decommissioned.

Under-replicated block: A failure of this health check may indicate a loss of DataNodes  (or) wrong replication factor applied to the job from client side. Use the HDFS fsck command to identify which files contain under-replicated blocks.

If the target replica set more than block replication in hdfs-site.xml then reset the replica of the file by using below command,
hdfs dfs -setrep -w 3 HDFS/FILE/LOCATION 

Refer the Fix: Under Replicated blocks in HDFS manually to fix the under replicated blocks manually.

Labels:

Comments

Post a Comment

Popular posts from this blog

Load epoch timestamp value into hive table

Problem: Load epoch timestamp into hive table. Solution:  Use BIGINT and load into temp table. Use UDF function of to_utc_timestamp() to convert it into specific timezone. Queries: CREATE EXTERNAL TABLE tags_temp ( user_id INT , movie_id INT , tag STRING, date_time BIGINT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY "|" STORED AS TEXTFILE LOCATION 'LOCATION TO THE FILE IN HDFS' ; CREATE EXTERNAL TABLE tags ( user_id INT , movie_id INT , tag STRING, date_time timestamp )STORED AS ORC TBLPROPERTIES ( 'orc.compress' = 'SNAPPY' , 'creator' = 'uvaraj' , 'created_on' = '2016-12-30' , 'description' = 'tags details' ); INSERT OVERWRITE TABLE tags SELECT user_id,movie_id,tag, to_utc_timestamp(date_time, 'UTC' ) FROM tags_temp;
Post a Comment
Read more

Fix: Under Replicated blocks in HDFS manually

Problem: Under replicated blocks in HDFS Solution: Execute the below command to fix the under replicated blocks in HDFS, sudo -u hdfs hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
Post a Comment
Read more