Linux BCC Instance Insufficient Space Issue Resolution
Background
In a BCC Linux instance, you might encounter "no space left on device" errors due to insufficient disk space.
Cause of problem
Primary causes of insufficient disk space include:
- Block space usage reaches 100%.
- Inode space usage reaches 100%.
- Deleted files remain inaccessible due to open handles, preventing the release of corresponding space.
- Block space usage reaches 100%.
Use the df -h command to check the disk partition usage.
df -h
As shown, the block space usage has reached 100%.
Solution:
- Delete unnecessary files.
- Expand the disk space.
inode space usage reaches 100%
Use the "df -i" command to check the inode usage.
As shown in the diagram, the inode usage for /dev/vdb1 has reached 100%.
Primary solutions are as follows:
- Delete files or directories that are consuming a high number of inodes.
Navigate to the directory where the inode usage has reached 100%.
Execute: for i in ./*; do echo $i; find $i | wc -l; done, to locate directories with high inode usage.
Enter the directory with the highest inode usage level by level, continue executing the above commands to locate files or directories occupying excessive space, then and then clean up the file or directory 。
- Modify inode Data Backup Required Beforehand.
Detach Disk Partition or mount directory: umount /mnt
Reformat the file system with the command: mkfs.ext4 /dev/xvdb -N 51376. In this example, 51376 inodes are specified, but adjust according to your actual requirements.
Verify the updated inode count.
Remount: mount /dev/vdb1 /mnt
- Resize Disk/Expand Disk
Deleted files not released due to handle
Run the command "du -sh" to identify significant discrepancies between file space usage and available disk space when inode usage is still low.
Run the following command to check if problematic directories contain files with unresolved handles.
lsof | grep delete If processes occupy handles of deleted files, terminate processes and free up disk space with the kill -9 command.
kill -9 pid