FILE SYSTEM FULL CHECKING IN SOLARIS

Problem:-- File System Full  

Symptoms

File Systems Full

The following errors could be seen when a filesystem is full:

On the system console:

<date> <hostname> vmunix <filesystem> : file system full

WARNING: <filesystem>: file system full

NOTICE: alloc: <filesystem> file system full no space left on device

vmunix: file system full

For example:

WARNING: /: filesystem full

There are several reasons why a filesystem gets full. An important thing to consider  is how you set up your filesystems during installation,;you need to take care how much space is used for each filesystem and think ahead.

With forward thinking it is less likely that your filesystems will get full, but will  not prevent a filesystem getting full. This document will show the most common reasons why a filesystem may become full and how to handle them.

Resolution

This document is written mainly to deal with the OS filesystems (such as root, var and usr), but it can be utilized to troubleshoot other filesystems.  There are many ways of finding what's filling up a filesystem, which can sometimes be a difficult process.  One problem is that a filesystem can be filled up by one or few very large files (which is generally easy to find) or by thousands of smaller files (which can be difficult to find and pinpoint the cause).

First you need to figure out which files are filling up your filesystem.

A very useful way to list the size of files in a filesystem is with the du command.

The following example lists files from largest to smallest on the root filesystem:

$ du -akd / | sort -nr | more

   or

$ du -akd / | sort -nr > /tmp/du.out

The latter will give you a file you can review at your convenience.

The -d option of the du command keeps du from crossing partition boundaries.

The  -a  option tells du to report file sizes (without this option du just reports the amount of space used in each directory. The  -k  option means that du will report in terms of kilobytes rather than 512-byte blocks. On Solaris 9 or later replace  k  with  h  if you prefer  human-readable  output, that is output in terms of kilobytes, megabytes or gigabytes depending on the number reported.

The -nr option of sort puts the files in reverse numerical order.

Of course, this can be used on filesystems other than root, just substitute the required path for  /  in the "du" command.

The command  du -skd /  summarizes the amount of kilobytes used for a filesystem, in the given case for the root filesystem. If this is different from what is reported by the df -k command,

Unmount any mounted filesystems and check the mount point directories for files.  Remove the files, or move them if you think you need them, and mount the filesystems again.

For the /tmp filesystem, you will have to boot the system into single-user mode to access the /tmp directory without having swap mounted over it.

For /var and /usr, you will have to boot the system from cdrom, mount the root filesystem, and then check the /var and /usr directories under the mounted root filesystem. These should normally be empty when /var or /usr is not mounted.

Standard filesystems to look at first will be:

Filesystem	Checks
/tmp	If /tmp is full or contains large files, a reboot will clean this directory. A default Solaris installation shares the diskspace for /tmp and swap as you can see in the output of the df command. Note: /tmp is not cleaned at boot time if /tmp is configured as a separate filesystem.
/dev	Large files may appear here when trying to write to a device using the incorrect device name.  For example /dev/rmt/o (letter 'o') instead of /dev/rmt/0 (digit 'zero' for a tape drive.  This is a very common problem if the machine does not have a tape drive attached and someone uses a tape command like tar or ufsdump.  That will just create a large file in /dev/rmt/.  So be sure to check the /dev directory for actual links not files.
/	Look for core files.  Check /.wastebasket and /lost+found directory for large files.  Check for a .CPR file in root, this is put there by power suspend/resume software.
/var	Third party packages sometimes leave tar files in /var/sadm/pkg directory. If /var is full (and is a separate filesystem) or /var directory is the one we determined is using up most space in root, check the following. Clearing out (but NOT deleting; the files should be truncated to zero length) the following files might gain you some space.  Use caution here because you will lose various log information.  For example, the utmp[x] and wtmp[x] files contain user access and accounting information: /var/cron/log /var/spool/lp/logs /var/adm/utmp /var/adm/utmpx /var/adm/wtmp /var/adm/wtmpx /var/log/syslog* /var/adm/messages.* NOTE: if you zero out the utmp, utmpx, wtmp or wtmpx files, you should reboot your machine. To zero out a file: # cat /dev/null > filename       NOTE: For Solaris[TM] 9 or greater, see logadm(1M) for a useful tool to manage log files. Check /var/saf  - check for _log and in tcp and zsmon directories. There will be _log files - you can zero them out with "cat /dev/null > filename".  If your system is being used as a printer host, check /var/lp/logs for files, they can be removed if they have been printed or left over from system crashes or printer problems.  Check /var/preserve.  Check /var/spool/* directory.  Subdirectories like "lp" or "mqueue" are used for spooling.  Check /var/crash for any system cores.  Also check /var/tmp for files not needed;         /var/tmp  is not cleaned up with a reboot. A word of caution regarding the /var/sadm directory.  This directory contains package and patch information and generally should not be touched.

Also it could be that you are running out of inodes and are getting the message  file system full .  In this case recreate a partition with more inodes.  The basic steps are:

• Remove unneeded files.
• Backup the partition.
• recreate using newfs -i nbpi /dev/<rfsname> where nbpi is chosen smaller then the  default for the disk size, and rfsname is the raw filesystem; e.g. /dev/rdsk/cNtNdNsN.  See man newfs(1M) for more information.
• Restore information back to the partition.

In the course of normal system operation, the root and usr filesystems (or directories) are mostly static (do not grow over time).  /var however, does grow over time (because it contains log files, package database, print and mail spoolers, etc.).  The name  var  is in fact an abbreviation for  varying  or  variable  as the  /var  filesystem is intended for files which vary in size and content over time (see the filesystem(5) manual page for more details about this).  It is good system administration practice to monitor log files to make sure they don't get too large.

If a filesystem suddenly fills up, that could have been caused by installing a new piece of software into a wrong directory.

Check any lost+found directory on any filesystem that is full.

Another approach would be to list files by their modification date (if the date of when the filesystem filled up is known).

# ls -lRt / | more will list all the files and sort them by the modification dates.

Helo Team,

Please find the sample scripts for Locating files, for removal if a filesystem becomes full, and also find the Related Doc attached.

Description
File system is full - how to locate files to remove

Steps to Follow

Locating files for removal if a filesystem becomes full:

Here is the syntax to find files that are greater in size than 1 MB:

   cd
   find . -size +1000000c -exec ls -l {} +

The  -mount  option to the find command can be used to restrict the search to the filesystem containing the directory specified.

To find files generated by NFS and remove them if they are older than seven days:

find / -name .nfs\* -mtime +7 -exec rm -f {} + -o -fstype nfs -prune

To search for core files starting at the root directory and delete them:

find / -name core -exec rm {} +

To search for core files that have not been accessed in seven days and display them to the screen:

find / -name core -atime +7 -print

To identify all files owned by a particular user and send a long listing of these files to the superuser:

find / -user  -ls |  mailx -s "users files" root@hostname

Look for files that have not been modified in 90 days in the /export/home directory:

find /export/home -mtime +90 -print

To find files that are greater than 400 blocks (512-byte blocks) and display the matching path names:

find /export/home -size +400 -print

Check /var/saf for a _log file and under /var/saf/tcp and /var/saf/zsmon for log files; they can be truncated with:

cat /dev/null > filename

Do an ls -Ll of the /dev directory to make sure that all entries are character devices or block devices, not files or use this find:

find /devices /dev -type f ! -size 0 -ls

**Specifically check the /dev/rmt directory, all entries should be links, not large files.

Check .wastebasket and the lost+found directory for large files.