How To Recover from File System Corruption Using FreeBSD's Single-User Mode

Introduction

Despite your best efforts, there are certain instances where your server might suffer file corruption and need recovery. Sometimes this happens when your VPS is powered off abruptly, or when a piece of software or hardware malfunctions suddenly.

Either way, there are steps you can take to try to recover your VPS. On FreeBSD many of these functions can be performed from single-user mode. In this tutorial we will go through the steps you can take to boot your FreeBSD droplet into single-user mode and how to use the tools there to attempt to recover a damaged filesystem.

Important Considerations and Risks

In any circumstance, good backups are the best way to prevent data loss. Implementing a reliable offsite backup solution and thoroughly testing it on a regular basis is the only way to guarantee the integrity of your important data.

While recovery options like fsck are often useful, there is no guarantee that it will work correctly and success is often simply a matter of luck. The fsck operation can occasionally cause data corruption on active disks. Problems can still occur in cases of severe damage though, so consider these last-resort methods for data recovery.

Booting to Single User Mode

To boot your droplet into single user mode you can run the following command:

sudo nextboot -o "-s" -k kernel

This will instruct your droplet to enter single user mode on the next reboot rather than attempting to load the full system. Once you are ready you can use the reboot command to reboot your droplet.

sudo reboot

After the reboot your droplet will no longer be accessible over the network via ssh. You will now need to access your droplet via the console in the control panel. Once the reboot has completed you will find yourself at a very minimal shell similar to the one shown below:

When you see the line:

Enter full pathname of shell or RETURN for /bin/sh:

You can press Enter to start a shell session.

Alternate Method

If your droplet is not accessible via ssh you can still boot your droplet to single user mode using this alternate method.

First, in the control panel click on the power cycle button to reboot your droplet.

Immediately after this, open your droplet’s console. After a few seconds you should see a screen like this:

At this menu select item 2 and press enter to continue.

Performing a Filesystem check

Now that your droplet is in single-user mode we can proceed. First, lets make sure that we know what device we are checking. Running the command below will show the filesystems currently configured.

cat /etc/fstab

The output you see should be similar to this:

The first item in the list shows a ufs filesystem which is what we are looking for. Now you can run the following command to perform a filesystem check on this disk.

fsck -yf /dev/gpt/rootfs

Checking the Results

Once your filesystem check is complete you can reboot your droplet using the reboot command to leave single-user mode and restart your dropet in normal (Multi-User) mode.

Connect to your droplet using an SSH client. If your droplet responds to your ssh connection and it was not before then this is a good sign. If you ran into specific problems earlier that alerted you to the possibility of corruption, you should try these operations again to see if they are successful.

An important thing to do is check the /lost+found directory. This is where fsck puts partially recovered files.

Sometimes, fsck is able to recover file data, but it cannot find a reference to the file on the filesystem. Basically, it is a file without a name. If fsck is faced with this situation, it places these files in the /lost+found directory so that you can manually try to figure out what the file is.

After running fsck, look to see if anything has been placed in that directory. Since the lost+found directory is only available to the root user we will first change to the root account with a sudo su command:

sudo su
cd /lost+found
ls

If there are files in this directory, you will want to see if you can identify them. Often, these are files that you had deleted anyways, but were still being used when the system crashed. It is worth checking them though to be sure.

If your filesystem is noticeably broken still, or if the boot fails when changing back to multi-user mode, your best option may be to use a recovery ISO to recover any needed files. To do this, you will need to open a ticket with the support team so they can boot your droplet into the recovery environment.

Conclusion

While filesystem corruption is never a good thing, it doesn’t necessarily mean that all of your important data was lost. The success of your recovery operations comes down to a number of factors, such as how quickly the filesystem noticed the corruption, how widespread the issue was, and what files were affected.

When it comes down to it, simple recovery using automated tools is largely a function of luck. That being said, in many cases, fsck recovery is successful and you can go back to using your server without too much of a headache. Remember, keeping good backups is the most important step you can take to prevent data loss.