Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Commands & Shells, Featured

Recovering Deleted /bin on CentOS

Submitted by on June 25, 2014 – 1:54 pm 4 Comments

So, as the saying goes, shit happens and the other day I accidentally deleted /boot on a CentOS 6 box. I have NetBackup running on the system, so this would not have been a problem (as long as I didn’t try to reboot). Unfortunately, I also blew away /bin (where ls, cat and all that useful stuff lives) and some of /etc, for good measure. Without some commands in /bin, NetBackup “bp” command is unable to find backup image records. Bummer.

Options in this case usually are a) boot the system from a CD, mount the boot disk and copy missing binaries from another, identical server; b) reinstall OS, load NetBackup agent and run a bare-metal restore. There may be some options in between, depending on the specifics. But I really did not feel like spending the rest of the day recovering this server.

Luckly, I was still logged into the server. It was a case when the plane lost all the engines, but was still in the air. The solution was to install YUM utilities and use the optional features of YUM to identify missing files and reinstall relevant packages. Here’s a summary of the commands:

Install yum-utils and yum-verify:

yum -y install yum-utils.noarch yum-verify

Run yum-verify to identify specific paths to the missing binaries. In my case, I deleted /bin, so the syntax I used was:
yum verify --verify-filenames='*bin/*'

The output goes something like this:
coreutils.x86_64 : A set of basic GNU tools commonly used in shell scripts
    File: /bin/arch
        Problem:  file is missing
    File: /bin/basename
        Problem:  file is missing
    File: /bin/cat
        Problem:  file is missing
    File: /bin/chgrp
        Problem:  file is missing

Then you just grab the names of the affected packages and reinstall them. For example:
yum -y reinstall coreutils

Or you can feed the output of yum-verify to yum-reinstall:
yum verify --verify-filenames='*bin/*' | grep "^[a-z].*:.*" | awk '{print $1}' | while read line ; do yum -y reinstall “${line}” ; done

Once I had the basics in /bin, I was able to use local NetBackup commands to restore the missing contents of /boot and /etc. Whew.

Print Friendly, PDF & Email

4 Comments »

Leave a Reply

%d bloggers like this: