Articles in Commands & Shells
Bash scripting: lists and random things
Imagine you have an HPC cluster with a hundred compute nodes named node001-node100. The two commands below will help you generate a list of node names – either all name on one line or one name per …
Fast IP Range Scanner in Bash
Just a simple script to scan an IP range and do a reverse DNS lookup. This can be useful for building an inventory of systems on your LAN. Just make sure to give your network …
Remounting Active NFS Filesystems
Here’s a scenario: you have an NFS client mounting a filesystem from server1. You need to migrate this data to NFS server2 and remount the filesystem from the client to point to the new server. The problem is that there is a user application on the client system that uses the NFS mount, preventing you from unmounting it.
Uninstall PostgreSQL from Ubuntu
Ok, so this is a very simple one: uninstall PostgreSQL running on Ubuntu. There are a bunch of pieces to the database server and just running “apt-get remove postgresql” will still leave you with with a bunch of leftovers
Making Rsync Faster
If you Google something along the lines of “make rsync faster”, the most common thing you’ll see is people saying “hey, I have a gigabit network connection and my rsync is crawling along at a hundred kilobytes per second.” Well, the issue here is not the network. Rsync needs time to analyze source and destination, generate checksums and compare timestamps, build a list of stuff to transfer and then, finally, start the copy process, one item at a time. You see the problem, I am sure.
A Faster Way to Delete Many Folders
This issue is particularly annoying when dealing with expansive directory structures mounted via NFS. Let’s say you have a directory with ten thousand folders and each of them has more subfolders and files. And you …
Simple Shell Random Password Generator
A very simple script to generate a random secure password of desired length. This can come in handy when setting up multiple user account and/or lacking imagination.
Generate Encrypted Password for /etc/shadow
Here is a simple way to generate an encrypted password string for use in Linux /etc/shadow, among other things. This is just in case you don’t wish to use the usual tools for changing the password (passwd, chpasswd).
Use wget with Your Rapidshare Account
If you have a Rapidshare premium account, you can use wget to download files. The process is two-fold: first, you need to use wget with your Rapidshare username and password to obtain and store the Rapidshare cookie; then, you can use wget with the cookie file to download a list of links.
Configure Postfix to Use Gmail for Sending
Here is a simple script to help you configure Linux postfix to use your Gmail account for sending emails. The script has been tested on OpenSuse 11.4 and 12.1, but should work find on any recent Linux disro. The only requirement is that you have postfix insalled.
Playing With Time in Bash Shell
The “date” command in Bash shell offers a remarkable array of features that can be very useful in performing many system administration tasks. As you will see below, it is easy to determine date, time, day of week for any interval of time. This can be very useful for system automation tasks with “cron” and “at”.
A Simple Process Monitoring Script
Let’s say you are running a data restore. Things are moving along, but network is congested and the backup server is overloaded. You really don’t feel staring at the restore status for the next several hours and just want to be notified when the process completes. The simplest method of monitoring for processes starting or ending on Unix systems is by using “ps” with a “while” loop.
Compress Old Log Files on Linux
Most log files located in /var/log are part of the log rotation and will be compressed automatically. However, in many cases various user applications maintain log files outside of /var/log. These logs are not managed …
Testing SSH Connectivity to Multiple Servers
Imagine a hypothetical scenario: you support hundreds of remote servers and you need to check which server you can access via SSH and which servers are not letting you log in. Doing this manually is a tedious process that many sysadmins choose to skip. The inevitable outcome is inability to quickly access a system when it really counts.
Writing a Linux Startup Script
As basic as the task of creating a startup script may sound, even experienced sysadmins sometimes run into problems with having services start at the right time during the boot process or stop during the shutdown. Two major reasons for this: the procedure is a bit convoluted due to linking. Also, writing startup scripts is not something you have to do very often these days.
Viewing “man” Pages in a Web Browser
The Unix “man” pages are an indispensable tool even for the most experienced of sysadmins. Unfortunately, in a terminal window the text may be hard to read and getting a hard copy to take with you down to the datacenter will really test your Unix skills. The following quick script will convert the “man” page of your choice into formatted text and open it in the Web browser.
Autorebooting Servers on Low or High Load
Recently I ran into a situation where I needed to reboot several HPC cluster compute nodes. A couple of systems were still running user jobs that I did not want to interrupt. I also didn’t want to sit around and watch those jobs, as there was not way of telling how long they could take to complete. The solution was to write a short script that looks something like this…
Testing RAM in Linux
A number of good and mostly free tools are available for testing RAM hardware. However, most of these applications are designed to operate in stand-alone mode that requires shutting down the OS. Still, tare a few ways to run memory integrity checks from inside the operating environment. Here we will look at using dd and memtester.
Using Expect with SSH and Su
We have discussed using expect with sudo in an SSH session. Another common way to run commands on remote servers as root is to use the su – root command, where sudo is not configured. The trick to automating su operations using expect is to launch the su command with the spawn ssh command.
Moving Average with Excel and Shell Script
In this example we will show you how to calculate simple moving average using Excel and a shell script. As more data is added to the file, the shell script and the Excel formula will automatically use the specified number of the latest data points to calculate the moving average.
Configuring Passwordless SSH
Passwordless SSH works by pre-sharing public encryption keys. It is important to understand that setting up passwordless SSH opens your servers to certain security risks. A compromised user account on one server may allow the attacker to gain access to multiple systems on the network. Try to avoid using passwordless access for accounts with elevated privileges.
Automating SSH and Sudo with Expect
Let’s imagine a hypothetical scenario: you have a list of a hundred Linux servers and you need to log into each one of them and remove a local user “roger” and his home directory. Doing this by hand will get tedious and, chances are, you will make a few typos and there will be some collateral damage.
Most Commented