Featured »

sy2zeb1vp2h8itlxfbbylw0jrm0[1]

Validating HTTPS Cache Peers for Squid

January 6, 2020 – 12:38 am

I have a squid proxy server that uses a long list of authenticated cache peers in a round-robin configuration. The process looks something like this:

The key to getting this setup working well is to weed …

Read the full story »
Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Featured, Networking, Squid »

Validating HTTPS Cache Peers for Squid

January 6, 2020 – 12:38 am
sy2zeb1vp2h8itlxfbbylw0jrm0[1]

I have a squid proxy server that uses a long list of authenticated cache peers in a round-robin configuration. The process looks something like this:

The key to getting this setup working well is to weed out unresponsive cache peers. In my case the proxies used as cache peers are ‘premium’ – I pay for the service. The vendor provides me with a regularly-updated list of working proxies. Or so they claim. I don’t how they’re checking this list, but almost always it requires some cleaning.

I wrote a little script that in a couple of minutes can go though a list of about three thousand proxies and select a few hundred that are sufficiently responsive. The script makes sure the proxy’s response meets these four conditions:

  1. The proxy responds to a request.
  2. The response arrives within the specified time window.
  3. The response contains an IP address.
  4. This IP address is not your own.

The last line of my /etc/squid/squid.conf contains this directive:

The peers.conf file looks something like this:

The script will go through the list of proxies and run about 200 curl instances at a time. Depending on your system’s resources, you may adjust this value (maxthreads). I am expecting the proxy to provide output within 5 seconds (timeout_01), but you may want to adjust this as well, depending on how picky you can afford to be.

The very last step would be to reload squid, making it re-read the configuration. Should something go wrong, the original peers.conf file will be preserved with the current date extension. Give this script a shot and, should everything work out well, add it to cron to run at least daily.

And here is the script (also available in my GitHub repo):

 

Updating PHP 5.6w to 7.1u on CentOS 6.10

December 30, 2019 – 9:53 pm
wvtchvkz4f4ua8we5yngswc1mkd[1]

This is mostly just a note to self. As I mentioned previously (probably more than once), I very much dislike systemd and will stick with CentOS 6 for as long as possible. Having said that, …

Analyzing atop Logs with atopsar

December 26, 2019 – 9:52 pm
0h7vjqy3irb9p63bjuuxvd0tq85[1]

I have discussed atop previously but concentrated primarily on how to run it and how to collect data. Now I’d like to spend some time talking about ways to analyze the data collected with atop.

Appending Filenames

December 22, 2019 – 9:51 pm
gk5arsj5ijgmshcbh73vvpkdh2k[1]

If you google how to append, say, current timestamp to the filename in Bash, almost every suggestion boils down to something really basic along the lines of mv $somefile $somefile_$(date +’Y%-%m-%d’). Technically, this is correct …

Verifying SNMP Connectivity on Multiple Hosts

December 19, 2019 – 9:50 pm
Photo-2019-10-07-14-39-03_0266[1]

I needed to check if SNMP was accessible on whatever live servers that existed in a particular subnet. Here’s a quick script to do this.

Selecting Time Ranges from Logs, Part 2

December 16, 2019 – 9:47 pm
4bbqfrqoa4vxlgrr02pg883ozks[1]

A couple of years ago I wrote an article about selecting specific time ranges from log files. I proposed two options: either convert all timestamps to epoch format (a CPU-intensive process) or rely on regex …

Bulk-Adding IPTables Rules

December 12, 2019 – 9:46 pm
b3gk9tc9x14ki5xa4bx8c4v1b66[1]

I’ve been using my mod of this handy script to block countries with iptables. One issue with the script is that it is adding rules one-by-one using the iptables -A syntax. This is the proper …

Automatically Block Frequent Visitors

December 9, 2019 – 9:45 pm
gea59uduencunzibzb6i3gxn36m[1]

I had a few similar scripts floating around this site, but this one is a bit more all-inclusive and better organized. The script will analyze your firewall/whatever access log and block particularly active visitors.

Windows 10 Upgrades

December 6, 2019 – 9:44 pm
img_0002[1]

I did the only sensible thing and disabled automatic updates on my two Windows 10 laptops. Microsoft’s belated foray into the world of CI/CD for releasing Windows patches suffers from limited automated testing.

Detecting Blurry Photos with ImageMagick

December 3, 2019 – 9:43 pm
ll7je7yy7kvxsiw8ibssm0nmqf0-2[1]

Here’s a typical scenario: I go to a birthday party where everyone knows I’m a shutterbug, so I have to bring my camera. As time goes by and blood alcohol concentration rises and attention to …

Working with iptables Logging

November 30, 2019 – 9:41 pm
gpxvtms6cvea4q504jtbxjfwcd5z-2[1]

Most commonly iptables is used to allow, block, or redirect connections. However, it also has a logging feature that can be very useful for network traffic analysis and system security.

Copying File Attributes Across Servers

November 29, 2019 – 9:40 pm
5wqsp02uml4to2k7pe8m63sd7q7[1]

To make long story short, had to fix someone’s chmod -R 777 /. A late-night copy-paste fail, it would seem. Needless to say, console access is required, as SSH will not work with permissions on …

CD/DVD-to-ISO Helper Script

November 27, 2019 – 9:39 pm
IMG_4678[1]

I can’t recall the last time I needed to convert a CD to ISO. I have four laptops and not one even has a DVD drive. It took me a while to find an external …

Analyzing Network Performance

November 25, 2019 – 9:37 pm
gly8stqifn615gp3xbc34h64byx-2[1]

Much of network performance analysis will be comparative in nature. Thus, seeing the output of multiple commands side by side can be quite useful. Bash has a useful little utility called pr and we’ll make …

Copying Data in a Restricted Environment

August 17, 2019 – 8:04 pm
o42ex2ebr1lhz7wxo84scrr9iuu

Consider this not-so-hypothetical scenario: you have some data on server_a that you would like to copy to server_b. Unfortunately, these two servers cannot communicate with each other. Nor do they have access to any common …

Raising Dead Services

August 15, 2019 – 2:23 pm
ktd9zk0j9t6vap82rmdxxip6jgg

I suppose you may say all of this can be just as easily done via systemd service config files. Well, yes and no. Yes, it can. And, no, not as easily. Not even close. Because …

IMDb Movie Title Parser in Bash

July 12, 2019 – 7:19 pm
42out6dayezwo4ntrduw5udzq0k-2

This is an update to the IMDb parser I wrote years back. From time to time IMDb makes small changes to their setup that break my script. This time they decided to start blocking curl, …

Sending Windows Logs to Remote Syslog

July 6, 2019 – 11:21 am
5aeqm3bh93xr6guzkfva9l1l83i

Nothing fancy here: just a quick note on directing Windows event logs and select application logs to a remote syslog server.

Plugging iPhone’s Privacy

July 3, 2019 – 11:19 am
0841lqjlj5ql1mfi3s4iyfznsny

Many recent publications , , suggest the iPhone is full of security holes threatening your privacy. The threat seems to be coming not so much from the phone’s operating system, but from the apps, …

Squeezing Video Files

June 30, 2019 – 11:16 am
sqsb8tfjfhkyxgnmnq8dqzdbbdm

All that crap I’ve been saving from YouTube, Facebook and whatnot tends to add up. As quality is not a huge concern here (not that it was very high to begin with), optimizing those video …

Finding Passwords in Logs and Shell History

June 27, 2019 – 11:15 am
odjxa5y2t4lq4jzrfzv1cf3pvzw

Sooner or later it will happen: you type something after which you expect a password prompt then, without looking, you type the password. However, you fat-fingered the first command, and your password ended up in …

Anatomy of Internet Bullshit

June 25, 2019 – 11:12 am
yyamsb8x7dou2n5x9abhju9dffi

Here’s an oldie from two years ago that reared its ugly head on Pocket: Starting Your Day on the Internet Is Damaging Your Brain, by Srinivas Rao. The author presents his personal opinion that reading …

Late Night Rant: College Admissions Scandal

June 23, 2019 – 11:11 am
vwnz1i4nkpz6wqtxr00pse2xwea

This is America: you’re either a duper or a dupee. I’m a duper. You guys are the dupees.
— Frank Reynolds (Danny DeVito), It’s Always Sunny in Philadelphia
How will the 2019 college admissions scandal work out? …