Automatically Validate HTTP Proxies

Submitted by Igor on August 5, 2009 – 4:14 pm6 Comments

Let’s say you downloaded a long list of Web proxy servers. Now you are stuck with the task of weeding out the proxies that are dead, slow, fake, or otherwise unusable. There are some applications out there that claim to validate proxy servers. The most common problem with these applications: they are excruciatingly slow. These apps also tend to get stuck once in a while. And, if your list of proxies is too long, these applications may crash altogether because of numerous memory leaks and other such examples of fine programming.

I would like to bring your attention to the following, hopefully, useful script that will go through some very long proxy lists in just a minute or two and will get rid of the trash. A few words about how it works are in order. I created a simple HTML page on my Web server (see $pvcurl variable below). This page contains a unique text string ($pvcstring variable).

The first step is to ping the proxy and see if it responds in a reasonable period of time. The ping commands are launched in background to speed up the process. If the proxy does respond, the next step is to use wget to see if you can download the $pvcurl and match the $pvcstring. If everything checks out, the proxy is added to the final list of good proxies. Just as the ping command, the wget threads are started in background mode with a 30-second timeout.

Download: proxy_validate.ksh

#!/bin/ksh

configure() {
	pvcurl="http://www.krazyworks.com/pvc.html"
	pvcstring="191628769290432845414226"
	wget_timeout=30

	proxyin="/tmp/proxylist.in"

	if [ ! -f "$proxyin" ]
	then
		echo "Proxy list $proxyin not found. Exiting..."
		exit 1
	fi

	proxyout="/root/proxylist.out"

	if [ -f "$proxyout" ]
	then
		rm "$proxyout"
	fi
}

cleanup() {
	killall wget
	for i in 1 2 3 4 5
	do
		if [ -f "/tmp/proxy_verify.tmp$i" ]
		then
			rm "/tmp/proxy_verify.tmp$i"
		fi
	done
}

wgetrun() {
	if [ `wget -q --timeout=$wget_timeout --tries=1 -O - "$pvcurl" | grep -c "$pvcstring"` -eq 1 ]
	then
		echo "${proxy}:${port}" >> "$proxyout"
	fi
}

pingrun() {
	ping -q -c 1 -W 5 $proxy >/dev/null 2>&1

	if [ $? -eq 0 ]
	then
		wgetrun &
	fi
}

verify() {
	sort "$proxyin" | uniq > "/tmp/proxy_verify.tmp1"
	mv "/tmp/proxy_verify.tmp1" "$proxyin"
	proxy_total=$(wc -l "$proxyin" | awk '{print $1}')
	
	i=1
	j=1
	cat "$proxyin" | while read line
	do
		echo "Processing proxy $i of $proxy_total"
		proxy=$(echo $line | awk -F':' '{print $1}')
		port=$(echo $line | awk -F':' '{print $2}')
		export http_proxy="${proxy}:${port}"
		(( i = i + 1 ))

		pingrun &
	
		if [ $j -eq 100 ]
		then
			if [ `ps -ef | grep -c [w]get` -gt 100 ]
			then
				sleep $wget_timeout
				killall wget
				j=1
			fi
		else
			(( j = j + 1 ))
		fi
	done

	echo "Waiting for threads to finish ($wget_timeout seconds)..."
	while [ `ps -ef | egrep -c "[w]get|[p]ing"` -gt 0 ]
	do
		sleep 5
	done
}

# RUNTIME

configure
cleanup
verify
cleanup

#!/bin/ksh

configure() {

pvcurl="http://www.krazyworks.com/pvc.html"

pvcstring="191628769290432845414226"

wget_timeout=30

proxyin="/tmp/proxylist.in"

if [ ! -f "$proxyin" ]

then

echo "Proxy list $proxyin not found. Exiting..."

exit 1

proxyout="/root/proxylist.out"

if [ -f "$proxyout" ]

then

rm "$proxyout"

}

cleanup() {

killall wget

for i in 1 2 3 4 5

if [ -f "/tmp/proxy_verify.tmp$i" ]

then

rm "/tmp/proxy_verify.tmp$i"

done

}

wgetrun() {

if [ `wget -q --timeout=$wget_timeout --tries=1 -O - "$pvcurl" | grep -c "$pvcstring"` -eq 1 ]

then

echo "${proxy}:${port}" >> "$proxyout"

}

pingrun() {

ping -q -c 1 -W 5 $proxy >/dev/null 2>&1

if [ $? -eq 0 ]

then

wgetrun &

}

verify() {

sort "$proxyin" | uniq > "/tmp/proxy_verify.tmp1"

mv "/tmp/proxy_verify.tmp1" "$proxyin"

proxy_total=$(wc -l "$proxyin" | awk '{print $1}')

i=1

j=1

cat "$proxyin" | while read line

echo "Processing proxy $i of $proxy_total"

proxy=$(echo $line | awk -F':' '{print $1}')

port=$(echo $line | awk -F':' '{print $2}')

export http_proxy="${proxy}:${port}"

(( i = i + 1 ))

pingrun &

if [ $j -eq 100 ]

then

if [ `ps -ef | grep -c [w]get` -gt 100 ]

then

sleep $wget_timeout

killall wget

j=1

else

(( j = j + 1 ))

done

echo "Waiting for threads to finish ($wget_timeout seconds)..."

while [ `ps -ef | egrep -c "[w]get|[p]ing"` -gt 0 ]

sleep 5

done

}

# RUNTIME

configure

cleanup

verify

cleanup

soccermaster1

Hi I have one year

I know that this letter is true and it is the menu that I’m ready to help

ATTENTION SIR,

Jordan International Bank Plc, London have acknowledged your contact information as the appointed representative and next beneficiary to Three Million, Two Hundred Thousand United States Dollar (USD $ 3.2 Million) deposited in our bank by late Dr.Adel Abolhassan,with Miss,Zainab Vanda Abolhassan,as the next-of-kin to it.

Miss. Zainab Vanda Abolhassan automatically inherits the said deposited fund with us as the next beneficiary because late Mr.Adel Abolhassan registered her name as the next-of-kin when the account was opened. The bank instructed her to appoint a trustee to act as proxy to her if she intends to reclaim the money now since her current status being a ward of the United Nations with refugee status restricts her from performing such transaction personally,

You are now requested to provide the bank with two legal authorizations for representative fund transfer called the (1A)Grant of Representation issued by Probate Registry and(2b) Change of Ownership Order from a legal attorney which must be legalized and validated by a Higher Court within the country where the next-of-kin presently resides for her to append signatory.

The Grant of Representation is mandatory to protect the interest of the bank, that of yours and Miss Zainab Vanda Abolhassan so that she can not testify in future that she did not authorize the transfer of her inheritance to your account. You are mandated to forward the requested documents without prolonged delay and provide the legal authorization documents must bear the signature of next-of-kin being Miss Zainab Vanda Abolhassan,

NOTE: On receiving the said demanded for legal authorizing documents for representative fund transfer, it will be verified after which we will credit the money to your provided Bank account through telex wire transfer which will reflect in your account after 48 Hours‏,
Adam

My internet service providor provide me pppoe(no user name no password & IP’s are automatically optained) internet service, can you tell me that how to share internet connection in another 2PC.
**I tried it by ICS,proxy, router, but no result
Sophia C

i am not sure what the hell is wrong with this site.
yesterday i bought a PREMIUM membership on megaupload and download megaupload manager,
then downloaded some files.

this morning i woke up and megaupload does not work, and says server can’t be found.

when i use the proxy (hide my asx) it works, but cannot download any file due to proxy.

looks like they just take the money then block you for not downloading any file, so they can save
some bandwidth for getting more PREMIUM users?
i live in Japan.
Ev dog

If there is any good people who knows a lot about computers or they are like a computer programmer please leave me their email. I want to know more about computers
cardskid22

ok well my friend got a new phone and didnt need her slvr l7 so she gave it to me. (we both have at&T) and so i put my chip in and stuff and i enter the unlock code and it says your number and my number comes up.. ok everything is fine.. BUT i have picture messaging on my plan and this phone wont send them or let me go on aim?? and sometimes a little in-call icon comes up on screen and im not in a call. also, i cannot acces the internet when it is in my plan and i can on other phones. can anyone help me?
_marky_mark_

Sometimes I get banned from a web site and most of the time it’s for silly reasons. Unfortunately, it’s very difficult to reset an ip address with windows 7. I tried entering the site through a proxy server, but the site still recognizes me. I’ve disabled cookies and the web site is in java. How does it still know my computer?
The web site is http://www.chat-avenue.com and the proxy server is http://www.freeproxyserver.net