Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Commands & Shells, Networking

Automatically Validate HTTP Proxies

Submitted by on August 5, 2009 – 4:14 pm 6 Comments

Let’s say you downloaded a long list of Web proxy servers. Now you are stuck with the task of weeding out the proxies that are dead, slow, fake, or otherwise unusable. There are some applications out there that claim to validate and unblock proxy servers. The most common problem with these applications: they are excruciatingly slow. These apps also tend to get stuck once in a while. And, if your list of proxies is too long, these applications may crash altogether because of numerous memory leaks and other such examples of fine programming.

I would like to bring your attention to the following, hopefully, useful script that will go through some very long proxy lists in just a minute or two and will get rid of the trash. A few words about how it works are in order. I created a simple HTML page on my Web server (see $pvcurl variable below). This page contains a unique text string ($pvcstring variable).

The first step is to ping the proxy and see if it responds in a reasonable period of time. The ping commands are launched in background to speed up the process. If the proxy does respond, the next step is to use wget to see if you can download the $pvcurl and match the $pvcstring. If everything checks out, the proxy is added to the final list of good proxies. Just as the ping command, the wget threads are started in background mode with a 30-second timeout.

Download: proxy_validate.ksh

#!/bin/ksh

configure() {
    pvcurl="https://www.krazyworks.com/pvc.html"
    pvcstring="191628769290432845414226"
    wget_timeout=30

    proxyin="/tmp/proxylist.in"

    if [ ! -f "$proxyin" ]
    then
        echo "Proxy list $proxyin not found. Exiting..."
        exit 1
    fi

    proxyout="/root/proxylist.out"

    if [ -f "$proxyout" ]
    then
        rm "$proxyout"
    fi
}

cleanup() {
    killall wget
    for i in 1 2 3 4 5
    do
        if [ -f "/tmp/proxy_verify.tmp$i" ]
        then
            rm "/tmp/proxy_verify.tmp$i"
        fi
    done
}

wgetrun() {
    if [ `wget -q --timeout=$wget_timeout --tries=1 -O - "$pvcurl" | grep -c "$pvcstring"` -eq 1 ]
    then
        echo "${proxy}:${port}" >> "$proxyout"
    fi
}

pingrun() {
    ping -q -c 1 -W 5 $proxy >/dev/null 2>&1

    if [ $? -eq 0 ]
    then
        wgetrun &
    fi
}

verify() {
    sort "$proxyin" | uniq > "/tmp/proxy_verify.tmp1"
    mv "/tmp/proxy_verify.tmp1" "$proxyin"
    proxy_total=$(wc -l "$proxyin" | awk '{print $1}')
    
    i=1
    j=1
    cat "$proxyin" | while read line
    do
        echo "Processing proxy $i of $proxy_total"
        proxy=$(echo $line | awk -F':' '{print $1}')
        port=$(echo $line | awk -F':' '{print $2}')
        export http_proxy="${proxy}:${port}"
        (( i = i + 1 ))

        pingrun &
    
        if [ $j -eq 100 ]
        then
            if [ `ps -ef | grep -c [w]get` -gt 100 ]
            then
                sleep $wget_timeout
                killall wget
                j=1
            fi
        else
            (( j = j + 1 ))
        fi
    done

    echo "Waiting for threads to finish ($wget_timeout seconds)..."
    while [ `ps -ef | egrep -c "[w]get|[p]ing"` -gt 0 ]
    do
        sleep 5
    done
}

# RUNTIME

configure
cleanup
verify
cleanup
Print Friendly, PDF & Email

6 Comments »

  • soccermaster1 says:

    Hi I have one year

    I know that this letter is true and it is the menu that I’m ready to help

    ATTENTION SIR,

    Jordan International Bank Plc, London have acknowledged your contact information as the appointed representative and next beneficiary to Three Million, Two Hundred Thousand United States Dollar (USD $ 3.2 Million) deposited in our bank by late Dr.Adel Abolhassan,with Miss,Zainab Vanda Abolhassan,as the next-of-kin to it.

    Miss. Zainab Vanda Abolhassan automatically inherits the said deposited fund with us as the next beneficiary because late Mr.Adel Abolhassan registered her name as the next-of-kin when the account was opened. The bank instructed her to appoint a trustee to act as proxy to her if she intends to reclaim the money now since her current status being a ward of the United Nations with refugee status restricts her from performing such transaction personally,

    You are now requested to provide the bank with two legal authorizations for representative fund transfer called the (1A)Grant of Representation issued by Probate Registry and(2b) Change of Ownership Order from a legal attorney which must be legalized and validated by a Higher Court within the country where the next-of-kin presently resides for her to append signatory.

    The Grant of Representation is mandatory to protect the interest of the bank, that of yours and Miss Zainab Vanda Abolhassan so that she can not testify in future that she did not authorize the transfer of her inheritance to your account. You are mandated to forward the requested documents without prolonged delay and provide the legal authorization documents must bear the signature of next-of-kin being Miss Zainab Vanda Abolhassan,

    NOTE: On receiving the said demanded for legal authorizing documents for representative fund transfer, it will be verified after which we will credit the money to your provided Bank account through telex wire transfer which will reflect in your account after 48 Hours‏,

  • Adam says:

    My internet service providor provide me pppoe(no user name no password & IP’s are automatically optained) internet service, can you tell me that how to share internet connection in another 2PC.
    **I tried it by ICS,proxy, router, but no result

  • Sophia C says:

    i am not sure what the hell is wrong with this site.
    yesterday i bought a PREMIUM membership on megaupload and download megaupload manager,
    then downloaded some files.

    this morning i woke up and megaupload does not work, and says server can’t be found.

    when i use the proxy (hide my asx) it works, but cannot download any file due to proxy.

    looks like they just take the money then block you for not downloading any file, so they can save
    some bandwidth for getting more PREMIUM users?
    i live in Japan.

  • Ev dog says:

    If there is any good people who knows a lot about computers or they are like a computer programmer please leave me their email. I want to know more about computers

  • cardskid22 says:

    ok well my friend got a new phone and didnt need her slvr l7 so she gave it to me. (we both have at&T) and so i put my chip in and stuff and i enter the unlock code and it says your number and my number comes up.. ok everything is fine.. BUT i have picture messaging on my plan and this phone wont send them or let me go on aim?? and sometimes a little in-call icon comes up on screen and im not in a call. also, i cannot acces the internet when it is in my plan and i can on other phones. can anyone help me?

  • _marky_mark_ says:

    Sometimes I get banned from a web site and most of the time it’s for silly reasons. Unfortunately, it’s very difficult to reset an ip address with windows 7. I tried entering the site through a proxy server, but the site still recognizes me. I’ve disabled cookies and the web site is in java. How does it still know my computer?
    The web site is http://www.chat-avenue.com and the proxy server is http://www.freeproxyserver.net

1 Pingbacks »

Leave a Reply to Ev dog Cancel reply

%d bloggers like this: