Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Commands & Shells

Wget examples and scripts

Submitted by on November 27, 2005 – 3:19 pm 27 Comments

Wget is a command-line Web browser for Unix and Windows. Wget can download Web pages and files; it can submit form data and follow links; it can mirror entire Web sites and make local copies. Wget is one of the most useful applications you would ever install on your computer and it is free.

You can download the latest version of Wget from the developers home page. Precompiled versions of Wget are available for Windows and for most flavors of Unix. Many Unix operating system have wget pre-installed, so type which wget to see if you already have it.

Wget supports a multitude of options and parameters. This variety may be confusing to people unfamiliar with Wget. You can view the available Wget options by typing wget –help or on a Unix box type man wget.

Here are a few useful examples on how to use Wget:

1) Download main page of Yahoo.com and save it as yahoo.htm

2) Use Wget with an HTTP firewall:

Set proxy in Korn or Bash shells

Set proxy in C-shell

Run wget for anonymous proxy

Run wget for proxy that requires authentication

3) Make a local mirror of Wget home page that you can browse from your hard drive

Here are the options we will use:

-m to mirror the site
-k to make all links local
-D to stay within the specified domain
–follow-ftp to follow FTP links
-np not to ascend to the parent directory

The following two options are to deal with Web sites protected against automated download tools such as Wget:

-U to mascarade as a Mozilla browser
-e robots=off to ignore no-robots server directives

4) Download all images from Playboy site

Here are the options we will use:

-r for recursive download
-l 0 for unlimited levels
-t 1 for one download attempt per link
-nd not to create local directories
-A to download only files with specified extentions

5) Web image collector

The following Korn-shell script reads from a list of URLs and downloads all images found anywhere on those sites. The images are processed and all images smaller than a certain size are deleted. The remaining images are saved in a folder with named after the URL. The url_list.txt file contains one URL per line.

This script was originally written to run under AT&T UWIN on Windows, but it will also work in any native Unix environment that has Korn shell.

6) Wget options

Print Friendly, PDF & Email

27 Comments »

  • Joey 01 says:

    how to use/write perl script in linux copy file from windows?

    I need to execute perl script from linux server copy a file from windows drive folder.
    example like:
    the file is located at c:abcdxyz.log
    i need to copy this xyz.log to a linux/unixbase server directory /var/opt/abc/xyz.log by using perl script on linux/unix base server.

    how to write the perl script to perform the activity above?
    any example?

    Pls help! Thank You.

  • Sonny says:

    I seen many operating systems in Linux that are free. What is the best and takes up the least amount of space. Please also tell me the recommended space needed for the os.

  • John G says:

    computers have a different operating system. Other choices are also available, such as UNIX and Linux. Why do you think Windows is so prevalent? What do the other systems have to offer that Windows does not? Who uses the other systems?

4 Pingbacks »

Leave a comment

Add your comment below. You can also subscribe to these comments via RSS

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url=""> 

This is a Gravatar-enabled weblog. To get your own globally-recognized-avatar, please register at Gravatar