Wget and User-Agent Header
Posted: Tue Sep 08, 2009 3:09 am
As you may already know, Wget is a popular (particularly in the Unix world) command-line downloader and Web crawler application. You can read more about Wget in one of my earlier posts on the subject. One issue with Wget is that some sites block it from accessing their content. This is usually done by adding Wget to the robots.txt on the Web server and by configuring the server to reject requests with the user-agent header containing “wget”...
Read more: http://www.krazyworks.com/wget-and-user-agent-header/
Read more: http://www.krazyworks.com/wget-and-user-agent-header/