wget huh?

default

wget -O/dev/null -d http://www.freebsd.org
View current headers, such as user agent

wget -i file-full-of-links.txt
wget each link in a file

wget http://somesite.there/a-folder/thisfile-{01..11}.mp3
Download files name thisfile-01.mp3 through thisfile-11.mp3 (with bash 4.0 and greater)


wget -P ../wgotted/ http://www.someWebsite.com/somePage.htm
Download a file to a specific folder

wget -P ./wgotted/ http://www.thatWebsite.com/pics/001S12.jpg thisName.jpg
Save to a specific folder as as choosen name

mirror page
wget -np -mk http://www.getme.com/thisFolder/
Mirror site from specific folder.
-np = no pareent
-m = mirror
-k = convert links relative to file structure 
-U = set user-agent (pretend to be another program)

wget -np -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' -mk http://www.some-chosen-site.com/folder/
If you get a 403 Forbidden error, you can try -U with
a user-agent of a browser that works to see the site.

wget -mirror -w 4 -p --convert-links -P ../wgotted/ruby_book/ http://www.rubycentral.com/book/

wget -mirror -w 4 -p --convert-links -P http://www.douzhe.com/down/ebooks/

wget -mirror -w 4 -p --convert-links -P ../wgotted/ebooks/douzhe http://www.douzhe.com/down/ebooks/

wget -r -l1 -H -P ../wgotted/ http://yeahyeah.com/sites/joey/excelReader/fetchPics.html

wget -r -nd -l2 -H --no-check-certificate -P ./wgotted/sites/ http://ok.com/fetchPics.php

wget -E -H -k -K -p -nd -o sedking-log.txt http://www.somepage.url
Grab all into single page, everything in one directory, best to make
a new directory and wget from it


Examples of trying to save files as a name with wget
----------------------------------------------------
paste -d: urls.txt names.txt | while read line ; do echo $line| awk -F: '{print "wget "$1" -O "$2}'| sh ; done

paste -d: urls.txt names.txt | while IFS=: read url file ; do wget "$url" -O "$file"; done

paste urls.txt names.txt | while read url file ; do wget "$url" -O "$file"; done

paste -d: urls.txt names.txt | while read line ; do echo $line| awk -F: '{print "wget http://"$1" -O "$2}'| sh ; done

awk -F: '{print "wget http://"$2" -O "$3}'
awk -F: '{print "echo wget http://"$2" -O "$3}'
awk -F: '{print "echo wget http://"$2" -O "$3"; echo bla-bla-bla"}'
----------------------------------------------------

default