6

I'm going to wget youtube search page where the URL is horrible. This is example when searching searchkeyword sorted by upload date:

wget http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24

and this one is page 2 of the search result:

wget http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24&page=2

The wget give me misleading page for both pages.

Improve this question

3 Answers 3

Reset to default
9

Wrapping the URL in single quotation marks should do the trick. Also, you will want to give a name to the downloaded file as well (otherwise it will have the weird name that wget guessed from the URL). So

wget 'http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24' -O nice_name.html

Explanation as per laebshade's comment (thanks for reminding me): The many funny characters in the URL will be interpreted by the shell as special commands or other control sequences. From the top of my head, ~ ! # ^ & * will all mean something else to the shell (depends on what shell you are using). You can use \ to escape them one by one, or quote the whole thing.

Improve this answer
2
  • 4
    It would be good to explain the reason for using quotes. The reason it's not giving you the right URL is because the URL contains an '&' (ampersand) character, which your shell interprets as a fork() to background; in other words, your shell sends the first part of the command wget http://youtube.com/results?search_type=videos& to the background, then treats the next part of the "command" as a new one, until it reaches another &, and so on and so forth. Commented Jul 17, 2011 at 14:38
  • @laebshade Very helpful explanation. :) Commented Aug 29, 2011 at 10:01
4

Don't use wget for downloading individual pages. It is better suited for things like recursive downloads of whole sites. Try curl instead.

Also, make sure you quote your arguments on the command line so that you don't run into shell globing problems with characters like ?.

curl 'http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24&page=2' \
    > results.html
Improve this answer
2
  • Instead of > results.html you can also use curl http:://url.com/randompage -o results.html Commented Jul 17, 2011 at 11:49
  • Perfect: Curl obviously has different magic embedded. ^wget^curl made it work in my case (pulling a JSON file of a google spreadsheet.) Hat's off to you Good Sir! Commented Jan 2, 2017 at 18:19
3

Use youtube-dl for downloading Youtube videos. It's a epic python script. It req's Python. If you have Ubuntu, install with this command: sudo apt-get install youtube-dl otherwise you can directly download it from the website: http://rg3.github.com/youtube-dl/

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.