need help with bizarre java.net.HttpURLConnection behavior

Question

I am attempting to download a jpg using HttpURLConnection and am encountering a very peculiar bug.

Here's the url: http://www.vh1.com/sitewide/promoimages/shows/m/my_antonio/video/supertrailer/seg_1/320x240.jpg

if you open it in a browser you will see the image.

However, when I use HttpURLConnection I don't get the image... What I get is a 301 which, quite strangely, redirects to http://wap.vh1.com

so

    con.setInstanceFollowRedirects(true);
//additional stream code here to go and get the stuff found in con

proceeds to go ahead and download the text from wap.vh1.com, rather than the jpg that you see in the browser.

I'm guessing that there is some header wackiness that's causing this, but I haven't the faintest idea what the host is expecting to see in order to redirect me to the same place as where it's redirecting the browser (and curl and wget and everything else I can think to point at it).

I'm just about ready to shoot myself, so, if you help me you will be preventing my 6 year old daughter from going fatherless.

Could you post your full code you are using to create the connection and fetch the URL? — Doug Porter
– Doug Porter, Commented Nov 23, 2009 at 22:49
there's a whole lot of it and it works for all other urls, so, I'm reluctant to clutter up the post with it... I'll add a few lines that are the most pertinent... — Yevgeny Simkin
– Yevgeny Simkin, Commented Nov 23, 2009 at 22:51
Seems like vh1 think your request is coming from a mobile device - are the other URLs that work for images? — martin clayton
– martin clayton, Commented Nov 23, 2009 at 22:55

ZZ Coder · Accepted Answer · 2009-11-23 22:54:21Z

4

The site redirects you based on user-agent. Add this before you open the connection,

conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.15) Gecko/2009101600 Firefox/3.0.15");

answered Nov 23, 2009 at 22:54

ZZ Coder

75.7k30 gold badges139 silver badges170 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

BalusC Over a year ago

I would just add that every "valid" desktop-client like useragent is allowed here and not explicitly Firefox on MacOS :)

EJB · Accepted Answer · 2009-11-23 23:04:05Z

1

It seems like the server interprets your request as coming from a mobile device, possibly based on the User-Agent header. That's why your redirected to the mobile site. Try setting the User-Agent explicitly.

answered Nov 23, 2009 at 23:04

EJB

2,41315 silver badges15 bronze badges

Comments

miku · Accepted Answer · 2009-11-23 23:01:07Z

0

For more flexibility you can utilize the http commons libraries, which have great debugging support for the wire through log4j ...

Also, user agents and more request parameters can be set easily.

For more information, see their tutorial.

answered Nov 23, 2009 at 23:01

miku

189k47 gold badges314 silver badges317 bronze badges

Comments

Jim Ferrans · Accepted Answer · 2009-11-23 23:25:33Z

0

While I can't help you with your specific problem, here's what I would do:

Download wireshark, sniff the HTTP request sent by your java application
Copy/Paste the request, and run it with telnet (or a tool such as WFetch)
Fiddle with the request headers and see if behavior changes.

(I'd suspect the site screens the request based on the User agent header or something similar)

edited Nov 23, 2009 at 23:25

Jim Ferrans

31k12 gold badges58 silver badges83 bronze badges

answered Nov 23, 2009 at 22:58

nos

231k60 gold badges436 silver badges516 bronze badges

1 Comment

Jim Ferrans Over a year ago

Another tool that can help in this sort of situation is the FireBug plug-in for Firefox.

Jim Ferrans · Accepted Answer · 2009-11-23 23:27:42Z

0

The java.net package doesn't support lot of the needed features out of the box (like automatically saving and sending cookies). Use Apache's httpClient instead

edited Nov 23, 2009 at 23:27

Jim Ferrans

31k12 gold badges58 silver badges83 bronze badges

answered Nov 23, 2009 at 23:02

jutky

3,9836 gold badges35 silver badges45 bronze badges

Collectives™ on Stack Overflow

need help with bizarre java.net.HttpURLConnection behavior

5 Answers 5

1 Comment

Comments

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

1 Comment

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related