Problems handling HTTP 302 in Java with HttpURLConnection

Question

I am trying to create a code to get the final redirected URL for the websites needed, but I am having trouble when handling the http response 302. It seems like it is not doing the request properly, otherwise I do not understand what's wrong. I have tested it forcing redirection with twitter.com and facebook.com and works great (responses 301 though), however when trying the URL indicated in "urlin" goes in an endless loop of http 302 responses.

This is my first post so I am not able to post the printout because it gives more than two links.

Here's the code:

/**
 * @param args
 */
public static void main(String[] args) {
    String urlin = "http://feeds.nashuatelegraph.com/~r/news/breaking/~3/jxDTXgSDSGc/jpmorgan-ex-workers-charged-in-london-whale-loss.html";
    String url = new String();
    try{
        System.out.println("URL to redirect: "+urlin);
        int iteration = 0;
        //Preparamos la conexión
        HttpURLConnection con =(HttpURLConnection) new URL(urlin).openConnection();
       // con.setRequestProperty("User-Agent", "Mozilla 5.0");
        con.setReadTimeout(20000);
        con.setInstanceFollowRedirects(false);

        //Definimos un booleano que hara de flag
        boolean redirect = true;
        //Iniciamos la busqueda de URL final
        while(redirect){
            System.out.println("\nIteration number: "+ ++iteration);
            con.connect();
            System.out.println("Connected URL: "+con.getURL().toString());
            int status = con.getResponseCode();
            System.out.println("status: "+status);
            //Tratamos el codigo de respuesta obtenido
            if (status != HttpURLConnection.HTTP_OK) {
                if (status == HttpURLConnection.HTTP_MOVED_TEMP
                        || status == HttpURLConnection.HTTP_MOVED_PERM
                        || status ==    HttpURLConnection.HTTP_SEE_OTHER){
                    redirect = true;
                    //Capturamos la nueva URL
                    String newUrl =     con.getHeaderField("location");
                    //Obtenemos la cookie por si se necesita
                    String cookies =    con.getHeaderField("Set-Cookie");
                    System.out.println("Cookies: "+cookies);
                    //Reabrimos la conexión
                    con = (HttpURLConnection) new URL(newUrl).openConnection();
                    if(cookies!=null){
                     con.setRequestProperty("Cookie", cookies);
                    }
                }
                //Tratamos los errores 400 y 404
                if (status == HttpURLConnection.HTTP_NOT_FOUND ||   status == HttpURLConnection.HTTP_BAD_REQUEST){
                    throw new Exception("Error 400 o 404"); 
                }
            } else {
                redirect = false;
                //Obtenemos la URL final
                url = con.getURL().toString();
            }
        }
    } catch (SocketTimeoutException e) {
        System.out.println("Se ha producido un timeout con URL: "+urlin);
    } catch (UnknownHostException e) {
        System.out.println("Direccion URL desconocida: "+urlin);
        e.printStackTrace();
    } catch (IOException e) {
        System.out.println("Error IOException al procesar el registro URL: "+urlin);
        e.printStackTrace();
    } catch (Exception e) {
        System.out.println("Error al procesar el registro URL: "+urlin);
        e.printStackTrace();
    } 
    if(!url.equals("")) {
        System.out.println("URL final: "+url);
    } else {
        System.out.println("URL final: "+urlin);    
    } 
}

I will appreciate all the advice you can give.

check this stackoverflow.com/questions/9143438/302-found-response — upog
– upog, Commented Sep 9, 2013 at 15:04

Carlo Pellegrini · Accepted Answer · 2013-09-09 15:52:34Z

0

Seems that the site you are scraping does multiple redirections, using cookies to identify the actual step.

In your code, you are intercepting only the Set-Cookie header, but you are discarding previously set cookies (i.e the cookies set by response n-2).

Also, the site does a http/https switch, which you might need to consider in order to send the appropriate cookie set.

answered Sep 9, 2013 at 15:52

Carlo Pellegrini

5,68643 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2761746 Over a year ago

Thanks for the answer! Handling this issues with the cookies is something I haven't ever done. Is there any example I can look for?? I haven't been able to find much about it.

Collectives™ on Stack Overflow

Problems handling HTTP 302 in Java with HttpURLConnection

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related