1

JSoup seems to work with just about all URLs I've tried but this one gives me the 400 error.

String url = "http://localad.walmart.com?storeref=3008&forceview=y";

Response response = Jsoup.connect(url.replaceAll(" ", "%20"))
    .method(Method.GET)
    .userAgent("Mozilla")
    .followRedirects(false)
    .timeout(5000)
    .data("pragma", "no-cache")
    .execute(); 

Error I get is:

Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=400, URL=http://localad.walmart.com?storeref=3008&forceview=y&pragma=no-cache
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
1
  • can you help with that situation? Do you have answer? Commented Dec 22, 2017 at 17:35

3 Answers 3

3

It is a little annoying when JSoup throws a fatal exception on a 400 Bad Request as this completely terminates any process running, even when wrapped in a try/catch. But hey.

There is a solution to this, on your initial connection URL, append the method;

.ignoreHttpErrors(true)

For example;

Jsoup.connect(url).ignoreHttpErrors(true).execute().statusCode();

Which then gives you the official '400' status code rather than throwing a fatal exception.

I know this is an old post, but posting for reference as I came across this thread while looking for a solution to this exact thing.

Sign up to request clarification or add additional context in comments.

Comments

0

400 is Bad Request.

You should try URLEncoder.encode(url, "UTF-8") instead of using replaceAll.

1 Comment

I did try that but because the URL has no protocol that throws a MalformedURLException
0

Moved on without relying on JSoup for resolving the intermediate (redirected URL). I needed the final redirect URL (which JSoup had no problems working with), so used the following code to get that.

import java.net.URI;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.ProtocolException;
import org.apache.http.impl.client.DefaultRedirectStrategy;
import org.apache.http.protocol.HttpContext;

public class MyRedirectHandler extends DefaultRedirectStrategy {

    public URI lastRedirectedUri;

    @Override
    public boolean isRedirected(HttpRequest request, HttpResponse response, 
        HttpContext context) {
    try {
        return super.isRedirected(request, response, context);
     } catch (ProtocolException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return false;
    }

    @Override
    public URI getLocationURI(HttpRequest request, HttpResponse response, HttpContext context)
        throws ProtocolException {

        lastRedirectedUri = super.getLocationURI(request, response, context);
        return lastRedirectedUri;
    }

Invoking code:

    DefaultHttpClient httpclient = new DefaultHttpClient();
    String url2 = "http://localad.walmart.com/walmart/new_user_entry.aspx?storeref=3008&forceview=y";
    MyRedirectHandler handler = new MyRedirectHandler();
    httpclient.setRedirectStrategy(handler);

    HttpGet get = new HttpGet(url2);
    httpclient.execute(get);

    String lastUrl = url2;
    if (handler.lastRedirectedUri != null) {
        lastUrl = handler.lastRedirectedUri.toString();
    }

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.