Fast and asynchronous way of making multiple http requests in JAVA

Question

I have a program that should make really fast http requests. Requests should be made asynchronously so that it won't block the main thread.

So I have created a queue which is observed by 10 separate threads that make http requests. If something is inserted in the queue then the first thread that gets the data will make the requests and process the result.

The queue gets filled with thousands of items so multithreading is really neccessary to get the response as fast as possible.

Since I have alot of code I'll give a short example.

main class

package fasthttp;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;

public class FastHTTP {

    public static void main(String[] args) {
        ExecutorService executor = Executors.newFixedThreadPool(10);

        for (int i = 0; i < 10; i++) {
            LinkedBlockingQueue queue = new LinkedBlockingQueue();
            queue.add("http://www.lennar.eu/ip.php");//for example
            executor.execute(new HTTPworker(queue));
        }
    }

}

FastHTTP class

package fasthttp;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.LinkedBlockingQueue;

public class HTTPworker implements Runnable {

    private final LinkedBlockingQueue queue;

    public HTTPworker(LinkedBlockingQueue queue) {        
        this.queue = queue;
    }

    private String getResponse(String url) throws IOException {

        URL obj = new URL(url);
        HttpURLConnection con = (HttpURLConnection) obj.openConnection();

        StringBuilder response;
        try (BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()))) {
            String inputLine;
            response = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                response.append(inputLine);
            }
        }
        return response.toString();
    }

    @Override
    public void run() {
        while (true) {
            try {
                String data = (String) queue.take();
                String response = getResponse(data);
                //Do something with response
                System.out.println(response);
            } catch (InterruptedException | IOException ex) {
                //Handle exception
            }
        }
    }
}

Is there a better or faster way to make thousands of http requests response processing asynchronously? Speed and performance is what I'm after.

Apache, Jersey and many other projects support asynchronous HTTP requests out of the box. See stackoverflow.com/questions/3142915/… for example. Google for "Java asynchronous HTTP". Your throughput is limited by the number of threads, since each thread does a blocking HTTP request. — JB Nizet
– JB Nizet, Commented Jun 26, 2015 at 8:17

lkallas · Accepted Answer · 2015-09-18 10:36:02Z

0

Answering my own question. Tried Apaches asynchronous http client but after a while I started using Ning's async client and I am happy with it.

answered Sep 18, 2015 at 10:36

lkallas

1,4165 gold badges24 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

navy1978 Over a year ago

Hi Ikallas,basing on the project :github.com/AsyncHttpClient/async-http-client there are different approaches , which one did you use ?

Andrea Ciccotta · Accepted Answer · 2020-11-21 18:19:30Z

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
import java.util.stream.Collectors;

import org.apache.http.client.methods.HttpGet;

import java.util.Iterator;

import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;

public class RestService {
    private final static Executor executor = Executors.newCachedThreadPool();
    private final static CloseableHttpClient closeableHttpClient = HttpClientBuilder.create().build();

    public static String sendSyncGet(final String url) {
        return sendAsyncGet(List.of(url)).get(0);
    }

    public static List<String> sendAsyncGet(final List<String> urls){
        List<GetRequestTask> tasks = urls.stream().map(url ->  new GetRequestTask(url, executor)).collect(Collectors.toList());
        List<String> responses = new ArrayList<>();
        while(!tasks.isEmpty()) {
            for(Iterator<GetRequestTask> it = tasks.iterator(); it.hasNext();) {
                final GetRequestTask task = it.next();
                if(task.isDone()) {
                    responses.add(task.getResponse());
                    it.remove();
                }
            }
            //if(!tasks.isEmpty()) Thread.sleep(100); //avoid tight loop in "main" thread
        }
        return responses;
    }

    private static class GetRequestTask {
        private final FutureTask<String> task;

        public GetRequestTask(String url, Executor executor) {
            GetRequestWork work = new GetRequestWork(url);
            this.task = new FutureTask<>(work);
            executor.execute(this.task);
        }

        public boolean isDone() {
            return this.task.isDone();
        }

        public String getResponse() {
            try {
                return this.task.get();
            } catch(Exception e) {
                throw new RuntimeException(e);
            }
        }
    }

    private static class GetRequestWork implements Callable<String> {
        private final String url;

        public GetRequestWork(String url) {
            this.url = url;
        }
        public String getUrl() {
            return this.url;
        }
        public String call() throws Exception {
            return closeableHttpClient.execute(new HttpGet(getUrl()), new BasicResponseHandler());
        }
    }
}

Collectives™ on Stack Overflow

Fast and asynchronous way of making multiple http requests in JAVA

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related