8

I have a list of approximately a thousand Java objects and am iterating a List container to process them, the same processing for every object. This sequential approach is taking a lot of time for processing, so I want to attempt to speed it up with parallel processing. I checked Java executor frameworks but got stuck.

I thought of one approach to implement my requirement. I want to specify some minimum fixed number of objects to be processed by each thread so that each does its processing in a quick manner. How can I achieve this? Should I use another approach?

For example:

List<Object> objects = new List<Object>(); 

for (Object object : objects) {
  // Doing some common operation for all Objects 
}
Improve this question

3 Answers 3

Reset to default
12

There are many options for processing a list in parallel:

Use a parallel stream:

objects.stream().parallel().forEach(object -> {
    //Your work on each object goes here, using object
})

Use an executor service to submit tasks if you want to use a pool with more threads than the fork-join pool:

ExecutorService es = Executors.newFixedThreadPool(10);
for(Object o: objects) {
    es.submit(() -> {
        //code here using Object o...
    }
}

This preceding example is essentially the same as the traditional executor service, running tasks on separate threads.

As an alternative to these, you can also submit using the completable future:

//You can also just run a for-each and manually add each
//feature to a list
List<CompletableFuture<Void>> futures = 
    objects.stream().map(object -> CompletableFuture.runAsync(() -> {
    //Your work on each object goes here, using object
})

You can then use the futures object to check the status of each execution if that's required.

Improve this answer
Sign up to request clarification or add additional context in comments.

17 Comments

The Executor service you have mentioned contains a pool of 10 threads. In my case suppose I am having a 1 thousand objects in a list or can vary . The each thread will process 1 object and for 10 threads will process 10 orders simultaneously. What would be the good approach to decide to create a pool of threads? As my list elements will vary
@SSV Then it's better NOT to batch in sub-lists if you know it takes some time to process each entry. If processing per-element takes exactly the same amount of time, it could make sense (then again you would have to use as many threads as there are cpus max), but the better choice would be not to batch as each element's processing time would probably differ.
@user2862544 that will have to be tested, although I believe the bottom two will have better performance given that they offer more control over thread pool size. I'd personally use the last one, but calling runAsync with an executor service (second argument). The first one is OK too, but it uses the default fork-join pool.
@user2862544 One shouldn't need to create a thread pool each time. There should be one shared thread pool with sufficient number of threads in it. Then all concurrent tasks will be submitted to the same executor service. It will just be important to carefully choose pool size and test according to anticipated load.
@user2862544 Absolutely. abc() will maintain the list iteration and will end up returning consistent results for its input. But what matters more is that you make sure that the maximum number of tasks to be submitted to the executor service at any given time doesn't cause an abnormal wait queue. And for this, you need to test in order to pick the right pool sizes.
|
9

You can use a ThreadPoolExecutor, it will take care of load balance. Tasks will be distributed on different threads.

Here is an example:

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Test {

    public static void main(String[] args) {

        // Fixed thread number
        ExecutorService service = Executors.newFixedThreadPool(10);

        // Or un fixed thread number
        // The number of threads will increase with tasks
        // ExecutorService service = Executors.newCachedThreadPool(10);

        List<Object> objects = new ArrayList<>();
        for (Object o : objects) {
            service.execute(new MyTask(o));
        }

        // shutdown
        // this will get blocked until all task finish
        service.shutdown();
        try {
            service.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    public static class MyTask implements Runnable {
        Object target;

        public MyTask(Object target) {
            this.target = target;
        }

        @Override
        public void run() {
            // business logic at here
        }
    }
}
Improve this answer

10 Comments

Thread pooling approach looks good but in my case Like I am having list of 1000 objects and creating a pool of large number of thread would be a gud approach to process each object?
What will you do to every single object?
I am thinking to create a sublist of elements of parent list like I am having a list of 1 thousand objects and like each sublist will have some fixed number of objects and then I will pass each sublist to execute method of executer service so that each thread process more number of objects rather than processing only one. Would this approach be good,
I need to fetch one attribute of an each object and pass that attribute of some third party rest api and will get the response and then on th basis of response , I will update the same attribute in each object
@SSV Seems the majority of the procedure is waiting for response from the third party api, then executor is good to handle this kind of asynchronous job. You do not need create sublists. It does not help. The threads in the executor will get resued. What you need to do is find a suitable thread number. Executors.newFixedThreadPool(thread number);
|
2

Split list into multiple sub-lists and use multi threading to process each sub-lists parallel.

public class ParallelProcessListElements {
    public void processList (int numberofthreads,List<Object>tempList, 
            Object obj, Method method){

                final int sizeofList=tempList.size();
                final int sizeofsublist = sizeofList/numberofthreads;
                List<Thread> threadlist = new ArrayList<Thread>();

                for(int i=0;i<numberofthreads;i++) {
                    int firstindex = i*sizeofsublist;
                    int lastindex = i*sizeofsublist+sizeofsublist;
                    if(i==numberofthreads-1)
                        lastindex=sizeofList;

                    List<Object> subList=tempList.subList(firstindex,lastindex );

                    Thread th = new Thread(()->{
                                try{method.invoke(obj, subList);}catch(Exception e) {e.printStackTrace();}
                            });

                    threadlist.add(th);
                }

                threadlist.forEach(th->{th.start();try{Thread.sleep(10);}catch(Exception e) {}});
    }

}


public class Demo {
    public static void main(String[] args) {

        List<Object> tempList= new ArrayList<Object>();
        /**
         * Adding values to list... For Demo purpose..
         */
        for(int i=0;i<500;i++)
            tempList.add(i);

        ParallelProcessListElements process = new ParallelProcessListElements();
        final int numberofthreads = 5;
        Object obj = new Demo();
        Method method=null;

        try{ method=Demo.class.getMethod("printList", List.class);}catch(Exception e) {}
        /**
         * Method Call...
         */
        process.processList(numberofthreads,tempList,obj,method);
    }

    public void printList(List<Integer>list) {
        /**
         * Business logic to process the list...
         */
        list.forEach(item->{
            try{Thread.sleep(1000);}catch(Exception e) {}
            System.out.println(item);
            });
    }
}
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.