How to implement a multithreaded pool in Java

Question

I have a scenario where I use threads.

Firstly I have a folder where there are files which get updated frequently. So, I wrote a thread which reads the contents of the folder and writes the file names to a static list and updates the list if new files come in.

Secondly i wrote another thread which takes the file names from the list and do some processing with the files.

These two threads run continuously, one checking for new files, one processing the new files.

Now I need to process three files at a time with three threads running. When one thread completes processing another thread takes another file name from the list and starts the process.

So I need some mechanism to have three threads and checking them whether they are alive or not and accordingly starts a new thread and the file list also gets updated frequently.

I also looked into ExecutorService but while the list get updated I could not provide it updated list.

Thanks, Sandeep

Not very clear, I'm afraid. Why three, and not an arbitrary number? Why check threads to see whether they're alive? — Vance Maverick
– Vance Maverick, Commented Feb 25, 2011 at 4:38
I meant to add -- it sounds like the natural data structure to use is a queue ( download.oracle.com/javase/6/docs/api/java/util/Queue.html ). To take advantage of this, you would have a consumer thread reading from the queue, doing work when it succeeded in taking items from it. — Vance Maverick
– Vance Maverick, Commented Feb 25, 2011 at 4:39

SimonC · Accepted Answer · 2011-02-25 04:48:38Z

2

Building on the existing answers, your code would look something like:

    final ExecutorService executor = Executors.newFixedThreadPool(2);

    final FilePoller poller = ...
    final FileProcessor processor = ...

    new Thread(new Runnable() 
      { 
        @Override
        public void run() 
        {
          while (true)
          {
            final File file = poller.pollForFile();
            executor.submit(new Runnable() { public void run() { processor.processFile(file); } } );
          }
        }
      });

Assuming your processors can keep up with the poller this would be fine, otherwise you'd want to put in some throttling mechanism before submitting to the executor.

answered Feb 25, 2011 at 4:48

SimonC

6,7481 gold badge27 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Cornelius Dol · Accepted Answer · 2011-02-25 04:38:41Z

0

Don't use a list; instead use a java.util.concurrent.ThreadPoolExecutor and just drop Runnable's representing the file to be processed into the executor instead of putting them into your global list.

answered Feb 25, 2011 at 4:38

Cornelius Dol

64.2k26 gold badges142 silver badges190 bronze badges

Comments

Peter Lawrey · Accepted Answer · 2011-02-25 09:39:00Z

0

Similar to @SimonC's suggestion but instead of a really long comment I have an answer.

final FilePoller poller = ...
final FileProcessor processor = ...

final ExecutorService executor = Executors.newFixedThreadPool(4);

executor.submit(new Runnable() { 
    public void run() {
        final File file = poller.pollForFile();
        executor.submit(new Runnable() { public void run() { processor.processFile(file); } } );
        // repeat, but wait for a free thread.
        executor.submit(this);
    }
  });
 // to stop the whole thing
 executor.shutdown();

answered Feb 25, 2011 at 9:39

Peter Lawrey

535k83 gold badges770 silver badges1.2k bronze badges

Comments

devsprint · Accepted Answer · 2011-02-25 13:15:40Z

0

How about watching for changes in the folder and spawn a thread/file, assuming that the notification change is giving you a list of changes in the folder?

answered Feb 25, 2011 at 13:15

devsprint

6815 silver badges19 bronze badges

Collectives™ on Stack Overflow

How to implement a multithreaded pool in Java

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related