Java multithreading issue - Consumer/Producer pattern

Question

Have tried to resolve this issue but it is proving difficult.

I have two threads, one is a producer, and the other a consumer. All in separate classes. The two threads run independently. The producer is pooling about 8-12 folders for inputs. It then translates all the files concurrently and place in a shared folder called 'readyToLoad'. Once the producer thread finishes, the consumer threads now go into the 'readyToLoad' folder and start processing the translated documents.

Now to the issue, while the consumer is processing the translated documents, it wouldn't allow the producer to put in more translated files into the 'readyToLoad' folder.

My question is, how do I prevent the consumer from locking the 'readyToLoad' folder? How can I manage this situation?

Apologies for the long text but I feel it will help understand where I'm having issues. Thanks for your help all.

UPDATE : added the consumer code (the one that performs loading and locks the file while loading is going on)

public class LoadManager {
protected static final Logger logger = KCLogger.getLogger();
 ArrayList<Loader> threads = new ArrayList<Loader>();
 KCBLConfig config;
 private static final ExecutorService service = Executors.newFixedThreadPool(10);
 public LoadManager(KCBLConfig config) {
    this.config = config;
 }

public void start() throws Exception {
    logger.log(Level.INFO, "Starting loading threads.");
    try {
        TreeMap<String, ConnectionHandler> connectionHandlers = config.getConnectionHandlers();
        Iterator i = connectionHandlers.keySet().iterator();
        while (i.hasNext()) {

            ConnectionHandler connectionHandler = connectionHandlers.get((String) i.next());

            for (LoadFolder loadFolder : connectionHandler.getKcXMLFolders()) {
               Loader loader = new Loader(loadFolder.getId(), loadFolder, config.getConnectionHandlers());
               Thread loaderThread = new Thread(loader);
                loaderThread.start();   
               //service.submit(loader);
              // service.shutdown();
               //service.awaitTermination(1, TimeUnit.MILLISECONDS);
                threads.add(loader);
            }
        }
    } catch (Exception e) {
        logger.log(Level.SEVERE, "There was an error starting the loaders. Stopping all loader threads.", e);
        this.stop();
        throw e;
    }
    logger.log(Level.INFO, "All loader threads created. There are " + threads.size() + " loader threads");
}

If I understans correctly, the consumer is locking the folder, and you don't want that. Then show us the code that the consumer uses to get files from this folder. I don't see how it could lock a folder without specific code to do that. — JB Nizet
– JB Nizet, Commented Jul 19, 2013 at 9:47

duffymo · Accepted Answer · 2013-07-19 09:34:04Z

2

The Java producer/consumer idiom is usually done with a BlockingDeque now.

I'd have separate folders for producers and consumers. Why take on that complexity?

answered Jul 19, 2013 at 9:34

duffymo

310k46 gold badges376 silver badges571 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Albert Over a year ago

you mean having a separate folder for producer to translate to e.g 'translatedFolder'? and another folder 'readyToUpload' folder for the consumer to process the translated files? And another function that performs a copy of every translated file into the 'readyToUpload' directory? Just trying to understand what you meant by separate folders..

duffymo Over a year ago

I'd have an in and an out box. I don't care what you name them.

Albert Over a year ago

Yes I do have these two folders. and that's exactly the name given to them.

duffymo Over a year ago

Each thread should only deal with one folder. Sounds like you want two threads to access the same one. I'd find a way to isolate them completely. Otherwise, what's the point?

JB Nizet Over a year ago

@duffymo: I'm not sure to follow your reasoning. The consumer has to get what the producer has produced somehow. So, if what the producer produces is files, the consumer has to read them somehow. How would having two separate folders help?

|

marc3l · Accepted Answer · 2013-07-19 10:00:43Z

0

You can just use a Monitor in Java http://en.wikipedia.org/wiki/Monitor_(synchronization) with the synchronized methods.

answered Jul 19, 2013 at 10:00

marc3l

2,6257 gold badges40 silver badges65 bronze badges

Comments

JB Nizet · Accepted Answer · 2013-07-19 10:26:58Z

0

If I understand correctly, you don't have any notification mechanism between the producer and the consumer. The producer stores files in a folder, and the consumer has an infinit loop which reads all the files from this folder and moves them somewhere else.

That's the problem. Here's how I would solve it.

The producer(s) and consumer(s) should all share a single BlockingQueue instance. Each time a producer has finished producing a file (the file is completely written to the shared folder), it adds the name of the file to the blocking queue.

The consumer(s) take file names from the queue. This is a blocking operation, so the consumer is blocked until a file name is available. When a consumer has got a file name from the queue, it reads this file (and this file only), and does whatever he wants with it.

This has several advantages:

no risk of having one file being accessed by a producer and a consumer at the same time
you can have as many producers and consumers you want
the consumers don't scan the directory for nothing endlessly
you can set a bound to the queue to avoid producing too many files, and let the chance for consumers to keep up with the pace of the producers.

answered Jul 19, 2013 at 10:26

JB Nizet

694k94 gold badges1.3k silver badges1.3k bronze badges

6 Comments

Albert Over a year ago

Have actually thought of BlockingQueue. I instantiated it somewhere in the code when I was doing testing, may be I didn't implement it correctly... I think there might be issue still because the producer is in a loop as well, continually pooling dirs and submitting to the shared folder.. is the block going to make the consumer stop immediately producer is ready to chunk files into the sharedDirectory?

JB Nizet Over a year ago

Could you rephrase the question? I don't understand it.

Albert Over a year ago

Sure. My question is, will the producer still able to write into the sharedFolder while the consumer is performing operation? i.e producer translating and putting files concurrently into the sharedFolder while the consumer is uploading the translated file..

JB Nizet Over a year ago

I don't see any reason why it couldn't. Reading a file and writing another file in parallel in the sam folder is supported by all the OSes I know about.

Albert Over a year ago

Brilliant! One more thing, do you have any example of BlockingQueue implementation? It doesnt have to tailor my requirement but if it does, then that's awesome :) Thanks for your help so far

|

Collectives™ on Stack Overflow

Java multithreading issue - Consumer/Producer pattern

3 Answers 3

6 Comments

Comments

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related