I currently wrote my program to use 32 threads and read 1 file per thread (so 32 .txt files). The multi-threading has nothing to do with CPU speed, but making 32 calls to BING's api per second is a lot faster than making 1. Each of the .txt files containing a list of search queries. I create a thread it reads one line at a time from it's file. Is it possible to create all 32 threads and point them to a single .txt file??
1
-
2I'm parsing big text files and I typically do it like this: one producer is doing the I/O and putting in a queue then as many consumers as I want fetch jobs from the queue. In your case your unique producer could read the txt file and put one entry per line in the queue.TacticalCoder– TacticalCoder2011-11-14 18:30:05 +00:00Commented Nov 14, 2011 at 18:30
Add a comment
|
3 Answers
Use the Producer-Consumer pattern. Have only one thread that reads file and pushes each line/command into ArrayBlockingQueue (thread safe read and write) using put().
All other 32 threads should read from the same queue object by calling take(). They will block if queue is empty, which is nice.
This solution is better because the disk is inherently single-threaded so you won't get much by reading the file concurrently.
answered Nov 14, 2011 at 18:31
Tomasz Nurkiewicz
342k72 gold badges713 silver badges680 bronze badges
1 Comment
chrstahl89
Thanks, that makes changing the input a lot easier than having 32 separate files.
You can synchronize the code that handles the file. Each time you write to the file you open, write and close it. Implement using a monitor and it should be fine.
Could also use java util logging as it is thread safe. If you implement a handler then logging api should take care of the thread safety issues.
1 Comment
G_H
There's no writing involved in the question and this doesn't have a thing to do with logging.
It depends on how these threads are implemented. If each has its own reader or input stream on the file, then concurrent reading probably isn't much of a problem. Unless the JVM implementation on the OS implicitly locks a file when an input stream is opened on it and the lock can't be shared across streams.
But you'd be doing a lot of unnecessary work, to be honest. It'd be much better to encapsulate the access to the file in a separate class and pass instances of that to your threads, then have that class do the necessary concurrency handling.
