Clojure : Get "OutOfMemoryError Java heap space" when parsing big log file

Question

all.
I want to parse big log files using Clojure.
And the structure of each line record is "UserID,Lantitude,Lontitude,Timestamp".
My implemented steps are:
----> Read log file & Get top-n user list
----> Find each top-n user's records and store in separate log file (UserID.log) .

The implement source code :

;======================================================
(defn parse-file
  ""
  [file n]
  (with-open [rdr (io/reader file)]
    (println "001 begin with open ")
    (let [lines (line-seq rdr)
          res (parse-recur lines)
          sorted
          (into (sorted-map-by (fn [key1 key2]
                                 (compare [(get res key2) key2]
                                          [(get res key1) key1])))
                res)]
      (println "Statistic result : " res)
      (println "Top-N User List : " sorted)
      (find-write-recur lines sorted n)
      )))

(defn parse-recur
  ""
  [lines]
  (loop [ls  lines
         res {}]
    (if ls
      (recur (next ls)
               (update-res res (first ls))) 
      res)))

(defn update-res
  ""
  [res line]
  (let [params (string/split line #",")
        id     (if (> (count params) 1) (params 0) "0")]
    (if (res id)
      (update-in res [id] inc)
      (assoc res id 1))))

(defn find-write-recur
  "Get each users' records and store into separate log file"
  [lines sorted n]
  (loop [x n
         sd sorted
         id (first (keys sd))]
    (if (and (> x 0) sd)
      (do (create-write-file id
                             (find-recur lines id))
          (recur (dec x)
                 (rest sd)
                 (nth (keys sd) 1))))))

(defn find-recur
  ""
  [lines id]
  (loop [ls lines
           res []]
    (if ls
      (recur (next ls)
               (update-vec res id (first ls)))
      res)))

(defn update-vec
  ""
  [res id line]
  (let [params (string/split line #",")
        id_        (if (> (count params) 1) (params 0) "0")]
        (if (= id id_ )
          (conj res line)
          res)))

(defn create-write-file
  "Create a new file and write information into the file."
  ([file info-lines]
   (with-open [wr (io/writer (str MAIN-PATH file))]
     (doseq [line info-lines] (.write wr (str line "\n")))
     ))
  ([file info-lines append?]
   (with-open [wr (io/writer (str MAIN-PATH file) :append append?)]
     (doseq [line info-lines] (.write wr (str line "\n"))))
   ))
;======================================================

I tested this clj in REPL with command (parse-file "./DATA/log.log" 3), and get the results:

Records-----Size-----Time----Result
1,000-------42KB-----<1s-----OK
10,000------420KB----<1s-----OK
100,000-----4.3MB----3s------OK
1,000,000---43MB-----15s-----OK
6,000,000---258MB---->20M----"OutOfMemoryError Java heap space java.lang.String.substring (String.java:1913)"

======================================================
Here is the question:
1. how can i fix the error when i try to parse big log file , like > 200MB
2. how can i optimize the function to run faster ?
3. there are logs more than 1G size , how can the function deal with it.

I am still new to Clojure, any suggestion or solution will be appreciate~
Thanks

jgriego · Accepted Answer · 2013-11-20 04:27:05Z

9

As a direct answer to your questions; from a little Clojure experience.

The quick and dirty fix for running out of memory boils down to giving the JVM more memory. You can try adding this to your project.clj:
```
:jvm-opts ["-Xmx1G"] ;; or more
```
That will make Leiningen launch the JVM with a higher memory cap.
This kind of work is going to use a lot of memory no matter how you work it. @Vidya's suggestion ot use a library is definitely worth considering. However, there's one optimization that you can make that should help a little.

Whenever you're dealing with your (line-seq ...) object (a lazy sequence) you should make sure to maintain it as a lazy seq. Doing next on it will pull the whole thing into memory at once. Use rest instead. Take a look at the clojure site, especially the section on laziness:

(rest aseq) - returns a possibly empty seq, never nil

[snip]

a (possibly) delayed path to the remaining items, if any

You may even want to traverse the log twice--once to pull just the username from each line as a lazy-seq, again to filter out those users. This will minimize the amount of the file you're holding onto at any one time.
Making sure your function is lazy should reduce the sheer overhead that having the file as a sequence in memory creates. Whether that's enough to parse a 1G file, I don't think I can say.

answered Nov 20, 2013 at 4:27

jgriego

1,2668 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Vidya Over a year ago

I was thinking boosting the memory would just treat the symptom rather than cure the disease, but your point on maintaining laziness is great.

Community · Accepted Answer · 2017-05-23 12:24:12Z

2

You definitely don't need Cascalog or Hadoop simply to parse a file which doesn't fit into your Java heap. This SO question provides some working examples of how to process large files lazily. The main point is you need to keep the file open while you traverse the lazy seq. Here is what worked for me in a similar situation:

(defn lazy-file-lines [file]
  (letfn [(helper [rdr]
                  (lazy-seq
                    (if-let [line (.readLine rdr)]
                      (cons line (helper rdr))
                      (do (.close rdr) nil))))]
         (helper (clojure.java.io/reader file))))

You can map, reduce, count, etc. over this lazy sequence:

(count (lazy-file-lines "/tmp/massive-file.txt"))
;=> <a large integer>

The parsing is a separate, simpler problem.

edited May 23, 2017 at 12:24

CommunityBot

11 silver badge

answered Nov 20, 2013 at 16:27

JohnJ

4,8432 gold badges30 silver badges40 bronze badges

Comments

Vidya · Accepted Answer · 2013-11-20 04:03:38Z

0

I am also relatively new to Clojure, so there are no obvious optimizations I can see. Hopefully others more experienced can offer some advice. But I feel like this is simply a matter of the data size being too big for the tools at hand.

For that reason, I would suggest using Cascalog, an abstraction over Hadoop or your local machine using Clojure. I think the syntax for querying big log files would be pretty straightforward for you.

answered Nov 20, 2013 at 4:03

Vidya

30.4k7 gold badges48 silver badges73 bronze badges

Collectives™ on Stack Overflow

Clojure : Get "OutOfMemoryError Java heap space" when parsing big log file

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related