Lazy concatenation of sequence in Clojure

Question

Here's a beginner's question: Is there a way in Clojure to lazily concatenate an arbitrary number of sequences? I know there's lazy-cat macro, but I can't think of its correct application for an arbitrary number of sequences.

My use case is lazy loading data from an API via paginated (offseted/limited) requests. Each request executed via request-fn below retrieves 100 results:

(map request-fn (iterate (partial + 100) 0))

When there are no more results, request-fn returns an empty sequence. This is when I stop the iteration:

(take-while seq (map request-fn (iterate (partial + 100) 0)))

For example, the API might return up to 500 results and can be mocked as:

(defn request-fn [offset] (when (< offset 500) (list offset)))

If I want to concatenate the results, I can use (apply concat results) but that eagerly evaluates the results sequence:

(apply concat (take-while seq (map request-fn (iterate (partial + 100) 0))))

Is there a way how to concatenate the results sequence lazily, using either lazy-cat or something else?

The lazy-cat macro evaluates each argument only as required. — Thumbnail
– Thumbnail, Commented Oct 27, 2014 at 19:04
Do you really need to concatenate the results, or do you just want to consume them lazily? — Diego Basch
– Diego Basch, Commented Oct 27, 2014 at 19:12
Having the results sequence concatened would make processing them easier. But it's not a strict requirement. I can also map over the results pages. — Jindřich Mynarz
– Jindřich Mynarz, Commented Oct 27, 2014 at 19:17

Alex · Accepted Answer · 2014-10-27 19:13:29Z

11

For the record, apply will consume only enough of the arguments sequence as it needs to determine which arity to call for the provided function. Since the maximum arity of concat is 3, apply will realize at most 3 items from the underlying sequence.

If those API calls are expensive and you really can't afford to make unnecessary ones, then you will need a function that accepts a seq-of-seqs and lazily concatenates them one at a time. I don't think there's anything built-in, but it's fairly straightforward to write your own:

(defn lazy-cat' [colls]
  (lazy-seq
    (if (seq colls)
      (concat (first colls) (lazy-cat' (next colls))))))

answered Oct 27, 2014 at 19:13

Alex

14k1 gold badge40 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jindřich Mynarz Over a year ago

Thank you for the clarification on apply and a suggestion for lazy-cat'.

pestrella Over a year ago

nice answer; btw, when using next this function eagerly evaluated 2 items from colls even when only requiring 1 evaluation, e.g. (first (lazy-cat' aseq)). Using rest instead gave the expected behaviour.

Collectives™ on Stack Overflow

Lazy concatenation of sequence in Clojure

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related