Idiomatic Way Handle and Concat Multiple HTTP Requests In Clojure?

Question

I'm working on a program that needs to make multiple calls to various microservices, possibly do some processing on those results, and then return a combination of those processes.

A very basic example might look like:

(def urls ["http://localhost:8080" "http://localhost:8080"])

(defn handle-http-request [url]
  (let [request (http-kit/get url)]
    (do-some-processing (:body @request))))

(defn home-page
  [request]
  (let [response (pmap handle-http-request urls)]
    (ring-resp/response {:buildings (first response) :characters (second response)})))

In this case i'm using pmap to handle running all my requests in parallel, and then return them as JSON that the UI making the request can handle. In the real environment there will be more URLs each fetching different data.

My question is whether this is an appropriate way to handle this problem? I've looked some core.async, and see it as a possible way to handle this but worry it might be overkill? My second concern is handling errors, it would appear that core.async might be able to more elegantly handle issues where a remote has timed out. Am I right in this assumption, or is using pmap okay in this situation?

Lastly, are there any accepted patterns or reading on how to handle microservice architectures like this? I see my problem as relatively specific, but feel the idea of a server making requests to many others and compiling those results is nothing new.

I wouldn't use core.async for this. Use either pmap or create a future for each request. Could you be more specific about what you want the output to be? And why? — Alan Thompson
– Alan Thompson, Commented Feb 3, 2021 at 4:42
@AlanThompson In this situation the output would probably be a map that's then made into a JSON string. My picture is that the whole process is kicked off by some UI requesting data, and that's an easy way to return it so that the UI can do what it needs to. — newBieDev
– newBieDev, Commented Feb 3, 2021 at 14:14

cfrick · Accepted Answer · 2021-02-03 08:46:07Z

The HTTP-Kit documentation provides an example for this using futures:

Combined, concurrent requests, handle results synchronously

(let [urls ["http://server.com/api/1" "http://server.com/api/2" "http://server.com/api/3"]
      ;; send the request concurrently (asynchronously)
      futures (doall (map http/get urls))]
  (doseq [resp futures]
    ;; wait for server response synchronously
    (println (-> @resp :opts :url) " status: " (:status @resp))
    )

Yet, often this is still not enough. Make sure you configure timeouts and you have an escalation strategy once the timeouts hit. The more other services you are going to hit, the more complex this task gets. Maybe have a look at libraries like Resilience4JClj, that provide configurable ways to deal with retrying, caching, timeouts, ...

Note: pmap is best suited to CPU intensive tasks. Or as the docs state:

Only useful for computationally intensive functions where the time of f dominates the coordination overhead.

Collectives™ on Stack Overflow

Idiomatic Way Handle and Concat Multiple HTTP Requests In Clojure?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related