How do you get multiple urls at the same time in a synchronus function

Question

I am getting data from the open weather map API. Currently the data is being retrieved synchronously which is slow. However, the function has to be synchronous as it is part of a library, but it can call an async function. How might I still make concurrent requests to increase performance? A solution that does not use reqwests works, but reqwests is preferred.

fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
                           coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
    let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
                      coordinates.get(0).expect("Improper coordinates").to_string() + "," +
                          coordinates.get(1).expect("Improper coordinates"), metric);
    let mut data: Vec<HashMap<String, String>> = Vec::new();
    for url in urls {
        let request = reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected");
        data.push(request);
    }
    return data;
}

harmic · Accepted Answer · 2022-12-12 22:17:30Z

1

If your program isn't already async, probably the easiest way might be to use rayon.

use reqwest;
use std::collections::HashMap;
use rayon::prelude::*;

fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
                           coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {

    let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
                      coordinates.get(0).expect("Improper coordinates").to_string() + "," +
                          coordinates.get(1).expect("Improper coordinates"), metric);
    
    let data : Vec<_>= urls
        .par_iter()
        .map(|&url| reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected"))
        .collect();
    
    return data;
}

answered Dec 12, 2022 at 22:17

harmic

31k6 gold badges75 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

cafce25 · Accepted Answer · 2022-12-12 03:00:55Z

The easiest is probably to use tokios new_current_thread runtime and blocking on the data retreival.

use std::collections::HashMap;
use tokio::runtime;
pub fn collect_data() -> Vec<HashMap<String, String>> {
    let rt = runtime::Builder::new_current_thread()
        .build()
        .expect("couldn't start runtime");
    let urls = vec!["https://example.com/a", "https://example.com/b"];
    rt.block_on(async move {
        let mut data = vec![];
        for url in urls {
            data.push(async move {
                reqwest::get(url)
                    .await
                    .expect("Url Get Failed")
                    .json()
                    .await
                    .expect("json expected")
            });
        }
        futures::future::join_all(data).await
    })
}

Jmb · Accepted Answer · 2022-12-12 08:14:12Z

You need an asynchronous runtime in order to call asynchronous functions. The easiest way to get one is to use the #[tokio::main] attribute (which despite the name can be applied to any function):

#[tokio::main]
fn get_combined_data(
    open_weather_map_api_url: String,
    open_weather_map_api_key: String,
    coordinates: Vec<String>,
    metric: bool,
) -> Vec<HashMap<String, String>> {
    let urls: Vec<String> = get_urls(
        open_weather_map_api_url,
        open_weather_map_api_key,
        coordinates
            .get(0)
            .expect("Improper coordinates")
            .to_string()
            + ","
            + coordinates.get(1).expect("Improper coordinates"),
        metric,
    );
    futures::future::join_all (urls.map (|u| {
        async move {
                reqwest::get(url)
                    .await
                    .expect("Url Get Failed")
                    .json()
                    .await
                    .expect("json expected")
            }
    })).await
}

Collectives™ on Stack Overflow

How do you get multiple urls at the same time in a synchronus function

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related