Scrape a data links of a page with JavaScript from R

Web traffic and API call:

So, if you start ,for example, with

http://burkinafaso.opendataforafrica.org/data/#menu=topic

You can see the list of all the top level links along with the counts of their datasets. If we were to click on Aide Publique (7) we would then see the 7 sections which, if you click on any, then present you with Select dataset.

If you monitor the web-traffic when doing that first click you will see the API POST request made to retrieve the data for Aide Publique (7):

If we further inspect the request we can observe the query string params in the url and the request payload:

The params is basically some info about us we can probably remove; and a little experimentation with the payload shows, if we exclude the payload, we actually get all the topics and not just Aide Publique (7).

The API response:

Now, the response is json and is an array but in R that means a list. Looking at part of one item in the list as an example:

A comparison of that info versus the actual dataset links e.g.

Clicking on the Select DataSet yields the end url of

http://burkinafaso.opendataforafrica.org/nthpfqd/aide-ext%C3%A9rieure-par-secteur-de-1995-%C3%A0-2006-en-millions-de-fcfa

A quick comparison with list item 1 (prior image of json) shows that this if we consider this new url decoded:

http://burkinafaso.opendataforafrica.org/nthpfqd/aide-extérieure-par-secteur-de-1995-à-2006-en-millions-de-fcfa

is of the format:

'http://burkinafaso.opendataforafrica.org/{item["id"]}/{item["title"]}'

meaning, in a loop over the json response object we can generate the final links by concatenating a base string with the current item id and current item title. We can also extract the title from the current item name. We can use purrr and map_df to handle the loop and final dataframe generation and httr to make the POST.

R:

library(httr)
library(purrr)

r <- content(POST("http://burkinafaso.opendataforafrica.org/api/1.0/meta/dataset"))

df <- map_df(r, function(item) {

  data.frame(title = item$name,
             url = paste0("http://burkinafaso.opendataforafrica.org/", item$id,'/',item$title),
             stringsAsFactors=FALSE)
})

View(df)

Py:

import requests
import pandas as pd

r = requests.post('http://burkinafaso.opendataforafrica.org/api/1.0/meta/dataset').json()
df = pd.DataFrame([(item['name'], f'http://burkinafaso.opendataforafrica.org/{item["id"]}/{item["title"]}') for item in r]
                  ,columns = ['Title','Url'])
print(df)