0

I try to scrape a ajax/js based website for a .csv download.

If I visit the site in a browser, I need to click a button that calls a js function "downloadChartCSV()". This function generates and downloads a .csv file. That the file I'm after.

How can I automate this?

I have tried:

  • requests-html, which can login and render the page, but I miss an option to call that onClick event. There seems to be an open issue about exactly this: Github Issue
  • Selenium which probably works, but I don't want a full browser that is flashing up.
Improve this question
4
  • 1
    Can you provide the link with the related button you want to scrape from? Maybe you can obtain the csv via an ajax url Commented Sep 8, 2020 at 21:07
  • The function does not call an ajax request, but instead crunches some data that is already present on the site. I'd love to share the link as an example, but you have to log in and it's behind a paywall. Commented Sep 8, 2020 at 21:20
  • Too bad.. but maybe my answer is helpful Commented Sep 8, 2020 at 21:23
  • 1
    @YannickFunk Your approach was really helpful. I could solve this by debugging the site. Turns out there was a url to an api endpoint that delivered all the data in a neat JSON package. I can call the api and skip all the javascript stuff. Im not sure if you can formulate this as an answer, since it is technically not an answer to the original question. But it solved my problem much more elegantly. Commented Sep 9, 2020 at 7:26

1 Answer 1

Reset to default
1

If you want to have the functionality of Selenium, but don't want a full browser flashing up: Use the PhantomJS Driver.

https://realpython.com/headless-selenium-testing-with-python-and-phantomjs/

This is a very good Introduction into it.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.