How can I upload chromedriver when I want to use selenium in aws lambda function?

Question

I want to make python crawler and it works in local. But I want to run crawler in regularly so I put my crawler in aws lambda.

I downloaded chromedriver and put somewhere in my directory and I can use it in local server. But in lambda, I don't know how to set the path and upload chromedriver file.

I tried absolute path but it didn't work. Should I upload the chromedriver for lambda function? If so, how can I do?

my code in localhost

    chrome_driver_path = "../chromedriver_win32/chromedriver.exe"
    driver = webdriver.Chrome(
        executable_path = chrome_driver_path,
        chrome_options=options
    )
    URL = "https://*****.co.kr"
    driver.get(URL)

You can have the driver as part of s3 bucket or else check whether you have any cdn's for chromedrivers located , So that you can refer by the path and use it , — redhatvicky
– redhatvicky, Commented Sep 30, 2019 at 19:38

Mrk Fldig · Accepted Answer · 2019-10-01 08:47:05Z

1

So you can't run .exe on lambda because it's Linux but you CAN run puppeteer as above and use that to parse the HTML.

Install puppeteer as above(npm i puppeteer --save). Bundle everything up.(You compress all your code and node_modules into a zip file) Deploy to AWS. Voila.

I HIGHLY recommend the Serverless framework as it takes the pain out of deployments, you can get it here

Do bear in mind if your crawling job is going to take more than 15 minutes you'll need to schedule this via cron on something like a t2.micro rather than Lambda because it'll timeout.

edited Oct 1, 2019 at 8:47

answered Sep 30, 2019 at 19:35

Mrk Fldig

4,5366 gold badges41 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

HoW_3W Why Over a year ago

Thank you. It takes more than 15 minutes so I cannot use lambda function anyway. I should find another way as you said.

Mrk Fldig Over a year ago

So ping me a link to another question, I can help you design that in about 20 minutes.

Horatiu Jeflea · Accepted Answer · 2019-09-30 14:31:21Z

0

Lambda cannot run executable files. Have you considered headless chrome?

https://github.com/GoogleChrome/puppeteer

https://pptr.dev/#?product=Puppeteer&version=v1.20.0&show=outline

answered Sep 30, 2019 at 14:31

Horatiu Jeflea

7,4747 gold badges42 silver badges69 bronze badges

3 Comments

Mrk Fldig Over a year ago

It can't run EXE files but it can run linux binaries? Perhaps rephrase your answer - so yes you can but you need to find the linux binaries and upload them with your lambda package?

Horatiu Jeflea Over a year ago

Yes, being linux it's harder to run exe, maybe convert it to binary. I did not know Lambda can run executables, not being its purpose and for security reasons. Good to know

Mrk Fldig Over a year ago

Yeah it's a bit funky, personally I'd recommend moving to a t2.micro for customers with this use case rather than Lambda, but saying that if he scrape quick enough it might be OK

Collectives™ on Stack Overflow

How can I upload chromedriver when I want to use selenium in aws lambda function?

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related