Questions tagged [selenium]
Selenium is a tools suite to automate web browsers, and web application testing
164 questions
3
votes
1
answer
131
views
Multi-Page Web Scraping Code Using Selenium with Multithreading
I have written a web scraping script using Selenium to crawl blog content from multiple URLs. The script processes URLs in batches of 1000 and uses multithreading with the ThreadPoolExecutor to ...
- 31
3
votes
1
answer
110
views
Scraping website with Python and Selenium to collect data from dynamic website
Summary:
The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
- 269
2
votes
1
answer
82
views
A selenium web scraper to package NBA data
I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
- 21
0
votes
2
answers
87
views
Application install code- Need to remove Nested Loops and make code more compact and readable. Code installs application and its version as per req
This code installs the application, according to the type of application selected like alpha, beta, and released, also it checks for latest version in each type.
...
- 29
2
votes
0
answers
101
views
Python script that refreshes network connection at random intervals
This is a followup to Python script that reboots the router every 600 seconds.
Recently censorship in China has increased because of the meeting of heads of states, and my previous script has been ...
- 3,704
11
votes
4
answers
2k
views
Python script that reboots the router every 600 seconds
This is a Python script I wrote to reboot the router every 600 seconds.
Why did I do it? Well, in case you don't know, I live in China, and I use VPNs to browse the international internet.
My ISP ...
- 3,704
0
votes
1
answer
126
views
Poetry Web Scraping in Python [closed]
I have a script that obtains urls that lead to a specific poem. This code current works and uses multiprocessing pools. I currently am getting restricted or blocked by some way from the website that I ...
- 3
5
votes
1
answer
101
views
Waits for one of two xpaths to be present, one inside of an iframe, one outside of it
I'm using selenium for testing, and I've written code that waits for one of two locations before it continues. This is really easy when both locations are within the same frame, because then I can ...
- 175
1
vote
1
answer
74
views
Python script for get minimum product price having maximum reviews
Technical task.
user visits amazon.com website
user fills out a search field with the product name and activates
search ==>
a page with search results is displayed.
user looks for the product ...
- 21
1
vote
1
answer
254
views
Selenium/Beautifulsoup web scrape of Grainger
This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
- 11
1
vote
3
answers
2k
views
To automate scheduling at the Italian Consulate
I'm a beginner developer in Python and I need some tips to improve my code, in performance (speed) and hardware usage. Is it possible to improve?
And I have a question: is it possible to keep the ...
user267072
2
votes
1
answer
67
views
scraping pollution data tables from government website
i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved:
...
3
votes
1
answer
239
views
Scrape PokeDex and display in tkinter
Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
- 33
3
votes
1
answer
202
views
Crawler/scraper for soccer match results
I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
1
vote
1
answer
92
views
Volunteer registration website Selenium testing
I have a volunteer registration website that I wrote some end-to-end tests for in Selenium. Selenium is of course software that lets you automate clicking around a browser, which makes it great for ...
- 1,445
1
vote
0
answers
274
views
Web scraper for pokemon prices using Selenium
I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
- 11
1
vote
1
answer
418
views
Selenium web scraper to scrape Apple tweets of the last 3 years
I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
- 11
3
votes
0
answers
127
views
Scraping housing dataset row by row from Estate Agent's website
I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame.
...
- 31
4
votes
2
answers
410
views
Scraping campsite availability from a webpage using vba with selenium
I wrote some code to extract the information from a table, but it takes an extremely long time.
The table is in the format of a calendar.
I need the information on an Excel sheet with column 1 as the ...
- 41
1
vote
1
answer
397
views
Web-scrape bonds information from nasdaqomxnordic
I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated!
...
- 13
4
votes
2
answers
654
views
Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python
I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths.
The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
- 55
1
vote
1
answer
114
views
Web Scraping DOL with Selenium
This is my code to extract data from the DOL website for a project I'm doing for my portfolio. The code does extract all of the data that I need, however it runs really slowly. I think it took like 15 ...
- 55
2
votes
0
answers
615
views
My first vba-selenium code
I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company.
The asset value is not available other than on ...
- 121
1
vote
2
answers
134
views
WaitFor method for website automation testing with Selenium WebDriver
I am trying to use Selenium for website automation testing tasks and I am new to Selenium testing framework. The the situation I faced is to wait the website components loading and then do the related ...
- 7,575
4
votes
1
answer
1k
views
Scraping a dynamic website with Scrapy (or Requests) and Selenium
I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here.
Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
- 427
2
votes
1
answer
1k
views
Extract flight arrival data from web
I'm trying to scrape arrival data from Flight Radar 24
My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process?
Here's my script:
...
- 51
1
vote
1
answer
73
views
Simple login test, translated from java to python
I'm reading on how to write proper testing suites here. So I'm trying to follow the selenium example in the docs which is in Java; I'm trying to translate it to Python since my app is written in ...
- 241
1
vote
0
answers
47
views
Organizing things together to form a minimum viable Scraper App (part 2)
This is a follow-up of my question over here.
Response to @Reinderien's answer:
I have corrected the more trivial issues highlighted in @Reinderien's answer below as follows. ...
- 427
1
vote
0
answers
103
views
Optimising this web-scraping code
A member of SO who I immensely respect just told me that the code below makes him uncomfortable.
...
- 199
2
votes
0
answers
2k
views
Python script to scrape google maps without API
I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance.
It uses selenium ...
- 21
2
votes
1
answer
565
views
Looping through search results in Selenium
This is a follow-up of my question over here.
I am extending the functionality of @Reinderien's suggested code to enable the automatic looping through of pages in the search result.
I've mainly ...
- 427
0
votes
1
answer
230
views
Unix batch job execution from BDD automation framework
As part of my testing, I have to connect to Unix server using putty and execute few unix jobs.
Feature file
Steps for test scenario:
...
- 89
3
votes
1
answer
317
views
Looping through keywords and multiple pages in Selenium
This is a follow-up of my question over here.
I adopted @Reinderien's suggested script to a second website below:
fudan.py
...
- 427
1
vote
1
answer
134
views
How to make use of a class of descriptors to save code and other issues
This is a follow-up to my question over here.
Having worked through most of the issues highlighted by @Reinderien in his detailed answer, I somehow got the code to work...
Some quick observations:
<...
- 427
3
votes
1
answer
256
views
Getting xpath indices to count forward to preserve table structure
Scraping tables from the web get complicated when there are 2 or more values in a cell. In order to preserve the table structure, I have devised a way to count the row-number index of its xpath, ...
- 427
-2
votes
1
answer
130
views
Can I refactor my code anymore? [closed]
I have written one method; however I feel I can refactor it even more but not sure what
Can someone help me here?
...
- 11
2
votes
0
answers
378
views
A thread-safe WebDriver pool to submit browsing jobs to
For a project, I need to make use of Selenium WebDrivers, and since they're so expensive to start, I decided to write a pool to manage them.
...
- 16.6k
2
votes
2
answers
357
views
Scrape and parse dictionary website and get meaning of each word in a text file
In this program, I have done automation using Selenium, to find english meaning of each sanskrit word from a dictionary website.
The function eng_meaning accepts ...
- 2,471
1
vote
0
answers
409
views
Selenium Webscraper for Odd data analyses
I have written the following code to achieve the following:
Regularly scrape all live matches on the betting site oddsportal.com
Pull odds data into data frame
Evaluate data frame for two odd ...
- 11
5
votes
1
answer
134
views
Web scraping data from website using Selenium
ApplicationDetail.txt:
URL: https://www.abc.com
applicationNo : 123456
Application Class:
...
- 51
2
votes
0
answers
605
views
Discord Bot Python. Selenium screenshots
I wrote my own Discord Bot which is taking screenshot of a specific website. This scrip is very simple and thought it should work fast, but it isn't. I read a lot, and I think I'm not able to improve ...
- 21
2
votes
1
answer
3k
views
How to make selenium in python faster
I'm making a bot for PlayStation 5.
And restock is coming soon so I want to make my bot faster.
The approximate duration is 4.90 seconds.
Is there any way to make this faster?
I found that I should ...
4
votes
0
answers
105
views
Python Selenium First Project - loop through pages and check/uncheck tick boxes
New to Python/Selenium and learning on the fly with my first basic selenium script.
Request user input - activate or deactivate.
Open's website, loops through shop URL as well as activate/deactivate ...
- 41
1
vote
1
answer
105
views
Reading configuration variables in a Python project
I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper)
I have created .env file at the root of the project ...
- 633
5
votes
1
answer
319
views
Automating TypeRacer using selenium
This is a personal project which I created solely to torture my friends ;)
I'm pretty sure you've heard about typeracer.
And if you have friends like mine who are faster than you, you'd surely be ...
- 2,800
1
vote
2
answers
210
views
open multiple tabs using selenium
I am using the following script to read search terms from a file and then search those terms in google (each search result in it's own tab).
...
- 585
2
votes
0
answers
107
views
C# Scraper - EventHandler
It might help to check my previous question out, its part of the same application. The only minor changes are with the entity class names. They now each have their own entities, and I use interfaces ...
- 1,355
8
votes
1
answer
287
views
Booking an East London Tennis Court
Description
I'm not sure if it Covid-19 but lately it is impossible to book a tennis court in my area on time. It's always full or maybe just don't check enough :)
To beat the queue and get notified ...
- 11.8k
6
votes
1
answer
510
views
Web Scraping using Selenium and Python
This is my first program code ever, and it actually works.
My goal is to scrape information from the website and store it in a database.
It is a site that has historical data on sporting events and ...
user230470
5
votes
1
answer
1k
views
Game scraper for Steam
I made a scraper for Steam that get different info about a Steam game, such as the price, the specs and the supported platforms. The reason I made this was because I have a super slow laptop, so ...
- 153