Web Scraping using python and Beautiful soup

Question

I'm trying to scrape this website and get the download links

https://gogohd.pro/download?id=OTk1OTk=&typesub=Gogoanime-SUB&title=Dragon+Ball+Super+Episode+100

using this piece of code

# import libraries
from requests_html import HTMLSession

# specify the url
URL = 'https://gogohd.pro/download?id=MTkzNTU3&typesub=Gogoanime-SUB&title=Chainsaw+Man+Episode+1' 

session = HTMLSession()
r = session.get(URL)

for link in r.html.links:
    print(link)

But it's not returning the links and is instead returning it empty. I tried replicating it with selenium but to no avail :(

KunduK · Accepted Answer · 2022-12-21 12:31:45Z

2

Use the chrome option with following css selecto using selenium.

from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=chrome_options)
driver.get("https://gogohd.pro/download?id=MTkzNTU3&typesub=Gogoanime-SUB&title=Chainsaw+Man+Episode+1")
time.sleep(3)
downloadLinks =[link.get_attribute('href') for link in driver.find_elements(By.CSS_SELECTOR,"div.dowload>a")]
print(downloadLinks)
print(f"Download links count: {len(downloadLinks)}")

output:

['https://gogodownload.net/download.php?url=aHR0cHM6LyAdrefsdsdfwerFrefdsfrersfdsrfer363435349AawehyfcghysfdsDGDYdgdsfsdfwstdgdsgtertseWVpYnU0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjM2MHAubXA0P3Rva2VuPWtQaXpiR0xjQ2lDaXdJY25xNnNRSHcmZXhwaXJlcz0xNjcxNjM2ODQ2JmlkPTE5MzU1Nw==', 'https://gogodownload.net/download.php?url=aHR0cHM6LyAdeqwrwedffryretgsdFrsftrsvfsfsr9seWVpYnAawehyfcghysfdsDGDYdgdsfsdfwstdgdsgtertU0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjQ4MHAubXA0P3Rva2VuPWpRUzd1UnA4U2Z0X0tUeWYtRGNXc1EmZXhwaXJlcz0xNjcxNjM2ODQ2JmlkPTE5MzU1Nw==', 'https://gogodownload.net/download.php?url=aHR0cHM6LyAdrefsdsdfwerFrefdsfrersfdsrfer363435349AdeqwrwedffryretgsdFrsftrsvfsfsrseWVpYnU0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjcyMHAubXA0P3Rva2VuPVYxb1ZsZDI3VGtoZGdNRjVURS1yYmcmZXhwaXJlcz0xNjcxNjM2ODQ2JmlkPTE5MzU1Nw==', 'https://gogodownload.net/download.php?url=aHR0cHM6LyAdeqwrwedffryretgsdFrsftrsvfsfsr9seWVpYnAdrefsdsdfwerFrefdsfrersfdsrfer36343534U0bmM3LmdvY2RuYW5pLmNvbS91c2VyMTM0Mi9lYzBiNzk3NmM1M2Q4YmY5MDU2YTYwNjdmMGY3ZTA3Ny9FUC4xLnYwLjEwODBwLm1wND90b2tlbj04clZFSUlOOGpicTYtZWx0bmNqT3VRJmV4cGlyZXM9MTY3MTYzNjg0NiZpZD0xOTM1NTc=', 'https://streamsss.net/d/aat4tegpjl4g', 'https://dood.wf/d/c2y9u6k23a2o', 'https://fembed9hd.com/f/gl1enu-757y2l2l', 'https://bodelen.com/afu.php?zoneid=2052717']
Download links count: 8

edited Dec 21, 2022 at 12:31

answered Dec 21, 2022 at 11:38

KunduK

33.4k5 gold badges19 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Syed Abdullah Over a year ago

I tried this code but i'm facing 2 problems. The page is giving an error 'USB: usb_device_handle_win.cc:1045 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)' and that I want it to be completely hidden, headless. Any help with that? I checked a similar project's code and they've usen ajax cdn links which I cant understand as well

KunduK Over a year ago

Those are warnings right. Not related to the links. Are getting same output as I on console?

Syed Abdullah Over a year ago

No im not getting the same output. it gives me count: 0 and doesnt output the link

KunduK Over a year ago

@SyedAbdullah : Not sure what problem at your end. Headless as well working fine to me.

Syed Abdullah Over a year ago

Okay i tried it a few times and sometimes, it works however most of the times it goes to the captcha screen and gives an error. I'll see if i can find a work around and thank you!

Collectives™ on Stack Overflow

Web Scraping using python and Beautiful soup

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related