I am using Ghost and BeautifulSoup to parse a HTML page. The problem that I have, is that the content of this page is dynamic (created with angularJS). At the beginning the html only shows something like "please wait! page loading". After a few seconds the content of the html appears. Using Ghost and BeatifulSoup I just get the HTML code of the loading page whith only 2 small divs. The URL stays the same. Is there a possibility to wait until the "real" content is loaded?
2 Answers
Load the page in a real browser (headless like PhantomJS is also an option) automated by selenium, wait for the desired contents to appear, get the .page_source and pass it to BeautifulSoup:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS()
driver.get("your url here")
# waiting for the page to load - TODO: change
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.ID, "content")))
data = driver.page_source
driver.close()
soup = BeautifulSoup(data, "html.parser")
Comments
Use phantomjs to open the page. Save it as a local file using phantomjs File System Module Api. Later use this local file handle to create BeautifulSoup object and then parse the page. See http://www.kochi-coders.com/2014/05/06/scraping-a-javascript-enabled-web-page-using-beautiful-soup-and-phantomjs/
