Walmart Price Scraping with Python 3

Question

I am very new to this concept, but I am trying to learn how to use python to manipulate HTML data.

I wrote a python (ver. 3.4.1) script which fetches the URL and returns some information, which I parse using BeautifulSoup (ver. 4).

import urllib.request
from bs4 import BeautifulSoup

response = urllib.request.urlopen('http://www.walmart.ca/en/ip/xbox-one/6000187109065')
html = response.read()

soup = BeautifulSoup(html)

print(soup.find_all('div', {"class" : "price-current"}))

In this example, I am attempting to obtain the price of the Xbox One. I chose this div because it is the one which displays the price to the user on the webpage. I am aware that there is a <span itemprop="price">$399.99</span> which is available to scrape. Scraping from that is fairly straight-forward, I am more curious as to why the price won't show up in the div that it is supposed to.

I hypothesize that it is something to do with HTML headers, or perhaps some kind of POST/GET data which is sent automatically when browsing with a standard web browser. Could anyone explain why the prices do not show up and what I would have to do to get them to show as expected?

For me, <div class="price-current"></div> is blank (Though it looks like that div was modified through script). Why not get it from <span itemprop="price">$399.99</span> — Mike Christensen
– Mike Christensen, Commented Aug 7, 2014 at 18:03
Well one reason would be if you plan to price scrape and the price displayed to the user changes, but the span element does not then you would never know. A second reason could be that only the span element's price changes and the one displayed on their site does not. I doubt they will do any price-matching based on some buried html-element. Thirdly, I want to learn more rather than immediately take the easy solution. — user3919479
– user3919479, Commented Aug 7, 2014 at 18:10

scandinavian_ · Accepted Answer · 2014-08-07 18:04:30Z

2

The price is loaded using javascript. This is the request:

Request URL:http://www.walmart.ca/ws/online/products
Request Method:POST

Form Data:
products:[{"productid":"6000187109066","skus":[[{"skuid":"6000187109066","status":"10"}]]}]
csrfToken:b08bfe580f3d9a0d893435fb

Since it includes an csrfToken, you will want to figure out how it is generated or provided before making the post request. They might be relying on session cookies as well.

It appears that walmart has an API though: https://developer.walmartlabs.com/

answered Aug 7, 2014 at 18:04

scandinavian_

2,5761 gold badge18 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3919479 Over a year ago

Thanks stranger, this is quite helpful. How did you trace this to that specific javascript request?

scandinavian_ Over a year ago

I used Chromes developer tools, under the network tab. Firefox has equivalent capabilities.

ilyazub · Accepted Answer · 2021-02-26 16:17:22Z

Disclaimer: I work at SerpApi.

You can scrape Walmart Product prices via google-search-results package from SerpApi. It's a paid API with a free trial.

Full example

import os
from serpapi import GoogleSearch

params = dict(
    engine="walmart",
    query="xbox one",
    api_key=os.getenv("API_KEY"),
)

search = GoogleSearch(params)
results = search.get_dict()

for organic_result in results["organic_results"]:
    product_id = organic_result["us_item_id"] or organic_result["product_id"]

    search.params_dict = dict(
        engine="walmart_product",
        product_id=product_id,
        api_key=os.getenv("API_KEY"),
    )

    product_result = search.get_dict()

    print(
        f"Price of '{product_result['product_result']['title']}': '{product_result['product_result']['price_map']['price']} {product_result['product_result']['price_map']['currency']}'"
    )

Output

Price of 'Microsoft Xbox One S 1TB Console (Certified Refurbished)': '429 $'
Price of 'Star Wars Jedi: Fallen Order, Electronic Arts, Xbox One': '39.82 $'
Price of 'Microsoft - Xbox One S 1TB All-Digital Edition Console with Xbox One Wireless Controller (Renewed)': '499.99 $'
Price of 'Need for Speed: Heat, Electronic Arts, Xbox One, 014633373233': '25 $'
Price of 'Call of Duty: Black Ops Cold War, Activision, Xbox One': '49.94 $'
Price of 'NBA 2K21, 2K, Xbox One, 710425596858': '29.96 $'
Price of 'Refurbished Microsoft Factory Xbox One X 1TB, 4K Ultra HD Gaming Console in Black, FMQ-00042, 889842246971': '499.99 $'
Price of 'Madden NFL 21, Electronic Arts, Xbox One': '29.66 $'
Price of 'Need for Speed Payback, Electronic Arts, Xbox One, 014633370058': '14.96 $'
Price of 'Nickelodeon Kart Racers, Gamemill, Xbox One, 856131008060': '14.96 $'
Price of 'Sonic Mania (Sega), Xbox One, 010086640939': '14.96 $'
Price of 'Minecraft Master Collection, Microsoft, Xbox One, 889842394979': '42.88 $'
Price of 'Mortal Kombat 11, Warner Bros., Xbox One, 883929668977': '15 $'
Price of 'UFC 4, Electronic Arts, Xbox One': '29.96 $'
Price of 'Grand Theft Auto V, Rockstar Games, Xbox One': '22.8 $'
Price of 'Man Eater, Deep Silver, Xbox One, 816819017517': '24.96 $'
Price of 'Crash N. Sane Trilogy, Activision, Xbox One, 047875881969': '39.99 $'
Price of 'Spyro Reignited Trilogy, Activision, Xbox One, 047875882423': '29.83 $'
Price of 'Call of Duty: Modern Warfare, Activision, Xbox One, 0047875884366': '47.9 $'
Price of 'Cyberpunk 2077, Warner Bros, Xbox One': '39.74 $'
Price of 'Mass Effect Andromeda, Electronic Arts, Xbox One, 014633734096': '6.91 $'
Price of 'THQ theHunter: Call Of the Wild (Xbox One)': '19.93 $'
Price of 'Wreckfest, THQ-Nordic, Xbox One, 0811994021649': '29.83 $'
Price of 'Assassin’s Creed Valhalla Xbox Series X|S, Xbox One Standard Edition': '36.51 $'
Price of 'NBA 2K20, 2K, Xbox One, 710425595264': '21.99 $'
Price of 'LEGO Worlds, Warner Bros, Xbox One': '15 $'
Price of 'Call of Duty: Black Ops 3 Zombie Chronicles Edition, Activision, Xbox One, 047875881228': '37.88 $'
Price of 'Jumanji The Video Game, Bandai Namco, Xbox One, 819338020792': '19.93 $'
Price of 'Red Dead Redemption 2, Rockstar Games, Xbox One': '34.99 $'
Price of 'HALO 5, Microsoft, Xbox One, 885370928518': '19 $'
Price of 'LEGO Jurassic World, Warner, Xbox One, 883929472727': '19.8 $'
Price of 'Plants vs. Zombies: Battle for Neighborville, Electronic Arts, Xbox One, 014633736007': '14.96 $'
Price of 'The LEGO Marvel Collection, Warner Bros., Xbox One, 00883929670499': '24.98 $'
Price of 'LEGO The Incredibles, Warner Bros, Xbox One, 883929633005': '18 $'
Price of 'Assassin's Creed: Origins, Ubisoft, Xbox One, 887256028459': '19.93 $'
Price of 'Spongebob Squarepants: Battle for Bikini Bottom Rehydrated, THQ-Nordic, Xbox One, 811994022165': '29.77 $'
Price of 'The SIMS 4, Electronic Arts, Xbox One': '33.84 $'
Price of 'Star Wars Battlefront 2, Electronic Arts, Xbox One, 014633735321': '14.96 $'
Price of 'Pac-Man Championship Edition 2 + Arcade Game Series, Bandai/Namco, Xbox One, 722674220705': '14.49 $'
Price of 'Microsoft Xbox One S 1TB All Digital Edition 3 Game Bundle (Disc-free Gaming), White, NJP-00050': '198.95 $'

Collectives™ on Stack Overflow

Walmart Price Scraping with Python 3

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related