Scrapy - handle exception when one of item fields is not returned

Question

I'm trying to parse Scrapy items, where each of them has several fields. It happens that some of the fields cannot be properly captured due to incomplete information on the site. In case just one of the fields cannot be returned, the entire operation of extracting an item breaks with an exception (e.g. for below code I get "Attribute:None cannot be split"). The parser then moves to next request, without capturing other fields that were available.

item['prodcode'] = response.xpath('//head/title').re_first(r'.....').split(" ")[1]
#throws: Attribute:None cannot be split . Does not parse other fields.

What is the way of handling such exceptions by Scrapy? I would like to retrieve information from all fields that were available, while the unavailable ones return a blank or N/A. I could do try... except... on each of the item fields, but this seems like not the best solution. The docs mention exception handling, but somehow I cannot find a way for this case.

Can you post more of your code with an example?

James
– James

2015-10-25 16:58:39 +00:00
Commented Oct 25, 2015 at 16:58 — James
– James, Commented Oct 25, 2015 at 16:58
Apologies James, this got answered in between. Thanks!

Turo
– Turo

2015-10-25 18:26:37 +00:00
Commented Oct 25, 2015 at 18:26 — Turo
– Turo, Commented Oct 25, 2015 at 18:26

alecxe · Accepted Answer · 2015-10-25 17:11:32Z

The most naive approach here would be to follow the EAFP approach and handle exceptions directly in the spider. For instance:

try:
    item['prodcode'] = response.xpath('//head/title').re_first(r'.....').split(" ")[1]
except AttributeError:
    item['prodcode'] = 'n/a'

A better option here could be to delegate the item field parsing logic to Item Loaders and different Input and Output Processors. So that your spider would be only responsible for parsing HTML and extracting the desired data but all of the post-processing and prettifying would be handled by an Item Loader. In other words, in your spider, you would only have:

loader = MyItemLoader(response=response)

# ...
loader.add_xpath("prodcode", "//head/title", re=r'.....')
# ...

loader.load_item()

And the Item Loader would have something like:

def parse_title(title):
    try:
        return title.split(" ")[1]
    except Exception:  # FIXME: handle more specific exceptions
        return 'n/a'

class MyItemLoader(ItemLoader):  
    default_output_processor = TakeFirst()

    prodcode_in = MapCompose(parse_title)

Collectives™ on Stack Overflow

Scrapy - handle exception when one of item fields is not returned

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related