Get href using css selector with Scrapy

Question

I want to get the href value:

<span class="title">
  <a href="https://www.example.com"></a>
</span>

I tried this:

Link = Link1.css('span[class=title] a::text').extract()[0]

But I just get the text inside the <a>. How can I get the link inside the href?

Could you give more details on what you're using & the code you're trying to parse? Might wanna try a::@href or a::href to select the attribute. — Robin
– Robin, Commented Jan 17, 2014 at 9:19

paul trmbrth · Accepted Answer · 2017-07-20 15:37:12Z

81

What you're looking for is:

Link = Link1.css('span[class=title] a::attr(href)').extract()[0]

Since you're matching a span "class" attribute also, you can even write

Link = Link1.css('span.title a::attr(href)').extract()[0]

Please note that ::text pseudo element and ::attr(attributename) functional pseudo element are NOT standard CSS3 selectors. They're extensions to CSS selectors in Scrapy 0.20.

Edit (2017-07-20): starting from Scrapy 1.0, you can use .extract_first() instead of .extract()[0]

Link = Link1.css('span[class=title] a::attr(href)').extract_first()
Link = Link1.css('span.title a::attr(href)').extract_first()

edited Jul 20, 2017 at 15:37

answered Jan 17, 2014 at 9:37

paul trmbrth

20.8k4 gold badges56 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

aisensiy Over a year ago

The "extract()[0]" looks bad, is it possible to extract a single value not a list?

paul trmbrth Over a year ago

There's this proposal: github.com/scrapy/scrapy/issues/568 (quite a lot of debate). With this PR: github.com/scrapy/scrapy/pull/624 . I personally would like to see .get() and .getall()

lufte Over a year ago

Where can I see all Scrapy's CSS extensions? Can't find it in the docs.

paul trmbrth Over a year ago

Scrapy's extensions are documented in Parsel docs (the selectors library): parsel.readthedocs.io/en/latest/…

jerinisready Over a year ago

you can simply use Link = Link1.css('span.title a::attr(href)').get()

|

Eddy · Accepted Answer · 2016-02-14 11:54:07Z

8

Link = Link1.css('span.title a::attr(href)').extract_first()

you can get more infomation from this

answered Feb 14, 2016 at 11:54

Eddy

811 silver badge3 bronze badges

Comments

Jorgesys · Accepted Answer · 2020-02-26 03:28:15Z

0

This will do the job:

Link = Link1.css('span.title a::attr(href)').extract()

Link will have the value : https://www.example.com

edited Feb 26, 2020 at 3:28

answered Feb 26, 2020 at 2:55

Jorgesys

127k26 gold badges347 silver badges276 bronze badges

Collectives™ on Stack Overflow

Get href using css selector with Scrapy

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related