How can I add "http://test.url/" to the result link.get('href') below, but only if it doesn't contain "http"
import urllib2
from bs4 import BeautifulSoup
url1 = "http://www.salatomatic.com/c/Sydney+168"
content1 = urllib2.urlopen(url1).read()
soup = BeautifulSoup(content1)
for link in soup.findAll('a'):
print link.get('href')
Use urlparse.urljoin:
>>> import urlparse
>>> urlparse.urljoin('http://example.com/', '/a/b')
'http://example.com/a/b'
>>> urlparse.urljoin('http://example.com/', 'http://www.example.com/a/b')
'http://www.example.com/a/b'
In Python 3.x, use urllib.parse.urljoin:
>>> import urllib.parse
>>> urllib.parse.urljoin('http://example.com/', '/a/b')
'http://example.com/a/b'
>>> urllib.parse.urljoin('http://example.com/', 'http://www.example.com/a/b')
'http://www.example.com/a/b'
Complete example
import urllib2
from bs4 import BeautifulSoup
import urlparse
url1 = "http://www.salatomatic.com/c/Sydney+168"
content1 = urllib2.urlopen(url1).read()
soup = BeautifulSoup(content1)
for link in soup.findAll('a'):
print urlparse.urljoin(url1, link.get('href'))
I would use urljoin
>>> from urlparse import urljoin
>>> urljoin('http://test.url/', '/relative/path')
'http://test.url/relative/path'
In your example you only need to do this when you find a relative url.
import urllib2
from bs4 import BeautifulSoup
url1 = "http://www.salatomatic.com/c/Sydney+168"
content1 = urllib2.urlopen(url1).read()
soup = BeautifulSoup(content1)
for link in soup.findAll('a'):
get = link.get('href')
if get.startswith('http'):
print get
In the spirit of BeautifulSoup, this works well with your original code.
If what you want is to preface the non-http sites with a http://test.url/ then you need to do this:
for link in soup.findAll('a'):
get = link.get('href')
if not get.startswith('http'):
print 'http://test.url/'+get
You're set either way.
for link in soup.findAll('a'):
currenturl = link.get('href')
if not currenturl.startswith("http"):
currenturl = "http://test.url/" + currenturl
print currenturl
This works:
url = "http://test.url/"
link_list = [link['href'] for link in soup.findAll('a')]
result_list = [url+i if 'http' not in i else i for i in link_list]
for link in result_link:
print link
