Beautiful Soup Basics
Posted on Jun 14, 2018 in Notes • 15 min read
Beautiful Soup¶
Some notes on using beautiful soup on dataquest.
In [20]:
# To get content from webpages via `get()`
import requests
from bs4 import BeautifulSoup
import pandas as pdp
First get content from html via requests.get¶
In [ ]:
page = requests.get('http://dataquestio.github.io/web-scraping-pages/simple.html')
Basic tags¶
In [2]:
# raw HTML content of the page
page.content
In [3]:
# Create an instance of BS class to parse our doc
soup = BeautifulSoup(page.content, 'html.parser')
# `prettify` method displays nicely formatted HTML
soup.prettify()
In [4]:
# Move through the structure one level down
soup.children # returns a list generator, requires the `list()` function
list(soup.children)
[type(item) for item in list(soup.children)]
Out[4]:
Search by tags¶
- use the
find_all()method and pass in tag name as astr
In [5]:
# Find first instance of a certian tag
soup.find('p') # returns bs4.element.Tag
# Find all instances of a certain tag
soup.find_all('p') # returns bs4.element.ResultSet
Out[5]:
In [6]:
# Access the text
soup.find_all('p')[0].get_text()
Out[6]:
Search by class & id¶
- use the
class_oridattribute offind_all()
In [7]:
page = requests.get('http://dataquestio.github.io/web-scraping-pages/ids_and_classes.html')
soup = BeautifulSoup(page.content, 'html.parser')
In [9]:
outer_text = soup.find_all(class_='outer_text')
Search by CSS selectors¶
- use the
select()method CSS selectors examplesp a— finds all a tags inside of a p tag.body p a— finds all a tags inside of a p tag inside of a body tag.html body— finds all body tags inside of an html tag.p.outer-text— finds all p tags with a class of outer-text.p#first— finds all p tags with an id of first.body p.outer-text— finds any p tags with a class of outer-text inside of a body tag.
In [10]:
# Finds all p tags that are inside a div
soup.select("div p") # returns a python list
Out[10]:
Navigating the Web Structure¶
- inspect HTML with Chrome Dev tools
- click the target text
- find the "outermost" element that contains all of the text
- explore the div more
- use
find()orfind_all()to navigate to the target - further explore the tags of the target information and act accordingly
- are they accessible simply by
classorid? USEfind().get_text() - are they within an attribute of a tag? USE
find()and access it as adict
- are they accessible simply by
find() and find_all() have to be called on bs4.element.Tag NOT bs4.element.ResultSet
In [28]:
page = requests.get("http://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168")
soup = BeautifulSoup(page.content, 'html.parser')
seven_day = soup.find(id="seven-day-forecast")
forecast_items = seven_day.find_all(class_="tombstone-container")
# Select the first element of the result set
tonight = forecast_items[0] # create a bs4.element.Tag
print(tonight.prettify())
- The name of the forecast item is in
class_="period-name" - The short description of the conditions is in
class_="short-desc" - The low temperature is in
class="temp temp-low"
In [12]:
period = tonight.find(class_="period-name").get_text()
short_desc = tonight.find(class_="short-desc").get_text()
temp = tonight.find(class_="temp").get_text()
print(period)
print(short_desc)
print(temp)
- The description of the conditions is in the
titleproperty of theimgtag
How to extract the title attribute from the img tag
- treat the
BeautifulSoupobject like a dictionary - pass in the attribute we want as a key
In [13]:
img = tonight.find("img")
desc = img['title']
print(desc)
Get ALL Information¶
generalise the process above
In [17]:
# Use CSS selector to get period_name classes with in all tombstone_container classes
period_tags = seven_day.select(".tombstone-container .period-name")
period_tags
Out[17]:
In [34]:
periods = [pt.get_text() for pt in period_tags]
type(periods)
Out[34]:
In [19]:
short_descs = [sd.get_text() for sd in seven_day.select(".tombstone-container .short-desc")]
temps = [t.get_text() for t in seven_day.select(".tombstone-container .temp")]
descs = [d["title"] for d in seven_day.select(".tombstone-container img")]
print(short_descs)
print(temps)
print(descs)
Wrangle results¶
In [33]:
import re
string = re.sub("e|l", "", "Hello people")
string
Out[33]:
Store the result in pandas.DataFrame¶
In [21]:
weather = pd.DataFrame({
"period": periods,
"short_desc": short_descs,
"temp": temps,
"desc": descs
})
weather
Out[21]:
Write to Excel¶
In [ ]:
writer = pd.ExcelWriter('file_name.xlsx', engine='xlsxwriter')
df.to_excel(writer)
writer.save()
Analysis¶
For example, we can use regex and the Series.str.extract method to pull out the numeric temperature values
In [24]:
temp_nums = weather["temp"].str.extract("(?P<temp_num>\d+)", expand=False)
weather["temp_num"] = temp_nums.astype('int')
temp_nums
Out[24]:
In [25]:
weather["temp_num"].mean()
Out[25]: