lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

lesson9Title

lesson10Title

lesson11Title

lesson12Title

pythonIntroCrawlingChapter2Title

lesson13Title

lesson14Title

lesson15Title

lesson16Title

lesson17Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter3Title

pythonIntroCrawlingChapter4Title

from bs4 import BeautifulSoup

# Example HTML document
html_doc = """
<div class="content">
  <p class="intro">Welcome to the CodeFiends!</p>
  <div class="section">
    <p>First section paragraph.</p>
    <p>Second section paragraph with <a href="http://example.com">a link</a>.</p>
  </div>
  <div class="footer">
    <p>Contact information: <a href="mailto:contact@example.com">contact@example.com</a></p>
  </div>
</div>
"""

# Create a BeautifulSoup object
soup = BeautifulSoup(html_doc, 'html.parser')

# Extract all p tags within the first div with class 'content'
print("Content Paragraphs:")
content_paragraphs = soup.select('div.content > p')

for p in content_paragraphs:
    print(p.text)

print('-' * 10)

# Extract all p tags, including links, within the div with class 'section'
print("Section Paragraphs:")
section_paragraphs = soup.select('div.section p')

for p in section_paragraphs:
    if p.a:
        print(f"{p.text} (Link: {p.a['href']})")
    else:
        print(p.text)

print('-' * 10)

# Extract the href attribute values from all a tags in the div with class 'footer'
print("Footer Links:")
footer_links = soup.select('div.footer a')

for a in footer_links:
    print(a['href'])

# Handling Nested HTML Elements

Nested HTML elements mean that an element is contained inside another element.

```html title="Nested Elements"
<div>
  <p>First paragraph.</p>
  <p>Second paragraph.</p>
</div>
```

In the example above, the `<div>` element contains two nested `<p>` elements.

Handling such elements is an essential skill in web scraping.

<br />

## Navigating Nested Elements

1. `Understanding Parent-Child Relationships`

   - HTML elements can have parent-child relationships.

   - For instance, the `<p>` elements inside a `<div>` are child elements of the `<div>`.

2. `Finding Elements on a Specific Path`

   - Use `find()` or `find_all()` to locate elements on a specific path.

   - Example: `soup.find('div').find('p')` finds the first `<p>` inside the first `<div>`.

<br />

### Example: Extracting Nested Elements

```python title="Extracting Nested Elements"
html_doc = """
<div>
  <p class="inner-text">
    First paragraph.
    <span>Text within a span</span>
  </p>
  <p class="inner-text">Second paragraph.</p>
</div>
"""

soup = BeautifulSoup(html_doc, 'html.parser')

# Extract all p tags inside the first div
for p in soup.find('div').find_all('p'):
    print(p.text)
```

<br />

## Using Attributes for Extraction

- Use tag attributes like class, ID, or other attributes to extract specific elements.

- Example: `soup.find_all('a', class_='external_link')` finds all `<a>` tags with the class 'external_link'.

<br />

## Using CSS Selectors

- In BeautifulSoup, you can leverage CSS selectors with the `select()` method.

- Example: `soup.select('div.content > p.paragraph')` finds `<p>` elements that are direct children of a `<div>` with class 'content', having class 'paragraph'.

<br />

## Example: Extracting Data from Complex Structures

```python title="Extracting Data from Complex Structures"
html_doc = """
<div class="content">
  <p class="paragraph">First paragraph in content.</p>
  <div class="inner-content">
    <p>Inner paragraph.</p>
  </div>
</div>
"""

soup = BeautifulSoup(html_doc, 'html.parser')

# Extract all p tags within the 'content' class div
content_paragraphs = soup.select('div.content p')
for p in content_paragraphs:
    print(p.text)
```

<br />

## Practice

Click the _`Run Code`_ button on the right side of the screen to see the scraping results or modify the code!

python_execution