lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

lesson9Title

lesson10Title

lesson11Title

lesson12Title

lesson13Title

lesson14Title

lesson15Title

lesson16Title

lesson17Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter2Title

pythonIntroCrawlingChapter3Title

pythonIntroCrawlingChapter4Title

# Key Takeaways So Far

So far, you've learned what crawling is, how it differs from scraping, and its legal and ethical responsibilities.

Before moving on to modules and HTML basics, let's review the key points card by card.

robots.txt is the standard file where a website tells crawlers what is allowed and blocked — always check it before crawling.

Key Takeaways So Far

In which file does a website state its allowed crawling scope?