- XPath selectors
- code_html.xpath('//img/@src')
- links = code_html.xpath('//a/@href')
- bs.find_all("<html_tag_label>")
- items.py, pipelines.py, settings.py, and spiders
- spiders/my_sypder.py.
- pipelines.py
- CrawlSpider
- process_item (item, spider)
- Scrapinghub
Chapter 4, Web Scraping with BeautifulSoup and Scrapy
by Sam Washington, Dr. M. O. Faruque Sarker, Jose Manuel Ortega
Learning Python Networking - Second Edition
- Title Page
- Copyright and Credits
- About Packt
- Contributors
- Preface
- Section 1: Introduction to Network and HTTP Programming
- Network Programming with Python
- Technical requirements
- An introduction to TCP/IP networking
- Protocol concepts and the problems that protocols solve
- Addressing
- Python network programming through libraries
- Interacting with Wireshark with pyshark
- Summary
- Questions
- Further reading
- Programming for the Web with HTTP
- Technical requirements
- Consuming web services in Python with urllib
- Consuming web services in Python with requests
- Handling forms with urllib and requests with Python 3.7
- Handling cookies with urllib and requests with Python
- Handling HTTP Basic and Digest Authentication with requests
- Summary
- Questions
- Further reading
- Section 2: Interacting with APIs, Web Scraping, and Server Scripting
- Application Programming Interface in Action
- Technical requirements
- Introduction to REST APIs
- Introduction to JSON and the JSON module
- Interacting with a JSON hybrid-REST API (Twitter)
- Introduction to XML
- Working with XML and a full REST API (Amazon S3 bucket) with the Boto module
- Summary
- Questions
- Further reading
- Web Scraping with BeautifulSoup and Scrapy
- Technical requirements
- Introduction to web scraping
- Extracting information from web pages and parsing HTML with BeautifulSoup
- Introduction to Scrapy components and architecture
- Scrapy as a framework for performing web crawling processes and data analysis
- Working with Scrapy in the cloud
- Summary
- Questions
- Further reading
- Engaging with Email
- Technical requirements
- Introduction to email protocols
- Sending emails with SMTP through the smtplib library
- Learning the POP3 protocol and retrieving emails with poplib
- Manipulating and retrieving emails on the server email using IMAP with imapclient and imaplib
- Summary
- Questions
- Further reading
- Interacting with Remote Systems
- Section 3: IP Address Manipulation and Network Automation
- Working with IP and DNS
- Technical requirements
- Principles of the IP protocol
- Retrieving the network configuration of a local machine
- Using Python to manipulate IP addresses and perform CIDR calculations
- The dnspython module as a tool for extracting information from DNS servers
- GeoIP lookups with pygeoip and python-geoip
- Summary
- Questions
- Further reading
- Implementing IPv6 and Address Manipulation
- Technical requirements
- Learning and understanding the IPv6 protocol
- Create an echo client and server with IPv6
- Understanding netifaces module for checking IPv6 support on your network
- Using the netaddr module as a network-address manipulation library for Python
- Understand ipaddress module as IPv4 and IPv6 manipulation library
- Summary
- Questions
- Further reading
- Performing Network Automation with Python and Ansible
- Section 4: Sockets and Server Programming
- Programming with Sockets
- Technical requirements
- Basics of sockets
- Working with UDP and TCP sockets in Python 3.7
- Working with IPv6 sockets in Python 3.7
- Non-blocking and asynchronous socket I/O
- HTTPS and securing sockets with TLS
- Summary
- Questions
- Further reading
- Designing Servers and Asynchronous Programming
- Technical requirements
- Building a multiprocessing-based TCP server
- Building asynchronous applications with asyncio and aiohttp
- Building asynchronous network applications with Tornado
- Building asynchronous network applications with Twisted
- Building asynchronous network applications with Celery
- Summary
- Questions
- Further reading
- Designing Applications on the Web
- Technical requirements
- Writing a web application with WSGI
- Existing web application frameworks (Django, Flask, and Plone)
- The MVC pattern and dynamic web programming with Python
- Creating RESTful web applications and working with Flask and HTTP requests
- Working with a database in Flask with SQLAlchemy
- Summary
- Questions
- Further reading
- Assessment
- Chapter 1, Network Programming with Python
- Chapter 2, Programming for the Web with HTTP
- Chapter 3, Application Programming Interface in Action
- Chapter 4, Web Scraping with BeautifulSoup and Scrapy
- Chapter 5, Engaging with Email
- Chapter 6, Interacting with Remote Systems
- Chapter 7, Working with IP and DNS
- Chapter 8, Implementing IPv6 and Address Manipulation
- Chapter 9, Performing Network Automation with Python and Ansible
- Chapter 10, Programming with Sockets
- Chapter 11, Designing Servers and Asynchronous Programming
- Chapter 12, Designing Applications on the Web
- Another Book You May Enjoy
Chapter 4, Web Scraping with BeautifulSoup and Scrapy
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.