In this section, we will explore Scrapy for deploying spiders and crawlers in the cloud.
Working with Scrapy in the cloud
by Sam Washington, Dr. M. O. Faruque Sarker, Jose Manuel Ortega
Learning Python Networking - Second Edition
- Title Page
- Copyright and Credits
- About Packt
- Contributors
- Preface
- Section 1: Introduction to Network and HTTP Programming
- Network Programming with Python
- Technical requirements
- An introduction to TCP/IP networking
- Protocol concepts and the problems that protocols solve
- Addressing
- Python network programming through libraries
- Interacting with Wireshark with pyshark
- Summary
- Questions
- Further reading
- Programming for the Web with HTTP
- Technical requirements
- Consuming web services in Python with urllib
- Consuming web services in Python with requests
- Handling forms with urllib and requests with Python 3.7
- Handling cookies with urllib and requests with Python
- Handling HTTP Basic and Digest Authentication with requests
- Summary
- Questions
- Further reading
- Section 2: Interacting with APIs, Web Scraping, and Server Scripting
- Application Programming Interface in Action
- Technical requirements
- Introduction to REST APIs
- Introduction to JSON and the JSON module
- Interacting with a JSON hybrid-REST API (Twitter)
- Introduction to XML
- Working with XML and a full REST API (Amazon S3 bucket) with the Boto module
- Summary
- Questions
- Further reading
- Web Scraping with BeautifulSoup and Scrapy
- Technical requirements
- Introduction to web scraping
- Extracting information from web pages and parsing HTML with BeautifulSoup
- Introduction to Scrapy components and architecture
- Scrapy as a framework for performing web crawling processes and data analysis
- Working with Scrapy in the cloud
- Summary
- Questions
- Further reading
- Engaging with Email
- Technical requirements
- Introduction to email protocols
- Sending emails with SMTP through the smtplib library
- Learning the POP3 protocol and retrieving emails with poplib
- Manipulating and retrieving emails on the server email using IMAP with imapclient and imaplib
- Summary
- Questions
- Further reading
- Interacting with Remote Systems
- Section 3: IP Address Manipulation and Network Automation
- Working with IP and DNS
- Technical requirements
- Principles of the IP protocol
- Retrieving the network configuration of a local machine
- Using Python to manipulate IP addresses and perform CIDR calculations
- The dnspython module as a tool for extracting information from DNS servers
- GeoIP lookups with pygeoip and python-geoip
- Summary
- Questions
- Further reading
- Implementing IPv6 and Address Manipulation
- Technical requirements
- Learning and understanding the IPv6 protocol
- Create an echo client and server with IPv6
- Understanding netifaces module for checking IPv6 support on your network
- Using the netaddr module as a network-address manipulation library for Python
- Understand ipaddress module as IPv4 and IPv6 manipulation library
- Summary
- Questions
- Further reading
- Performing Network Automation with Python and Ansible
- Section 4: Sockets and Server Programming
- Programming with Sockets
- Technical requirements
- Basics of sockets
- Working with UDP and TCP sockets in Python 3.7
- Working with IPv6 sockets in Python 3.7
- Non-blocking and asynchronous socket I/O
- HTTPS and securing sockets with TLS
- Summary
- Questions
- Further reading
- Designing Servers and Asynchronous Programming
- Technical requirements
- Building a multiprocessing-based TCP server
- Building asynchronous applications with asyncio and aiohttp
- Building asynchronous network applications with Tornado
- Building asynchronous network applications with Twisted
- Building asynchronous network applications with Celery
- Summary
- Questions
- Further reading
- Designing Applications on the Web
- Technical requirements
- Writing a web application with WSGI
- Existing web application frameworks (Django, Flask, and Plone)
- The MVC pattern and dynamic web programming with Python
- Creating RESTful web applications and working with Flask and HTTP requests
- Working with a database in Flask with SQLAlchemy
- Summary
- Questions
- Further reading
- Assessment
- Chapter 1, Network Programming with Python
- Chapter 2, Programming for the Web with HTTP
- Chapter 3, Application Programming Interface in Action
- Chapter 4, Web Scraping with BeautifulSoup and Scrapy
- Chapter 5, Engaging with Email
- Chapter 6, Interacting with Remote Systems
- Chapter 7, Working with IP and DNS
- Chapter 8, Implementing IPv6 and Address Manipulation
- Chapter 9, Performing Network Automation with Python and Ansible
- Chapter 10, Programming with Sockets
- Chapter 11, Designing Servers and Asynchronous Programming
- Chapter 12, Designing Applications on the Web
- Another Book You May Enjoy
Working with Scrapy in the cloud
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.