Chapter 6: Serverless ETL
Technologies
Serverless tenology is exciting because it doesn’t exist without Cloud computing. e Cloud-
Native terms arise because they are a “native” capability of the distributed, elastic compute envi-
ronment provided. e word serverless means that servers are not top of mind in solving problems.
If you like to solve problems quily, as I do, then you will love serverless. A thought turns into
code, whi turns into a solution.
One way to get started with the examples in this apter is to wat the following screencast server-
less cookbook with AWS and GCP. e source code is in the Github Repo hps://github.com/noahgi/serverless-
cookbook²⁹¹.
Video Link: hps://www.youtube.com/wat?v=SpaXekiDpFA²⁹²
AWS Lambda
AWS Lambda is a building blo for many things on AWS. It is a great place to start when diving into
serverless tenology. First, let’s dive into what occurs under the hood when you use AWS Lambda,
as shown in the diagram. A house with a lightbulb in the garage can turn on many ways, the light
swit, or the garage door open event. A Lambda responds to many signals as well.
²⁹¹https://github.com/noahgift/serverless-cookbook
²⁹²https://www.youtube.com/watch?v=SpaXekiDpFA
138 Chapter 6: Serverless ETL Tenologies
Events
Learn how to AWS Lambda as a Garage Lightbulb in the following screencast.
Video Link: hps://www.youtube.com/wat?v=nNKYwxf96bk²⁹³
An excellent way to get started with a first Lambda function is via a simple example.
Learn how to build a Marco Polo Lambda function in the following screencast.
Video Link: hps://www.youtube.com/wat?v=AlRUeNFuObk²⁹⁴
You can find the code for the example below.
gist for Marco²⁹⁵
1 def lambda_handler(event, context):
2 if event["name"] == "Marco":
3 return "Polo"
#### Invoking AWS Lambda from the CLI
²⁹³https://www.youtube.com/watch?v=nNKYwxf96bk
²⁹⁴https://www.youtube.com/watch?v=AlRUeNFuObk
²⁹⁵https://gist.github.com/noahgift/3b51e8d800ea601bb54d093c7114f02e
139
A convenient tri is to use an AWS Cloud Shell or AWS Cloud9 environment to invoke an AWS
Lambda. How could you do this?
1 aws lambda invoke --function-name MarcoPolo9000 --payload '{"name": "Marco" }' out.t
2 xt | less out.txt
e real example is in Github here²⁹⁶.
AWS Step Functions
Note you could also ain many of these functions together using AWS Step Functions²⁹⁷. You can
see an example of the workflow of a ain of Lambda functions in the diagram.
²⁹⁶https://github.com/noahgift/serverless-cookbook/blob/main/marco-polo-lambda.py
²⁹⁷https://aws.amazon.com/step-functions/
140 Chapter 6: Serverless ETL Tenologies
Screen Shot 2020-11-30 at 7 46 07 PM
141
Here is the code for the example.
gist for Polo²⁹⁸
1 def lambda_handler(event, context):
2 if event["name"] == "Polo":
3 return "Marco"
Notice how ea function emits an output than then goes to the next operation. e following code
example is in Github²⁹⁹.
1 {
2 "Comment": "This is Marco Polo",
3 "StartAt": "Marco",
4 "States": {
5 "Marco": {
6 "Type": "Task",
7 "Resource": "arn:aws:lambda:us-east-1:561744971673:function:marco20",
8 "Next": "Polo"
9 },
10 "Polo": {
11 "Type": "Task",
12 "Resource": "arn:aws:lambda:us-east-1:561744971673:function:polo",
13 "Next": "Finish"
14 },
15 "Finish": {
16 "Type": "Pass",
17 "Result": "Finished",
18 "End": true
19 }
20 }
21 }
You can wat a Marco Polo Step function in the following screencast.
Video Link: hps://www.youtube.com/wat?v=neOF0sxmYjY³⁰⁰
Another excellent reference is the Web Scraping Pipeline Github Project³⁰¹
²⁹⁸https://gist.github.com/noahgift/f2f5f2bc56a3f39bf16de61dbc2988ec
²⁹⁹https://github.com/noahgift/serverless-cookbook/blob/main/marco-polo-step-function.json
³⁰⁰https://www.youtube.com/watch?v=neOF0sxmYjY
³⁰¹https://github.com/noahgift/web_scraping_python
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.