Historical Data

If your pipelines have been up and running for a while, historical cloud costs can help you bootstrap your budget. CSPs offer ways to stratify your bill across different services and filter by tags, labels, and projects to help you zero in on the expenses that relate to your pipelines. Figure A-2 shows a very modest cloud bill where you can see different services broken out.

Figure A-2. The world’s smallest cloud bill, stratified by service type

If you haven’t looked at a cloud bill before, it’s definitely an eye-opening experience to see the myriad things you get charged for. Digging into the details can help you get a sense of how costs break down. As you learned in Chapter 3, cloud storage costs include storing the data at rest as well as events to request, create, and delete objects. Figure A-3 shows the breakdown of storage costs for the S3 costs in Figure A-2.

Figure A-3. Simple storage service cost breakdown

In this case, the primary cost driver is data storage, and you can see that there were so few service requests that they didn’t add to the overall cost at all.

Combining these billing details with your knowledge of the pipeline will help you create a mental model of how the cloud costs spread out across the system. Even a rough sense of the ratio of compute to storage costs can be helpful when trying to estimate future expenses.

This mental model will also help you evaluate forecasting data. Major CSPs offer forecasting tools as part of their billing services, using historical usage to predict future cloud bills. These forecasts are not always correct. Your knowledge of the system plus the historical usage data can help you determine whether a forecast is reliable.

As an example of how this mental model can help with budgeting, a project I worked on had a one-time event where we were tuning autoscaling settings. Over a few weeks, we ran large test ingestion jobs to assess different autoscaling rules, increasing our cloud compute costs.

When we looked at the billing forecast, this one-time increase was forecast as a month-over-month increase, making it appear as though our future cloud compute spend was going up. This was an important point to clarify to company leadership when reviewing the cloud budget, as the forecast provided by the CSP was not accurate.

Estimating for New Projects

If you are working on a new project or if it’s difficult to isolate pipeline costs from other projects in a cloud bill, you can come up with a rough cost estimate. CSPs offer cost estimation tools, such as the AWS pricing calculator, the Azure pricing calculator, and the Google Cloud pricing calculator, that can help with estimating.

Using the benchmarking process from Chapter 1 will help you get a sense of compute needs, which you can scale up for the volume of data and compute complexity you expect over the budget period.

Other costs will vary based on architecture. If you’re ingesting primarily to cloud storage, you can estimate storage and data transfer costs based on expected data size. Standing up a small prototype environment can help estimate costs for databases and other services, something IaC practices can help you with. Running this environment for some test ingestion events will populate billing data you can use to seed your estimation.

Whether you use existing cloud bills or come up with an estimate, you’ve now developed a baseline cost estimate for the pipeline. This might be the limit of what you can provide, which is quite valuable information.

You can think of this as a detailed receipt for the time period you’ve looked into. A budget goes beyond this to provide an expectation of what costs will look like in the future. In the next section, you’ll learn what to look for to determine how your costs might change, enabling you to come up with a budget.

Changes That Impact Costs

In the simplest scenario for a budget, your pipeline continues operating exactly as it has in the past; the same data load, the same data processing logic, and the same infrastructure. In this case, the billing history probably gives you a decent estimate of what to expect in the future.

I have yet to work on a pipeline where this is the case, which I suppose is fortunate because otherwise I’d be very bored. Realistically, you’ll be experiencing some kinds of changes over the budget time period that can impact cloud costs. In this section, you’ll learn some typical changes in data pipelines that can make cost forecasting challenging and ways to address the ambiguity they introduce.

Some of these changes you may be able to estimate, while others are entirely unforeseeable. Remember that a budget is about communicating; if you see changes on the horizon but can’t come up with a cost estimate, make note of these changes as possible risks. Even if you can estimate the cost impacts of changes, it is worthwhile to show that you’ve considered upcoming changes as a part of your budget.

Budget Summary

The budget summary is a short synopsis of the budget. This gives the budget audience a succinct description of what the budget covers, the total cost, assumptions, and risks.

Taking a look at rows 1 through 6 in Table A-1, you have the following sections:

Project

This tells you what the budget covers. In this case, the budget is for the HoD bird identification pipeline introduced in Chapter 6.

Timeline

The timeline refers to the time period the budget covers. In this example, the budget covers the fourth quarter of 2023. It’s important to make sure you present the time period in a way that everyone will understand. For example, Q4 2023 could refer to the fourth quarter of the calendar year, which is October through December. This could be different from the fourth quarter of the fiscal year, so make sure the way you present the timeline is clear to all potential recipients.

Total

This is the total budget for the time period.

Assumptions

Anything you are assuming could impact the budget should be spelled out here. In the example, an assumption is that the current data load from existing customers will be the same over the budget period. There are always going to be assumptions when you are forecasting costs. List major ones here to raise awareness of what you are basing your budget on.

Risks

As you learned in “Changes That Impact Costs”, there are several things that can impact the budget that you might not be able to plan for. In this section, you can communicate what future changes could impact costs.

Don’t try to enumerate every possible scenario, such as detailing that every API you interact with could change. Rather, think about what is on the horizon that you can’t estimate costs for. In the example, the uncertainty of what new customer data will look like is cited. In this case, you know there is a push to bring new customers on board, but you don’t know how that will impact cloud bills because future customer data is unknown.

Cost-saving measures

This is where you can show off the cost-efficient measures you’ve taken. If you can put numbers to this section, all the better; for example, “Saved 40% on compute with reservations and improving data processing efficiency.”

Changes Between Previous and Next Budget Periods

Rows 7 through 14 in Table A-1 communicate important metrics about the system you are budgeting for. This will vary from project to project and can include metrics for system capacity, performance, number of customers supported, or SLAs.

Rows 7 through 10 show the baseline metrics covering the last budget period. Rows 8 through 14 show the expected change over the next budget period. Notice that the expected growth is presented as a range, which helps to further communicate the uncertainty around these impacts.

The metrics in Table A-1 are presented at a system level, but it can also be desirable to break down these metrics in different ways. For example, presenting a per-customer cost distribution can be helpful to inform how to charge for the service.

Cost Breakdown

Rows 15 to the end of Table A-1 are for breaking down the total cost into individual cloud cost categories across different environments. Showing the comparison with the previous budget period gives a sense of how costs are evolving and can also be an opportunity to discuss how costs differed from the prior budget expectations.

The sections you include will depend on where you incur costs. In this example, at a high level are the production, testing, and development environments. Production is, as expected, the largest contributor to the budget and gets its own distinct section. The testing and development environments are combined, as they contribute a fraction of the costs. Make sure to include a budget for all the test and development environments you run, recalling the DEV, TEST, STAGING, and PROD environment example in Chapter 5. In that case, you have PROD and three lower environments to budget for.

There is no DR for this pipeline, so that section is absent. If you have DR, how you represent it in a budget will depend on whether you have a hot, warm, or cold failover system. A hot system, where a production replica is running all the time, will give you a consistent baseline cost to use for a budget. Warm and cold systems are a bit less straightforward, since these only come online at capacity in response to an incident.

In the case of warm and cold DR, you can represent the cost in a budget on a per-incident basis, including an assumption of how long you expect the DR system to be in operation before switching back to the mainline. For example, if it costs $1,000 to bring a cold DR system online and run it for one day, you can represent this as $1,000 per incident in a budget. When estimating DR costs, don’t forget the cost of data replication and egress where appropriate.

Within the environments, the breakdown of cost categories, as shown in rows 18 through 22 in Table A-1, depends on your architecture. In the example, the major sections include compute, egress, storage, and databases. An “Other” category rolls up remaining costs that are too small to deserve their own line item. In this case, “Other” would include the Lambda function developed to alert on night heron content.