Chapter 24. Partnering with Engineers to Enable FinOps
FinOps is a team sport, and very little gets done without the engineers. In this chapter, we’re going to talk about how to instill a culture of FinOps by building up engineering motivation to be cost-efficient rather than dictating a mandate. Mandates only provide short-term action while they are top of mind for leadership. Ultimately, the pressure from mandates lessens and things go back to business as normal, with cost inefficiency creeping back in.
In this chapter, we’ll look at why a software engineer’s day-to-day life makes considering cost optimization difficult and we’ll cover the transitions that engineers tend to make as they begin to integrate cost considerations into their daily workflows.
Many engineers have come from a world where cost considerations weren’t historically part of their job. Now, in the cloud, they are being asked to consider cost as a new constraint. Conveying the context of why the cost of variable infrastructure is now an important piece of their job and its impact on the business is key; otherwise, it may get deprioritized.
In a mature practice, engineers and developers are at the pointy end of the FinOps spear and their early involvement, alignment, and buy-in are crucial to success.
Integrating Us with Them
When revisiting the first edition of this book, we noticed how much it used an us versus them tone when it came to FinOps teams and engineering teams. Some of the phrasing implied that engineers were somehow outside of the work of FinOps and that they didn’t generally care about cost. This is far from the truth in companies who are mature in their cloud—and FinOps—transformation. Lightly engaged engineers who only do reactive cost optimization reflect an immature FinOps practice and maturity, where inefficiencies frequently are baked into the development process and FinOps processes are limited to cleaning up the waste after the fact reactively. Instead, mature practices partner with engineers early on in service and architecture design to proactively avoid inefficiencies in the way services are deployed on cloud.
The personas driving FinOps have shifted in the last few years. In the State of FinOps 2022, the team responsible for driving FinOps overwhelmingly reports to the CTO or CIO and is staffed with—and increasingly led by—engineers. Once the cost-conscious culture is instilled, the work of FinOps itself is performed by engineers around the organization, leaving more focus on other functions for the central FinOps team. As we mentioned in Chapter 15, many critical functions of FinOps—like usage reduction and optimization—are not typically done centrally by the FinOps team; rather, those responsibilities are distributed directly to engineering teams in various business units or groups. The engineers who are deploying services know their infrastructure better than anyone and are best suited to optimize it.
What’s on the Mind of the Engineer?
Engineering motivations stand in contrast to finance team motivations. The latter tend to focus on business viability, gross margins, and future financial stability of the business through the lens of dollars spent against budgets and forecasts. Engineers are motivated to innovate, deliver value through software improvements, and generally make things more efficient across a range of areas including security, performance, uptime, and reliability.
In Chapter 3, we talked about how to increase the motivation of teams and reduce the loss of trust. Recall the action scale introduced, where there is balance between the motivation and the effort required. There is sometimes a misperception that engineers aren’t motivated to focus on cost efficiency; in reality, their FinOps work simply competes with other critical constraints and considerations.
To balance the action scale, FinOps practitioners need to work with engineers (and engineering leadership, who set their goals and incentives) to make cost efficiency as big a consideration as security, performance, reliability, or sustainability (as discussed in Chapter 19). Focusing on only one measure of success is not an option.
A perceived lack of engagement from engineers is not typically a lack of motivation. Due to the long list of areas for which they are responsible, they must make regular trade-off decisions. If FinOps requires a large—or unclear—amount of time, then it may not fit into the engineering schedule unless prioritized by leadership.
It’s important for FinOps to look for ways to reduce the load on engineers, especially with automation. As discussed in Chapter 8, putting FinOps data in the path of the engineer will enable faster decision making and reduce cognitive load lost to context switching. We don’t want engineers to spend high amounts of time (or brain power) to figure out how their actions are affecting spend projections, not only taking them away from other priorities but also reducing the likelihood that they will tend to their FinOps responsibilities. FinOps reports should give engineers a clear outline of what actions they need to consider and the precise data supporting those actions, decreasing the load and time commitment needed for engineers to consider cost.
Constraints and the Solving of Hard Problems
Humans don’t like to be constrained. But constraints are required for innovation.
To create something great requires two things, a plan and not quite enough time.
L. Bernstein
Cost guardrails can often be met with a sentiment of “We went to the cloud to have fewer constraints, and now you are adding more constraints back on me.” A further refrain we hear from engineering teams is “Tell me where you want to go, but don’t tell me how to get there.” By collaborating to set mutually agreed-upon cost guardrails in the form of budgets and forecasts—and providing trusted data on cloud spend and usage to teams managing infrastructure—FinOps can provide engineers the constraints needed to tell them where they need to go in terms of cost, while leaving to them the innovation of how to achieve that cost efficiency. It also helps if leadership provides the why of their cost efficiency work to the larger business.
A clear vision for the desired outcome delivered via a clear set of constraints is the goal. Without constraints, the development process can be too freeform and may not ultimately align with the desired outcome. Conversely, too many constraints means there is not enough room for creative engineering to occur, as it leaves no room for innovation. Some of the best solutions to cloud efficiency come by allowing room for engineers to be innovative, with clear cost constraints but not overly prescriptive guidance for achieving them.
Rather than allowing engineers to choose any available cloud architecture and then cleaning up associated cost inefficiencies later, early cost constraints reduce unnecessary waste and the need to clean up future expensive technical—and cost—debt. However, even an architecture designed to be cost-efficient at the start can almost always be optimized over time, as code efficiencies are introduced, new cloud features are released, and real-world load evolves.
Engineers like to solve hard problems. This is evident from the number of engineering success stories you hear at major cloud providers’ annual conferences. Most of these talks cover how companies have used the cloud to scale, be more innovative, add new features, and introduce completely new business opportunities to an organization.
Engineers who build cloud solutions that fit within the boundaries of cost constraints should be celebrated as innovative. Give engineers a platform to share the technical difficulties they overcome with creative engineering solutions to meet their cost constraints.
Benjamin Coles, Engineering Manager in FinOps at Apple
Chapter 13 explored different levels of cloud budgeting and how some teams new to cloud can push back on the concept of having a budget. Ironically, there has always been a budget for IT spending in most organizations with related hardware constraints, but that spending was abstracted away via hardware that was purchased sometimes years in advance. Dollars themselves are being introduced as the new constraints, rather than quantities of available computing power as was historically the case.
Principles for Enabling Cost-Efficient Engineering
After introducing the concept of dollars as a constraint and the Toyota versus Porsche analogy in his talk at FinOps X 2022, Gabe Hege shared a set of principles for both business and technical people to remember when working on a FinOps transformation. Gabe is an engineer who works in a FinOps enablement team at a Fortune 100 company, but who reports to a finance director who ultimately reports to the CFO.
As a cautionary tale, Gabe quipped that the “only limitation of cloud is how deep your pockets are. But there’s a big difference between limitless and free. Cheap used a lot is expensive.”
The following sections summarize Gabe’s six principles for FinOps teams to work more effectively with their engineering partners.
#1: Maximize Value Rather Than Reduce Cost
Optimizing everything isn’t always the answer: you may have a local optimization opportunity, but not one that globally helps the business. The engineering perspective is that finance is hyperfocused on spend. If finance is able to reframe themselves as being focused on business value, then engineers can relate more closely to that and will be more open to trade-off conversations aligning with the FinOps principle of decisions being driven by the business value of cloud. Engineers are comfortable making trade-offs—it’s a core part of their job.
Gabe’s take on this: “You want to enable them to maximize value. It’s not just about spending less, it’s about spending on the right things. For example, in a manufacturing example you might decide to use a more expensive material in order to get more out of it.”
Aim to show engineers how they can add value to the company, and then ensure that they are recognized for doing so by enabling them to say, “We were able to get performance increases while also driving cost decreases.” Anyone can throw more money at a performance challenge, but the really hard problem is to enable cost-efficient performance.
#2: Remember That We Are on the Same Team
Teams need to collaborate closely for FinOps to flourish. In his own organization, Gabe related the partnership to a bowling analogy: “Finance is the bumpers and engineering is the bowling ball. Together, we are aiming to knock down the most pins in the least amount of turns.”
For many, the push to start or accelerate adoption of FinOps comes out of the finance side of the house, who push their engineering counterparts to get on top of costs and chase them with reports telling people to go reduce waste. Too often this comes with an accusatory tone and without enough understanding of how the cloud actually works to understand why engineers were making the choices they made.
#3: Prioritize Improving Communication
Gabe went on to say that “a lot of the terminology used in finance doesn’t register with engineers. Likewise, engineering terminology doesn’t register well with those used by finance. It’s so important to be on the side of the engineer and not be against them. I look at FinOps work as a partnership where we also must help the nonengineers, like those in finance, understand what it looks like to partner with engineering. And to help both understand how the FinOps culture will meld into their daily lives: FinOps is not a separate thing, it should blend in until it’s a habit just as security work must become embedded into the process.” The importance of a common language around FinOps and cloud was covered in Chapter 4.
He proposed that organizations consider moving some engineering talent into finance, and alternatively have finance participate in some engineering activities so they can more directly understand the constraints engineering has to consider during trade-off conversations.
#4: Introduce Financial Constraints Early in the Product Development
If you introduce strong financial constraints at the end of the development cycle, do not expect them to be in the next release. Quick wins might move the needle in the right direction, but they are unlikely to get you to the long-term goals (see the Afterword of the book for more on prioritizing long-term gains over short-term wins). Financial constraints take time to implement, especially if introduced late in the game, as they may not be compatible with fundamental choices made early on, requiring rearchitecting. The cost to rearchitect may ultimately not be worth the cloud cost savings when compared to the engineering time required to do so and its impact on delivery schedules of other releases. As Gabe said in his FinOps X talk: “If finance only shows up at the point of a release hitting staging or production, then it’s too late. You need to bake cost constraints into the beginning, otherwise you’re often asking engineering to build a new product.”
#5: Enablement, Not Control
Requirements are the main thing engineers focus on. Their job is first to cut out the noise of the requirements to determine what really needs to be built. Then they examine the constraints to determine how to build it. If you have overly rigid restraints, you end up with what Gabe calls technicians: those who are told to simply “turn the valve this much” without ever innovating. On the flip side, you have what Gabe called artists, who go too far in crafting artisanal and usually overly expensive solutions to a problem. There’s a classic episode of The Simpsons where Homer is given free rein and an unlimited budget to design the perfect car. What comes out the other end is an expensive monstrosity. Without finding an appropriate level of cost guardrails and other constraints, innovation can actually suffer.
In the middle there are enough constraints to be creative, but it’s not so rigid that you can’t find an elegant solution. Simple guardrails like a total cost or a list of approved cloud service SKUs or approved regions can create innovation. But if you do that, you need to provide other parameters where engineers have flexibility. For example, if you stay within this budget and avoid these services or regions, then you can do whatever you want.
On this principle, Gabe ended by sharing that “we are so worried about artists but we also don’t want technicians. Too many constraints turn into restraints. Enable people to do the good behavior you want. And remember that all engineers do not know everything about the cloud; education is key.”
#6: Leadership Support Isn’t Helpful, It Is Essential
Leadership is the single most important factor of the success of a FinOps transformation, or any organizational transformation for that matter. If your engineering and finance leaders don’t agree on the importance and the desired outcomes, then the cultural change is doomed to fail. Again, it’s not that you need mandates from leadership but that you need leaders to constantly send signals that cost—and its ultimate outcome of business value—is important at all stages of development. The message should not be a mandate such as “reduce cost by 10%,” but rather a constant drumbeat that “cost efficiency is always important and is a core part of your job.”
Here Gabe referenced his former job as a security engineer, saying, “You can’t just do occasional security sprints; you’re always baking security into every part of the development lifecycle. If a new feature breaks your cost constraints then it’s a bug, just as it would be with security. You have to apply that level of discipline.”
Data in the Path of the Engineer
Engineers—especially developers—often operate in cycles of deep focus, often referred to as flow. Interrupting one of these cycles can greatly impact the engineer’s performance. It’s for this reason that you should interrupt engineers only when the timing of the response or action needs to be immediate; using real-time communication methods such as chat, video call, or in-person walk up will pull engineers away from their train of thought and disrupt their focus. If the action you need them to make can wait, then consider how you can inform them of the task in a way that won’t be disrupting.
Returning to the concept of data in the path of the engineer introduced in Chapter 8, having FinOps data exist in the places your engineering teams are already working is a key contributor to reducing the effort needed by engineers to engage with FinOps. Integrating with engineers’ existing processes avoids the effort required for them to open a new set of reports and run additional rituals around FinOps. However, injecting FinOps into engineering team reports and processes unannounced should be avoided, as this can really interrupt the flow of engineering team rituals. Ideally, engineers should help guide where inserting FinOps data will most help them within their existing work processes.
If leadership mandates to their engineers that they must clean up cloud waste immediately and cut costs by 10%, it’s likely they will see quick results. Mandates are a quick way to get action from engineers. But they are often not sustainable and can create a negative culture of whiplash. As soon as the mandate goes stale and is no longer being reinforced by leadership, it falls out of the minds of engineering teams, who fall back into focusing on delivery and optimization of other key metrics. Whereas the negative feelings toward FinOps are long-lasting.
Cost mandates are point-in-time impacts that can be effective in the short term, but you will run into the same creeping waste again over time. The broader message that cost metrics are important to the business should be continuously reiterated as part of the way engineering operates.
When you start to build in at the cultural level that cost efficiency is a core value, it becomes part of the workflow of services. Aim for rehearsed practice and building muscle memory. FinOps becomes part of the whole story. Culture doesn’t go away—it just becomes part of the day-to-day.
Models for Partnering with Engineering Teams
Each FinOps team is going to partner with their company’s engineers in different ways, driven by the mix of knowledge and skills of the members on the FinOps team.
Direct Contribution
FinOps teams with engineering skills often take a more proactive approach by directly helping engineers design cloud services to be more efficient and work on tools that avoid creating waste up front. This is a powerful method of building a collaborative environment between engineers and FinOps teams that can be more effective at keeping the amount of usage optimization to a minimum. For very large companies, this type of partnership can have its limitations, as the number of engineering teams actively working on cloud can overwhelm the capacity of the FinOps team.
Indirect Collaboration
Not all FinOps teams are staffed with engineers, but this does not mean that they will be ineffective at partnering with the engineering teams around them. This type of partnership will often see strong collaboration between the FinOps team and the Cloud Center of Excellence (CCoE) or Technical Architecture Group (TAG) within the organization. Lean on the TAG and/or CCoE teams to help drive all engineering teams to design more efficiently and use their deep cloud knowledge to help FinOps understand current cloud spend, predict future costs, and clearly identify opportunities for optimization.
Indirect Collaboration with Targeted Contribution
As a hybrid approach to the partnership with engineering teams, take the indirect collaboration model but use the engineering skills of the members of the FinOps team to target specific areas of the business. The FinOps members could work directly with your engineering teams with the largest spend or focus on the least cloud mature engineering teams within your organization to uplift them. The FinOps team members performing this direct collaboration may float around the organization as the need changes or be dedicated to a specific engineering domain. At Atlassian, there are cost engineers who embed within other teams to perform this function.
Picking a model to build your partnership should be done by first assessing what skills are in the FinOps team, how well established your CCoE/TAG teams are within engineering, and the maturity of cloud adoption across your organization by all engineering teams. Choosing one model does not lock you into that way of partnering forever, and as your cloud footprint scales, the hybrid model becomes more and more likely to be where you end up.
Conclusion
It is our hope that you’ve gotten a better understanding of the engineering persona and the central role it plays in FinOps. Partners from all sides need to listen and break down challenges that aren’t necessarily engineering-specific. We can’t lob a challenge over the fence and expect it to be pushed to completion. Engineers have a huge set of competing priorities they are juggling. When cost awareness is introduced, additional cognitive load comes with it. The method and timing of communication with engineers can greatly impact the perception of FinOps being a partnership versus a mandate.
Friction around making cost-efficiency improvements typically comes from one of these areas:
Other priorities being set by their leadership
Tight deadlines to deliver features or services
Concerns that changes might have negative consequences for performance
Unclear expectations for engineers, or the value of the effort is not well communicated
To summarize:
Building a partnership with engineers is more important than getting a list of FinOps tasks completed.
Cost savings is not the goal of the partnership, but rather efficient value for money spent.
Align with engineering teams on the financial constraints early in the development process to reduce wasted effort later.
Leadership should aim to avoid mandates (i.e., short-term cost focus periods) but instead encourage teams to continuously monitor and manage the needs of FinOps as part of the workflow and culture in the organization.
If your FinOps team doesn’t have the skills to review/recommend engineering designs/changes, collaborate with the architecture team(s) within your organization.
The next chapter looks at some other types of partnership in the organization, interacting with teams managing other IT frameworks.