Team Skills and Functions

We’ve established that the size of a team should be limited to about 4 to 8 people to keep it effective. But, who are the people on these teams and what should they be good at doing? Establishing the roles of your microservice teams is an imprtant part of desgining the template for the teams in your operating model.

Most microservice experts suggest that teams should be built as “cross-functional” teams. In this type of team, membership includes people with different types of expertise (or function) all working towards the same goal. That expertise can span across not only technology domains - grouping testers, designers and developers together, but also across the business domain - bringing business analysts, product owners and operations people into the team.

The cross-functional team is a well established principle in the Agile domain and is called out specifically in the Scrumm methodology in the definition of a Scrumm team. The primary advantage of building a team this way is that you have all the perspectives necessary to make decisions autonomously. If you have the right people involved, the coordination costs can be contrained to the boundaries of the group. Since we already established an upper limit on team size, this means that the team can move at high velocity with authority. This is a powerful combination and makes for a highly effective team.

But, who are the right people in the context of a microservices team? There are some roles that will be obvious for the work we are doing based on the scope of ownership we defined earlier. We are certainly going to need someone to design the API. We’ll also need someone to develop the implementation as well as someone to test and release the finished implementation. Those could be the same person, or that could be a group of people, but either way those are roles we know we need to fill.

But, what about beyond that core? Do we need a data architect on the team to design the internal data model? Do we require a UX designer to build a front end for the Microservice? Should we have someone from the business team on each of our microservices squads? Does every team need a security officer? What about HR and finance - can we squeeze them in as well?

The principle of the cross-functional team is a sound one, but in practice its a tricky thing to implement and optimize. Especially when we are limited by the practical constraints of limited people with the skills we need and the upper boundaries for an effective team. We’ll need to articulate clearly who needs to be on the team (and who doesn’t)

Some of the rhetorical questions about team membership we asked earlier were probably easy for you to answer. It’s unlikely you’ll need a finance expert on a Microservices team. But, why is that? The build and run costs for a Microservice certainly matter from an organizational perspective. In fact, in some companies the decisions that the finance team makes may be the most impactfull decisions and shape all of the technical decisions as a result.

But, in most cases the finance team is unlikely to be involved in the stream of decisions that exist in the scope of ownership that we defined for our microservice teams. This is the principle that we can keep in mind when we define team roles. The skills we need on our teams are the ones that have a direct relationship on the decisions that are made and the output that is produced. In other words, we want the people on the team to be limited to the people we need to make the best decisions.

So far, we’ve defined a scope of ownership that our teams will have over individual microservices. Later in this book we will also define a constrained scope of responsbilities for a Microservice that limits it to providing an API interface. This is in contrast to some architectural models in which the microservice includes the front end, user experience that the service enables.

This combination of scope and style constraints, coupled with our princple of limiting membership in our teams to people directly involved in our bounded output allows us to articulate exactly the kinds of skills we need on our team as follows:

• Interface Design (API Design)

• Development

• Data Architecture

• Product Management

• Quality Assurance

• Site Reliability Engineer (SRE) / DevOps

In our operating model, every team that owns and operates microservices must have these skills in the team at a minimum. How those skills are distributed amongst team members is a different matter which is constrained by the size of our team as well as by the talent and expertise of the people in our organization.

Talent Shapes

One of the biggest variables that you’ll need to face when you design your microservices teams is the variance in experience and talent within your organization and industry. Microservices architectures are built on cultural principles of moving fast and making autonomous, authoritative decisions. If the people making these decisions are inexperienced, there is a good chance they’ll make a lot of bad decisions before they learn enough to make better decisions.

Generally speaking, giving your teams the freedom to fail is a good thing. The iterative cycle of experimentation, evaluation and improvement is a core part of the methods and culture that Microservices is built upon. A good Microservice architecture should be designed to accomodate mistakes and errors at design, development and at run-time. But, in practice too many or the the wrong kinds of mistakes can be costly at scale and even fatal for the organization.

For example, if team members are making many mistakes at a high frequency, the rate of output of the team will go down, which inhibits your ability to showcase value from your microservice investment. Equally, If the team is making mistakes that will cause problems in production, the risk of a declining customer experience is increased. Worst of all, an inexperienced team may end up making many mistakes in the design and development of a service that will not be realized until much later when the service needs to be updated or improved.

One of the nice things about microservices is that the blast radius of problems is often limited by the scope of the service itself. For example, if a team does a really bad job of writing the code for a single microservice, we can always throw the code away and start again. In fact, this is a good heuristic measure we will use later when we talk about “right sizing” servcies - if you can’t afford to start over, the service has probably gotten too big.

But, if many of our teams are making mistakes at scale, it doesn’t matter that the problems are contained with the microservice. That’s why its vital that your team design template takes into account the types of experts you have access to so you can design for speed, safety and scale appropriately.

The truth is that not every organization has access to the same type of talent. Companies like Google and Netflix share stories of how they compete for the best talent in the world and are willing to pay the costs of high salaries and potentially turnover to attract and maintain the top 1% of talent in the world. Your company may not have the same level of commitment to attracting top talent universally. Does that mean you can’t have a Microservices architecture?

We don’t think so. The key is to design your teams in a way that you can distribute the experience and talent that you do have accordingly. In particular, you will need to play to your strengths and put people together who can collectively achieve the output your are looking for. This means mixing together appropriate levels of experience as well as domain expertise.

Generally speaking, there are two basic skill profiles that work well in a microservices team: “t-shaped” and “m” or “comb-shaped” skills. A “t-shaped” person is someone who has in-depth experience in one domain and shallow experience across a broad range of other domains that are also relevant to the team’s output. Team member with t-shaped skills can act authoritatively within their own area of expertise, but can also easily understand the implications and depencencies to other domain areas, which can reduce the cost of co-ordination within the team.

The “m-shaped” person is a team member who has deep expertise in more than one releveant domain. This is a person who can act authoritatively and independantly across a larger scope of team activities. A complementary group of “m-shaped” people can work together at a very high velocity as their shared skill sets enable them to solve problems faster due to their individual expertise.

TK t-shaped and comb-shaped diagram

One of the big differences between t-shaped and m-shaped skills is that it can be more difficult to find people with deep expertise across the range of skills needed to succeed in a Microservices team. This type of expertise is unique to people with many years of experience and can be costly to acquire.

In order to design the right team template for your system, you’ll need to have a good sense of what the talent in your company looks like. Do you have mostly “m-shaped” people or “t-shaped” people? How experienced are your developers, testers and product owners? Do you have experienced talent across the board, or do you have a few “A” level people who work with a supporting case of steady performers?

You don’t need to do a skills inventory or conduct a survey. But, it is important to get an honest perspective on how talent is acquired and distributed in your organization as this will determine what type of team design template is the right fit for you.

The Hierarchial, T-Shaped Team

[[t-shaped-team] .The T-Shaped Team image::img/x.png["TBD"]

This type of team is ideal for organizations who have lots of people to throw at a problem with highly varying degrees of specialization and experience. In our experience, this is the case for most large Enterprises and established organizations who are used to operating at large scale and have to support a large variety of existing applications.

The blueprint of this team is made up of the following roles: * 4 Microservice Developers (experience ranging form junior to senior) * 1 QA Lead * 1 SRE * 1 Microservice Lead * 1 Co-ordinator or Coach

This is a fairly hefty team, weighing in at 8 people and hits the upper limits of what most experts would deem acceptable for a high velocity, microservices team. But, the larger team size allows us to implement a team with a larger variety of roles and experience within it.

The five Microservice Developers in this team blueprint have the responsibility of writing the server code, implementing the data architecture and operationalizing the Microservice and getting it ready for deployment. They also are responsible for any support and maintenance activities that may be required over the life of the service. This set of developers can include a few developers with junior level experience as long as they are paired with intermediate and senior level engineers who can make decisions based on proven experience. We’ll take a closer look at the development activities that the Microservice developers will need to own in Chapter Seven.

The QA Lead in this team is responsible for the overall quality of the microservice implementation. This responsibility includes ownership of integration and contract tests as well as ensuring that the build and CI/CD pipelines exercise the right types of tests and the level of coverage is adequate. We’ll dive into more detail of the testing activities that the QA lead will be responsible for in Chapter Six.

Because this is a relatively large team, a Scrumm Master is needed to off-load some of the co-ordination costs and to keep the team aligned and moving towards their shared goal. We’ve used the term Scrumm Master here because most of the teams we’ve been working with operate in companies that use the Scrumm methodology. But, if you don’t use Scrumm feel free to replace this role with whichever project and product management role works for your organization.

Finally, the Microservice Lead owns the overall deliverables and owns the authority to make all of the decisions related to the outputs that the team produces. This is a senior level role and should be filled with the top tier of engineering and product talent available within the organization.

In practice, a good microservice lead will delegate responsibility for tasks within the team in order to scale the decision making work appropriately - but it is up to the lead to decide who should own which parts of the responsibility. The Microservice Lead is responsible for the design of the microservice’s interface, the design of the internal data structure and the overall security and realiability of the implementation. The lead also acts as a product owner to ensure that the service is fulfilling the jobs to be done as outline in the SEEDS process that we introduced in chapter three.

How to avoid the Distributed Monolith Anti-Pattern

We recommend sticking to the following principles to walk the line between consistent acceleration and suffocating coordination costs:

Commodities are the things in a market that are “fungible” or can be easily replaced with indistinguishable alternatives. For example, electricity is a good example of a product that is a commodity. Ideally, you should find the functions and services in your system that are commodity-like. In system architecture it’s difficult to find things that are pure commodities, but the goal is to find the things that don’t benefit a lot from innovation and differentiation.

For example, logging is quite often a commodity feature in services architectures. Especially the kind of logging used to track messages across a large system of services. Microservice teams would probably be happy to incoporate a library, tool or service that does that work for them.

When you are working in a single team its easy to make a decision as a group about how you are going to solve your problems. But when your services start to grow in scale you’ll need some way of getting everyone to move in the same direction. When you have libraries and services that all you teams should be using, how do you make sure that happens?

The traditional way to solve this kind of problem has been to mandate a set of rules that everyone has to follow. “you MUST use this logging library if you build a microservice.” But, in our experience designing a set of shared services this way comes with some pretty big validation, enforcement, maintenance and co-ordination costs. There will be times, when this kind of absolute rule is necessary - addressing security and compliance risks is the obvious example.

Better is to allow a tooling market to develop in your microservices system in which teams have the option to utilize the tools that will enable them to succeed. This mirrors our earlier advice in this chapter of using choice architecture and “nudges” to get the types of behaviour you want without introducing unnecessary bottlenecks and co-ordination costs. This works as long as you’ve established a clear set of governance rules as described earlier that teams are responsible for adhering to.

The primary pains of shared libraries come from the problems that arise when they break or need to change. For example, if you make a change to your common logging service that requires every microservice in your system to push out an update - you have a big problem.

Our advice is to adhere to loose-coupling principles as strongly as possible and maintain compatability with older implementations until you absolutely cannot. There should be very few cases where you a microservice team is forced to re-compile or re-develop a solution solely because you’ve made a change to a common component.