Options

This set of requirements is, in fact, quite unique. When you typically speak to a client about their requirements for a system, they will not mention non-functional requirements , such as scalability. The reason is that they will assume that whatever functionality they ask for will be scalable, reliable, usable, extensible, secure, accessible, and all the other things that, as a software architect, you need to explicitly consider. The reason for this is that most modern systems are almost all of those things, out of the box. Taking ASP.NET as an example (since our example code is in .Net), you can create an ASP.NET website that will, with zero lines of code, be secure, usable, extensible, and accessible; depending on how you deal with the hosting, it can also be very scalable.

As the person designing the system, it then falls to you to identify if these requirements involve sacrifices elsewhere. The devil is in the details. What exactly do we mean by usable? What does reliable mean? Does it mean that the system should have zero downtime? Exactly what, about the system, would need to be extensible: Do the users of the system require the ability to extend the system, or do we just mean that a change to the functionality is quick and easy?

In truth, when we go to extremes with any of these nonfunctional requirements, we often sacrifice others. Let’s take making the system both usable and accessible as an example; should we need to make the system accessible to people with very poor eyesight, we may increase the size of controls and text, place fewer items on a screen, and perhaps incorporate some form of voice control; however, when you gave that system to a person with perfect eyesight, they may not describe the system as being usable.

In our case, the client has realized that their requirement of user numbers is not standard and therefore requires special mention. Our job at this stage is to highlight to them what, if anything, we would need to sacrifice in order to provide that scalability.

In previous chapters, we’ve approached the problem by first exploring what the system would look like without automation.

Manual Process

Admittedly, this scenario probably lends itself less to a manual process than any that we have discussed so far in this book; however, let’s complete the thought experiment and see where it leads.

The requirement here, in its simplest form, is that users can post details of their workout, and other users can comment on that – the other points are merely constraints and concessions. If we start here, we may envisage a notice board in each gym where people would perhaps fill out a card such as shown in Figure 4-1.

Here, we’re asking the user to populate the length of the workout, calories burned, comment, etc. We may then imagine that once this is posted, other users would add Post-it notes to the card. This seems to work well to an extent, but we have a problem with making this scalable: even within the same gym. For example, after a certain number of posts, the board would be full; further, the comments themselves would obstruct the posts.

Let’s imagine a slightly different scenario then: a printed sheet of the workouts for that week. After a member has completed their training, they would still fill out a card, but this time, they would hand the card to a member of staff in the gym. The cards would be piled up, and once every day, someone would type them up and print them out. We could follow a similar principle for comments: each time a member wished to comment, they would fill out a card and hand it to a member of staff, who would include the comments in the printout for the following day.

If we wanted to make this work for all the sites then we would, perhaps, have the member of staff post the comments to a central location, who would type out the posts and comments and send the sheets to every gym.

We did say initially that this would not lend itself to a manual process, but I suspect this mental picture will help when thinking about the design of the system.

CQRS

We’ve already discussed the concept of CQRS when we were looking at event sourcing in Chapter 2. However, in this case, we’re examining CQRS in its own right. Let’s first remind ourselves of what CQRS is, and then we can explore whether it will help us with our current problem.

CQRS stands for Command Query Responsibility Segregation . According to a 2011 post by Martin Fowler, this was something that was brought into the public domain by Greg Young.

The idea behind this pattern is, as the name suggests, that you segregate parts of your application that read data from those that write it. This provides some huge benefits, but it is far from without cost; we’ll explore both here.

Benefits

Let’s imagine that we decided to use a typical transactional database for our social media design. Figure 4-2 illustrates a familiar pattern.

The data store in this case is very likely ACID compliant (see Chapter 3). What this means is that each time you write data to a table, the application (and the user) must wait while the data is written, indexes are updated, constraints are checked, etc. This process is lightning fast, and so it can manage a high throughput. At the same time, you’re reading information from those same tables and very possibly updating records too. Let’s have a look what our relational data structure might look like for our social media solution.

Figure 4-3 shows a potential data structure (in fact, it doesn’t really matter what the data structure is for the purposes of this discussion).

As posts are being written to the database, comments may be being added. As we’ve already said, modern databases are incredibly good at performing these types of tasks quickly, but when your user base gets to very big numbers, that might not be quickly enough. If that is the case, then we need to ask what we can do about that and what is important, and what is not.

Since we’re listing the benefits of CQRS here, let’s think about how using CQRS can help us here. One way it could help is if we created a separate data store and wrote to that instead. To be clear, this need not necessarily be a separate database (although it can be); for example, we could introduce a system such as that shown in Figure 4-4.

You would then have an offline process that simply took the elements of the queue tables and wrote them to the Post and Comment tables. Although this does provide some benefit, more typically, the Command side writes to a different store.

Note

I am very deliberately using the term “data store,” rather than database, the reason being that there are many viable places to store such data and they are not all in a typical database.

The principle here is simply that you read from a read-only location. The location that you write to is obviously not write-only, because you would never be able to extract that information, but effectively you write to a buffer location. It may be that you decide to combine this approach with event sourcing; however, you could simply have your write location as a message bus.

As with everything relating to software, this is a trade-off.

Consistency Models

Data consistency refers to the state of the data across the system. For example, if I make a change to the data, a system that is said to be strongly consistent would show the same data state to all the users of that system; additionally, the user that made that change initially should see that change reflected.

Figure 4-5 illustrates what a strongly consistent system would look like. At a given point in time, all three users shown in Figure 4-5 would need to see the same information when looking for Field1.

This kind of model is typical when dealing with a system that has a single data store. It is bread and butter for most RDBMSs – a user updates a field, and all other users see that change. However, let’s imagine the scenario in Figure 4-6.

Looking at Figure 4-6, it’s difficult (although not impossible – see the “Distributed Transactions” section in Chapter 3) to imagine a situation where that setup would be strongly consistent.

Note

There are dozens of consistency models. In this section, I’m going to group them into three categories and cover a broad explanation of that category; however, in doing so, we will miss some of the nuance of the individual models.

Strong or Strict Consistency

As we’ve already alluded to, strongly consistent data is data that is reflected across an entire system in such a way that as the data is written, it is visible to all other parts of the system in the order in which it was written.

In this model, we pay a performance price in order the see the same view of the data across the system.

Sequential or Causal Consistency

In causal consistency, we no longer expect to see the data as it is written across the system; however, we do expect to see the data in the order in which it was written across the system. For example, if I write three pieces of data, [a, b, c], I would expect to see them written in that order.

In this model, we get better performance than with strong consistency, but we need to maintain the order; this means that an additional payload must travel with the data itself that gives a dependency graph of data. This, therefore, increases the amount of data being passed around and, in order to maintain the order, can result in blocking calls.

Weak or Eventual Consistency

The eventual consistency model allows a situation where I can write some data and that the data will be available for reading at some stage in the future, but in no guaranteed order.

This is by far the fastest model, but we pay a high price in that you can end up in a situation where either you are unable to see data that has been written or you are able to see it in a state where it is not final; for example, imagine the following customer data:

• Customer Record: Customer Id, Customer Address Id.

• Customer Address Record: Address

In eventual consistency, we can have a situation where either the address exists without the customer or the customer exists without the address. In fact, the only guarantee that we have is that when we have finished processing (i.e., eventually), everything will be created correctly.

For our purposes, eventual consistency seems like an excellent option – if a post is made on the site, it doesn’t really matter if that post is not immediately visible, nor does it matter if the comments don’t appear at the instant they are posted.

Schema Creation

Looking at our target architecture, there are only two conceptual parts to it (excluding the databases); these are the Client and the Process Data Service.

Updating the Database

When deciding how we should update the database, there are some architectural decisions to consider.

Checking the Data

Listing 4-2 gives a SQL script that will display the posts, along with the count of comments next to them. This will be useful later on.

As with previous chapters, all the code can be found in the following GitHub repo:

https://github.com/Apress/software-architecture-by-example

Web Service

Since we need a web service, our first step is to create one to wrap the database update calls.

Appendix A covers the finer details of creating an API in .Net – so here we’ll just list the changes once the API is created (essentially, once the default API project has been created).

Let’s now discuss how we might access the MongoDB to update it – spoiler: this will essentially be filling in the methods that we’ve used before.

Accessing MongoDB

In order to access the MongoDB, we’re going to create a wrapper project.

As you will see from Figure 4-9, we have a very simple project, consisting of an interface and implementation class.

I’ve listed these two methods together in Listing 4-6, as they, essentially, do the same thing – firstly, they instantiate an object (e.g., Comment or Post) with appropriate values. Following this, we get a typed collection from Mongo; that is, we ask the DB for a collection of Posts or Comments; and then, once we have one, we add to that collection and return the ID.

In Listing 4-7, we can see that we first return the collection as before using the GetCollection method; then, instead of inserting, we call FindOneAndDeleteAsync; this takes the first record that it finds according to the criteria provided, returns the record, and then deletes it (so that we don’t re-read the record).

The next thing that we’ll need to do is to create our client application.

The Client

The client application, for this example, will simply be a console application. We’re basically going to call the web services – as we discussed earlier, given the required deployment model, we have little choice on this.

There’s quite a lot of code here, but it’s all very simple: we display the menu and then respond to the keypress. Options 1 and 2 create a single Post or Comment; option 3 creates a Post and then a Comment for that Post; options 4 and 5 are for the purpose of testing load, and they create many posts and comments.

The first thing we do here is use the HttpClientFactory to create an HttpClient object. The HttpClient object allows us to make HTTP calls outside of the application.

Following this, we call PostAsync to invoke the HttpPost method of the controller.

From the code in Listing 4-10, we can see that the only real difference is that the CreateComment method accepts a parameter, which it passes to the /comment endpoint. Also, CreateComment does not return a value, whereas CreateSinglePost does; you’ll notice that we’re accepting this parameter as optional and, where it isn’t supplied, we pick a random Post.

Listing 4-11 shows that when a post is created, we add that post to an in-memory list of posts.

The final piece of the puzzle is the offline process that reads from the MongoDB and writes that information to SQL Server.

Process Data Service

The ProcessDataService is a very simple console application.

As we can see, all we’re doing here is establishing a SQL Server connection, an instance of the MongoDBWrapper, and passing both into a separate method: ReadPosts.

In Listing 4-13, we can see that the ReadPosts method is a loop that is exited only where GetNextPost returns null; that is, where there are no further posts available. Following this, a transaction is established (for more discussion on transactions, please see Chapter 3). We then insert the post into the SQL table and subsequently call ReadComments (see Listing 4-14). Following this, we commit the transaction.

The ReadComments in Listing 4-14 method is very similar to the ReadPosts method that we saw earlier; we read the comments from the Mongo database and try to insert them into the relational database; we do this using the same transaction, meaning that the post and comments will either get written together or not at all.

Figure 4-11 shows the data that has been transferred from the MongoDB across to the relational database.