Microservices: Brownfield: Reporting

 icon-arrow-left Microservices: Brownfield: Transactions

The support reporting for our micro-services architecture system can be a little complex. At difference to a monolithic architecture where you may have a few databases to report from, in the micro-services architecture system, you will have a tons of databases to report from since each micro-service will have its own database.

The report of the data will come from the be split across multiple micro-services and since there is no a central database where you could extract this information you may need to join data across databases. Also, in the micro-service architecture system, reporting can be slow. 

One way to facilitate the reporting is to have a dedicated reporting micro-service which calls all our micro-services and takes care of collecting and consolidate the data. The only disadvantage is when we are reporting large volumes of data or we wish to obtain a report in real-time.

Another way is to have a data dump which is having the micro-services dumping the data to a central database that later can be use to do reporting. 

 

Share
Leave a comment

Microservices: Brownfield: Transactions

 icon-arrow-left Microservices: Brownfield: Migration: Database | Microservices: Brownfield: Reporting icon-arrow-right 

When moving from a monolithic system to a micro-service architectured system, we need a different approach when dealing with transactions.

Transactions are useful:

  • They ensure data integrity.
  • They allow us to updates several records as part of one transaction.
  • If one or more updates (and/or creates) fails, we can roll the entire transaction back.

In monolithic transactions are simple. We can have one process which is updating and creating records. These records are part of the transaction; therefore, the same process can either commit the transaction or roll it back if there are any issues.

In micro-services, transactions spanning are complex because there are several processes. This means that several micro-services are involved in complete one transaction. Since our transaction is distributed along multiple micro-services, it becomes a complex procedure to observe and solve problems; therefore, it becomes complex to roll back.

For example, we can have a order being place. This process will take several micro-services working together.

If one of these micro-services fails when trying to create or update a record, we will need to rollback the entire transaction.

How to handle fail transactions:

  • Option 1: Try again later.
    • The part of the transaction that failed is put into a queue so another service can pick it up and process.
      • Transaction will eventually be completed.
      • It relies on other instances not failing with the same part of transaction.
  • Option 2: Abort the entire transaction.
    • We detect our transaction has failed, then we issue an Undo transaction to all the micro-services involved so they undo any creates or updates
      • Problems:
        • Who issue the undo transaction?
        • What happens when the undo transaction fails itself.
      • One way to overcome this problem is to use a transaction manager software.

        • This software use a two phase commit.
        • Phase 1: All micro-services involved indicates to the transaction manager if they are fine to commit to their part of the transaction.
        • Phase 2: If they are fine to commit, then the transaction manager tells all participating micro-services to commit the transaction.
        • If any of the micro-services doesn’t respond or responds with a “no to committing” then the transaction manager tells to all the participating micro-services to rollback the transaction.
        • Problem using transaction manager?
          • We are heavily dependent of it. 
          • It delays the processing of our transactions. Potential bottleneck.
          • Complex to implement.
          • More complex when we need to communicate with a monolithic system.
            • This can be accomplish with the message queue.

 

Share
Leave a comment

Microservices: Brownfield: Migration: Database

 icon-arrow-left Microservices: Brownfield: Migration | Microservices: Brownfield: Transactions icon-arrow-right 

In this section, we are going to go over splitting the monolithic database into databases that will be used on each micro-service. In this way, each micro-service will have its own database which makes it easier to maintain and part of the whole micro-services concept.

As establish on the previous articles related with micro-services, we want to avoid shared databases. We want our micro-services to be as independent as possible. In this way, they can be independently changeable and deployable. A shared database limits us and makes our micro-services dependent.

The approach to split our monolithic database into micro-services databases is similar to split the code into bounded contexts as explained on the previous article, Microservices: Brownfield: Migration.

We split seams in the database which are related to seams in the code. In other words, we can take the tables that are related with our code and move them (or recreate them) into the new database. In our case, all the tables needed for the account functionality, will be taken from the shared database into the single database exclusive used for the account micro-service.

Note that in the process of moving from monolithic to micro-services, we may have to modify our the data layer of our monolithic system to access multiple databases.

A question may cross your mind which is, what do we do when we have a table which is linked across seams? For example, you may have a promotion which is linked to an order. So you have two services, the promotion service and the order services working together. Then, we must provide API calls which allow us to fetch the data for that relationship. In our example case between the promotion and the order, we will have the Order service fetching specific data from the Promotion service. 

Remember that we are refactoring our database into multiple database. We must worry about data referential integrity. This means that if we delete an account of a customer, for example, we might have to take care of orders related with that customer. Those orders exists in the Orders service. We would do this by calling the method in the Orders micro-service which would instruct in our example case to delete or disable specific orders related with the specific account ID that was deleted in the Account service. We must ensure that our micro-services talk to each other in order to keep the data referential integrity. 

In the case where we have static tables that are required by all micro-services, The best action is

  • Make that data into a configuration file available to all micro-services.
  • Or, have a specific micro-service just for these static tables.

The same principles apply when you have valid shared data that is read and written by multiple services. You move the data to a configuration file or you create a micro-service that can be used by the other micro-services.

 

Share
Leave a comment

Microservices: Brownfield: Migration

 icon-arrow-left Microservices: Brownfield: Introduction |  Microservices: Brownfield: Migration: Database icon-arrow-right 

 

Assuming you have done the previous step of reorganizing your code, now can get ready for the migration to actual micro-services. Your monolithic architecture should be looking more or less as follow:

As you can see we have:

  • Code grouped based on business domain or function.
  • Clear boundaries with clear interfaces.

We begin by converting one of these groups into a micro-service. This will allow us to learn in progression of migration, the use of new technology, the use of distributed transactions, the implementation of separate data, and allow us to get comfortable with the whole process of migration.

This provide an advantage since you can switch between the old functionality (in the monolithic architecture) and the new micro-service functionality. If you encounter any issue with the new micro-service functionality, you can switch back to the old functionality of the monolithic architecture. Note that you may have to keep maintaining the two variations of the same code until you only use the micro-service alone.

One thing that you may have to do is to prioritize what parts of the system you can migrate to a micro-service with the least impact. There are different ways to prioritize:

  • By level of risk.
  • By dependencies.
  • By technology utilized.
  • By complexity.

Since we are following an incremental approach when doing our migration, we can ensure that our micro-services are going to work with our existing monolithic system and that they will run  alongside. Also, this helps with the learning curve since we are introducing a new way to work. Remember to keep monitoring both the monolithic system and the micro-services for any kind of impact. Always keep reviewing and improving your infrastructure in order to support the new distributed transaction done by the micro-services.

If we do everything well, we should be ending with our new micro-service architecture as shown below:

Share
Leave a comment

Microservices: Brownfield: Introduction

   icon-arrow-left Microservices: Technology |  Microservices: Brownfield: Migrationicon-arrow-right 

 

The term Brownfield (used in IT) means that we are going to do development and deployment of new software systems while taking into account previous existing legacy software systems. The term Brownfield originates from the contemporary civil engineering where in some cases required that new buildings were designed and erected while having in consideration other structures and services already in place. In our case, transforming a monolithic system into a micro-service system can be considered Brownfield.

 

Approach

So we have an existing monolithic architecture and needs to be migrated to a micro-services architecture.

  1. Decide on the initial approach to take in order to ready your system for the migration from the monolithic architecture to the micro-services architecture.
  2. Take a look to the actual migration about how to migrate a monolithic system to a micro-service system.
  3. Our monolithic system may have one large database; therefore, we will need to do a database migration from this large database into multiple databases to support our micro-services.
  4. Take a look into the effect our micro-services architecture may have over the transactions within your system and reporting within your system.

Our existing system is a monolithic architecture that most likely have grow to a huge size. The most likely, the business section of our monolithic architecture is large and complex. 

We can assume that our current system lacks most of the micro-services design principles; therefore, the first step is to figure out how can we begin introducing those principles.

Let’s begin by analyzing the code of our system and try to identify the seams that form part of it. We need to do an identification and separation that reflects the business domains within our business. We may be able to identify modules, classes, and methods that are related with specific parts of our organization so we can group them based on the business domain.

Just for your information, the separation doesn’t have to be exactly based on the domains of your organization, it can be also done based on the functions within your organization. 

You may find code that overlaps from one department to another. In this cases, we must refactor the code so it can be split into the bounded context. 

Begin by creating a module for each bounded context, then begin moving the code around in increments. Then, ensure to validate your code refactoring by implementing unit tests and integration tests that validate your changes.

After every release, review the code again, then refactor again, and keep repeating this process until you can have bounded context clearly defined. 

When you finish having well define bounded contexts then you are going to be able to create micro-services boundaries. This means that we are going to transform these bounded contexts into micro-services.

Share
Leave a comment