If your application is having problems loading, it’s time to think of solutions to solve them. Once you have the number of users you can handle, the application performance becomes slower and an error occurs. Network requests started being time-out, data queries began to take some time to process, and the page loading speed became slower and slower. Congratulations @@@. Your application starts to scale. If you don’t scale quickly, other competitors can copy your ideas and build a more perfect app. At that time, even if it was the first person, it was not the destination.

The price of scaling

Before you start scaling vertically, deep or from the inside out, a very important rule should be followed. You should not deploy optimal test versions, or try scale when you don’t really need it. The implementation of scaling solutions introduces the following issues:

• More functions will take more time
• The system becomes complicated with many pieces, and the variables increase
• The code becomes harder to test
• Finding and resolving bugs is also becoming harder

You should only accept scaling if your app has reached its limit. Keeping the system simple, do not introduce a lot of complications from scaling if not really needed.

Find Bottlenecks with metrics

Every application / system is different, to determine which scaling solution to deploy, you need to determine where the bottlenecks are. This is the time to check your resource tracking system or create it if you don’t already have one. No matter the technology stack you are using there will always be tools available for monitoring your resources. If you are running applications on IaaS platforms such as AWS, Azure, GCP, there will be great applications for resource tracking.

These tools illustrate the performance of your resource usage through graphs and other illustration methods. These tools can detect resources that are overloaded, or existing resources that cannot handle new tasks. If nothing is displayed in the resource pool, you can view logs to find the most resource-consuming tasks. You should host the database at another server, and in that case you should also monitor the usage of resources for that server.

By thinking about how users use your application and logical thinking about possible errors, bottleneck determination can be relatively simple. For example, with Twitter, the platform is primarily for people to read and write tweets. If Twitter’s tracking service announces that loading the database is having problems, their development teams will start optimizing from this partition. In this article, we will divide the solution to scaling database.

Scaling application

What is the bottleneck now? How bottleneck, we can start deploying solutions to solve these problems. Remember that simplicity is the key, we always want to avoid unnecessary hassles.

The first goal of scaling solutions is to reduce loading, time to handle complex tasks. How scaling techniques often work in the following ways:

• Use data that has been previously searched
• Eliminate client-side requests for previously owned data
• Storing results from repetitive tasks
• Eliminate complex tasks in the request and response loop

Many techniques scale scaling for some form of hosting. In the past, memory was expensive and scarce, today, it is no longer expensive to add it to the server. Memory is faster to access data than a disk or network, in this day and age, where users have lots of choices, along with the minimum attention span, their speed and performance. Crucially for the survival of your application.

Methods of scaling database

Cache Database Queries

Query database caching is one of the easiest improvement methods for you to handle database loading. Typically, an application will have several queries that are used multiple times to build the main thread in the application. Instead of using it one more time each time we receive a request to retrieve data, we can simply cache them in memory on the web server. The first request will retrieve data from the database, then cache that result on the server, future requests just need to read the data from the cache. This will increase the performance of the system as well as spending less time in retrieving data. As well as increasing availability, if a database is not available at a time, the cache can continue to provide services to the application, making it more difficult for the application to fail. There are many tools you can use to analyze log queries, so you can check out the queries that are still time-consuming and the queries are frequently used.

Obviously, the data that the cache in the cache can become obsolete or quickly become obsolete. You will have to pay attention to what data you have chosen to store and when to use it. For example, an online newspaper will have a new daily newspaper every 24 hours, instead of requesting that data from the database every time a user visits the website, they can store that data. on the web server for 24 hours and serve directly from the server. Product or business requirements determine what can and can be cached.

Database Indexes

Data indexing is a technique to improve the speed of data retrieval operations on the database table. Indexes are used to quickly locate data without having to search every table row every time the table is accessed. Typically, the data structure for a database index will be a binary search tree. This allows to reduce the time complexity of data access from linear time O (n) to Olog time (n).

Depending on the number of rows in a table, this can significantly save the time that queries use indexed columns. For example, if you have 10,000 users and your application has profile pages that search for users by their username, a non-indexed query will check each row in the user table until it finds one. see records that match the username passed into the query. That can take up to 10,000 checks -> O (n). By creating an index for the username column, the database can reduce the query complexity to (Olog (n)). In this case, the maximum number of checks would be 14 instead of 10,000!

Efficient indexing reduces the load on the database by increasing efficiency, which also provides a significant increase in performance leading to a better user experience. Creating an index adds another data set that will be stored on the database, so make careful judgment when deciding which fields to index. Even with the existing storage space used, indexing tends to be very valuable, especially in modern developments where cheap memory and performance are indispensable to survive.

Session Storage

Many applications handle sessions by storing session IDs in cookies, then storing actual data for each key / value pair of each session in a table on the database. This can become a large amount of reading and writing to your database. If your database is overloaded with session data, it’s a good idea to rethink how and where you store that data. Moving session data to an in-memory caching tool like redis or memcached can be a good choice. This will remove the session data load from your database and also increase the access speed because in memory is faster than the continuous disk storage that most databases use. However, because the internal memory is volatile memory, there is a risk of losing all session data if the buffer system is not working.

You might also consider changing your session deployment to storing session information in the cookie itself, which will move the media to maintain session state off the server and on the client. JWT wave is the most popular implementation for this model. This will reduce your database of all session data and eliminate the dependency of server-side sessions, although it presents its own challenges.

Master Slave Replication

If your database is still overloaded from reading even after saving general queries, creating efficient indexes and processing session archives, then copying may be the next best step.

With master-Slave clone, you have a unique database that you write. It is copied into some (how much you need) the slave database you read from there, with each slave database on another machine (refer to the diagram below). This will load off the master database and distribute it across multiple servers. The model also improves write performance when the master database is dedicated to recording, while significantly increasing read speed and reducing latency when slave databases are spread across different regions.

Because each slave database is on another machine, writing to the master database that needs to be transmitted through the slaves can lead to inconsistent data. If you need to immediately read the data written to the database, assuming you are updating a record and want it to be displayed immediately, you can choose to read from the main database. Slave-master copying is an incredibly powerful extension solution, but it comes with its complexity. It is wise to implement this solution after using all of the simpler solutions and ensure optimization in the application.

• This architectural model is often referred to as master-slave replication, but it is a term that has received criticism for many years and is in the process of being replaced in the tech community.

Database Sharding

Most of the extended solutions to date have focused on reducing the load through the management of reads into the database. Database sharding is a horizontal scaling solution for load management by managing read and write to the database. It has an architectural model involving the partitioning (partitioning) of the master database into multiple databases (segments), faster and easier to manage. There are two types of database sharding techniques – vertical sharding and horizontal sharding (refer to the diagrams below). With horizontal partitioning, the tables are taken and placed on different machines with each table having the same columns, but different rows. Vertical partitions are more complex, involving splitting a board across multiple machines. A table is split and put into new, separate tables. Data is kept in a vertical partition independent of data in all other partitions, each table containing both separate rows and columns.

Both sharding techniques facilitate horizontal scaling, also known as scaling, which allows the practice of adding more machines to the system for distribution / transmission. Horizontal scaling often contrasts with vertical scaling, also known as scaling, which involves upgrading the hardware of an existing server. Database scaling is relatively simple, although any non-distributed database will have limits on computing and storage capabilities, so it is possible to freely scale up to help the system. Your system is a lot more flexible.

A fragmented database architecture can also significantly increase the query speed of your application, as well as increase resilience to errors. When submitting a query on an unprotected database, it may have to search every table row that can be greatly delayed. Additionally, by sorting a table into multiple tables, queries must pass through much fewer records to return results. Because each table is in a separate server, the impact of the server becoming unavailable will be minimized. With a protected database, the impact of a shutdown is likely to affect only a single segment, compared to an unprotected database where a power outage is likely to cause the entire application. unavailable.

Having a segmented database architecture provides some pretty big benefits, however, it is complex and has high implementation and maintenance costs. This is definitely an option you want to consider after using up other scaling solutions because the branching ineffective implementation can be quite serious.

Conclude

Above are 5 scaling database solutions you can consider to increase the efficiency of your system. You should also note when to actually use them and always remember that we can KEEP IT SIMPLE, STUPID. Happy coding !!!

Source

https://medium.com/swlh/5-database-scaling-solutions-you-need-to-know-e307570efb72