Hello everyone, after a few days of being fed properly. Then I can also give a little knowledge.
So today, I want to stand up to share some of the things I learned, hoping to help everyone.
Let’s get it !!
In this article, we will discuss ways to reduce the costs of using Amazon S3.
First, let’s take a look at the factors that influence the monthly cost you will pay for Amazon S3:
- Size of data stored per month (GB).
- Request and retrieve data (eg PUT, COPY, POST, LIST, GET, SELECT or other requests).
- Number of conversions between different classes.
- Data retrieval size and required quantity.
- Data transfer fee.
One of the most important factors that we need to keep an eye on when it comes to cost are storage classes. Each storage class has a different price, so make sure you remember and understand their storage classes and their use cases. Let’s take a quick look:
Amazon S3 provides users with 5 different layers of storage:
- S3 Standard
- S3 Intelligent – Tiering
- S3 Standard – Infrequent Access
- S3 One Zone – Infrequent Access
- S3 Glacier
- S3 Glacier Deep Archive
Each S3 object can be assigned to a class, so in a buckket, we can see simultaneously there are many objects with their respective class.
Since I studied on the Linux Academy taught in the N.Virginia region, I showed the current price list for everyone to see:
The S3 Standard class is typically used for frequently accessed data. Although the cost per Gb is high, there is no need to pay for the number of requets. Therefore this storage class is best suited for objects that are read or written multiple times per month.
The third layer in the table is S3 Standard-IA (S3 Standard – Infrequent Access). This class has a lower storage cost, but the cost of access is higher. According to AWS, it is intended for persistent data storage but does not often get access to data that needs immediate access.
S3 Standard-IA should be used if the object is accessed on average less than once per month. Why ? Because that is the frequency with which both S3 Standard and S3 Standard-IA have similar overhead. And that is also the recommended minimum amount of time to hold objects in class S3 Standard-IA. If kept for less than 30 days, the rest will be charged.
Usually, it is difficult to know how often the object is accessed. AWS created S3 Intelligent-Tiering to solve this problem. This layer dynamically moves data between the S3 Standard and S3 Standard-IA classes. And that minimizes the cost of S3 to the object. If an object is being held for more than 30 days, the ** S3 Intelligent-Tiering ** class will be cheaper than S3 Standard and S3 Standard-IA . In this case then this should be your first choice.
Class S3 One Zone-IA is similar to S3 Standard-IA. However, instead of storing data in 3 (or more) AZs (Availability Zone), data is only stored in one AZ. Therefore, data may not be available if access to the AZ is unsuccessful. S3 One Zone-IA should be used if risk is acceptable.
The last two layers in the table are S3 Glacier and S3 Glacier Deep Archive . They have the lowest cost per GB. But access costs are high. Hence they are used for archival purposes.
Objects in Glacier are not immediately available. If we want to access any of the contents of an object in Glacier, we will have to wait until the retrieval finishes. For batch retrieval, this interval can be up to 5 to 12 hours. With other retrieval mode, it may be faster but more expensive. For this reason, Glacier should only be used for rarely visited objects. For example, Glacier is ideal for backing up any long-term data that is not frequently accessed.
The difference between S3 Glacier and S3 Glacier Deep Archive is that the Deep Archive is intended for even less visited objects. Example: Objects are accessed every 6 (or more) months. The storage cost of S3 Glacier Deep Archive is lower. Objects in this class need to be stored for at least 180 days. Otherwise, that minimum period will be charged.
A little bit about the classes is like that, now I come to the note:
1. Select the correct class when initializing object:
Our first step is to analyze the access types for the data. We will have to think about the intended usage for each new object that will be created in S3. Each object in S3 must have a specific access pattern and have an S3 class that best fits it. We cannot define a default class for each group in S3 but we can assign it to each object.
Once we determine the best class for each new object in S3, all we need to do is place this class while uploading this object to Amazon S3. Can be done using the AWS CLI , AWS Console or AWS SDK . Therefore, each new object will have the correct class. This is possibly the best way to save money and be the most time effective.
2. Adjust the S3 class appropriately for existing objects:
Now that we have the correct classes for new objects (to be created), we can focus on the objects that have been created. The procedure is similar to that described in section 1. Begin with analyzing the data access patterns for the existing object. Then decide the best class for each object. And finally, assign that class to the object. This will allow us to optimize all S3 groups and reduce the cost of using S3.
How do I check if this already works? We can use AWS Cost Explorer to check the daily cost of S3. AWS displays consumption for each service, including Amazon S3.
Note that it can take a long time to update every class for an object after it has been created. That is why we have to define the classes before objects can be created. Depending on the number of objects you have, it can be a considerable amount of time changing the object. We should focus on large (or very frequently accessed) objects to update their storage classes first.
We can also use S3 Storage Class Analysis . This is a tool to analyze access types of S3 objects. It monitors the objects in a bucket, showing the amount of data stored in the group, the amount of data being retrieved, and how often data is accessed. Note that we will ohari pay a small fee to use this tool. But on the contrary, we can know that objects are accessed frequently or not. After you know this data, we can update the S3 storage class accordingly. For example, if you find that most of the objects in a group are only accessed once per year (and you don’t need immediate access), then adjust the storage class of these objects to S3 Glacier. Deep Archive .
3. Delete unused S3 objects
We pay for the amount of data stored in S3. So, if we remove unused objects, the cost is reduced significantly.
So, how do I check the content of an S3 group?
Example: We could enumerate objects per group. This will display the object name (or key) without downloading the object’s contents. We can use the AWS Console, AWS CLI or SDK.
4. Using S3 Lifecycle:
Amazon S3 provides a tool to dynamically change the storage class of any object. For example, you can convert from the S3 Standard class to S3 Glacier after a few days of object creation. So you can convert each object to the most suitable storage class. This in turn helps reduce costs.
How does S3 Lifecycle work? We set rules for each group. Each rule has a transition period. It counts the number of days since the object was created (or deleted). And after this time, the rule also passes the storage class to the object. Note that it is always possible to convert objects to a longer storage class, but not to a short term storage class.
It is also possible to set life cycle rules for entire groups or based on prefixes. So we don’t need to convert each of our objects individually. S3 Lifecycle Management is one of the most useful tools to save costs on S3.
5. Notice when object s3 expires:
Amazon S3 Lifecycle Management can also set an expiration policy. When an object expires, AWS will automatically delete it.
If you keep the logs (or any other temporary data) as S3 objects, it’s a good idea to time them. We can set the file logs to expire 30 days after their creation, and they will be deleted automatically.
6. Compress S3 objects:
We should compress any object before uploading it. Just create a compressed file (for example: ZIP, GZIP, …), this file will be smaller than the original file. Then upload the archive to S3. The amount of data used in S3 will be lower. Then Amazon S3 usage costs will be reduced.
Note that, to view the original file, you will have to download and unzip it. But in return we can save a lot of space in S3.
7. Package S3 objects:
We also pay for the amount of operations performed in Amazon S3. If you have to download multiple S3 objects at the same time, you should package them into one large object (eg TAR, ZIP or equivalent).
Some storage classes have minimal storage charges for objects. For example, the minimum storage charge per object for S3 Standard-IA and S3 One Zone – Infrequent Access is 128KB, and the minimum cost per object for the S3 Glacier and S3 Glacier Deep Archive classes is 40 Kb.
So a small 1 KB object (with class S3 Standard-IA) will be counted as 128 KB. Packing multiple files together gets minimal cost benefit.
9. Limit for Object Versions:
Object Versions S3 is a very useful tool. Every time you change the contents of an Object, AWS stores its previous version as well. But what if we had a 1 MB Object with 100 instances, we would have to pay for 100 MB of storage.
But you can use lifecycle policies to automatically delete the previous versions after a while.
10. Use Bulk Retrieval Mode for S3 Glacier:
Amazon S3 Glacier has 3 types of retrieval:
S3 Glacier Deep Archive has two types of retrieval:
Retrieval time : This means how quickly S3 creates an object’s content. The faster we access the objects, the more expensive it will be to operate it. If we could wait a few hours for the retrieval, we could save even more money. So try to use Bulk Retrieval Mode if possible.
- Change Region Some Regions are much more expensive than others. So prices for Amazon S3 services vary as well. Therefore, we should consider switching to the lower priced area.
Another factor to consider is the cost of data transmission between AWS regions. Data sent to the same region is free, but sending data to another region will be charged on Gb.
Above are some tips to reduce the cost of using Amazon S3 services.
If people find this article good, please Upvote , Like and Share to give me more motivation to do many other posts.