![]() Impact of log sizes on S3 Glacier storage costs This was achieved by taking advantage of log aggregation and compression. Thus, the primary objective of the solution was to minimize the overall number of lifecycle transitions required to archive the data. Because Primex had a large number of small files to transition into S3 Glacier Flexible Retrieval (450 million), the cost to archive them via an effective lifecycle policy was comparatively high. ![]() Lifecycle transition costs are directly proportional to the number of objects being transitioned. With the desire to reduce costs of storing this seldom-accessed data, Primex considered archiving the data to S3 Glacier Flexible Retrieval (formerly S3 Glacier). The log files represented just over 40 TB of storage on the S3 Standard storage class, at an average object size of approximately 88 KB. archiving use caseĪpplications running over an IoT device fleet generated 450 million log files over a period of 3 years. Primex, a sophisticated platform for monitoring vaccines and healthcare assets, took full advantage of the available AWS resources around object archiving to develop a solution that ultimately ended up reducing object transition costs into the S3 Glacier storage classes by over two orders of magnitude. The data archiving use case detailed in this post originated directly from an AWS customer, Primex Inc. Effective archival strategies serve to reduce storage costs while upholding high availability and durability guarantees for your data. Additionally, S3 also provides bucket/object lifecycle policies that can automatically take transition actions or expiration actions. Amazon S3 Glacier storage classes (S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive) are secure, durable, and extremely low-cost Amazon S3 cloud storage classes for data archiving and long-term backup. To transition objects into the S3 Glacier storage classes, S3 offers the Amazon S3 Intelligent-Tiering storage class which is the only cloud storage class that delivers automatic storage cost savings when data access patterns change, without performance impact or operational overhead. In this blog post, I highlight some of the cost-related considerations of archiving millions of log files to the Amazon S3 Glacier storage classes. ![]() Many customers elect to write logs directly to S3 in various formats, while others prefer to stream logs via AWS services such as Amazon Kinesis. As such, there is a need to retain logs permanently with high availability and durability guarantees. ![]() In some applications, log files serve as the ultimate source of truth and are essential for governance and audit purposes. These logs are later used in business intelligence to provide useful insights and generate dashboards, analytics, and reports. The logs may contain information on runtime transactions, error/failure states, or application metrics and statistics. In distributed architectures, there is often a need to preserve application logs, and for AWS customers preservation is often done via an Amazon S3 bucket. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |