For successful corporate operations, data access is essential. However, data is frequently blocked or corrupted due to device problems, cyberattacks, and natural disasters. Businesses and IT specialists are forced to work hours to recreate and recover data that has been destroyed. Data replication is the best strategy to protect data and assure reliable data access. Having a replica can also help enterprises with a large number of locations access data more quickly.
Recently, AWS announced a new feature for S3 Batch Replication which comes in handy in this situation. It makes replicating existing data from a source bucket to one or more destinations simple. You can use this feature to replicate an unlimited number of objects in a single job.
In this blog, we will explore S3 Batch Replication, its use cases, when to use it, how to use S3 Batch Replication and its pricing model. We will also demonstrate step-by-step instructions on how to replicate existing objects using S3 Batch Replication.
In this blog, we will cover:
- What is S3 Batch Replication?
- Cross Region Replication VS Same Region Replication
- How does it work?
- Use Cases of S3 Batch Replication
- Why Use S3 Batch Replication?
- When to use S3 Batch Replication?
- Pricing of S3 Batch Replication
- Hands-on
- Conclusion
What is S3 Batch Replication?
Amazon Simple Storage Service (S3) Replication is an elastic, fully managed, inexpensive technology that replicates objects between buckets. S3 Replication provides the highest flexibility and functionality in cloud storage, enabling you to address your data sovereignty and other business requirements.
You can use Amazon S3 Replication to automatically replicate S3 objects across different AWS Regions using S3 Cross-Region Replication (CRR) or between buckets in the same AWS Region using S3 Same-Region Replication (SRR).
It allows you to replicate data to multiple destination buckets in the same AWS Region or in other AWS Regions.
The metrics and notifications provided by S3 Replication allow you to keep a close watch on the replication process.
| Then | Now |
| S3 batch replication replicates newly created objects to 2 or more AWS buckets within the same AWS account and different AWS accounts. | S3 batch replication now replicates existing objects to 2 or more buckets within the same AWS account and different AWS accounts. |
| Backfill data to newly formed buckets with newly uploaded objects. | Backfill data to newly formed buckets with existing objects. |
| Cannot retry objects that failed to copy to the destination. | Can retry objects that failed to copy to the destination. |
| Objects were replicated with the same account between different regions. | Now, Objects can be replicated cross-accounts as well. |
Cross Region Replication(CRR) VS Same Region Replication(SRR)
Cross-Region Replication and Same Region Replication, both allow you to replicate data at a bucket level, a shared prefix level, or an object level using S3 object tags. Moreover, both these features allow you to alter account ownership for the object replicas to safeguard data from any accidental deletion. However, there are a few key fundamental differences to make note of.
How does it work?
Use cases of S3 Batch Replication
Complemented by CRR and SRR, S3 Batch Replication can handle any size of data and provides a fully managed solution for data sovereignty and compliance, disaster recovery, and performance improvement. You can begin with S3 Batch Replication with only a few clicks in the S3 console or a single API request.
- Backfill newly created buckets —If a new multi-region storage initiative demands, you can use Batch Replication to set up new buckets and backfill them with existing objects from another bucket.
- Retry Application – You can use Batch Replication to retry replication for a variety of reasons, failure to replicate objects, initially, when objects replicated in one destination need to be replicated elsewhere, or when replicating replica objects from another source
- Migration – Batch replication can help you with the migration of existing objects across buckets, irrespective of the Region or the account they belong to, without affecting metadata and version ID.
Why use S3 Batch Replication?
- For disaster recovery purposes, customers may choose to duplicate their data to a new AWS Region. To do so, they’ll need to migrate existing data into the new destination bucket.
- Data replication for globally expanding companies. Consider the case of a UK-based animation business that has recently opened a new studio in Singapore. To reduce latency for their employees, they will be required to duplicate all internal files and in-progress media files to the APAC (Singapore) Region.
- Customers experiencing mergers and acquisitions are obligated to hand over ownership of present data from one AWS account to another.
- Customers end up implementing sophisticated methods to replicate existing objects between buckets. Furthermore, duplicating objects between buckets do not maintain object metadata like version ID and object creation time.
S3 Batch Replication proposes a new potential through S3 Batch Operations that eliminates the need for customers to brainstorm solutions for replicating existing objects between buckets. It offers an easy way to copy present objects from a source bucket to multiple destinations. This capability empowers you to replicate any number of objects with a single job.
When to use S3 Batch Replication?
- Replicate existing objects: use S3 Batch Replication to replicate objects that were included in the bucket before the replication rules were set.
- Replicate objects that previously failed to replicate: retry replicating objects with the S3 Replication rules that could not be replicated before, owing to factors like insufficient permissions.
- Replicate objects that have already been replicated to a different destination: you may need to keep numerous copies of your data in different AWS accounts or Regions. S3 Batch Replication can replicate previously replicated objects to new destinations.
- Replicate replicas of objects generated by a replication rule: S3 Replication produces replicas of objects in destination buckets. Object replicas cannot be replicated via live replication. S3 Batch Replication alone can replicate these replica objects.
Pricing of S3 Batch Replication
- S3 Data transfer cost: pricing includes cross-region data transfer OUT from S3 within AWS.
- S3 Storage cost: pricing includes storage of objects in the destination S3 bucket.
- Batch Operations job cost: pricing per batch operation job in a particular region is $0.25
- Batch Operations manifest cost: pricing is $0.015 per 1 million objects in the source bucket.
- Batch Operations object cost: pricing for 1 million objects processed is $1.00.
Hands On
Let’s do S3 batch replication. We will be using S3 and IAM services in this demonstration. First, we have to create a source bucket. You can give it a unique name as we have given “workfallbucket”. Upload some objects into the bucket as shown below.
Click on the bucket that you just created and navigate to the management tab. For creating S3 batch replication, we first have to set up the replication rule in the source bucket. For creating the replication rule, remember to enable the versioning on both the source and destination buckets.
As you will scroll down a little, you will see Replication rules there. Click on create replication rule.
You will be redirected to the new window to create the replication rule.
Provide a unique name to the replication rule. Keep the status for the rule as enabled.
For choosing the rule scope, select Apply to all objects in the bucket. If you choose “Limit the scope of this rule”, you have to provide the prefix to filter the objects in the bucket.
For Destination, create another bucket with a unique name as we have given”destinationbucketworkfall”.
Select the bucket name from the dropdown as shown below.
Select the destination bucket.
Note: Create the source and destination buckets in different regions.
Create the IAM role for S3 replication “S3_Replication_Role_for_Workfallbucket”.
Go to IAM. Create a policy with the below configuration. Name it as S3_Replication_Policy.
Click on create role. Select S3 from services.
Now, select the policy you just created in the Add permissions page.
Select S3_Replication_Policy as shown below.
Here you can review your role and click on create.
Click on create role. The role is created successfully.
Choose the role from the dropdown as shown below.
For this implementation, there is no need for Encryption. You can leave it unchecked.
For the Destination storage class, you can leave it blank for this implementation. Using the Destination storage class, you can change the storage class for replicated objects.
Leave this unattended..
Click on save. We will receive a pop-up with the message “Replicate existing objects “. You can replicate the existing objects of the source bucket using this.
Click on “Yes, replicate existing objects” and click on Submit.
As you will click on submit. You will be redirected to the Batch Operations job page. Now we will create the Batch operations job.
Select “Wait to run the job when it’s ready”. Using this you can manually run the job and if you want to review the job details or manifest file, you can do that before running the job.
Select “Save Batch Operations manifest”. Provide the destination where you want your manifest to be saved.
Specify the folder within the bucket where you want your manifest to be saved.
In the completion report section, provide the path where your completion report will be saved. Specify the folder within the bucket for the exact location.
For the S3 batch operations job, you have to create the S3 batch operation role.
The image below shows the creation of the S3 batch operations policy.
Mention the following permissions in the S3_BatchOperations_Policy. Provide the source bucket ARN and manifest and completion report bucket ARNs.
Click on create policy as shown below.
Create a role for S3 batch operation.
Provide the IAM role that you just created during batch job creation and click on Create job.
You can see the job has been created successfully. The status of the batch job is in the preparing state.
Now the status has been changed to “Awaiting your confirmation to run”. Here you can see the total number of objects that would be replicated is 5. There is a unique job id that is associated with every job.
Select the job that is just created and click on “Run job”.
You will be redirected to the page where you can review the job details.
After you’ve gone over everything, go to the bottom of the page and select “Run job.”
Now you can see the status of your job as Ready.
Now the job gets completed. All the objects in the source bucket are replicated to the destination bucket.
When you open your destination bucket, you will see the objects as shown below.
By following this step-by-step implementation you’ll be able to perform the S3 Batch Replication successfully. You can try performing the replication for the failed jobs objects and also newly uploaded objects or replicating objects to new buckets cross-account or in the same-account.
Conclusion
In this blog, we have explored Cross-Region Replication vs Same Region Replication, S3 Batch Replication, its use cases, when to use it, how to use S3 Batch Replication, and its pricing module. It allows you to replicate data to multiple destination buckets in the same AWS Region or in other AWS Regions. It also backs two-way replication between two or more buckets in the same or different AWS Regions. We also demonstrated step-by-step instructions on how to replicate existing objects using S3 Batch Replication. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.
Meanwhile …
Keep Exploring -> Keep Learning -> Keep Mastering
This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.
