When dealing with file uploads, you must be aware that files are uploaded in buffers stored in memory and if the file is larger than the allocated memory in your VM, it may run out of memory and the application might crash. For example, if your allocated memory is 2GB, if you try to upload a file 2GB in size, the memory will max out.
This calls for a change in how we carry out file uploads. As a result, multipart uploads have come to the rescue. We shall look at how we can write code on both the server and the client to accept multipart file uploads. On the client side, we shall use React and NestJS on the server side. We shall also use an S3 bucket for storing our files.
NestJS is the most rapidly growing NodeJS framework in TypeScript and ReactJS. As it helps to break down the most complex of structures and work with individual components. AWS S3 is used for scalable object storage in the cloud and provides data availability, security, and performance that are unmatched in the market. These objects are often stored in binary format. We shall use the aws-sdk npm package to interact with Amazon S3 in this blog.
In this blog, we will demonstrate how to
- configure AWS S3 buckets
- write server-side applications using NestJS
- write client-side applications using ReactJS
Hands-on
In this hands-on, we will be implementing the following:
- Creating a multipart upload – This makes a call to AWS S3 to create a unique uploadId which will be used to trace the parts that are being uploaded back to the original file for consolidation purposes.
- Uploading parts – This is the main part where the file chunks are uploaded and part of the metadata in this operation is the uploaded.
- Multipart upload completion – This is where the file is consolidated back into one complete object in S3 using the various parts that were uploaded.
Let’s configure the AWS S3 bucket:
We shall begin by creating an S3 bucket as shown in the following image. You can type in the search bar in the top navigation; “S3”.
Give your bucket a name, select a region, and use all the other default values/configurations.
Next, we need to Edit the CORS configuration for our S3 bucket as shown below.
We now need to create a new User Group and Assign them a policy that grants access to S3. Click on Create Group.
Select The checked permission policy as shown below.
We can create a new user and add them to the User Group with Full S3 access. Click on Add users and give your user a name.
You can skip the tags and go straight to click on Create user.
Server Side Application(NestJS)
Code repository – https://github.com/workfall/multipart-upload-backend
Back on the server-side code (NestJS), we will use microservice architecture.
maint.ts
shared-service.service.ts
This is the part that aws-sdk comes into play.
shared-service.controller.ts
Client Side Application (React)
Code repository – https://github.com/workfall/multipart-upload-frontend
On the client side, we shall consume the endpoints we created and also most importantly determine the number of chunks a given file will have. Which will also be used to get pre-signed URLs for each part being uploaded.
After all the chunks have been uploaded we have to send a request to S3 to complete the upload of the file and put together the chunks to form one file. If not, S3 will assume that the upload is incomplete and you won’t see your file in the AWS console.
FileUploadComponent.tsx
Uploader.tsx
This file has all the logic to deal with uploading the file. It gets the chunks of the file, gets a pre-signed URL for each part, and gives it a part number. The parts are sent as chunks using the pre-signed URL.
App.tsx
Conclusion
In this blog, we looked at how to perform multi-part uploads to Amazon S3 using ReactJS frontend and NestJS backend. The multi-part upload method with pre-signed URLs is ideal for very large files and it is also secure because the pre-signed URL is usually short-lived. Summary:
- Create file chunks (client-side)
- Create multipart upload
- Obtained pre-signed URL for each part/chunk
- Complete muti-part upload
Note: Always remember to clear incomplete parts for multipart uploads in order to prevent extra charges on AWS. You can do that in the Storage Lens tab in the S3 console. We will come up with more such use cases in our upcoming blogs.
Meanwhile…
If you are an aspiring developer and want to explore more such topics, here are a few of our blogs for your reference:
NestJS:
- How to Deploy NestJS Microservices to AWS Elastic Beanstalk?
- How to Architect NestJS Microservices with AWS Elastic Beanstalk?
ReactJS:
- Angular vs React: Which one to choose and when?
- How to build and deploy dApps using Solidity Smart contract, WEB3.JS & React.JS?
AWS S3:
- How to ETL API data to AWS S3 Bucket using Apache Airflow?
- How to easily replicate existing S3 objects using S3 batch replication?
Keep Exploring -> Keep Learning -> Keep Mastering
At Workfall, we strive to provide the best tech and pay opportunities to kickass coders around the world. If you’re looking to work with global clients, build cutting-edge products and make big bucks doing so, give it a shot at workfall.com/partner today!
