Assume that you are developing a distributed application and looking for a solution to transmit a large volume of data, at any level of throughput, without losing messages or requiring other services to be available, you can think about Amazon SQS! Using Amazon SQS you can decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. RedBus, BMW, EMS, NASA, and more organizations are using the AWS SQS service! It is a fully managed message queue service that allows you to decouple and scale microservices, distributed systems, and serverless applications. Send any amount of data at any speed without missing messages or requiring the availability of other services, along with improving the system’s overall fault tolerance.
In our blog How to build a Serverless Workflow with AWS Step Functions, we discussed AWS Step functions, their features, benefits, use cases, etc. In this blog, we will discuss Amazon SQS in detail. In part 2 of this blog, we demonstrated the step-by-step instructions on how to orchestrate Queue-based Microservices with AWS Step Functions and Amazon SQS.
In this blog, we will cover:
- What is Amazon SQS?
- Benefits
- Types of Queues in Amazon SQS
- The functionality of Amazon SQS
- Using Amazon SQS with other AWS Web Services
- Security in Amazon SQS
- Amazon SQS vs Amazon SNS
- Pricing
- Companies using Amazon SQS
- Conclusion
What is Amazon SQS?
Queues are a great way to mix and match software structures. They allow non-simultaneous communication across separate systems, which is particularly beneficial when the system’s output is uneven. Amazon’s version of queues is called Amazon SQS or Simple Queue Service.
For instance, if you have something along the lines of:
System A – sends out signals in large bursts regularly.
System B – consumes messages continuously but at a slower rate.
A queue in this architecture would allow System A to produce messages as quickly as it can while System B digests them at its own pace.
Amazon SQS is a hosted queue that allows you to link and disconnect distributed software systems and components. It is safe, durable, and available. SQS removes the complexity and expense of managing and operating message-oriented middleware, allowing developers to concentrate on unique work. SQS allows you to transmit, store, and receive messages at any volume between software components without losing messages or necessitating the availability of other services. SQS provides two different types of message queues. Standard queues provide high output, best-effort ordering, and delivery at least once. SQS FIFO queues are meant to ensure that messages are processed only once, in the sequence in which they are received.
Benefits
Cut the time spent on administrative tasks: All continuous operations and underlying infrastructure required to deliver a highly reliable and efficient message queuing service are managed by AWS. With SQS, there are no upfront expenditures, no need to acquire, install or configure messaging software, and no lengthy infrastructure setup and upkeep.SQS queues are dynamically produced and scale automatically, allowing you to easily construct and scale applications.
Deliver messages on time: Use Amazon SQS to send any amount of data at any throughput without missing messages or requiring the use of other services. SQS allows you to disconnect application components, allowing them to operate and fail separately, improving the system’s overall fault tolerance. Every message is duplicated and stored across various availability zones to ensure that it is available whenever it is needed.
Keep Sensitive data secure: You can use Amazon SQS to send sensitive data between applications by encrypting each message body using server-side encryption (SSE). The integration of Amazon SQS SSE with AWS KMS allows you to manage the keys that safeguard SQS messages as well as keys that protect your other AWS resources from a single location. To help you fulfill your legal requirements, AWS KMS logs every use of your encryption keys to AWS CloudTrail.
Scale flexibly and cost-effectively: Amazon SQS uses the AWS cloud to scale up and down based on demand. You don’t have to worry about capacity planning or pre-provisioning because SQS adapts easily to your application. There is no limit to the number of messages that can be stored in a queue, and conventional queues can handle almost any amount of data. Costs are calculated depending on consumption, which saves money over the “always-on” model of self-managed messaging middleware.
Types of Queues in Amazon SQS
Amazon SQS offers two queue types for different application requirements:
1. Standard Queues
Standard queues can handle practically an infinite number of transactions per second for each API action.
- At-Least-Once Delivery: A message is delivered at least once, but it is sometimes delivered multiple times. For redundancy and high availability, Amazon SQS saves copies of your messages on multiple servers. When you receive or delete a message, one of the servers that stores a copy of the message may be unavailable.
- Best-Effort Ordering: Messages are sometimes delivered in a different order than they were sent. Standard message queues can be used in a variety of circumstances as long as your application can handle messages that come several times and out of order, such as:
- Separate live user queries from background tasks that require a lot of time: Allow users to upload media while it is being resized or encoded.
- Assign tasks to many worker nodes: Process a large number of requests for credit card validation.
- Messages to be processed in batches: Schedule the addition of many entries to a database.
- A standard queue takes every effort to keep messages in order, but several copies of a message may be delivered out of order. If your system demands that messages be delivered in a specific order, we recommend using a FIFO (First-In-First-Out) queue or including sequencing information in each message so that you can reorder them when they arrive.
FIFO Queues
FIFO queues can handle 300 messages per second by default (300 send, receive, or delete operations per second). FIFO queues may support up to 3,000 messages per second when batching 10 messages per operation (maximum). If you need more throughput, you can use the Amazon SQS console to select high throughput mode for FIFO, which supports up to 30,000 messages per second with batching and up to 3,000 messages per second without batching.
- Exactly-Once Processing: A message is sent only once and is available until it is processed and deleted by the recipient. There aren’t any duplicates in the queue.
- The order in which messages are transmitted and received is tightly maintained with first-in-first-out delivery (i.e., First-In-First-Out).
- When the order of processes and events is crucial, or duplicates cannot be tolerated, FIFO queues are used to improve messaging between applications.
- Ensure that commands submitted by the user are performed in the correct order.
- Send price adjustments in the correct order to display the correct product price.
- Stop a student from enrolling in a course without first creating an account.
The functionality of Amazon SQS
Infinite queues and messages: Create an unlimited number of Amazon SQS queues in any region with an unlimited number of messages.
Message Payload Size: Message payloads can be up to 256KB in size and can be in any format. Each request is billed as a 64KB “chunk” of payload. A single API call with a 256KB payload, for example, will be billed as four requests. The Amazon SQS Extended Client Library for Java, which uses Amazon S3 to store the message payload, can be used to send messages larger than 256KB. SQS is used to send a reference to the message payload.
Messages can be sent, received, or deleted in batches of up to 256KB or 10 messages. SQS can be even more cost-effective for customers who use batching since batches cost the same as single messages.
Long polling: Reduce unnecessary polling to save money while getting fresh messages as soon as feasible. Long-poll queries can take up to 20 seconds to receive the next message if your queue is empty. Long poll requests are charged at the same rate as ordinary poll requests. Messages in queues can be kept for up to 14 days. Messages can be sent and read at the same time.
Message locking: A message is “locked” while it is being processed when it is received. This prevents the message from being processed by other machines at the same time. The lock will expire if the message processing fails, and the message will become available again.
Queue sharing: Anonymously or with specified AWS accounts, securely share Amazon SQS queues. IP address and time of day can also be used to limit queue sharing.
Protect the contents of messages in Amazon SQS queues with server-side encryption (SSE) keys controlled by the AWS KMS. As soon as Amazon SQS gets a message, SSE encrypts it. The communications are encrypted and are only decrypted when they are forwarded to an approved consumer by Amazon SQS.
Dead Letter Queues (DLQ): Use Dead Letter Queues to handle messages that have failed to be processed by a consumer. When a message’s maximum receive count is reached, it is sent to the DLQ associated with the original queue. Set up distinct consumer processes for DLQs to aid in the analysis and understanding of why messages are stuck. The source queue’s DLQs must be of the same kind (standard or FIFO).
Using Amazon SQS with other AWS Web Services
Amazon SQS message queuing can be combined with other AWS services like EC2, ECS, Lambda, and S3 to improve the scalability and reliability of distributed applications. Some typical design patterns are shown below:
Work queues can be used to decouple components of a distributed application that may not all perform the same amount of work at the same time.
Buffer and Batch Operations: Enhance your architecture’s scalability and reliability by smoothing out temporary volume surges without losing messages or raising latency.
Request Enqueuing: Enqueue the request to move slow operations off of interactive request pathways. You can distribute identical copies of a message to many queues at the same time by combining SQS and Simple Notification Service (SNS).
Prioritization: Work should be prioritized using various queues. Because message queues decouple your processes, scaling up the send or receive pace of messages is a simple matter of adding another process.
Resiliency: When a component of your system fails, it does not have to bring the entire system down with it. Message queues isolate components of your system, allowing messages to be added to the queue and processed when the system recovers.
Security in Amazon SQS
Data Protection
- In Amazon Simple Queue Service, the AWS shared responsibility model applies to data protection.
- AWS is in charge of safeguarding the global infrastructure that supports the whole AWS Cloud.
- You are in charge of keeping your material hosted on this infrastructure under your control. An endpoint for Amazon SQS is a logical entity inside an Amazon Virtual Private Cloud (Amazon VPC) that only enables access to Amazon SQS. Requests are routed to Amazon SQS, and replies are routed back to the VPC.
Identity and Access Management in Amazon SQS
- Credentials are required for access to Amazon SQS, which AWS can employ to authenticate your requests. These credentials must provide access to AWS resources like Amazon SQS queues and messages.
You can log in to AWS using any of the following identities:
- Username for the AWS account’s root user
- IAM user
Logging and Monitoring in Amazon SQS
- AWS CloudTrail, a service that keeps track of Amazon SQS calls made by a user, role, or AWS service is connected with Amazon SQS.
- CloudTrail records API calls to Amazon SQS queues as events, including calls from the Amazon SQS dashboard and code calls to Amazon SQS APIs.
Monitoring Amazon SQS queues using CloudWatch
- Amazon CloudWatch to observe and analyze metrics for your Amazon SQS queues because Amazon SQS and Amazon CloudWatch are connected.
- At one-minute intervals, CloudWatch metrics for your Amazon SQS queues are automatically collected and published to CloudWatch. These statistics are gathered for all queues that are active according to CloudWatch’s standards. If a queue includes any messages or is accessed by any action, CloudWatch considers it to be active for up to six hours.
Compliance Validation in Amazon SQS
- Compliance and Security Quick Start Guidelines to go over architectural concerns and show you how to set up baseline security and compliance environments on AWS.
- Whitepaper on Architecting for HIPAA Security and Compliance
- This collection of workbooks and guides from AWS may be relevant to your industry and location.
- The AWS Config service evaluates how well your resource configurations adhere to internal policies, industry best practices, and government standards.
- AWS Security Hub provides a comprehensive picture of your security state within AWS, allowing you to ensure that you’re following industry standards and best practices.
Resilience in Amazon SQS
- AWS Regions and Availability Zones form the foundation of the company’s global infrastructure. Multiple physically separated and isolated Availability Zones are joined by low-latency, high-throughput, and highly redundant networking in AWS Regions. With Availability Zones, you can create and run applications and databases that seamlessly failover from one zone to the next.
Data Encryption
- One can send sensitive data in encrypted queues using server-side encryption (SSE). The contents of messages in queues are protected by SSE utilizing keys controlled by AWS Key Management Service (AWS KMS).
- To handle customer master keys (CMKs) for server-side encryption, Amazon SQS interfaces with the AWS Key Management Service (SSE).
- The data keys that encrypt and decrypt the messages in Amazon SQS are validated and secured using CMKs.
Amazon SQS vs Amazon SNS
SNS is a distributed publish-subscribe system and SQS is a distributed queuing system.
Amazon SNS
- Publisher role in PubSub
- Publishing messages can deliver to many subscribers (fan-out) of different types (SQS, Lambda, Email)
- All the consumers can be of different types.
- It involves a push mechanism with entities such as topics and broadcasts.
Amazon SQS
- Queuing service for message processing
- An application must poll to discover new events, and messages are typically processed by a single consumer application.
- All the consumers are supposed to be of identical types
- It involves a pull mechanism using polling with entities such as a Queue.
Pricing
You can use Amazon SQS for free to get started. Every month, all customers can make 1 million Amazon SQS requests for free.
- Pay only for what you use
- No minimum fee
Amazon SQS charges are metered as:
- API Actions: Every Amazon SQS action counts as a request.
- FIFO Requests: API actions for sending, receiving, deleting, and changing the visibility of messages from FIFO queues are charged at FIFO rates. All other API requests are charged at standard rates.
- Contents of Requests: A single request can have from 1 to 10 messages, up to a maximum total payload of 256 KB.
- Size of Payloads: Each 64 KB chunk of a payload is billed as 1 request (for example, an API action with a 256 KB payload is billed as 4 requests).
- Interaction with Amazon S3: When using the Amazon SQS Extended Client Library to send payloads using Amazon S3, you incur Amazon S3 charges for any Amazon S3 storage you use to send message payloads.
- Interaction with AWS KMS: When using the AWS Key Management Service to manage keys for SQS server-side encryption, you incur charges for calls from Amazon SQS to AWS KMS.
Companies using Amazon SQS
NASA: More than 140,000 still photos, audio recordings, and videos are available in the NASA Image and Video Library, which documents NASA’s more than a half-century of achievements in exploring the great unknown. Amazon SQS is used to separate incoming jobs from pipeline operations, and Amazon Simple Notification Service is used to start the processing pipeline when new content is added.
BMW: AWS is being used by the BMW Group for their connected car application, which collects sensor data from BMW 7 Series automobiles and provides drivers with constantly updated map information. BMW used Amazon SQS, Amazon S3, Amazon DynamoDB, Amazon RDS, and AWS Elastic Beanstalk to create its new car-as-a-sensor (CARASSO) service in just six months.
Change Healthcare: Change Healthcare uses Amazon Web Services (AWS) to manage millions of confidential transactions from its clients every day, including Amazon EC2, Amazon S3, Amazon SQS, and Amazon SNS. while remaining in full compliance with HIPAA and other healthcare industry laws.
RedBus: For monitoring, notifications, and intercommunication, RedBus is expanding its AWS service to incorporate Amazon SQS and SNS. “Amazon SQS is a very good solution for enabling messages between our applications and external applications,” says the author. – RedBus CTO Charan Padmaraju.
Conclusion
In this blog, we have explored Amazon SQS, its benefits, features, types of Queues in SQS, its functionality, and security. With Amazon SQS one can send any amount of data at any speed without missing messages or requiring the availability of other services, along with improving the system’s overall fault tolerance. We will demonstrate the full implementation of Amazon SQS with step-by-step instructions in our upcoming blog. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.
For any further queries, feel free to post your comments, we are happy to help!
Meanwhile …
Keep Exploring -> Keep Learning -> Keep Mastering
This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.
