Since the term was coined in 2011, Microservices have been making waves among forward-thinking application development corporations. Microservices help decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. When your application is decoupled you need them to communicate in a seamless way to avoid any delay, this is where Queue-based Microservices come into the picture. Using AWS Step Functions and Amazon SQS you can easily orchestrate Queue-based Microservices. In this blog, we will simulate inventory verification requests from incoming orders in an e-commerce application as part of an order processing workflow. We will see how to use AWS Step Functions, AWS Lambda, and Amazon SQS to design and run a serverless workflow that orchestrates a message queue-based Microservice.
In this blog, we will demonstrate step-by-step instructions on how to orchestrate Queue-based Microservices with AWS Step Functions and Amazon SQS.
AWS Step Functions – Recap
AWS Step Functions is a serverless function orchestrator that makes it easy to coordinate multiple AWS service components of distributed applications and microservices using visual workflows. It is a reliable way to coordinate components and step through the functions of your application.
When your distributed applications become complex, the complexity of managing them also grows. With its built-in operational controls, Step Functions manages sequencing, error handling, retry logic, and state, and provides a graphical console to arrange and visualize the components of your application as a series of steps. This makes it easy to build and run multi-step applications. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order.
Amazon SQS – Recap
Queues are a great way to mix and match software structures. They allow non-simultaneous communication across separate systems, which is particularly beneficial when the system’s output is uneven. Amazon’s version of queues is called Amazon SQS or Simple Queue Service.
Amazon SQS is a hosted queue that allows you to link and disconnect distributed software systems and components. It is safe, durable, and available. SQS removes the complexity and expense of managing and operating message-oriented middleware, allowing developers to concentrate on unique work. SQS allows you to transmit, store, and receive messages at any volume between software components without losing messages or necessitating the availability of other services.
Hands-on
In this blog, we’ll look at how to utilize AWS Step Functions, Amazon SQS, and AWS Lambda to create a serverless workflow that orchestrates a message queue-based microservice. The storing of inventory verification requests from incoming orders in an e-commerce application will be our use case for SQS in this hands-on. As part of an order processing workflow, this hands-on will mimic inventory verification requests from incoming orders on an e-commerce platform. Inventory verification requests will be sent to a queue on SQS via Step Functions. We’ll use AWS Step Functions to create the process. A microservice will be used to verify inventories in the workflow. A queue is used by many microservices to receive requests. To illustrate the microservice in this hands-on, we’ll utilize an AWS Lambda function. A task state is used by the state machine to place a message on an SQS queue. A callback pattern is set for this task state. When you add to the end of a sentence, it’s called an appendix Step Functions will add a task token to the JSON payload and wait for a callback if you add waitForTaskToken to your resource. By contacting the Step Functions API, the microservice can return a response to Step Functions. Your inventory microservice will be an AWS Lambda function that utilizes a queue to buffer requests. It will verify inventory after retrieving a request and then deliver the result to Step Functions. A callback pattern is created when a job in Step Functions is set up in this manner. Callback patterns allow you to integrate asynchronous tasks in your workflow, such as the inventory verification microservice of this hands-on.
To implement this, we will do the following:
- Login to your AWS console and navigate to the dashboard.
- Navigate to the SQS service dashboard.
- Create a new standard queue with the configurations provided.
- Navigate to the Step Functions service dashboard.
- Create a new state machine with the workflow code provided in this blog.
- Navigate to the IAM dashboard to create a new role.
- Create a new role with the described policies for SQS and Step Functions in this blog.
- Navigate to the Lambda function dashboard.
- Create a new function attaching the created IAM role and adding the code provided in this blog.
- Attach the SQS trigger as the source trigger on the Lambda function dashboard.
- Navigate back to the Step Functions dashboard and start the execution to test the configurations.
- Check the execution result for success or failure.
- Explore the Execution event history to check the execution results.
- Finally, explore the Graph inspector on the state machine dashboard and test the workflow execution results.
Log in to the AWS account and navigate to the console.
Search for the SQS service and click on it to navigate to its dashboard.
On the dashboard, click on Create queue.
You will be navigated to the page as shown in the image below. Choose Standard.
Enter a name for the queue and make the configuration changes as shown in the image below.
Keep the Access policy as default.
Enter tags for your queue if any and click on Create queue.
On success, you will see the message as shown in the image below.
Search for the Step functions service and navigate to the dashboard.
On the dashboard, you can have a look at the pricing structure for Step Functions.
How it works has an interactive demo added to the dashboard.
Expand the left navigation pane and click on State machines. Click on Create state machine.
Select the options as shown in the image below for the workflow and the type of the workflow.
In case you are confused with which type to choose, you can expand the Help me decide and make a call for the same.
Now, scroll down to the definition.
Copy the below JSON and add it in the editor.
{
"Comment": "An example of the Amazon States Language for starting a callback task.",
"StartAt": "Check Inventory",
"States": {
"Check Inventory": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "<INSERT SQS QUEUE URL HERE>",
"MessageBody": {
"MessageTitle": "Callback Task started by Step Functions",
"TaskToken.$": "$$.Task.Token"
}
},
"Next": "Notify Success",
"Catch": [
{
"ErrorEquals": [ "States.ALL" ],
"Next": "Notify Failure"
}
]
},
"Notify Success": {
"Type": "Pass",
"Result": "Callback Task started by Step Functions succeeded",
"End": true
},
"Notify Failure": {
"Type": "Pass",
"Result": "Callback Task started by Step Functions failed",
"End": true
}
}
}
The graph will then change to as shown in the image below.
Now, navigate back to the SQS queue dashboard. Copy the URL of the newly created queue.
Paste it beside the QueueURL key in the editor pane. Scroll down and click on Next.
Enter a name for the state machine and select create a new role under permissions.
Add tags if any needed for your state machine.
Once done, scroll to the bottom and click on Create state machine.
On success, you will see the message as shown in the image below.
Now, search for the IAM service and navigate to its dashboard.
Select Roles from the left navigation pane.
Click on Create role on the dashboard.
Select Lambda under choose a use case and click on Next:Permissions.
Search for the AmazonSQSFullAccess policy and check the box.
Search for the AWSStepFunctionsFullAccess policy and check the box. Click on Next: Tags.
Add tags if any needed for your IAM role. Click on Next: Review.
Enter a role name for the newly created role. Click on Create role.
On success, you will see the message as shown in the image below.
Now, search for Lambda service and navigate to the dashboard.
Click on Create function.
Select Author from scratch, enter a name for the function and select the runtime as Node.js 14.x
Select Use an existing role and from the roles dropdown, choose the newly created role.
Scroll down and click on the Create function.
On success, you will see the message as shown in the image below.
Now, navigate to the code editor pane and paste the below code in the pane.
console.log('Loading function');
const aws = require('aws-sdk');
exports.handler = (event, context, callback) => {
const stepfunctions = new aws.StepFunctions();
for (const record of event.Records) {
const messageBody = JSON.parse(record.body);
const taskToken = messageBody.TaskToken;
const params = {
output: "\"Callback task completed successfully.\"",
taskToken: taskToken
};
console.log(`Calling Step Functions to complete callback task with params ${JSON.stringify(params)}`);
stepfunctions.sendTaskSuccess(params, (err, data) => {
if (err) {
console.error(err.message);
callback(err.message);
return;
}
console.log(data);
callback(null);
});
}
};
Click on Deploy to deploy the new code.
On success, you will see the message as shown below.
Scroll up and click on Add trigger. Make the configurations as shown in the image below. Click on Add.
On success, you will see the message as shown in the image below.
Navigate to the State machine dashboard and click on Start execution.
A modal will appear, click on Start execution.
The process of the workflow will begin and will take some time to be configured.
On Success, you’ll see the states turned to Green.
If you scroll up, you can check the Execution output tab to see the output of the execution.
Click on any of the state and you’ll see the Details for the same.
Under execution event history, you’ll see the flow of execution.
Expand one and you’ll see the output for the same.
In the Step output, you’ll see the Success message for the execution.
Conclusion
In this blog, we saw how to use AWS Step Functions and Amazon SQS to design and run a serverless workflow that orchestrates a message queue-based microservice. We first created an Amazon SQS Queue to simulate the storage of inventory verification requests from incoming orders in an e-commerce application. Then, we created a workflow with a state machine to describe how we wanted e-commerce orders to be processed. We then created a new role on the AWS IAM console to allow AWS Step Functions and SQS full access policies to the newly created role. We then created a microservice queue-based execution with AWS Lambda Functions. We will discuss more use cases for the services used in our upcoming blogs. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.
Meanwhile …
Keep Exploring -> Keep Learning -> Keep Mastering
This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.
