Integrating a messaging queue

SNS is a messaging system following the Pub/Sub pattern where you fire a message from one source and consume from many resources parallely. On the other hand, the other important messaging systems are queues, where you can fire messages from multiple sources and consume them from single or multiple consumers, but one by one.

AWS also provides such a service called SQS (Simple Queue Service). In SQS, you create queues and send messages to them. After that, you can poll messages to your customer and consume them. Before we finish this chapter, we can also see briefly how we can use SQS in our application. However, we will not consume the messages we will push to the queue we will create. Our application does not have such a use case, and to consume SQS messages, you need to have an always up and running consumer component installed on a normal instance.

In this section, we will replicate the messages from SNS to SQS. We've already created a SNS topic for registered users. We can also set the SQS topic as a subscriber to our topic, so whatever notification that comes to SNS will be also stored at the SQS queue. After that, it is up to you to consume those messages or not. SQS automatically deletes messages that have been in a queue for more than maximum message retention period. The default value for the retention period is 4 days. However, you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14 days).

First, let's create our queue and the policy attached to it:

"UserRegistrationQueue": { 
  "Type": "AWS::SQS::Queue" 
}, 
"UserRegistrationQueuePolicy": { 
  "Type": "AWS::SQS::QueuePolicy", 
  "Properties": { 
    "PolicyDocument": { 
      "Version": "2012-10-17", 
      "Statement": [ 
        { 
          "Effect": "Allow", 
          "Principal": "*", 
          "Action": "SQS:SendMessage", 
          "Resource": { 
            "Fn::GetAtt": [ 
              "UserRegistrationQueue", 
              "Arn" 
            ] 
          }, 
          "Condition": { 
            "ArnEquals": { 
              "aws:SourceArn": { 
                "Ref": "UserRegistrationSnsTopic" 
              } 
            } 
          } 
        } 
      ] 
    }, 
    "Queues": [ 
      { 
        "Ref": "UserRegistrationQueue" 
      } 
    ] 
  } 
} 

AWS::SQS::Queue has a lot of options for fine-tuning and also dead letter queues, and you should adjust them according to your needs, but with the default options, it will just work. The dead letter queue is a concept which many queue systems implement and it is intended to store messages which are not successfully processed in normal ways. So, you can use dead letter queues as a error recovery system.

The second resource adds an access policy to the queue. Just like Lambdas, we need to let SQS know that another identity, SNS in this case, wants to access to it for publishing messages. That's why we allow the SQS:SendMessage permission to SNS if the incoming message belongs to our topic's ARN.

As the last step, we will add the following subscription definition to Subscription to the UserRegistrationSnsTopic resource:

{ 
  "Endpoint": { 
     "Fn::GetAtt": [ 
       "UserRegistrationQueue", 
       "Arn" 
     ] 
  }, 
  "Protocol": "sqs"
} 

After deploying our application, we can navigate to the SQS section of the AWS Console and see our queue there.

You can click on the queue and Queue Actions and then on the View/Delete Messages button. A new window will open where you can see the incoming messages to your queue:

Now, you can create a new user via your API and see that the information about new users is arriving to your queue.