Messages

A message is a distinct piece of information transmitted by one client that can be received by one or more other clients. Each SQS message is identified by an ID value generated by the service. A message’s ID uniquely identifies that message within a queue, and the combination of a queue URL and a message ID can uniquely identify any message in SQS.

There is no limit to the number of messages that can be sent to SQS queues, nor to the number that may remain stored in queues pending delivery. Messages cannot be left in the service indefinitely; they will be automatically deleted after 15 days or so.

Warning

SQS does not include a timestamp with messages to indicate when a particular message was sent. This makes it difficult to check whether your queues contain messages that are nearly 15 days old. SQS is not intended for long-term data storage, so you must be prepared for very old messages to be deleted.

Messages have a maximum size of 256 KB. This size limit is not large by the standards of alternative messaging systems, and it clearly indicates Amazon’s intentions for how its messaging service should be used. SQS is not intended to serve as a means for distributing files or transmitting large data objects; it is intended to carry short notifications or instruction messages that refer to data resources stored outside the SQS service, for example, in the Simple Storage Service (S3).

The data content of SQS messages must be unicode text characters that are legal for standard XML documents, as defined in the XML 1.0 character specification.^[3] Legal unicode characters include those in the following ranges:

#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

If you need to transmit binary information, or you are in doubt about whether your messages will conform to these character set limitations, you should encode your message data using Base 64.

Sending Messages

The SendMessage operation creates a new message in an SQS queue and makes it available for retrieval. Table 8-5 shows the SendMessage request parameters.

The SQS Query API allows messages to be sent using either HTTP GET or POST requests. The size limits for messages sent using GET is only 8 KB due to the fact that the entire message text must be included in the request’s URI, and URIs cannot become too long. POST requests store the message body along with the other parameters in the request’s body and therefore do not suffer from the same message size limitation.

Warning

External factors, like web proxies between your clients and S3, may impose a URI limit much shorter than 8 KB for GET requests. If your messages are likely to be longer than a couple of thousand characters, we highly recommend you send them using POST requests.

Note

In addition to supporting standard HTTP POST request messages, the SQS Query API also allows messages to be sent using a specialized POST request format constructed as follows:

• The request’s URI is constructed in the same way as for a GET request, but it does not include the message’s contents.

• The message’s content is sent in the request’s body.

• The request uses the content type text/plain.

• The request description string does not include the MessageBody parameter.

It is unclear what advantages this nonstandard POST request format provides over the standard POST format that works as expected and is already supported by our AWS module. We will not implement this alternative approach in this book.

Table 8-5. SendMessage request parameters

Parameter Name	Value	Required?
Action	SendMessage	Yes
MessageBody	The data content of the message can be up to 256 KB and must be UTF-8 encoded text that is compatible with XML.	Yes

Here is an XML document returned by the operation, including the message’s unique identifier value in the MessageId element.

<SendMessageResponse xmlns='http://queue.amazonaws.com/doc/2007-05-01/'>
  <MessageId>0WQR23VAFKFCYM4ZC9MM|72AD0F0NE09VCMT3AJ61|M57DGQ7TB0T3MFY3J1X0</MessageId>
  <ResponseStatus>
    <StatusCode>Success</StatusCode>
    <RequestId>06ea96d8-008a-41e7-8bb9-f3b6da5d6a65</RequestId>
  </ResponseStatus>
</SendMessageResponse>

Example 8-5 defines a method that sends a message to an SQS queue and returns the identifier of the newly created message. The message data can optionally be Base-64 encoded prior to being sent if the method parameter encode is set to true.

Example 8-5. Send message: SQS.rb

def send_message(queue_url, message_body, encode=false)
  message_body = encode_base64(message_body) if encode

  parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
    {
    'Action' => 'SendMessage',
    'MessageBody' => message_body
    })

  response = do_query(HTTP_METHOD, URI.parse(queue_url), parameters)

  xml_doc = REXML::Document.new(response.body)
  return xml_doc.elements['//MessageId'].text
end

Here we demonstrate the vital difference between sending a message to a queue using a GET versus a POST request. We have included some of the debug logging information to highlight the difference in the URI between the two requests. Notice that the GET request requires a long URI even when the message is small, while the URI for the POST request is always quite short.

# Send a message using a GET request
irb> SQS::HTTP_METHOD = 'GET'
irb> sqs.send_message(queue_url, 'Hello SQS!')
REQUEST
=======
Method: GET
URI: http://queue.amazonaws.com/A1MU5FWLQSN7CU/testing?Action=SendMessage
     &Signature=kT3RopxqdtNrv6Tk1yOS4lcK3g0%3D
     &MessageBody=Hello+SQS%21
     &Version=2007-05-01
     &AWSAccessKeyId=ABCDEFGHIJ1234567890
     &Timestamp=2007-08-29T03%3A11%3A53Z
     &SignatureVersion=1
. . .
=> "0WQR23VAFKFCYM4ZC9MM|72AD0F0NE09VCMT3AJ61|M57DGQ7TB0T3MFY3J1X0"

irb> SQS::HTTP_METHOD = 'POST'
irb> sqs.send_message(queue_url, 'Hello SQS!')
REQUEST
=======
Method: POST
URI: http://queue.amazonaws.com/A1MU5FWLQSN7CU/testing
. . .
=> "1S5HB19QG48H4V6JYSEE|1SE7TJVAQZBXRPC11VS0|1SH8K2VEBJ99XCYY2JH0"

Receiving Messages

The ReceiveMessage operation allows an SQS client to receive messages that have been sent to a specific queue. This operation is slightly misnamed in the API, because it can actually receive more than one message at a time. The response for this operation includes a listing of messages with the ID and content data for each message.

Receiving a message from a queue does not delete the message from that queue. The receive action triggers the message to become invisible to subsequent ReceiveMessage requests for a period of time; this is called the message’s visibility timeout. The timeout to apply to received messages can be set with a parameter in the ReceiveMessage operation, or if no explicit timeout value is provided, the message is assigned the queue’s default visibility timeout. For more information on the visibility timeout and life cycle of messages, refer back to The Message Life Cycle.”

The ReceiveMessage operation allows you to specify a maximum limit for the number of messages you wish to receive; however, there is no guarantee that SQS will provide the number of messages requested, even if the queue contains more than enough messages. As we discussed in SQS Architecture,” SQS only checks a subset of its distributed servers for available messages. This means that any one receive request may return fewer messages than are actually available, and that multiple ReceiveMessage operations may be necessary to receive all of the available messages.

Table 8-6. ReceiveMessage request parameters

Parameter Name	Value	Required?
Action	ReceiveMessage	Yes
NumberOfMessages	The maximum number of messages to receive; this value may range from 1 to 256 inclusive. If this parameter is not provided with the request, a maximum of one message will be returned.	No
VisibilityTimeout	The visibility timeout that will be applied to any messages received by this operation; this value is specified in seconds and may be from 0 to 86,400. If this parameter is not provided, the queue’s default visibility timeout setting will be applied to the messages.	No

Here is an XML document returned by the operation, which includes a Message element for each message received from the service. The Message element includes two subelements: MessageId, which contains the message’s unique identifier, and MessageBody, which contains the message’s data content.

<ReceiveMessageResponse xmlns='http://queue.amazonaws.com/doc/2007-05-01/'>
  <Message>
    <MessageId>0NMA4SD7PN4TN0MF9ZZS|...|260BDAS2K7RNYSSR6GY1</MessageId>
    <MessageBody>Hello SQS!</MessageBody>
  </Message>
  <ResponseStatus>
    <StatusCode>Success</StatusCode>
    <RequestId>413154fa-18da-4002-81fe-3df6dbee0cf8</RequestId>
  </ResponseStatus>
</ReceiveMessageResponse>

Example 8-6 defines a method that receives messages from a queue and returns them as an array of hash dictionary objects. Each message is represented by a hash containing named mappings to the message ID and body content text. If the visibility_timeout parameter is provided with a duration value, the received messages will be made invisible to subsequent requests for the specified period of time.

Example 8-6. Receive message: SQS.rb

def receive_messages(queue_url, maximum=1, visibility_timeout=nil)
  parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
    {
    'Action' => 'ReceiveMessage',
    'NumberOfMessages' => maximum,
    'VisibilityTimeout' => visibility_timeout
    })

  response = do_query(HTTP_METHOD, URI.parse(queue_url), parameters)

  msgs = []

  xml_doc = REXML::Document.new(response.body)

  xml_doc.elements.each('//Message') do |msg|
    msgs << {
      :id => msg.elements['MessageId'].text,
      :body => msg.elements['MessageBody'].text
    }
  end

  return msgs
end

To receive the messages we sent earlier, we will run the receive_messages method and store the results in the variable msgs, so we can have a closer look at them. In our examples we have sent two messages, so we will try receiving up to two messages in this request.

irb> msgs = sqs.receive_messages(queue_url, 2)
=> [{:body=>"Hello SQS!", 
    :id=>"0NMA4SD7PN4TN0MF9ZZS|1SE7TJVAQZBXRPC11VS0|260BDAS2K7RNYSSR6GY1"}]

irb> msgs.length
=> 1

Notice that our receive request returned only one message, not two as we might have expected. This is because one of our messages is stored on SQS servers that were not sampled during the operation. If we run the command a number of times, we will eventually receive all the available messages, although before long we will also start receiving redelivered messages because the visibility timeout for the original message will expire.

Deleting Messages

The DeleteMessage operation outlined in Table 8-7 removes a specific message from a queue. The message is deleted immediately, regardless of its visibility state or whether it has been delivered to any SQS clients.

You should delete messages from SQS as soon as possible once a message has reached the end of its useful life; messages will persist and may be redelivered if you do not remove them from the service.

Note

This operation returns a successful result in all cases in which a message with the specified ID is no longer present in the queue. This means the action will succeed when you delete a message that was already deleted, or even if you delete a message that never actually existed in the queue in the first place.

Table 8-7. DeleteMessage request parameters

Parameter Name	Value	Required?
Action	DeleteMessage	Yes
MessageId	The identifier of the message to delete	Yes

Here is an XML document returned by the operation. The response contains no useful information beyond the status code.

<DeleteMessageResponse xmlns='http://queue.amazonaws.com/doc/2007-05-01/'>
  <ResponseStatus>
    <StatusCode>Success</StatusCode>
    <RequestId>7e53e4f7-07f3-40ca-8d7b-f4adda882289</RequestId>
  </ResponseStatus>
</DeleteMessageResponse>

Example 8-7 defines a method that deletes a specific message from a queue. The method returns true if the request succeeds.

Example 8-7. Delete message: SQS.rb

def delete_message(queue_url, message_id)
  parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
    {
    'Action' => 'DeleteMessage',
    'MessageId' => message_id
    })

  response = do_query(HTTP_METHOD, URI.parse(queue_url), parameters)

  return true
end

We can test the new method by sending a new message and storing the message’s ID in the variable msg_id. Once we have this ID, we can delete the message. We can even delete the message multiple times; the delete operation will always succeed even though the message is only actually deleted the first time.

irb> msg_id = sqs.send_message(queue_url, 'Doomed message')
=> "1QJNM0VK5203GVAS2KP7|A617CAQHX0CS7N69WY71|RW2145BC0M6GHC5ZPAC1"

irb> sqs.delete_message(queue_url, msg_id)
=> true

irb> sqs.delete_message(queue_url, msg_id)
=> true

Peeking at a Message

The PeekMessage operation described in Table 8-8 makes it possible to retrieve the data content of a message regardless of its visibility state, provided you know the message’s identifier. Unlike the ReceiveMessage operation, you can use the PeekMessage operation to view a message’s contents while the message is invisible, and peeking at a message does not trigger the message to become invisible.

You can only perform this operation when you know the message’s identifier in advance. For this reason, the PeekMessage action cannot be used as a replacement for the ReceiveMessage action. For a message receiver client to be able to peek at a message, it must first learn its identifier value, which it can do only by receiving the message.

The PeekMessage operation is most useful for tracking the progress of messages. For example, a message-sending client that knows the message’s identifier could determine whether or not a message has been processed by peeking at the message on the queue to see if it has been deleted.

Table 8-8. PeekMessage request parameters

Parameter Name	Value	Required?
Action	PeekMessage	Yes
MessageId	The identifier of the message to peek at	Yes

Here is an XML document returned by the operation. The document includes a single Message element representing the peeked message. The Message element includes two subelements: MessageId, which contains the message’s unique identifier, and MessageBody, which contains the message’s data content.

<PeekMessageResponse xmlns='http://queue.amazonaws.com/doc/2007-05-01/'>
  <Message>
    <MessageId>1ZBAAYNJ59Y3ACC1SKWD|...|M57DGQ7TB0T3MFY3J1X0</MessageId>
    <MessageBody>Hello SQS!</MessageBody>
  </Message>
  <ResponseStatus>
    <StatusCode>Success</StatusCode>
    <RequestId>288526a0-fa04-4d2c-bbe2-2541c36bf702</RequestId>
  </ResponseStatus>
</PeekMessageResponse>

Example 8-8 defines a method that peeks at a message and returns its data content and identifier. It is fairly pointless to return the message’s identifier, because you must already know this value to perform the operation; however, this result structure will be consistent with the data structure returned by the receive_messages method.

Example 8-8. Peek at message: SQS.rb

def peek_message(queue_url, message_id)
  parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
    {
    'Action' => 'PeekMessage',
    'MessageId' => message_id
    })

  response = do_query(HTTP_METHOD, URI.parse(queue_url), parameters)

  xml_doc = REXML::Document.new(response.body)

  return {
    :id => xml_doc.elements['//MessageId'].text,
    :body => xml_doc.elements['//MessageBody'].text
  }
end

To demonstrate message peeking, we will obtain message identifiers for the messages we sent previously and compare what happens when we peek at a message versus receiving it.

# Find a message identifier
irb> msg_id = sqs.receive_messages(queue_url).first[:id]
=> "1ZBAAYNJ59Y3ACC1SKWD|A617CAQHX0CS7N69WY71|M57DGQ7TB0T3MFY3J1X0"

# Peek at the message to retrieve its identifier and contents
irb> sqs.peek_message(queue_url, msg_id)
=> {:body=>"Hello SQS!", 
    :id=>"1ZBAAYNJ59Y3ACC1SKWD|A617CAQHX0CS7N69WY71|M57DGQ7TB0T3MFY3J1X0"}

To see where the peek operation can be very handy, we can try running the receive_messages method a few times, until we no longer receive any results; at this point all of our messages have become invisible to ReceiveMessage operations. Although we cannot receive these messages, we can still use a PeekMessage operation to view the message contents.

irb> sqs.receive_messages(queue_url, 10).length
=> 1
irb> sqs.receive_messages(queue_url, 10).length
=> 1
irb> sqs.receive_messages(queue_url, 10).length
=> 0

# All messages are invisible, yet we can still peek at the contents
irb> sqs.peek_message(queue_url, msg_id)
=> {:body=>"Hello SQS!", 
    :id=>"1ZBAAYNJ59Y3ACC1SKWD|A617CAQHX0CS7N69WY71|M57DGQ7TB0T3MFY3J1X0"}

We can determine when a message has been removed from a queue by checking for a 404 Not Found error response when we peek at the message.

irb> sqs.delete_message(queue_url, msg_id)
=> true

irb> sqs.peek_message(queue_url, msg_id)
AWS::ServiceError: HTTP Error: 404 - Not Found, AWS Error:
MessageNotFound - No such message with requested id

Controlling the Visibility of Messages

The ChangeMessageVisibility operation detailed in Table 8-9 modifies the visibility timeout duration for a specific message. A message’s visibility timeout can be adjusted at any time, regardless of the message’s prior visibility status.

Note

Alterations to the message’s timeout value take effect immediately, but they are one-off adjustments that apply only until the next visibility state change.

This operation can be used in a range of scenarios:

Defer the redelivery of a received message

If a message recipient will not have time to finish processing a message before it is scheduled to be redelivered, the recipient can defer the redelivery by changing the message’s visibility timeout to a higher value.

Mark a message for immediate redelivery

If a message recipient is unable to process a message, it can reset the message’s visibility timeout to 0 seconds. Doing this will make the message immediately visible again, so it can be received and processed by another message receiver.

Defer the first delivery of a message

A message sender can defer the initial delivery of a message by setting its visibility timeout to a high value before any receivers have accessed it. If the sender is the only one who knows the message’s ID, it will effectively be locked for the duration of the visibility timeout.

The sender cannot rely completely on this strategy, because there is no way to know for sure whether a particular message has already been received. A message’s visibility timeout cannot be specified in the SendMessage operation; it must be changed with a follow-up operation. Therefore, it is possible that the message will be delivered before the sender can apply a visibility timeout adjustment to lock the message.

Table 8-9. ChangeMessageVisibility request parameters

Parameter Name	Value	Required?
Action	ChangeMessageVisibility	Yes
MessageId	The identifier of the message that will have its visibility timeout changed.	Yes
VisibilityTimeout	The new visibility timeout value to apply to the message, measured in seconds. This value must be between 0 and 86400 inclusive.	Yes

Here is an XML document returned by the operation. The response contains no useful information beyond the status code.

<ChangeMessageVisibilityResponse xmlns='http://queue.amazonaws.com/doc/2007-05-01/'>
  <ResponseStatus>
    <StatusCode>Success</StatusCode>
    <RequestId>104a8898-77db-4808-bd95-cd7728093f89</RequestId>
  </ResponseStatus>
</ChangeMessageVisibilityResponse>

Example 8-9 defines a method that changes a message’s visibility timeout setting and returns true provided the request was successful. If no value is provided for the visibility_timeout parameter, the message’s visibility timeout is set to 0 seconds, making the message visible immediately.

Example 8-9. Change message visibility: SQS.rb

def change_message_visibility(queue_url, message_id, visibility_timeout=0)
  parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
    {
    'Action' => 'ChangeMessageVisibility',
    'MessageId' => message_id,
    'VisibilityTimeout' => visibility_timeout
    })

  response = do_query(HTTP_METHOD, URI.parse(queue_url), parameters)

  return true
end

The best way to demonstrate this method is to call the receive_messages method repeatedly, until all our messages are in the invisible state. We will then reset the visibility timeout of one message to 0 seconds, making it visible immediately, and confirm that we can receive that message that has been made visible.

# Locate a specific message ID
irb> msg_id = sqs.receive_messages(queue_url).first[:id]
=> "0MXR9SQQT1JHFPK37JCN|72AD0F0NE09VCMT3AJ61|K6KT2VSF12BR9S52HCC0"

# Send receive requests until all your messages are invisible
irb> sqs.receive_messages(queue_url, 10).length
=> 1
irb> sqs.receive_messages(queue_url, 10).length
=> 0

# Reset the visibility timeout for one message
irb> sqs.change_message_visibility(queue_url, msg_id, 0)
=> true

# Confirm that you can now receive that message
irb> sqs.receive_messages(queue_url, 10)
=> [
{:body=>"Hello SQS!", 
 :id=>"0MXR9SQQT1JHFPK37JCN|72AD0F0NE09VCMT3AJ61|K6KT2VSF12BR9S52HCC0"}]

---

^[3] See the character set section in Extensible Markup Language (XML) 1.0 (Fourth Edition) - http://www.w3.org/TR/REC-xml/#charsets