SQS is a scalable, fast, and fully managed distributed queuing service from AWS. AWS SQS helps you build loosely coupled applications. This allows the application components to run independently with SQS managing the message flow between them. SQS can act as a buffer between the writers and the readers of the messages, and each queue can support multiple writers and readers.
Minimum SQS message size is 1 KB, maximum message size is 256 KB. Messages in SQS queues are stored for 4 days; however, you can increase the retention period from 1 minute to 14 days. You can create an unlimited number of queues, and a queue can contain unlimited number of messages.
AWS SQS supports long polling, so instead of querying SQS every 10 seconds with no results, you can send the request to the SQS. It responds whenever there is a message present in that specific queue.
There is no upfront cost to pay, and you only pay for what you use. Note that due to the distributed nature of the queues, SQS does not guarantee first in, first out delivery of messages.
However, SQS does guarantee the delivery of your messages at least once. As AWS SQS stores your messages on multiple servers, it is possible that a message was not deleted on a temporarily unavailable server. This can result in your message being received and processed again. Hence, ensure your application is designed to take care of this situation.
- Installing AWS Java SDK.
In your Maven dependency section, add the following dependency for AWS Java SDK Version 1.9.28.1:
<dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-java-sdk</artifactId> <version>1.9.28.1</version> </dependency> - Creating a queue.
Execute the following sample program to create a queue called
thumbnailjobinput:// Create queue. public static void CreateQueue() { // Create BasicAWSCredentials with Access Key Id and Secret Access Key. BasicAWSCredentials credentials = new BasicAWSCredentials( "Access Key Id", "Secret Access Key"); // Create SQS client. AmazonSQSClient sqsClient = new AmazonSQSClient(credentials); // Set endpoint. sqsClient.setEndpoint("sqs.ap-southeast- 1.amazonaws.com"); // Create queue. sqsClient.createQueue("thumbnailjobinput"); } - Insert a message in the queue.
Execute the following sample program to insert messages into the queue named
thumbnailjobinput. Here, we are inserting messages with a user ID, and a background thumbnail creation job that monitors this queue, processes the message and creates a thumbnail image for that specific user ID.// Insert message. public static void InsertMessage() { // Create BasicAWSCredentials with Access Key Id and Secret Access Key. BasicAWSCredentials credentials = new BasicAWSCredentials( "Access Key Id", "Secret Access Key"); // Create SQS client. AmazonSQSClient sqsClient = new AmazonSQSClient(credentials); // Set endpoint. sqsClient.setEndpoint("sqs.ap-southeast- 1.amazonaws.com"); // Insert message. sqsClient.sendMessage(new SendMessageRequest().withQueueUrl( "thumbnailjobinput") .withMessageBody("123")); } - Receive the message in your program.
Execute the following sample program to retrieve one message at a time from a queue named
thumbnailjobinput:// Receive message. public static void ReceiveMessage() { // Create BasicAWSCredentials with Access Key Id and Secret Access Key. BasicAWSCredentials credentials = new BasicAWSCredentials( "Access Key Id", "Secret Access Key"); // Create SQS client. AmazonSQSClient sqsClient = new AmazonSQSClient(credentials); // Set endpoint. sqsClient.setEndpoint("sqs.ap-southeast- 1.amazonaws.com"); // Prepare the request. ReceiveMessageRequest request = new ReceiveMessageRequest( "thumbnailjobinput"); // Set number of messages to retrieve from queue. request.setMaxNumberOfMessages(1); // Get messages. List<Message> lstMessage = sqsClient .receiveMessage("thumbnailjobinput") .getMessages(); // Iterate through messages. for (Message message : lstMessage) { // Get the message body. String msg = message.getBody(); } } - Alternatively, you can also retrieve and delete message from the queue. The following sample program illustrates this:
// Receive and Delete message. public static void ReceiveAndDeleteMessage() { // Create BasicAWSCredentials with Access Key Id and Secret Access Key. BasicAWSCredentials credentials = new BasicAWSCredentials( "Access Key Id", "Secret Access Key"); // Create SQS client. AmazonSQSClient sqsClient = new AmazonSQSClient(credentials); // Set endpoint. sqsClient.setEndpoint("sqs.ap-southeast- 1.amazonaws.com"); // Prepare the request. ReceiveMessageRequest request = new ReceiveMessageRequest( "thumbnailjobinput"); // Set number of messages to retrieve from queue. request.setMaxNumberOfMessages(1); // Get messages. List<Message> lstMessage = sqsClient .receiveMessage("thumbnailjobinput") .getMessages(); // Iterate through messages. for (Message message : lstMessage) { // Get the message body. String msg = message.getBody(); // Process the message. // Delete the message after processing message. String receiptHandle = message.getReceiptHandle(); // Prepare request. DeleteMessageRequest deleteRequest = new DeleteMessageRequest(); // Set queue name. deleteRequest.setQueueUrl("thumbnailjobinput"); // Set receipt handle. deleteRequest.setReceiptHandle(receiptHandle); // Delete message. sqsClient.deleteMessage(deleteRequest); } }
AWS Java SDK helps you access AWS services from Java applications. You have to create a queue before inserting any data into it. When creating queues, you need to provide unique name for each one of them. SQS assigns each queue a URL that must be used used in all subsequent operations on that queue. SQS returns a message ID for each message in the response to the send message request.
The default polling method for messages is the short or standard polling. In short polling, SQS samples a subset of servers and returns messages from them. Hence, a request need not return all the messages in the distributed queue. If you continue to retrieve the messages, then SQS will sample all the servers and return your messages. SQS also supports long polling which reduces the number of empty responses when the queue is empty. Additionally, in long polling SQS queries all the servers for your messages. We can enable long polling in SQS by setting parameters in ReceiveMessage, CreateQueue, and SetQueueAttributes API calls. In our example, we are using the default polling method, that is, short polling.
While retrieving messages from the queue, you can also specify the number of messages to retrieve. In our example, we specify a value of 1 for the number of messages in our receive message request. Each time a message is received from the queue, you receive a receipt handle with it. This receipt handle is used for deleting the message or changing the message visibility.
As the message remains in the queue, SQS blocks it with a visibility timeout, which is the time period during which other processing components in your application cannot receive and process the same message. The default visibility timeout for messages in the queue is 30 seconds, that is, if you don't delete the message after retrieving and processing it within 30 seconds, then SQS makes the message visible again. You can also extend or terminate a message's visibility timeout through the API.
SQS provides support for message attributes. These attributes can be used to provide metadata about your message, for example, timestamps. This metadata is useful to processing component to decide on how to process the message without processing the message body.
SQS provides a batch message processing functionality for receiving, sending, and deleting messages, and also for changing the message visibility timeout values. These operations can process 10 messages in a single call. As more work is carried in the batch request, it makes more efficient use of threads and connections; thereby, improving the throughput. Overall costs are also reduced because the number of requests to process the messages is significantly reduced.
Amazon SQS supports dead letter queues. As a best practice, if some messages are not processed successfully, then they should be sent to a dead letter queue. This effectively isolates such messages and avoids repeated processing failures. You can analyze these messages to determine the reason for the processing failures. Dead letter queues should be located in the same region as your other queues.
You can use SQS access policies and/or IAM policies for setting up your access permissions. The main difference between the two is that SQS policies can be used to set up cross-account access to your queues. In addition, Amazon SQS is integrated with CloudWatch, so you can collect and analyze metrics such as the number of empty receives while polling for new messages. You can also create alarms and send notifications if the threshold is exceeded for a specific SQS metric, for example, the number of messages received. API calls to SQS can be tracked via CloudTrail Logs.