Amazon Simple Notification Service (SNS) and message batching

Kevin Burns

Kevin Burns

— 4 min read
Amazon Simple Notification Service (SNS) and message batching
AWS SNS Message batching

Amazon Simple Notification Service (SNS) is a fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications.

SNS offers high scalability, flexibility, and reliability for sending messages between application components. One of its key features is batch publishing, which can enhance the performance of you application and help lower costs of your SNS API calls, especially for high-volume messaging scenarios.

This article will attempt to explain how to implement SNS batching with a simple example using Python, along with its pros and cons.

Setting Up SNS for Batching in Python

To get started with SNS batch publishing:

  1. Create an Amazon SNS topic
  2. Subscribe endpoints to the topic (e.g., Amazon SQS queues, HTTP/HTTPS endpoints, or email addresses)
  3. Use the AWS SDK for Python (Boto3) to publish messages to the topic in batch mode

Here is a basic setup using Boto3 with additional error handling and I am also creating the topic for testing the message batching, of course if you already have a topic use that one instead.

import boto3
import os
from botocore.exceptions import ClientError

# Create an SNS client
sns = boto3.client('sns')

# Function to create an SNS topic
def create_sns_topic(topic_name):
    try:
        response = sns.create_topic(Name=topic_name)
        return response['TopicArn']
    except ClientError as e:
        print(f"Error creating SNS topic: {e}")
        return None

# Function to publish batch messages
def publish_batch_messages(topic_arn, messages):
    try:
        response = sns.publish_batch(
            TopicArn=topic_arn,
            PublishBatchRequestEntries=messages
        )
        return response
    except ClientError as e:
        print(f"Error publishing messages: {e}")
        return None

# Main execution
if __name__ == "__main__":
    # Create an SNS topic
    topic_name = 'my-topic'
    topic_arn = create_sns_topic(topic_name)
    
    if topic_arn:
        print(f"Topic created: {topic_arn}")
        
        # Prepare a batch of messages
        messages = [
            {
                'Id': '1',  # Unique ID for each message in the batch
                'Message': 'Hello, World!',
                'Subject': 'Greeting'  # Optional
            },
            {
                'Id': '2',
                'Message': 'Batch publishing demo',
                'Subject': 'Demo'
            }
        ]

        # Publish messages in batch
        response = publish_batch_messages(topic_arn, messages)
        
        if response:
            print(f"Messages published successfully to {topic_name}: ")
            print(response)
    else:
        print("Failed to create SNS topic. Check your AWS credentials and permissions.")

This example demonstrates creating a topic, preparing a batch of messages, and publishing them using the publish_batch method.

Alternatively you already have the Topic Arn created

import boto3
import os
from botocore.exceptions import ClientError

# Create an SNS client
sns = boto3.client('sns')

# Function to publish batch messages
def publish_batch_messages(topic_arn, messages):
    try:
        response = sns.publish_batch(
            TopicArn=topic_arn,
            PublishBatchRequestEntries=messages
        )
        return response
    except ClientError as e:
        print(f"Error publishing messages: {e}")
        return None

# Main execution
if __name__ == "__main__":
    # Use the existing SNS topic ARN
    topic_arn = os.environ.get('SNS_TOPIC_ARN')
    
    if not topic_arn:
        print("SNS_TOPIC_ARN environment variable is not set. Please set it with your existing topic ARN.")
    else:
        print(f"Using existing topic ARN: {topic_arn}")
        
        # Prepare a batch of messages
        messages = [
            {
                'Id': '1',  # Unique ID for each message in the batch
                'Message': 'Hello, World!',
                'Subject': 'Greeting'  # Optional
            },
            {
                'Id': '2',
                'Message': 'Batch publishing demo',
                'Subject': 'Demo'
            }
        ]

        # Publish messages in batch
        response = publish_batch_messages(topic_arn, messages)
        
        if response:
            print("Messages published successfully:")
            print(response)
        else:
            print("Failed to publish messages. Check your AWS credentials and permissions.")

Pros & Benefits of SNS Batching

  1. Improved Performance: Batching allows multiple messages to be sent in a single API call, reducing network overhead and improving throughput. This is particularly beneficial for high-volume messaging scenarios.

  2. Cost: SNS pricing is based on the number of API calls and the total size of published messages. By reducing the number of API calls through batching, you lower your SNS usage costs by a factor of 10 if you batch the maximum amount of 10 messages per request.

  3. Simplified Error Handling: The batch API provides detailed success and failure information for each message in the batch, allowing for error handling and retries.

The Cons

While batch publishing offers several advantages, it's important to consider the following:

  1. Maximum Batch Size: SNS limits each batch to a maximum of 10 messages. Larger batches will need to be split into multiple requests. You can create a additional helper function that can split the multiple messages into batches of 10.

  2. Message Size Limits: Each message in a batch must adhere to the standard SNS message size limit.

  3. Partial Batch Failures: If some messages in a batch fail to publish, you'll need to handle retries for those specific messages.

  4. Order of Delivery: SNS does not guarantee the order of message delivery, even within a batch. If message ordering is critical, you may need to implement additional controls for this.

  5. Increased Complexity: Implementing batching adds some complexity to your code compared to single-message publishing

Best Practices for SNS Batch Publishing

  1. Optimize Batch Size: Balance between maximizing the batch size (up to 10 messages) and maintaining reasonable response times for your application.

  2. Implement Robust Error Handling: Always check the response from publish_batch and handle any failed messages appropriately.

  3. Use Message Deduplication: If your use case requires exactly-once delivery, consider using FIFO (First-In-First-Out) topics with message deduplication.

  4. Monitor and Adjust: Use Amazon CloudWatch to monitor your SNS metrics and adjust your batching strategy as needed.

Conclusion

Batching in SNS is a powerful feature for improving the efficiency and cost-effectiveness of your messaging infrastructure, especially for high-volume scenarios.
By understanding its benefits and limitations, and following best practices, you can effectively leverage this capability in your Python applications.
As with any architectural decision, consider your specific use case, performance requirements, and scalability needs when deciding to implement batch publishing.