AWS SNS – SMS Monitoring Monthly Usage | Finding your AWS SNS SMS monthly usage using CloudWatch Insights and custom bash script with SNS Usage Reports

We all know how SNS keeps track of SMSMonthToDateSpentUSD metric under CloudWatch Metrics letting us know the current USD value
the account is currently using. However, at times we are often looking to get more insights into how many messages were sent out in a calendar month, and as of now, we do not have any metrics that expose this information. To get this, we would need
to dig into a few things and programmatically come up with a solution. There are two ways in which we can approach this with some caveats:

1. The simple way – using CloudWatch Insights

PreReq: SMS logging is enabled in a region.

SNS will create two log groups to mark success and failure logs in a region with CloudWatch Group named: sns/[aws-region]/[account-id]/DirectPublishToPhoneNumber and
sns/[aws-region]/[account-id]/DirectPublishToPhoneNumber/Failure. Alternatively, searching for “DirectPublishToPhoneNumber” under CloudWatch LogGroups shall yield both LogGroups if they had been previously created. If you don’t see DirectPublishToPhoneNumber/Failure LogGroup that will be since there are no failure logs created yet. We can programmatically iterate through the logs to get this information against the total parts and the price of each message over a month’s time.

Under CloudWatch Insights console, select the above two CloudWatch LogGroups, and the timeframe from the start of the current month in UTC. Hit run using this sample CloudWatch Insight query:

fields @timestamp, @message, `delivery.priceInUSD` as PriceInUSD, `delivery.numberOfMessageParts` as MessageParts
| sort @timestamp desc
| stats sum(delivery.numberOfMessageParts) as MessagesSentPerDay, sum(delivery.priceInUSD) as PricePerDayUSD by bin(1d) as Date

This will iterate through the CloudWatch log streams over the given time to aggregate the value of numberOfMessageParts and priceInUSD, and would look something like this:

Caveat: The default message retention period for these logs are 30 days, and hence this will not work for queries prior to 30 days unless you have explicitly changed this. If you haven’t use the UsageReports option below:

2. More robust way – using SMS Usage Report

PreReq: SMS Usage Report is enabled in a region.

When enabled, UsageReports will generate .csv.gz files for individual year/month/days providing additional insights for the SMS usage. We can parse these files created by SNS under S3 bucket to extract the “TotalParts” – giving us more accurate representation
of the number of SMS’ sent in any calendar month.

This script will fetch all the folders in a S3 bucket configured with usage report, extract the contents and spit out the total number of messages sent using TotalParts from the objects residing inside the buckets. This approach is more robust than the above, since this relies on the total part rather which SNS took to send the message than the individual CloudWatch Log Streams – which would be a single entry for a longer message.

#!/bin/bash
set -e

usage="
Usage: $(basename "$0") [-b BUCKET_NAME] [-r REGION] [-y YEAR] [-m MONTH]

Calls S3 to retrieve and parse the usage report to calculate number of messages sent in a calendar month
where:
    -b  Takes the input SMS usage report S3 bucket name
    -r  (Optional) Overrides region of S3 bucket. Defaults to current region as per aws configure
    -y  (Optional) Year for SMS report in YYYY format. Defaults to current year
    -m  (Optional) Month for SMS report in MM format. Defaults to current month"


# Set current month and year
YEAR=$(date +'%Y')
MONTH=$(date +'%m')
REGION=$(aws ec2 describe-availability-zones --output text --query 'AvailabilityZones[0].[RegionName]')

# Get usage report bucket from command line argument
while getopts b:y:m:r: opts
do
   case ${opts} in
      b)
        USAGE_REPORT_BUCKET_NAME=${OPTARG}
        ;;
      \?)
        echo "Please set the UsageReport bucketname in this script using -b flag. Example: ./sms-usage-reports.sh -b my_bucket"
        exit 1
        ;;
      y)
        YEAR=$OPTARG
        ;;
      \?)
        echo "usage: ./sms-usage-reports.sh -b my_bucket -y YYYY"
        ;;
      m)
        MONTH=$OPTARG
        ;;
      \?)
        echo "usage: ./sms-usage-reports.sh -b my_bucket -y MM"
        ;;
      r)
        REGION=$OPTARG
        ;;
      \?)
        echo "usage: ./sms-usage-reports.sh -b my_bucket -r us-east-1"
        ;;
   esac
done

# mandatory arguments
if [ ! "$USAGE_REPORT_BUCKET_NAME" ]; then
  echo "Error: -b argument must be provided to state the SMS usage report bucket name"
  echo "$usage" >&2; exit 1
fi

echo "Searching for SMS usage report for $MONTH/$YEAR in $REGION region."


echo "Creating new directory under /tmp/SMSUsageReports"
mkdir -p /tmp/SMSUsageReports
cd /tmp/SMSUsageReports

$(aws s3 sync --only-show-errors s3://$USAGE_REPORT_BUCKET_NAME/SMSUsageReports/$REGION/$YEAR/$MONTH .) && $(find . -type f 2> /dev/null | xargs gunzip 2> /dev/null)
TOTALSMS=$(find . -type f | cat */*  | awk -F ',' '{count[$8]++} END {for (word in count) print word, count[word]}' | grep -v  TotalParts | awk -F " " '{print $2}')
echo "Total SMS sent: ${TOTALSMS:-0}"


rm -r /tmp/SMSUsageReports && echo "Deleted: /tmp/SMSUsageReports"

Usage:

chmod +x sms-usage-report.sh
./sms-usage-report.sh -b 
Searching for SMS usage report for 03/2021 in us-west-2 region.
Creating new directory under /tmp/SMSUsageReports
Total SMS sent: 10
Deleted: /tmp/SMSUsageReports

The script supports the following flags to override the default values:

Usage: sms-usage-report.sh [-b BUCKET_NAME] [-r REGION] [-y YEAR] [-m MONTH]

Calls S3 to retrieve and parse the usage report to calculate number of messages sent in a calendar month
where:
    -b  Takes the input SMS usage report S3 bucket name
    -r  (Optional) Overrides region of S3 bucket. Defaults to current region as per aws configure
    -y  (Optional) Year for SMS report in YYYY format. Defaults to current year
    -m  (Optional) Month for SMS report in MM format. Defaults to current month

Hope this helps!