Cloud Intelligence Dashboards: How to Manage AWS Costs at Enterprise Scale

Bishr Tabbaa

·

Follow

Published in

Towards AWS

·

6 min read

·

Jul 31, 2024

147

1

I have been working with AWS Independent Software Vendor (ISV) customers for several years, and a common theme is how to effectively manage costs when multiple workloads span multiple regions and multiple accounts. This blog post covers Cloud Intelligence Dashboards which are an essential tool that can help you visualize, analyze, and manage costs and other activities at enterprise scale. This blog post also continues an earlier blog I wrote on how to save $1 Million per year which covered principles, practices, and tools for AWS cost management.

Cloud Intelligence Dashboard Overview

The Cloud Intelligence Dashboard (CID) is an open-source, AWS-led, community-driven initiative to provide visibility and analytics into AWS cloud activities. The CID dashboards consist of three primary components. First, there are the data sets sourced from AWS Cost and Usage Reports (CUR), AWS Config, AWS Trusted Advisor, S3 Storage Lens, Cost Anomaly Detector, and other services exported to Amazon S3. Second, there is a data lake and data pipeline built with S3 and AWS Glue crawlers and catalog schemas. Third, there are data analytics visuals built with Amazon Athena queries and AWS QuickSight dashboards, secured with AWS IAM or an external SSO SAML provider. The CID reference architecture can be visualized below, and it is a canonical example of how an AWS data lake can be created for other use cases as well.

Cloud Intelligence Dashboard Modules

The CID dashboards are pluggable modules that can be amended and extended with new features. The CID community has built several CID modules.

CUDOS

Cost Usage and Dashboard Operations Solutions (CUDOS) is a key, foundational module and dashboard that focuses on CTO and engineering team personas. It provides you high-level details as well as operational insights with the ability to drill down to resource-level granularity. You can find auto-generated cost optimization recommendations and actionable insights that can be used by your FinOps practitioners, Product Owners, and Engineering teams within the CTO organization. It allows you to quickly identify spikes and uncover uncertainties in your AWS usage by highlighting particular resources that can be optimized. Out-of-the-box benefits of the CUDOS dashboard include (but are not limited to):

• Use the built-in tag explorer to group and filter cost and usage by your tags.
• View resource-level details such as Lambda function or individual S3 bucket costs.
• Get alerted to service-level areas of focus such as top 3 On-Demand database instances by cost.

When our AWS teams work with customers, we regularly rely upon CUDOS visualizations and insights when making data-driven recommendations.

CID

CID is another foundational module that focuses on CFO team personas. It is a customizable and accessible dashboard to help create the foundation of your cost management and optimization (FinOps) initiatives. Executives, directors, and other individuals within the CFO’s line of business or who manage cloud financials for an organization will find the Cloud Intelligence Dashboard easy to use and relevant to their use cases. Little to no technical knowledge or understanding of AWS Services is required. Out-of-the-box benefits of the CID include (but are not limited to):

• Create chargeback or showback reports for internal business units, accounts, or cost centers.
• Track how Savings Plans (SP), Reserved Instances (RI), and Spot Instance usage has impacted your unit metrics such as your average hourly cost of Amazon EC2.
• Keep track of which accounts or internal business units receive savings and when RIs and SPs expire.

Modernization

Modernization is another foundational module that helps your organization combine DevOps and IT infrastructure with Finance and the C-Suite to grow more efficiently and effectively on AWS. This dashboard lets you set and track modernization and optimization goals such as percent OnDemand, Spot adoption, and Graviton usage. By enabling every line of business to create and track usage goals, and your cloud center of excellence to make recommendations organization-wide, you can grow more efficiently and innovate more quickly on AWS. Out-of-the-box benefits of the KPI dashboard include (but are not limited to):

• Track percent on-demand across all your teams.
• See potential cost savings by meeting certain KPIs and goals for your organization.
• Quickly locate cost-optimization opportunities such as infrequently used S3 buckets, old EBS snapshots, and Graviton eligible instance usage.

Note that there are other dashboard extension modules related to the Compute Optimizer, Trusted Advisor, Cost Anomaly, Health Events, and Extended Support Cost Estimator. All of these optional modules can add value depending upon where and when you walk upon your AWS cloud journey. For my ISV customers with long-lived EKS and RDS workloads enduring for years, I have found that the Extended Support dashboard can improve discoverability and focus attention around those specific resources that need to be migrated to the latest compatible version of Kubernetes, MySQL, and PostgreSQL. For organizations especially interested in resilience, performance, and security, I also recommend investigating the Trusted Advisor and Health Events modules.

Cloud Intelligence Dashboard Installation

The deployment and installation process consists of 4 main steps. Consult the workshop for the latest detailed instructions and CloudFormation artifact references.

1. Enable AWS Organizations at the Management (Payer) Account. Then decide whether there will be a separate Data Collection Account or whether the CID components related to S3, Glue, Athena, and Quicksight will all reside in the local primary Management (Payer) account. One factor in this decision is whether you have one or multiple Management (Payer) Accounts. Many customers have only one, however for those customers (like mine) that have grown through M&A and have multiple distinct subsidiaries, it can make sense to have a separate Data Collection Account to aggregate data across the payer roots.
2. Deploy a S3 bucket for aggregated CUR in the destination Data Collection Account. Log into the destination Account. Choose a Region. Launch a CloudFormation stack. Enter the source account IDs. Review the configuration, and then create the stack. This step takes 10–15 minutes.
3. In each source Management (Payer) Account, launch a CloudFormation stack that creates the CUR and replication rules. You will specify a stack name as well as the destination Data Collection Account ID used in step 2. Review the configuration, and then create the stack. This step takes about 5 minutes. It will take about 24 hours for your CUR to populate and then replicate to the destination account where you will deploy the dashboards. We also recommend creating a Support Case with these attributes (e.g. Service=Billing, Category=Other Billing Questions, Severity=Important Question, Subject=Backfill CUR for CID, Message=Hello, please backfill the CUR (name=cid) with 12–36 months of data). requesting a backfill of your CUR with 12–36 months of data. The case must be created from the Source (Management/Payer) account. You can continue, however the dashboards will be empty until the CUR data is available. Or you can return to the next step after 24 hours, and you will have data for the current month.
4. The final step is to deploy the dashboards in the Destination (Data Collection) Account. Log into the Destination Account, specify your QuickSight owner user, as well as the S3 path to the CUR data

Conclusion

Cloud Intelligence Dashboards (CID) provide AWS customers with a series of in-depth and customizable dashboards for comprehensive cost and usage details to monitor, analyze, and optimize costs as well as operations related to performance, security, and resilience. If you are an AWS customer operating multiple workloads in multiple regions and multiple regions, I highly recommend installing CID in collaboration with your AWS account team.

Enjoy the article? Have more questions? Reach out in the comments. I am grateful for your feedback. Follow me on Medium, LinkedIn, and Twitter for more updates.

References

• https://aws.amazon.com/blogs/aws-cloud-financial-management/a-detailed-overview-of-the-cost-intelligence-dashboard/
• https://aws.amazon.com/blogs/mt/visualize-and-gain-insights-into-your-aws-cost-and-usage-with-cloud-intelligence-dashboards-using-amazon-quicksight/
• https://catalog.workshops.aws/awscid/en-US
• https://github.com/aws-samples/aws-cudos-framework-deployment
• How to Save $1 Million a Year on your AWS Cloud Bill
• How to Become an AWS Solutions Architect