Introduction

Toast has always invested in the cloud from its early stages as a startup to its current status as a publicly traded company. Figuring out how to manage cloud infrastructure costs is a typical problem faced by many companies as they scale. Over time, companies typically see a rising cloud bill as their customer base increases while at the same time they lack visibility into their cloud investments. 

It is important for companies to determine the root cause for rising cloud costs. While business growth is a typical driver, some cost increases could be due to improper sizing of production resources and/or inefficiencies in lower environments.  It is also often difficult to clarify the ownership for these rising cloud costs and there are also usage and rate inefficiencies hidden amongst the many line items on invoices. 

For Toast, overcoming these issues has been a journey, for in the early startup days we initially focused on releasing new features and stabilizing the platform. 

As Toast’s customer base and expenses grew, the focus began to shift toward building a team that could proactively track and monitor our cloud costs. This eventually led to adopting a FinOps mindset which, as described by the FinOps Foundation, is an “operational framework and cultural practice which maximizes the business value of cloud, enables timely data-driven decision making, and creates financial accountability through collaboration between engineering, finance, and business teams.” At Toast we have embraced these principals and applied them in our own way.  

FinOps Journey

In the early days of Toast’s cloud adoption, our FinOps strategy was in its infancy. We typically tracked monthly costs and compared that to our growing customer counts to understand our monthly cost per customer. We would use that growth rate as a rough model to predict future cloud costs. A core group in the technical infrastructure team also managed our cloud spend by making up-front discounted product purchases and commitments to help reduce monthly costs. 

As we grew our customer base and our cloud footprint, we began to shift to a more proactive approach for managing cloud costs. We did this by first looking at areas where we could eliminate waste, such as over provisioned resources, idle databases, and other unused resources. Another key focus area included ensuring that assets were tagged properly so we could understand which services were the largest cost drivers;  a concerted, standardized tagging effort also helped get us visibility into cloud spend by different departments. 

 At Toast, part of our core values include operating as one team and embracing an ownership mindset. We applied these values to managing our cloud costs by expanding our cross-Toast collaboration via a partnership between members of the finance team, the technical infrastructure team, key lead software engineers and program management to build a FinOps culture. Goals include providing data visibility and transparency to senior engineers and leaders in each line of business, matching cloud spend to cost centers, determining how one cost center consumes another cost center’s services inside of Toast, and providing education about cloud costs and finance terms. In order to jump start our move in this direction, Toast switched over to using CloudZero, a leading cloud cost optimization platform, to help make cloud cost data available to leaders and engineers in each line of business.  With this data, development teams can see the direct cost impacts of the engineering decisions that they make and help ensure that we spend our resources on what matters the most.  

Having a solid cloud cost optimization platform in place is important, but it is not enough to fully develop a FinOps culture.  Many companies, including Toast start out with a top-down approach to managing cloud costs.  To really evolve our FinOps strategy, we saw an opportunity to leverage a bottom-up approach as well.  We needed our lead engineers on board with helping to keep our Cloud costs growing at a sustainable pace, even as we increased our number of customers and expanded our product offerings.

For the bottom-up approach the FinOps core team partnered with our enablement team to build out training both in FinOps and in the use of CloudZero. The training helped engineers understand the difference between Costs of Goods Sold (COGS) and Operating Expenses (OPEX) and why this information is important to the finance team. Engineering leads can now directly see the costs of the services that they build and can discover optimization opportunities that a centralized team would not typically see, since individual teams know their own services the best.  Having visibility to cost information also helps make sure teams don’t spend valuable engineering time on smaller optimizations that have very little pay back. 

We also have an ongoing effort to ensure resources are tagged correctly and that our cloud spend is correctly attributed to the correct cost center. The attribution is not limited to how much one department directly uses cloud services, but also how one department consumes another department's services inside of Toast.  For example, if one department builds a reporting platform, and another department uses that platform to run reports, the costs for using the platform should flow to the proper department.  With this information available, each line of business has the tools to manage their own budget and it also helps to decentralize our approach for managing cloud costs as we scale. 

From the top-down perspective, we recognized the need to drive more accountability at the line of business level.  We included cloud costs as part of our top-level engineering goals and regularly tracked our progress against forecasts during metrics reviews. The core FinOps team kicked off monthly cloud cost meetings with key stakeholders in each line of business to review monthly spend trends. The goal of the meetings is to understand potential cost drivers, compare actual costs to budgets and help adjust forecasts as needed. 

Our bottom-up and top-down approach to managing cloud costs also requires the proper tooling.  Cloud providers typically have their own native FinOps tools but these typically don’t have the wide range of features that are needed to manage a complex cloud architecture. Tools such as CloudZero support integrations with other SaaS vendors, so that there can be a single pane of glass to view the cost data for different cloud providers as well as monitoring, logging and messaging tools. We leveraged CloudZero’s AnyCost adapter to push these various sources of data into CloudZero.   

We also needed a FinOps tool with fine grained role based access, so that engineers are only looking at the data in the areas that they are responsible for.  Many FinOps tools provide role based access at the cloud provider account level which didn’t work for us as we needed to handle some shared accounts. Using CloudZero we now have a tool that provides role-based access based on the creation and management of user groups.   By combining tags, account info, and other resource metadata, CloudZero CostFormation® (their allocation engine), can bucket cloud costs by any organizational framework, including COGS or OPEX which are critical GAAP (Generally Accepted Accounting Principles) classifications for our finance team.

We also leverage Cloud Zero’s anomaly detection feature, wherein automated alerts are sent to the relevant engineers when costs spike in their departments. This helps minimize the business impact of unexpected cloud cost events. In addition, CloudZero supports analytics dashboards that the finance team uses to view line item costs such as usage, fees and discounts that helps them more proactively optimize savings opportunities. The analytics dashboard also provides a way to show monthly costs by department by service, and can be used to show trends over time.  

Conclusion

Building a FinOps culture is a gradual process, but it is important to move in this direction for all companies, not just at Toast. A few key takeaways from our journey include: 

• Tagging assets to unlock the ability to track cloud costs back to products, services or departments

• Leverage the proper tools to support a solid FinOps foundation

• Use a bottoms-up approach so teams have access to cost data and can make the trade-offs between spend, costs and quality as they build out their cloud architecture.

We started this process focusing on one cloud vendor and we will keep building our FinOps muscle as we scale. Our processes will continue to evolve based on what we’ve learned so far, and these continuous improvements will help us provide better insights and further refine our FinOps models. 

____________________________

^{This content is for informational purposes only and not as a binding commitment. Please do not rely on this information in making any purchasing or investment decisions. The development, release and timing of any products, features or functionality remain at the sole discretion of Toast, and are subject to change. Toast assumes no obligation to update any forward-looking statements contained in this document as a result of new information, future events or otherwise. Because roadmap items can change at any time, make your purchasing decisions based on currently available goods, services, and technology.  Toast does not warrant the accuracy or completeness of any information, text, graphics, links, or other items contained within this content.  Toast does not guarantee you will achieve any specific results if you follow any advice herein. It may be advisable for you to consult with a professional such as a lawyer, accountant, or business advisor for advice specific to your situation.}