Cloud Cost Management

How We Saved $10,000/Year by Re-Architecting Our Azure Data Pipeline When you're building data pipelines, it’s easy to default to managed services for simplicity. But sometimes, managing part of your own stack is the smarter (and cheaper) move. Our Scenario We initially built our data pipelines using: Azure Data Factory (ADF) for ETL Azure Data Lake Storage (ADLS) for storing raw and processed data Power BI for reporting Our data sources were SAP, MySQL, and PostgreSQL, and as volumes increased, the costs started stacking up. The Problem High operational costs due to daily ADF pipeline runs Growing need for low-latency queries and faster dashboards Increasing costs for storage + transformation + querying in the Azure ecosystem The Solution: Customizing the Architecture We re-architected the pipeline using reserved Azure VMs to host: Apache Spark (for ETL and transformations) ClickHouse (as our analytical DB for blazing-fast queries) Metabase (for dashboarding and reporting) The Impact Saved over $10,000 per year by reducing pay-per-use costs Gained full control over Spark optimizations Improved query performance significantly Simplified BI stack with Metabase + ClickHouse This transformation showcases how the right architecture, rather than tool substitutions, can drive substantial cost efficiencies and performance enhancements in data engineering. #DataEngineering #CostOptimization #Spark #ClickHouse #Metabase #ETL #Architecture #Azure #BigData Sumit Mittal

It's astonishing that $180 billion of the nearly $600 billion on cloud spend globally is entirely unnecessary. For companies to save millions, they need to focus on these 3 principles — visibility, accountability, and automation. 1) Visibility The very characteristics that make the cloud so convenient also make it difficult to track and control how much teams and individuals spend on cloud resources. Most companies still struggle to keep budgets aligned. The good news is that a new generation of tools can provide transparency. For example: resource tagging to automatically track which teams use cloud resources to measure costs and identify excess capacity accurately. 2) Accountability Companies wouldn't dare deploy a payroll budget without an administrator to optimize spend carefully. Yet, when it comes to cloud costs, there's often no one at the helm. Enter the emerging disciplines of FinOps or cloud operations. These dedicated teams can take responsibility of everything from setting cloud budgets and negotiating favorable controls to putting engineering discipline in place to control costs. 3) Automation Even with a dedicated team monitoring cloud use and need, automation is the only way to keep up with the complex and evolving scenarios. Much of today's cloud cost management remains bespoke and manual, In many cases, a monthly report or round-up of cloud waste is the only maintenance done — and highly paid engineers are expected to manually remove abandoned projects and initiatives to free up space. It’s the equivalent of asking someone to delete extra photos from their iPhone each month to free up extra storage. That’s why AI and automation are critical to identify cloud waste and eliminate it. For example: tools like "intelligent auto-stopping" allow users to stop their cloud instances when not in use, much like motion sensors can turn off a light switch at the end of the workday. As cloud management evolves, companies are discovering ways to save millions, if not hundreds of millions — and these 3 principles are key to getting cloud costs under control.

I reduced our Annual AWS bill from ₹15 Lakhs to ₹4 Lakhs — in just 6 months. Back in October 2024, I joined the company with zero prior industry experience in DevOps or Cloud. The previous engineer had 7+ years under their belt. Just two weeks in, I became solely responsible for our entire AWS infrastructure. Fast forward to May 2025, and here’s what changed: ✅ ECS costs down from $617 to $217/month — 🔻64.8% ✅ RDS costs down from $240 to $43/month — 🔻82.1% ✅ EC2 costs down from $182 to $78/month — 🔻57.1% ✅ VPC costs down from $121 to $24/month — 🔻80.2% 💰 Total annual savings: ₹10+ Lakhs If you’re working in a startup (or honestly, any company) that’s using AWS without tight cost controls, there’s a high chance you’re leaving thousands of dollars on the table. I broke everything down in this article — how I ran load tests, migrated databases, re-architected the VPC, cleaned up zombie infrastructure, and built a culture of cost-awareness. 🔗 Read the full article here: https://lnkd.in/g99gnPG6 Feel free to reach out if you want to chat about AWS, DevOps, or cost optimization strategies! #AWS #DevOps #CloudComputing #CostOptimization #Startups

The best performance improvements also save money. Here, you can learn how to use a Search Index to save money and improve your system. We struggled to run a query in the Database (like every big system). We have a big VM hosting a massive database with mostly all the data we need.  One of the products we host is a directory of people you can hire; you can search by: • Name • State  • Skills  • Many other fields 𝗪𝗲 𝗵𝗮𝘃𝗲 𝘀𝗼𝗺𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀 𝘁𝗵𝗲𝗿𝗲: • When you search by name, people misspell words all the time, so we have to use the “LIKE” statement, and with millions of records, the performance was taken a hit. • Skills was a separate table, and you could choose multiple skills to search, which means expensive JOINs. • We paid a lot to vertically-scale the DB to hold the heavy load. The solution was a new index, but not a database index; we used Azure AI Search. Azure AI Search provides advanced search capabilities and is particularly good at handling large datasets and complex queries. 𝗛𝗼𝘄 𝘄𝗲 𝗱𝗶𝗱 𝗶𝘁: 1. We created a JSON representation of the information we want to save to the index. It is simple: you make your classes and save them to the index like a JSON document.  2. Every time the information changes, we send a message to a Queue 3. An Azure function reacts to the message and refreshes the information in the index 4. Our API search endpoint translates the request coming from the UI and queries the index.     𝗪𝗵𝗮𝘁 𝗯𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝘄𝗲 𝗴𝗼𝘁: • We could downscale the DB; this is how we saved hundreds of dollars monthly. • The new service was a fraction of that saving, so we put money back into our pocket. • We avoid scaling our DB horizontally (trust me, this is complex) • Azure AI Search often follows a pay-as-you-go model, meaning you only pay for what you use. • Our search is 10x faster now. • Supports complex queries, fuzzy searches, and auto-completion. • Offers AI-driven capabilities like semantic search Don’t leave performance and cash on the table!

Cloud is no longer just an infrastructure decision. It’s proving to be a financial strategy for many organizations I speak to in our region; but in the era of generative AI, that strategy is being stress-tested.   We’re seeing IT leaders increase their GenAI cost projections by more than 3x in just a few months. At the same time, nearly 24% of cloud spend is estimated to be wasted due to overprovisioning and reactive management.   In the Middle East and Africa, where digital transformation is accelerating at national scale, this matters even more. Public cloud spending in #MEA continues to grow at double-digit rates annually. Governments, banks, telcos and energy companies are investing heavily in AI-driven services to enhance citizen and customer experiences.   This is why FinOps is no longer optional. It’s foundational to organizations' success and growth.   At IBM, we see FinOps as a cultural shift — not a cost-cutting exercise. This is a shift that brings engineering, finance and business teams into one operating model focused on maximizing business value from every cloud dollar.   A practical FinOps journey starts with three fundamentals: 🔎 Inform – Visibility & Accountability You cannot optimize what you cannot see. True cost allocation, forecasting, and TCO transparency create proactive control — not reactive alerts. ⚙️ Optimize – Usage & Rates Rightsizing. Elastic scaling. Commitment-based discounts. Automation that ensures workloads consume exactly what they need — no more, no less — without risking performance. 🔁 Operate – Continuous Improvement This is where AI changes the game. With GenAI embedded into FinOps practices, leaders can ask questions like: “Why is spend trending above forecast?”, “Where are anomalies?”, “What is the unit cost per transaction?” And get real answers — instantly.   Solutions like IBM Cloudability provide granular financial visibility, while IBM Turbonomic applies AI-driven automation to continuously balance performance and cost in real time.   For organizations across MEA pursuing AI at scale, FinOps becomes the control tower. It ensures: • Every dollar ties to measurable business value • Multi-cloud environments are managed consistently • Automation replaces manual firefighting   The real competitive advantage tomorrow will not just be adopting AI, it is orchestrating AI with the power of financial intelligence.   #FinOps #Cloud #AI #MEA #DigitalTransformation #IBM

If your cloud bill feels overwhelming, you’re not alone. With its mix of data visualizations, summaries, and trends, deciphering your bill can be challenging. However, gaining clarity is key to managing cloud costs effectively—especially as expenses rise due to the high storage demands and processing power needed to support AI and GenAI technologies.    I recently shared some tips with Morning Brew's Billy Hurley around some of the common cloud billing challenges (https://deloi.tt/4esEO0z). In fact, taking a closer look at your bill can help pinpoint major cost drivers, such as high transfer fees or over-provisioned resources. Also using tools to monitor and analyze trends in computing, storage, and data transfer can help guide informed decision-making on resource allocation.    For example, developers might inadvertently run expensive prompts in loops or leave GPU-intensive workflows active longer than necessary. Implementing usage quotas and automated alerts can mitigate these issues. Additionally, matching storage tiers to specific workloads—reserving premium tiers for mission-critical tasks while opting for basic tiers for less demanding needs—can lead to substantial savings.    If you’re interested in optimizing your cloud resources or managing cloud costs, please reach out. We can help you make the most of your hybrid cloud investment! 

Most architects are still treating Microsoft Fabric capacity and financial planning as separate workflows. Microsoft just made that a significant career risk. In architecting global data estates, I consistently see organisations make the same mistake, they treat Reserved Instances (RIs) as a technical architecture decision. They are not. They are a high-stakes financial instrument. If you want to protect your EBITDA whilst scaling in 2026, you must understand the Engine vs. Coupon distinction: Fabric Capacity is the Engine - It generates the bill based purely on uptime. It is a technical component that doesn't know RIs exist. Reserved Instances are the Coupon - They are a financial offset, prepaid credit in Azure that pays off that bill at a discount. Purchasing an RI will not change your performance, your scaling, or your architecture. It only reduces the bill. Many developers rush into a three-year commitment to chase the maximum discount. In a high-velocity AI environment, this is often a strategic error. 1-Year Commitment: ~40% discount. 3-Year Commitment: ~41% discount. For a 1% difference, you are trading 1,000+ days of architectural flexibility for a negligible gain. Unless you are 100% certain of your base load, choose flexibility. How to architect your billing independently from your topology: Think in Base Load - Buy by aggregate demand, not by SKU. Preserve Cash Flow - Use the "pay monthly" option to keep the annual discount without the upfront capital hit. Manage the Risk - Remember that RIs are use it or lose it, paused capacity still burns your credit. I have condensed these strategies into the cheatsheet below to help you de-risk your Fabric spend. Does your current data strategy treat billing as an afterthought or as a core architectural pillar? #MicrosoftFabric #DataStrategy #FinOps #CloudEconomics #OnyxMethodology

Cloud computing infrastructure costs represent a significant portion of expenditure for many tech companies, making it crucial to optimize efficiency to enhance the bottom line. This blog, written by the Data Team from HelloFresh, shares their journey toward optimizing their cloud computing services through a data-driven approach. The journey can be broken down into the following steps: -- Problem Identification: The team noticed a significant cost disparity, with one cluster incurring more than five times the expenses compared to the second-largest cost contributor. This discrepancy raised concerns about cost efficiency. -- In-Depth Analysis: The team delved deeper and pinpointed a specific service in Grafana (an operational dashboard) as the primary culprit. This service required frequent refreshes around the clock to support operational needs. Upon closer inspection, it became apparent that most of these queries were relatively small in size. -- Proposed Resolution: Recognizing the need to strike a balance between reducing warehouse size and minimizing the impact on business operations, the team developed a testing package in Python to simulate real-world scenarios to evaluate the business impact of varying warehouse sizes -- Outcome: Ultimately, insights suggested a clear action: downsizing the warehouse from "medium" to "small." This led to a 30% reduction in costs for the outlier warehouse, with minimal disruption to business operations. Quick Takeaway: In today's business landscape, decision-making often involves trade-offs.  By embracing a data-driven approach, organizations can navigate these trade-offs with greater efficiency and efficacy, ultimately fostering improved business outcomes. #analytics #insights #datadriven #decisionmaking #datascience #infrastructure #optimization https://lnkd.in/gubswv8k

FinOps project management – the Taiichi Ohno way Every failed FinOps project follows the same pattern: leadership is uninvolved, decisions are forced top-down, and engineers feel like they’re being policed rather than supported. People love dashboards and automation. But nobody wants to sit down and answer the real question: Who actually owns the cost problem? The fix? Make FinOps everyone’s problem Taiichi Ohno, the father of the Toyota Production System, had a simple rule - go to the source. In FinOps, that means dragging leadership out of their meetings and into the reality of cloud spending. Here’s a few things you can actually manage a FinOps project the right way: 1. Get leadership to walk the floor. Instead of reports, make them sit with engineers burning through compute. Let them see waste in action. 2. Make cost ownership non-negotiable. If finance, IT, and engineering all think someone else owns the problem, the project is already dead. Assign real accountability, not just “stakeholder meetings.” 3. Ask the “why” five times. Ohno’s famous method. Why is cloud spend high? Because we overprovision. Why do we overprovision? Because teams don’t trust auto-scaling. Why? Because it once failed. Why? Because no one tested it. Why? Because we never made it a priority. Now you have the real issue. 4. Forget meetings, do workshops. Meetings are for status updates. Workshops are for solving problems. Get hands-on, build real FinOps strategies with the teams. 5. Kill off half the KPIs. FinOps teams track 50+ metrics, but executives don’t care. Keep only the ones that drive real decisions. Everything else? Noise. 6. Make waste visible. Ohno forced Toyota workers to physically see inefficiencies. In FinOps, that means showing execs how much money is lost on unused services - in real dollars, not percentages. 7. Tie cloud costs to business impact. Nobody cares about savings if they don’t understand how it helps the business. Translate costs into things leadership understands - market expansion, faster products, competitive edge. 8. Don’t let the project become a hobby. Leadership loves FinOps until something else feels more urgent. Keep pushing, keep proving value, keep reminding them why this matters. 9. If FinOps is a "project," it’s already failing. This isn’t a one-time initiative - it’s a cultural shift. If you treat it like a fixed-length project, it’ll fall apart the moment your team moves on. Still think you can’t convince leadership, and its just a waste of time? Fine. But why not at least make an attempt? Nothing in this world has ever changed unless someone tried doing something that looked impossible - right up until they pulled it off. #FinOps #ProjectManagement

In most Azure projects, the default position for logs is to create a Log Analytics Workspace and turn everything on. This has a huge impact on the cost profile of your solution and can stick out like a sore thumb when it comes to FinOps optimization reviews. Lets say your workspace was ingesting 50GB of data per day - thats a high number but not unheard of in Enterprise environments where you may have multiple VMs, multiple AKS or Azure PaaS based workloads with the full network security stack running. With that figure, you're spending around €4,000/month on logs. That's around €50,000 per year on logs that you may not really need. The likely breakdown is: 👉 AzureDiagnostics: ingesting verbose logs from all services. 👉 ContainerLog: using the old ContainerLog table (not ContainerLogV2).  👉 AzureActivity: every write operation, across all environments including dev/test, shipped to the same workspace. There are ways to get this reduced: ✅ Migrate to resource-specific tables (away from the default AzureDiagnostics), typically 40–60% volume reduction for the same diagnostic data. ✅ Switch to ContainerLogV2, 50% reduction in container log volume, same signal ✅ Run a query to show the highest used tables. Then run another query to show if these have been queried in the last 30 days. ✅ Move infrequently queried tables to Basic tier. 20GB per day of Analytics data costs approx. €1500. 20GB per day of Basic data costs approx. €300. Now, there's a slight caveat - if you have created Dashboards for observability based on KQL Queries, then this counts as a query against a table. You'll need to work out if those specific tables are needed for longer than the 8 days retention offered by Basic Tier. What are your top 3 Log Analytics tables by ingestion volume — and when were they last queried? https://lnkd.in/diCXCwWs #Azure #AzureMonitor #LogAnalytics #FinOps #Observability #MVPbuzz