Culinary Event Coordination

How do we build data pipelines that don't break under pressure? A pipeline that not only scales but is also resilient to failure? I've been designing a solution using an Event-Driven Architecture (EDA) on AWS, and it directly tackles these challenges. This architecture's goal is to move data from an external CRM, through a processing-and-optimization phase, and into a Redshift data warehouse, all while being fully automated and fault-tolerant. Here is the step-by-step flow: 1. Ingestion & Event Trigger: The pipeline kicks off when a raw .csv file lands in an S3 bucket. This action immediately triggers an s3:ObjectCreated event, which is sent to a central EventBridge bus. 2. The Decoupling "Firewall": This is where the magic happens. A rule on EventBridge routes the new file event to an SQS Queue. This queue acts as a crucial buffer. It doesn't matter if we get 10 files or 10,000, the queue holds them, preventing the system from being overwhelmed. 3. Intelligent Transformation: A "Transform Lambda" polls this queue for jobs. When it finds one, it retrieves the raw CSV, transforms it into the highly-optimized Parquet format, and saves it to a separate 'processed' S3 bucket. 4. The Event Chain: The new Parquet file's creation triggers its own custom event ("ParquetFile.Created") back to the EventBridge bus. A second rule sees this event and invokes the "Load Lambda." 5. Final Load & Notification: This Load Lambda executes a COPY command, loading the fast, columnar Parquet data into Redshift. Upon success, it publishes a message to SNS, and the BI Team gets an immediate email: "The data is fresh and ready for analysis." The Business & Technical Wins This isn't just an engineering exercise; this design delivers key benefits: Superior Resilience: The SQS queue ensures no data is ever lost. If a downstream process fails, the message is safely retried without bringing the entire pipeline to a halt. Component Decoupling: Each service (ingest, transform, load) is independent. We can update, scale, or fix one part without breaking any other, a must for agile development. Performance & Cost: We use serverless components (Lambda, S3, SQS), so we pay only for what we use. Plus, converting to Parquet makes Redshift queries significantly faster and more cost-effective. Total Automation & Observability: The pipeline is "hands-off" from start to finish. The final SNS alert provides a clear feedback loop to stakeholders, building trust in the data.

There are two types of event manager. The difference is in how they plan. The first knows things will go wrong and builds their schedule to absorb it. They have pessimistic timings and float built into every dependency. They keep a gold copy nobody touches without authorisation. They have a weather trigger pre-written with the team. They've already decided who stops the show. They've already had the conversation about what happens if the power drops. The second is excellent at the job. Passionate, hard working, capable. But their plan is a best case document. And best case documents don't survive contact with bump in day. The difference between them is not talent. It's not even experience. It's whether they were ever taught to build a schedule as a risk management tool. Not as a timeline document. This isn't a personality type. It's a craft. The planners who absorb shocks weren't born more organised. They were taught a method. Here's how to know which one you are right now. #1. Open your current event schedule. Is there float built into the critical path, or does every task sit end to end? Resilient standard: use the PERT framework to set your critical path timings. #2. Look at your weather plan. Is it a written protocol with trigger points and decision rights, or a line in the risk register that says "monitor weather"? Resilient standard: named thresholds (wind speed, rainfall mm, lightning distance), named decision-makers, named comms tree. #3. Check your handover plan. If you're taken off site at hour ten, does your deputy know what decisions are theirs to make? Resilient standard: a RACI and decision matrix that pre-delegates decision authority to make the calls. If any of those made you uncomfortable, you're not alone. Most people in events are never taught this. They learn it the hard way, mid event, when something breaks and there's no protocol to fall back on. That's why I built the BTS Schedule Health Check. Two minutes, free, scored out of 100. It tells you which of those gaps is most likely to bite you. Which one of those three checks made you uncomfortable? 📬 Score your next schedule in two minutes (free): https://lnkd.in/gWfVxHNJ 🔔 Follow Iain Morrison for event scheduling and leadership content that works on site.

"We'll use events and CDC for loose coupling." This statement sounds good in a design doc. In reality, it's a top reason for "silent" production failures. An upstream system (Service A) produces data. A downstream system (Service B) needs that data. To avoid a "tight coupling," the engineer has Service B "listen" for changes from Service A. Maybe Service B uses Change Data Capture (CDC) to stream changes from Service A's database. Or it just consumes from a generic event log. Service A doesn't even know Service B exists. This feels like a win for loose coupling. It's actually a time bomb. And then reorg happens, team gets changed completely. The failure happens 3-6 months later. The team for Service A changes their data contract. They rename a field. They refactor the code and stop producing a specific event. Why? Because they forgot Service B was silently listening. The dependency was implicit. It wasn't obvious in their code. Service A's tests pass. They ship their change. Weeks later, Service B breaks. The data is corrupt. The system is down in production. The damage is done, and it takes days to trace and fix the problem, which happened due to a change made a month ago. That's why, stop relying on silent event streams for critical data flows. Use an explicit command instead. This doesn't mean it must be a synchronous API call. It can (and often should) still be an event. But Service A must explicitly publish a well-defined event. The code in Service A should literally say: event_publisher.send("OrderProcessed_v1", data) Now, the dependency is explicit. When the Service A team refactors their code, they see this line. They can't forget it. They are forced to think: "Who consumes OrderProcessed_v1? Oh, Service B. We are moving to v2, so we need to tell them." This conversation happens during development. Not during a production fire. Don't confuse "loose coupling" with "implicit dependencies." One is a good design goal. The other is a production incident waiting to happen. If you are preparing for mid-senior/staff SDE, EM system design HLD interviews, and need help, DM me COACH.

Stop coordinating microservices with central controllers. ⚠️ That is how systems become tightly coupled and fragile. As systems grow, workflows start to span multiple services and bounded contexts. Orders trigger payments. Payments trigger inventory. Inventory triggers notifications. The instinct is to control it all from one place. That is where things break into long dependency chains, cascading failures, and rigid flows that cannot evolve. ✅ The right approach is event-driven choreography. Instead of telling services what to do, let them react to what has happened. A service publishes a domain event. Other services subscribe and decide independently how to respond. No direct calls, no central coordinator, no hidden dependencies, just events flowing through the system. In .NET systems, this is commonly built using libraries like MassTransit with brokers such as Kafka, RabbitMQ, or Azure Service Bus. Consumers implement interfaces like IConsumer and react to events such as OrderPlacedEvent. The system evolves through reactions, not instructions. The benefits are immediate: ⇢ Decoupling: Services do not know about each other, only about events ⇢ Resilience: Failures are isolated and retries happen asynchronously ⇢ Extensibility: New services can subscribe without changing existing ones ⇢ Scalability: Workloads distribute naturally across consumers ⚠️ There are tradeoffs to understand. Debugging becomes harder. There is no single flow to follow. Eventual consistency is required. You cannot assume immediate state alignment. Poorly designed events create hidden coupling and break autonomy. This is where discipline matters. Define events as stable contracts. Keep them immutable. Make consumers idempotent so retries are safe. Invest in observability. Tracing and monitoring are not optional in event-driven systems. Also understand this: Choreography is not always the answer. Orchestration still has its place when workflows require strict control, visibility, or complex coordination. Most real systems use both. Event-driven choreography is not removing control, it distributes it so your system stops behaving like a pipeline when done right. It starts behaving like an ecosystem where services listen, react, and adapt, which is how modern systems scale. P.S. Event-driven systems are not just messaging, they define boundaries, contracts, and autonomy in your architecture from day one. --- ♻️ Share with your network if this helped ➕ Follow me [ Elliot One ] 🔔 Enable notifications to stay updated

Ever tried keeping Salesforce data in sync with an external system, only to run into polling delays, missed deletes, or performance bottlenecks? I’ve found Change Data Capture (CDC) to be a game-changer for event-driven integrations. With CDC, every record create, update, delete, or undelete fires a “change event” into Salesforce’s event bus. External systems subscribe once and get only the changes they need—no more round-the-clock polling. Some favorite use cases: Sales Cloud → ERP sync: Account and Opportunity changes flow in real time to your finance system. Service Cloud → Ticketing: Case updates automatically create or update tickets in Jira or ServiceNow. On-platform automation: Complex recalculations or external callouts happen asynchronously via CDC triggers, not inside the user’s save. Pro tip: Leverage the ChangeEventHeader—it tells you exactly which fields changed, when, and even who triggered the change. Use changeOrigin to avoid feedback loops when syncing bi-directionally. How are you using CDC in your org? Share your experiences or questions below!

Design for Scale - 2: The Power of Events in Distributed System Events are fundamental building blocks in modern distributed systems, yet their importance is often underappreciated. To understand their power, we must first distinguish events from commands and queries. Events represent immutable facts - things that have already occurred. In contrast, commands express intentions that may or may not succeed. While this distinction can be subtle, it's crucial for system design. Interestingly, we can also treat commands and queries themselves as event streams in different contexts, representing the historical record of customer interactions with our system. This event-centric thinking unlocks elegant solutions to traditionally complex problems. The most common type of event is Change Data Capture. I worked on a quota enforcement tracking resource usage system for millions of customers. The initial approach using scheduled batch queries placed enormous stress on the database. However, by recognizing that data volume was high but change velocity was relatively low, we pivoted to an event-driven approach: establish baseline counts and track mutations through events. This transformation converted a challenging scaling problem into simple in-memory counting. The durability of events provided built-in reliability - if processing failed, we could replay the event stream. We further optimized by buffering rapid add/delete operations in memory, allowing them to cancel out before writing to the quota system, dramatically reducing write pressure. Events can elegantly address the notorious distributed transaction problem through the Saga pattern. Instead of struggling with complex transaction coordination across heterogeneous datastores, we can listen to committed events from the primary system and reliably propagate changes. This approach transforms a difficult distributed transaction problem into a more manageable event-based synchronization challenge. This pattern isn't new - many database systems internally use similar approaches like write-ahead logs or commit logs for replication and synchronization. Events also provide a powerful foundation for system validation and auditing. Independent systems can cross-check correctness and completeness by consuming the same event streams. This pattern has proven successful even in language models for improving result accuracy. But events encompass more than just data changes. Metrics, application logs, audit trails, and user interactions all represent valuable event streams. This broader perspective enables creative solutions to seemingly intractable problems. Treating events as first-class citizens in distributed system design leads to more scalable, reliable, and maintainable architectures. Whether handling data mutations, system operations, or user interactions, event-driven approaches often simplify complex problems while providing built-in reliability and auditability. Befriend with your events!

Publishing a message to a message broker and hoping it lands in the right queue without understanding your exchange routing logic is a recipe for silent data loss or message duplication in production. In distributed microservices, RabbitMQ exchanges act as the primary traffic cops. Producers do not publish messages directly into queues; they publish to an exchange, which evaluates the message attributes against binding rules to determine where that data belongs. If you analyze the six topologies from the diagram, choosing the wrong exchange type will completely break your event-driven flow: 𝗗𝗶𝗿𝗲𝗰𝘁 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲: Exact string matching. The routing key on the message must explicitly match the binding key of the queue. This is your go-to for predictable, point-to-point unicast routing where a specific service instance handles a specific task type. 𝗧𝗼𝗽𝗶𝗰 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲: Pattern-based routing using wildcards. By leveraging dots as delimiters (e.g., orders.eu.processed), you can route messages using * (matches exactly one word) or # (matches zero or more words). This is crucial for publish-subscribe patterns where consumers want to filter sub-streams of data dynamically without requiring a separate exchange for every single topic variant. 𝗙𝗮𝗻𝗼𝘂𝘁 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲: Pure broadcasting. It completely ignores routing keys and duplicates incoming messages across every single queue bound to it. If you need to trigger multiple separate microservices simultaneously whenever a core business event happens, like notifying billing, shipping, and notification engines at once this is the most performant approach. 𝗛𝗲𝗮𝗱𝗲𝗿𝘀 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲: Attributes over keys. It bypasses the routing key entirely, evaluating the message headers array against the binding arguments. Using arguments like x-match: all or x-match: any, it routes based on complex, multi-attribute metadata. While highly flexible for multi-tenant isolation, keep in mind it carries a higher CPU processing overhead than basic key matching. 𝗗𝗲𝗳𝗮𝘂𝗹𝘁 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲: The built-in, nameless direct exchange. Every queue you create is automatically bound to it using the queue’s name as the routing key. It serves as a convenient shorthand for simple architectures, letting you simulate sending a message directly to a specific queue by matching its exact name. 𝗗𝗲𝗮𝗱 𝗟𝗲𝘁𝘁𝗲𝗿 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲 (𝗗𝗟𝗫): The architectural safety net. This is a standard exchange designated to collect messages that have been rejected by consumers, suffered a negative acknowledgment (nack), or dropped due to Time-To-Live (TTL) expiration. Implementing a proper DLX pattern is non-negotiable for system resilience, allowing you to isolate and audit failing messages without clogging up your main processing pipelines. Your event-driven backbone shouldn't be built on guesswork. Align your exchange choices with your system’s fan-out requirements, performance targets, and error-handling tolerances.

If you’re only measuring event ROI by who showed up, you’re leaving pipeline on the table. Here’s what most teams miss: Your pre- and post-event outreach isn’t logistics. It’s a demand gen campaign. So treat it like one — and attribute it like one. OneScreen’s recent event is the perfect example. They pulled three opportunities from the outreach campaign alone, from people who never walked in the door. Here’s how that works: The first email (sometimes the second) is focused on getting them to the event. Then you flip it: “Can’t make it? Let’s still talk and I’ll keep you on the list for what’s next.” That second track hooks the people a registration count would’ve written off. If that were a standard nurture sequence, you’d attribute every opp it generated to the campaign. So why are you not attributing it to the event? Same logic for no-shows. They registered. They’re interested. Your post-event follow-up — “here’s what you missed, let’s grab time” — is a second swing at pipeline. Count it. Here’s the reframe: the event isn’t a single point in time. It’s everything you do around it: the lead-up, the day-of, the follow-through. The room is just the middle. Measure the whole motion. Not just the moment.

Salesforce added a whole new type of Flow and I feel like nobody's talking about it. It's called the Automation Event-Triggered Flow. Dropped in Winter '26. Just sitting there in your Flow Builder right now. So for context, the flow most of us live in is Record-Triggered. A record changes, a flow runs. That's the bread and butter. But real work doesn't always start with a record change. A client uploads a document. Someone abandons a cart. An email bounces. An SMS fails to deliver. These things happen all the time and we've been duct-taping our way through them with helper checkboxes, hidden fields, and "hey can a dev write some Apex for this." Automation Event-Triggered Flows let you skip all that. You pick a business event from a pre-built library, Salesforce tells your flow when it happens, and you build your logic from there. No fake fields. No Apex. No workarounds. It's not the same thing as Platform Event-Triggered Flows either. Those require you to define and publish your own custom events. These come with a ready-made event library out of the box. File uploads, abandoned carts, email engagement, SMS failures, lifecycle changes. Salesforce publishes the events, you just subscribe. The file upload one was the first big use case. Before Winter '26, you couldn't react to a file being attached without Apex. Now there are actually two declarative paths: Automation Event-Triggered Flows (GA since Winter '26) and Record-Triggered Flows on ContentDocument/ContentVersion (beta in Spring '26). Different tools, different strengths. I put together a full breakdown covering what it is, how it compares to every other flow type, every event type currently available, step-by-step setup, the gotchas nobody warns you about, and when you should actually use it vs when you shouldn't. A full detailed guide is attached below. #Salesforce #SalesforceAdmin #SalesforceFlow #SalesforceAutomation #Trailblazer

When building scalable business automation in Salesforce, choosing the right tool isn’t about preference — it’s about architecture. Here’s a structured breakdown of when to use Flows, Apex, and Platform Events together in a real production-grade environment: 1️⃣ Use Record-Triggered Flows for Deterministic Logic Flows should handle operations that are: Declarative Synchronous Predictable Limited to the current transaction Best Use Cases: Field updates (Before Save) Lightweight record creation Validation logic Branching logic using Decisions Orchestrating subflows for modularity Why: Flows run on the platform’s optimized automation engine and avoid unnecessary Apex for simple tasks. 2️⃣ Use Apex for Complex, Optimized, or High-Volume Patterns Apex is needed when you require: Heavy record processing Complex loops Multi-object DML Transaction control (Savepoints, Rollbacks) Custom error handling External integrations Reusable service-layer logic Best Use Cases: Logic requiring Maps/Sets for performance Custom validation across multiple related objects Large data processing (Batch, Queueable, Schedulable Apex) Why: Code gives you precise control over Governor Limits and performance, especially with large volumes. 3️⃣ Use Platform Events for Asynchronous, Decoupled Architecture Platform Events are ideal for: Long-running operations Retry mechanisms Cross-system orchestration Multi-step business processes Event-driven integrations Best Use Cases: Notifying external systems Bulk updates without blocking UI Decoupling Flows that shouldn’t run in the main transaction Triggering logic asynchronously after a transaction commits Why: Platform Events eliminate bottlenecks created by synchronous Flow or Apex logic. 4️⃣ The Ideal Architecture (Modern Salesforce Pattern) The ideal enterprise-grade Salesforce design follows this pattern: Flow → Platform Event → Apex (Async) → Final Flow Breakdown: Use Flow for the initial transaction (lightweight). Publish a Platform Event to handle heavy/slow tasks. Process the event using Queueable Apex for integrations/data work. Trigger a Final Flow to update UI-facing records after async work completes. Benefits: No CPU timeouts No recursive Flow loops Faster user experience Better error recovery Scalable for large orgs Final Insight: “Salesforce performance problems rarely come from limits — they come from choosing the wrong tool for the job.” The best architecture isn’t about code vs no-code — it’s about synchronous vs asynchronous and tight vs decoupled design. #Salesforce #Apex