Understanding the Difference: "Wait on Completion" vs. "Parallel Execution" in Microsoft Fabric Pipelines / ADF

---

When designing pipelines in Microsoft Fabric or Azure Data Factory (ADF), understanding the execution flow of activities is critical. One setting that often causes confusion is "Wait on Completion"—especially in how it differs from parallel execution.

In this post, I’ll explore these concepts, highlight their differences with examples and visuals, and explain how to choose the right execution approach for your pipeline.

---

What is "Wait on Completion"?

The "Wait on Completion" setting controls whether a pipeline waits for an activity to finish before triggering the next activity.

• Enabled (Default): The pipeline triggers Activity A and waits for it to complete before moving on to Activity B.

• Disabled: The pipeline triggers Activity A and then immediately moves to Activity B without waiting for Activity A to complete.

Here’s a quick summary of how this works:

With "Wait on Completion" Enabled

• Activity B begins only after Activity A has completed.

• Ideal for tasks where Activity B relies on the result of Activity A.

With "Wait on Completion" Disabled

• Activity B starts as soon as Activity A is triggered, even if Activity A is still running.

• Useful for tasks where Activity B does not depend on the outcome of Activity A.

---

Is It the Same as Parallel Execution?

At first glance, disabling "Wait on Completion" may seem like creating parallel execution. However, it’s not the same. Let’s clarify the difference:

Asynchronous Execution (Wait Disabled)

• Activity B starts as soon as Activity A is triggered.

• There is still a sequential dependency: Activity B won’t start until Activity A is triggered.

• Their execution overlaps, but they are not truly independent.

Parallel Execution

• Activities are configured as independent branches in the pipeline.

• Both activities start at the same time, with no dependency or sequence between them.

• True parallel execution must be explicitly defined in the pipeline design.

---

Visual Comparison

To illustrate this, let’s consider two activities:

• Activity A: Sending an email.

• Activity B: Logging the email status in a database.

With "Wait on Completion" Enabled

Activity B waits until Activity A is complete before starting.

|--- Activity A ---| (email sent)
                   |--- Activity B ---| (log written)

With "Wait on Completion" Disabled

Activity B starts as soon as Activity A is triggered, even while Activity A is still running.

|--- Activity A ---| (email in progress)
    |--- Activity B ---| (log written)

Parallel Execution

Both activities start at the same time and run independently.

|--- Activity A ---| (email sent)
|--- Activity B ---| (log written)

---

Use Cases for Each Approach

---

Why the Difference Matters

Disabling "Wait on Completion" introduces an execution overlap that speeds up sequential pipelines by eliminating unnecessary wait times. However, it still ensures a logical order: Activity A must trigger before Activity B starts. This contrasts with true parallel execution, where activities run independently and simultaneously, without triggering dependencies.

---

How to Decide Which to Use

1. Use "Wait on Completion" (Enabled):

   • When the next activity depends on the completion of the current activity.

   • Example: Loading data from a source and then transforming it.

2. Disable "Wait on Completion" (Asynchronous Execution):

   • When tasks can overlap but still require a sequence of triggers.

   • Example: Sending notifications and logging them simultaneously.

3. Use Parallel Execution:

   • When tasks are entirely independent and can run at the same time.

   • Example: Processing different datasets concurrently.

Tips

In a nutshell when "Wait on Completion" is disabled without explicitly setting up parallel activities, the triggers are still sequential (Activity A starts first, and then Activity B is triggered immediately after). However, the execution of the activities can happen in parallel (they can overlap), since Activity B doesn't wait for Activity A to finish before starting.

So, the activities are triggered sequentially but executed concurrently (parallel execution) as soon as they are triggered.

---

Can You Tell Just By Looking at the Pipeline Layout?

Here’s an important thing to note: Just by glancing at the layout of activities in the pipeline canvas, you can't always immediately tell if the execution is sequential or parallel. The visual layout might make it look like activities are in sequence or parallel, but the actual behaviour depends on configuration settings like "Wait on Completion".

• Sequential Triggering: Activities can appear sequential in a straight line. But if "Wait on Completion" is disabled, they can still run asynchronously (overlap in execution).

• Parallel Execution: Activities in multiple branches might seem like they run in parallel, but for them to truly execute in parallel, they must be set up as independent tasks with no dependencies or explicit parallel configuration.

So, the layout of activities alone doesn’t tell the whole story. It’s important to check how activities are configured in terms of triggers, dependencies, and the "Wait on Completion" setting to fully understand how they will execute.

---

Final Thoughts

Choosing between "Wait on Completion" and parallel execution depends on your pipeline’s requirements. By understanding their differences, you can design pipelines that are both efficient and reliable. Microsoft Fabric gives you the flexibility to optimize execution for various scenarios, so experiment with these settings to find what works best for your projects!

Have questions or your own tips to share? Leave a comment below—I’d love to hear your thoughts!

Thanks for Reading !!! ❤️

---