TL;DR

• Two models on a fake e-bike shop dataset: price (regression) and loan default (classification).

• Linear Regression had the lowest held-out MAE for price - with only 817 usable bikes, the simple model performed best on the primary KPI.

• The default classifier was close to 90% accurate but caught only about 1% of defaults.

• Retraining with class balancing lowered accuracy but caught roughly 70% of defaults.

• The worked example at the end explains why accuracy is not enough when the important rows are rare.

The method, end to end

Every model in this project follows the same six steps. each phase below maps back to it:

Two questions, one pipeline shape. What will this bike sell for? → regression (Phase 4). Will this loan default? → classification (Phase 5). Same steps, run twice.

Phase 1 · What a model actually is

A model is a function learned from historical examples instead of a set of rules written by hand.

You show it past rows where you already know the answer ,this bike sold for £2,000, that one for £500 ;and it finds a pattern. For a new row, it applies that pattern. In code, that's it:

model.fit(examples, answers)     # here's what happened before
model.predict(new_case)          # now guess this one

Supervised = you have the answer column already (paid back or not).

Unsupervised = no answer column, just "find groups" or "find weird rows." This project is supervised only.

A data engineer may not choose the final algorithm, but still needs to understand what goes into it, what comes out, and how it will run in production.

The model families used in this project

These are the model families the notebooks actually compare:

K-Means, DBSCAN and Isolation Forest are useful unsupervised tools, but they are not part of these two Pedalflow pipelines, so they are outside this walkthrough.

Regression vs classification - The rule that saves you

"Regression" means number. "Classification" means yes/no. Except Logistic Regression, which does yes/no despite the name .

Both Linear and Logistic Regression sum weighted inputs. Linear outputs a number. Logistic squashes the sum to 0–1 so it reads as a probability.

The one rule that actually matters

Look at the target column, not the algorithm name. 
Number target → regression. Yes/no target → classification.

predicted_price_gbp could be 1432.57, so it is regression. The loan result is class 0 or 1, so it is classification. The associated probability tells us how strongly the model leans toward class 1.

You pick the question by picking the target column. The algorithm follows.

Phase 2 · Meet Pedalflow

You are here: raw CSVs. Nothing cleaned yet.

Pedalflow is a fake e-bike shop I made up for this walkthrough. Two datasets: used-bike resale (regression) and loan applications/ebike_finance (classification).

Spreadsheet 1: a used bike for sale

One row = one bike. Target: Resale_Price_GBP.

Nearly new, good condition, sold for £2,413. Across the file, prices run from £120 to about £11,000, with a typical bike around £1,174.

Spreadsheet 2: someone asking to finance a bike

One row = one loan application, captured before approval. Target: IsDefault (0 = paid, 1 = defaulted). Every feature must be knowable at decision time — no future payment history.

~10% of loans default. That imbalance drives everything in Phase 5.

All data is synthetic and generated from a fixed seed. The raw rows are reproducible, but exact model metrics can still move with library versions and time-dependent features such as Bike_Age, which the notebook calculates from the current year.Phase 3 · Loading and cleaning the data

Bike file (regression)

Load — nothing new if you read CSVs for a living:

df = spark.read.csv("ebike_resale.csv", header=True, inferSchema=True)

900 bikes in, blanks and all. Four gaps in seven rows - and those blanks are the next decision.

Three cleaning steps:

1. Nulls. Four gaps in seven sample rows. Easy fix: dropna() — 900 rows become 817. But first check: are nulls random, or do cheap bikes skip inspection? Dropping blindly can bias the training set.

900 rows in → 817 rows saved → 83 rows dropped for missing values.

One line of code, nine per cent of the data gone. Saved as pedalflow.ml.ebike_features so every later step reads the same 817 rows.

1. Feature engineering. Replaced Year_Manufactured with Bike_Age. "Made in 2019" is weak; "7 years old" is useful. Domain knowledge beats algorithm choice here.

2. Scaling. Battery capacity is in hundreds, while gears are under 20. This matters for distance-based models such as KNN and can matter for some linear or gradient-based models. Tree models are much less sensitive to scale. Dividing each numeric column by its standard deviation keeps the comparison fair across the model families used here.

Evaluation limitation

The current notebook calculates standard deviations before the train/test split. That lets the held-out data influence preprocessing slightly. A production pipeline should fit scaling parameters on the training rows only, then reuse those exact parameters for test and inference data.

No text columns in this file - no encoding needed. The target column (Resale_Price_GBP) stays out of the feature table.

Scaled values aren't human-readable - that's fine. The model doesn't need pounds and years; it needs consistent numbers.

Result: 817 rows, 11 columns, saved as pedalflow.ml.ebike_features.

Loan file (classification)

df = spark.read.csv("ebike_finance.csv", header=True, inferSchema=True)

Six of the nine columns here are words, and not one of them means anything to a model until it becomes a number. IsDefault is the answer we're trying to learn - 0 means they paid.

Different problems:

1. Nulls: none. Same dropna() line, zero rows lost. Don't drop nulls by reflex — check first.

2. No scaling, no new columns in this notebook.

3. Encoding: seven text columns → numbers. Models do arithmetic, not English.

What do 0, 1, 2 and 3 mean? They are category codes, not weights or scores. For example, the notebook currently maps Married, Single and Divorced to three numbers based on how often each value appears. A code of 2 does not mean “twice as important” as a code of 1.

There is one important catch: scikit-learn receives these columns as ordinary numbers. A Random Forest can therefore split a column with a rule such as Marital_Status_idx <= 1.5. That introduces an artificial order even though marital status has no natural ranking. The numbers are intended as IDs, but the current models can still use their numeric order.

A limitation in the current notebook

The frequency encoding is a convenient demo shortcut, not the design I would use in production. Education should use an explicit order. Unordered columns such as employment type, marital status and plan purpose should normally use one-hot encoding, or a model configured to handle categorical values directly. The mapping should be fitted on training data, saved and reused so test data cannot influence it and new data cannot silently change the codes.

Binary Yes/No columns can safely use a fixed 0/1 mapping. Existing numeric columns such as age, income and credit score stay numeric. The target, IsDefault, is kept out of the feature table.

The loan notebook does not scale its numeric features. That is acceptable for the tree-based models, but it puts KNN at a disadvantage because large-range columns such as income dominate its distance calculation. Treat the KNN result as illustrative unless it is rerun with suitable preprocessing.

Scaling ≠ encoding

Scaling (bikes): shrinks numbers, keeps order. Encoding (loans): assigns arbitrary integers to categories. Different operations, different files.

Words became numbers; the label stayed out. Output: 40,000 rows → pedalflow.ml.finance_features.

The two side by side

Save feature tables to Unity Catalog - not notebook variables. Training and scoring must read identical columns built the same way, or the model drifts silently.

Phase 4 · Train: predict bike price

You are here: ebike_features → pick a winning regressor.

The feature table deliberately excludes the answer. The training notebook reads Bike_ID and Resale_Price_GBP from the source file, then uses a Databricks FeatureLookup to join those labels back to ebike_features by Bike_ID. The loan notebook follows the same pattern with AgreementID and IsDefault.

How do you measure error on rows you haven't seen?

Hold out a test set. Same idea as validating a pipeline against a known-good sample:

817 bikes, all with known sale prices
  |
  +-- 653 bikes  TRAINING   model sees the prices, learns from them
  |
  +-- 164 bikes  TESTING    prices hidden from the model completely

653 train, 164 test. Model never sees test labels during training. Loans split 32k/8k with stratify=y so ~10.6% default rate holds in both halves.

Regression KPIs: how we choose the best price model

In ML these are usually called evaluation metrics. From a delivery point of view, they are the KPIs used to decide whether one model is better than another. The primary KPI should be chosen from the business problem before comparing models; otherwise it is easy to pick whichever number makes a preferred model look good.

Why MAE is primary: the business question is “how many pounds away was the estimate?” MAE answers that directly and gives equal importance to overpricing and underpricing. RMSE is checked as a guardrail because the dataset contains a few high-value collector bikes, where a large miss matters. R² confirms that the model adds value over a simple mean-price baseline.

Selection rule: compare every model on the same 164 held-out bikes, select the lowest MAE, then confirm that RMSE is acceptable and R² is positive. Using that rule, Linear Regression is the selected model for this run.

Why MAE is primary: the business question is “how many pounds away was the estimate?” MAE answers that directly and gives equal importance to overpricing and underpricing. RMSE is checked as a guardrail because the dataset contains a few high-value collector bikes, where a large miss matters. R² confirms that the model adds value over a simple mean-price baseline.

Selection rule: compare every model on the same 164 held-out bikes, select the lowest MAE, then confirm that RMSE is acceptable and R² is positive. Using that rule, Linear Regression is the selected model for this run.

Seven tools, same 817 bikes

Train seven algorithms, log metrics to MLflow - same loop as any batch job you'd audit later:

model_variants = {
    "Linear Regression": LinearRegression(),
    "Decision Tree": DecisionTreeRegressor(),
    "Random Forest": RandomForestRegressor(),
    "XGBoost": xgb.XGBRegressor(),
    "LightGBM": lgb.LGBMRegressor(),
    # ...and two more
}

for name, model in model_variants.items():
    model.fit(X_train, y_train)              # learn from the 653
    preds = model.predict(X_test)             # guess the 164
    mae = mean_absolute_error(y_test, preds)  # how far off, on average
    r2 = r2_score(y_test, preds)              # score against Larry
    mlflow.log_metric("mae", mae)             # store the result
    mlflow.log_metric("r2", r2)

Seven tools, same split. Log everything - otherwise you're comparing screenshots from memory.

Linear Regression had the lowest MAE on this held-out split. The synthetic price formula includes non-linear depreciation and a few collector-bike outliers, so this result does not prove that bike pricing is truly linear. It shows that, with only 817 usable rows and this preprocessing, the simpler model generalised best according to the chosen KPI. LightGBM was close (£174 vs £164), while the single Decision Tree generalised poorly (R² 0.146).

MAE is the average absolute error in pounds. R² compares squared prediction error with a baseline that always predicts the test-set mean: 1 is perfect, 0 matches that baseline, and a negative value is worse.

Meet Larry

Larry always predicts the mean price (£1,174). R² = 0.725 means our model beats Larry by a wide margin. If you can't beat Larry, you don't have a model — you have an expensive average.

Side note: Random Forest ranked Distance_To_Service_Hub_KM third in importance — a column I invented with no real link to price. Don't trust feature-importance charts blindly.

Every run logged automatically. The straight line wins — but look how close LightGBM gets.

Phase 5 · Train: predict loan default

You are here: finance_features → classifier. Same train/test split pattern as Phase 4.

About 90% of the loans are repaid. A baseline that predicts “no default” for every application is therefore correct about 89.4% of the time, even though it catches no defaults:

def guess(applicant):
    return "will pay it back"   # every single time

When the important class is rare, accuracy is not enough on its own.

Classification KPIs: how we choose the useful model

The KPI comes from the cost of making the wrong decision. In this example, approving a loan that later defaults is more costly than sending a good application for additional review. That makes recall for the default class the primary KPI.

Selection rule: first require recall to improve substantially over the 0% baseline. Then check precision and the confusion matrix to confirm that the number of false alarms is operationally affordable. Finally, check ROC-AUC to confirm that the probability ranking contains useful signal.

Important limitation

The demo does not define a maximum review workload or an agreed financial cost for false positives and false negatives. For that reason, the balanced Random Forest is the selected demonstration model, not a production-approved winner. A production team should agree those limits before training and encode them as an acceptance gate.

The notebook trains on 32,000 rows and tests on 8,000 held-out rows. Of those test rows, 846 are defaults. It compares two Random Forest models:

default_model = RandomForestClassifier(
    n_estimators=100,
    max_depth=8,
    random_state=42,
)

balanced_model = RandomForestClassifier(
    n_estimators=100,
    max_depth=8,
    class_weight="balanced",
    random_state=42,
)
default_model.fit(X_train, y_train)
balanced_model.fit(X_train, y_train)

class_weight="balanced" gives more influence to the rare default rows during training. This produces a newly trained model; it is not the same fitted model with a different display threshold.

The exact figures are logged in MLflow, but the direction is clear: the default model catches only a handful of the 846 defaults, while the balanced model catches roughly seven out of ten. Its overall accuracy is lower because it also sends more good applications for review.

How selection works in the current notebooks

The notebooks log every candidate's metrics, then manually set best_name to Linear Regression or the balanced Random Forest. They do not automatically calculate a winner. That is acceptable for a transparent demo; a production workflow should turn the agreed primary KPI and guardrails into a repeatable selection and approval step.

The four boxes

Every prediction lands in one of four groups:

Precision asks: of the applications we flagged, how many really defaulted? Recall asks: of all applications that defaulted, how many did we catch? Improving recall usually creates more false alarms, so the final operating point is a business decision.

Worth remembering

For this use case, a lower-accuracy model can be more useful if it catches substantially more defaults. Report the confusion matrix, precision and recall alongside accuracy.

AUC: does the model rank risk well?

AUC asks whether a randomly chosen default is usually scored as riskier than a randomly chosen non-default. It uses probability scores rather than one fixed approval threshold. In this experiment both Random Forest variants are around 0.80 AUC, so both contain useful ranking signal even though their class predictions differ sharply.

AUC is useful, but it does not choose the business threshold for you. Log AUC together with precision, recall and the confusion matrix.

Phase 6 · Save and score

You are here: models selected → save the fitted model → load it from another notebook → score sample rows.

What the notebooks actually save

The training loop logs metrics to MLflow and attempts to register every non-baseline candidate with fe.log_model(). On this workspace, model artifact upload can fail even when the metrics were recorded successfully. The working handoff therefore saves only the selected model to a Unity Catalog Volume with joblib:

payload = {
    "model": best_model,
    "model_name": best_name,
    "feature_names": list(X_train.columns),
}
joblib.dump(payload, "/Volumes/pedalflow/ml/data/models/model.joblib")

Saving feature_names matters. The inference notebook uses that list to select the same columns in the same order and fails clearly if a required feature is missing.

Registry versions

If registration succeeds, a higher version number only means “saved later.” It does not mean the model is better. Promotion should be based on logged metrics and an explicit approval process.

What inference currently does

Each 03_inference.ipynb notebook loads the saved joblib payload, reads selected rows from the feature table, restores the expected column order and displays predictions. The classification notebook also displays probability_of_default.

payload = joblib.load(MODEL_PATH)
model = payload["model"]
expected = payload["feature_names"]

features = spark.table(FEATURE_TABLE).toPandas()
X = features[expected]
features["prediction"] = model.predict(X)

This proves that the artifact can be handed from training to inference. It is not yet a production batch pipeline: the current notebooks score a small sample and do not write ebike_predictions or finance_predictions tables. Real-time serving is also not implemented here.

What got built

Pulling the map from the top of this post back out, filled in with what actually exists in the workspace right now:

Two takeaways: the selected model is not always the fanciest algorithm, and the primary KPI must match the business question—not whichever number looks best after training.

Worked example: why 89.5% is not enough

Pipeline version — 8,000 rows, 846 corrupted, 7,154 clean:

Step 1: Mark everything "fine" → 7,154/8,000 = 89.4% accurate, 0 catches. That's your floor.

Step 2: The default trained checker remains close to 89.5% accurate but catches only about 1% of corrupted rows. For the purpose of finding corruption, it barely improves on doing nothing.

Step 3: Retrain with class balancing. Accuracy falls to roughly 74%, but recall rises to roughly 70%. “Worse” accuracy, much better coverage of the problem.

Step 4: Same trap as a dashboard showing "99% rows OK" while missing the duplicate-key failures you built the job to find.

Rule

Rare target → ask of the bad rows, how many did we catch? Not "% correct overall."

Takeaways for your team

In standup terms:

• Feature engineering is most of the work. Nulls, encoding, scaling — same as building a gold table. model.fit() is two lines.

• Feature table = contract. Same columns, same order, same logic for train and score. Save to Unity Catalog.

• Log every run. MLflow metrics or you're arguing from notebook output.

• Version ≠ best. If registry upload succeeds, select versions by metrics and approval status—not by the highest number.

• Always report the baseline. Larry for regression, "always approve" for classification.

• Pin reproducibility. random_state=42, stratify=y — same reason you pin package versions.

• Batch vs real-time is infra, not ML. Know which you can support before promising it.

Code, data, and reproducible numbers: Machine-Learning-w-DBX.

Thanks for Reading .Keep in touch 👍

Understanding Blue-Green Deployments

Blue-green deployment is a technique that reduces downtime and risk by running two identical production environments called "Blue" and "Green." At any time, only one environment serves live traffic while the other remains idle. In traditional web applications and Azure Function Apps, this works by maintaining two complete infrastructure stacks with load balancers switching traffic between them.(Azure web app deployment slots are shown here below)

Why This Matters: Creating Parallel Production Environments

The Azure web app/function app Deployment Model Applied to Power BI

Azure App Services and Function Apps achieve zero-downtime deployments through deployment slots - maintaining two identical production environments where one serves live traffic while the other stands ready for updates. When deployment is complete, traffic switches seamlessly between environments, providing instant rollback capability.PowerBI currently missing native deployment slots, but the operational requirement remains the same. create parallel production instances that allow safe updates without impacting live business operations.

Building Production Twins for Power BI

This solution implements the deployment slot concept within PowerBI workspaces by creating identical copies of production assets:

• Active Production: Current reports and semantic models serving business users

• Staging Production: Parallel versions with deployment prefixes, ready for validation and switchover

This approach mirrors Azure's blue-green deployment pattern, adapted for PowerBI's workspace-centric architecture.

The Core Business Value

Organizations achieve the similar operational benefits that Azure deployment slots provide for web applications:

• Continuous Availability: Production reports remain accessible throughout deployment cycles, eliminating service interruptions during updates.

• Risk Mitigation: New versions undergo complete validation in production-identical environments before user exposure, reducing deployment-related incidents.

• Rapid Recovery: Rollback operations execute immediately by redirecting users to previous versions, minimizing business impact when issues arise.

• Deployment Confidence: Teams can execute updates during business hours with the assurance that failures won't disrupt operations.

What We Achieve

This implementation transforms PowerBI deployments from high-risk process into reliable, automated operations. Business users experience consistent service availability while IT teams gain the operational maturity and deployment confidence that modern DevOps practices provide for other enterprise applications.The parallel environment approach brings enterprise-grade deployment practices to PowerBI, ensuring analytics infrastructure operates with the same reliability standards as mission-critical business applications.Below mention high level flow of PowerBI artifacts movement.

Our approach adapts blue-green deployment principles to PowerBI's constraints using prefix-based naming conventions. Instead of separate infrastructure, we create parallel versions of reports and semantic models within the same or different workspaces, distinguished by prefixes.(If this is unclear, continue reading through the entire article)

The strategy works as follows:

• Blue Environment (Current Production) marked as 1 : Reports and semantic models with standard production names (e.g., "Prod Model A", "Prod Report A").

• Green Environment (New Version): Based on production workspace artifacts (1), the solution creates reports and semantic models with environment-specific prefixes (e.g., Dev-Model A, Dev-Report A(2), Test-Model A, Test-Report A(3), Prod_New_Model A, Prod_New_Report A (4)) by copying artifacts and binding them to their respective workspaces.

When deployment completes successfully, the green environment becomes the new production standard, and the blue environment serves as a rollback option.

🏗️ Solution Architecture

The solution combines several technologies to create a robust, automated deployment pipeline:

• Docker Containers: Provide consistent deployment environments and easy scalability across multiple target environments.

• Azure DevOps Pipelines: Orchestrate the build and deployment process with proper credential management.

• Fabric CLI: Handle workspace operations like copying reports and semantic models.

• PowerBI REST APIs: Manage report-to-dataset bindings and resolve object identifiers.(I initially attempted to implement this functionality using Fabric CLI's set (fab set) command, but came no success and reverted to the basics)

• Service Principal Authentication: Enable secure, automated access to PowerBI workspaces.

The above image(click on the image to zoom) explains flow that demonstrates how code changes flow from development through testing to production environments using containerization and automated deployment strategies.

The diagram illustrates a multi-stage deployment process that begins with main branch code changes being processed through Azure DevOps and pushed to Docker Hub as container images. The pipeline then supports both manual and automated deployment paths, where images are pulled and deployed across three distinct environments: Development, Testing, and Production.

The workflow shows how containers are dynamically created using docker run --rm commands for each environment, with built-in authentication, copying, and rebinding processes. Each container follows a complete lifecycle from creation to destruction, with the system maintaining separate workspaces for different environments. The architecture also incorporates PowerBI/Fabric production components and DevOps variable libraries for configuration management, demonstrating a robust continuous integration and deployment (CI/CD) strategy that ensures code quality and environment isolation throughout the software delivery lifecycle.

Two Main Deployment Phases

The architecture consists of two distinct phases that optimize both developer productivity and end-user simplicity:

Phase 1: Image Building (One-Time Setup , already cooked 🍳& I am explaining how I did and what it does) When developers push changes to the main branch of the Prod PowerBI devops repo, Azure DevOps triggers an automated build process that creates a Docker image containing all deployment scripts, Fabric CLI tools, and dependencies. The completed image is stored in Docker Hub, ready for use across all environments.

Phase 2: Deployment Execution (Adhoc Operations) 🎯 This is where end users interact with the system through two simple steps to utilize the image:

• Configure Parameters: Update Azure DevOps variable groups with workspace and report specifications & Service Principal credentials

• Execute Deployment: Trigger the second pipeline, which automatically pulls the pre-built Docker image and runs three parallel containers (Dev, Test, Prod_New) that handle authentication, copying, and rebinding processes

Once the blue path establishes the Docker image through the build pipeline and the green path deploys identical containers bound to their respective workspace environments, developers can seamlessly begin their modification workflow. Starting from the development environment, teams can follow familiar PowerBI deployment pipelines to systematically promote artifacts from one stage to another.

This solution elegantly integrates with existing PowerBI deployment architecture, maintaining consistency with established practices while providing a containerized foundation that ensures environment consistency and deployment reliability

Following image illustrates the workspace artifact progression in Production: image (1) captures the baseline state prior to container deployment, with images (2), (3), and (4) showing the successive states following DevOps pipeline execution for the container releases.

Finally, we can configure the PowerBI app setup as below by assigning custom report names and managing component visibility and swapping behavior as required, replicating the flexible configuration capabilities found in function app deployments.The point is is maintaining identical reports and models across environments, enabling developers to work on reports and models without disrupting user activity or causing downtime.

Container Setup

This section touches on Docker image creation and registry publishing concepts. While I'm walking through the entire process for comprehensive understanding, end users only need to focus on the deployment execution—the image creation is a one-time setup that's already handled.

Here's the Dockerfile structure: The container includes Python 3.11 slim package as the base runtime, the Microsoft Fabric CLI for workspace operations, and the requests library for PowerBI REST API interactions. The entrypoint script handles authentication and delegates to the main deployment logic.

FROM python:3.11-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates curl git build-essential \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install Fabric CLI and required Python packages
RUN pip install --no-cache-dir --upgrade ms-fabric-cli requests

# Copy deployment scripts
COPY deploy.py /app/deploy.py
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

For comprehensive technical documentation, including complete scripts and configuration files, refer to references section.

The ready-to-use Docker image is available on Docker Hub at https://hub.docker.com/repository/docker/nalakan/pbi_blue_green_dep_v1/general.

Build Pipeline (Image creation Process when there is a change to Prod source artifacts)

The build pipeline triggers automatically when code changes are pushed to the main branch:

trigger:
  branches:
    include:
      - main
  paths:
    include:
      - Dockerfile
      - entrypoint.sh
      - deploy.py
      - azure-pipelines-build-image.yml

pool:
  vmImage: 'ubuntu-latest'

variables:
  - name: imageRepository
    value: 'nalakan/pbi_blue_green_dep_v1'
  - name: imageTag
    value: '$(Build.BuildId)'

steps:
- checkout: self

# Fix line ending issues for shell scripts
- script: |
    echo "Normalizing .sh files to LF line endings..."
    find $(Build.SourcesDirectory) -type f -name "*.sh" -exec sed -i 's/\r$//' {} +
  displayName: Normalize shell scripts

- task: Docker@2
  displayName: Build and push image to Docker Hub
  inputs:
    command: buildAndPush
    repository: $(imageRepository)
    dockerfile: Dockerfile
    containerRegistry: dockerhub-connection
    tags: |
      $(imageTag)
      latest

Image Deployment Pipeline (Container Creation and Cross-Workspace Artifact Deployment)

This pipeline is manually triggered and leverages Azure DevOps variable groups for configuration management,service principal authentication and creating containers and deploying artifacts across multiple workspaces as illustrated in the referenced image.

trigger: none   # Manual execution only

variables:
  - group: Fabric-ServicePrincipal    # Service principal credentials
  - group: Fabric-DeploymentSettings  # Workspace and item names
  - name: imageRepository
    value: 'nalakan/pbi_blue_green_dep_v1'
  - name: imageTag
    value: 'latest'

stages:
- stage: Deploy
  jobs:
  - job: DeployToEnvs
    displayName: Deploy Dev / Test / Prod_New via Docker image
    steps:
    - script: docker pull $(imageRepository):$(imageTag)
      displayName: Pull image

    # Deploy to Development
    - script: |
        docker run --rm \
          -e FABRIC_CLIENT_ID=$(FabricClientId) \
          -e FABRIC_CLIENT_SECRET=$(FabricClientSecret) \
          -e FABRIC_TENANT_ID=$(FabricTenantId) \
          $(imageRepository):$(imageTag) \
            --source-workspace "$(ProdWorkspaceName)" \
            --target-workspace "$(DevWorkspaceName)" \
            --report-name "$(FabricReportName)" \
            --semantic-model-name "$(FabricModelName)" \
            --prefix Dev-
      displayName: Deploy to Dev

That's it ! The entire process(deploying the artifacts via containers) typically completes in under 10 minutes (These results are based on Azure DevOps free tier usage, so the numbers may not accurately represent enterprise-level performance benchmarks) deploying your parameterized reports across all three environments with zero manual intervention. Each environment gets properly prefixed items (Dev-, Test-, Prod_New_) that are fully functional and independently testable.

The beauty of this approach is that once the initial setup is complete, deployments become a simple parameter change and button click. No complex commands, no manual copying, no risk of forgetting to rebind reports to datasets. Everything just works as a charm.

Conclusion

This PowerBI DevOps solution uses blue-green deployment patterns with Docker, Azure DevOps CI/CD, and Fabric CLI integration. It complements Microsoft's standard deployment pipelines by adding blue-green deployment functionality that isn't natively available in Power BI. The approach enables zero-downtime deployments and instant rollbacks across development, test, and production environments while maintaining data consistency.

Acknowledgments 🙏

Thanks to Leon and Dennes @ Onyx Data UK for sparking the idea for the blue-green deployment for PowerBI. The implementation methodology and tools,techniques used shown here in this article is how I approached bringing their initial idea to life.

Resources

https://github.com/nalakan/PowerBIMeetsDocker

https://hub.docker.com/repository/docker/nalakan/pbi_blue_green_dep_v1/general

https://microsoft.github.io/fabric-cli/

Thanks for reading 📖! Hope this post added something valuable to your knowledge and sparked new ideas to explore further

Here’s what we’re setting out to achieve in final phase…

Understanding RAG

If you've been keeping an eye on AI trends, you've probably heard of RAG ; short for Retrieval-Augmented Generation. It's a method for giving large language models (LLMs) access to your own data, so they can answer questions with accurate, up-to-date, and context-rich information instead of relying only on what they were trained on.

How RAG Works

RAG creates a powerful synergy between your data and LLM capabilities. When a user submits a question, the system searches your data repository to find relevant information. The user question is then combined with matching results and sent to the LLM through a carefully crafted prompt. The LLM uses both the question and retrieved context to generate comprehensive, data-driven answers.

The Core Components: Embeddings, Indexes, and Vector Databases

Embeddings are numerical representations of text created by specialized AI models. These models convert words, sentences, or documents into high-dimensional vectors (arrays of numbers) that capture semantic meaning. Similar concepts produce similar vectors "car" and "automobile" become nearly identical number sequences despite being different words.

Indexes serve as the foundation for efficient data retrieval. They're specialized data structures that enable fast, accurate searches across your information repository. RAG indexes combine multiple search methods:

• Keyword searches for exact term matching

• Semantic searches for conceptual similarity

• Vector searches for nuanced content relationships

• Hybrid approaches that merge these capabilities

Vector Databases are storage systems optimized specifically for managing and searching embeddings. Unlike traditional databases that rely on exact matches, vector databases excel at similarity searches across millions of vectors in milliseconds.

Typical RAG Workflow

1. Document Processing: Your documents are broken into chunks and converted into embeddings

2. Storage: These embeddings are stored in a vector database (your searchable index)

3. Query Processing: User questions are converted into embeddings using the same model

4. Retrieval: The vector database finds the most semantically similar content

5. Generation: Retrieved context plus the original question are sent to the LLM for accurate, contextual answers(Most often using a different model )

This approach ensures responses are grounded in your actual data while leveraging the LLM's natural language capabilities.

RAG patterns that include Azure AI Search

Azure AI Search provides scalable search infrastructure that indexes diverse content types and enables retrieval through APIs, applications, and AI agents. The platform offers native integrations with Azure's AI ecosystem including OpenAI services, AI Foundry, and Machine Learning while supporting extensible architectures for third-party and open-source model integration.

Let’s dive into practical implementation from here onwards.This would include both UI based approach and code first approach.

Note : I have omitted minor or commonly known details, focusing instead on explaining and elaborating the RAG pattern and it’s core setup and flow.

Approach 1 : Low / No-Code Solution

1. Create an Azure OpenAI resource
   The initial step is to create an OpenAI resource in Azure, as illustrated below.

   

2. Navigate to the Azure AI Foundry portal and create an embedding deployment. You may select any base model of your choice; in this example, the ‘text-ada’ model has been selected.(text-embedding-ada-002 is part of gpt-3 model family.)

   

3. Jump back to Azure portal and create an Azure AI Search Resource (When allocating AI search mindful of the cost of the resource utilization)

   

4. Connect the document folder within the Fabric ‘Lakehouse’ files area and vectorize the data.(Once we navigate to the created AI search resource click as below.

   

5. Select Fabric OneLake files as source for the vectorization.Since the document I am using is purely text based document ( I used Microsoft Whitepaper on AI governance as my source file which consists 32 pages) and this was uploaded to Fabric Lakehouse files section by creating a folder

   

   

6. Select ‘RAG’

   

7. Next, specify the Fabric ‘Lakehouse’ URL along with the folder path where the files are located.

   

8. After completing these steps, navigate to the ‘Indexes’ section in the Azure AI Search portal. You will notice that an index is created within a short period of time.

   

9. Now we can create a new agent in Azure AI foundry and attach the created Azure AI search resource as a knowledge tool and play with the data in agent playground.

   

For the search type I have selected Hybrid search (Vector+Keyword)

Finally Try in agent playground 🏐

Showcase - [ 01 ] with Agents playground UI

It can be observed that the reference has been applied correctly, ensuring that no external sources were used as knowledge.(“AI Governance whitepaper.pdf”)

Tool calling succeeded, as confirmed within the thread status.

Approach 2: Code-First Implementation

Azure AI Foundry SDK / AI Projects client library

We'll primarily use the AI Projects client library is part of the Azure AI Foundry SDK, and provides easy access to resources in your Azure AI Foundry Project. ( The 'azure.ai.agents.models' package is a sub-module under this client library that handles agent-related operations.)

Install the libraries - (Note that the dependent package azure-ai-agents will be install as a result, if not already installed, to support .agent operations on the client.)

!pip install --upgrade azure-ai-projects azure-identity load_dotenv
!python -m pip install --upgrade pip

This code loads configuration values (like client ID, secret, tenant ID, and endpoint) from an external .env file (api_settings.env) into Python variables.

import os
from dotenv import load_dotenv
load_dotenv('api_settings.env')
CLIENT_ID = os.getenv("CLIENT_ID")
CLIENT_SECRET = os.getenv("CLIENT_SECRET")
TENANT_ID = os.getenv("TENANT_ID")
PROJECT_ENDPOINT = os.getenv("PROJECT_ENDPOINT")

The script below demonstrates the following steps.

• Authentication and Setup – The script retrieves Azure credentials (Tenant ID, Client ID, Client Secret) from environment variables and authenticates using ClientSecretCredential. It then initializes the AIProjectClient with the given project endpoint.

When using the Azure AI Projects library with Entra ID authentication, we must authenticate via a service principal or app registration. To enable this, assign the Azure AI User role (built-in Azure RBAC role) on the Azure AI Foundry resource to your identity. This ensures your service principal has permission to access AI projects using Microsoft Entra ID.

• Connection Discovery – It queries the list of configured project connections and identifies the Azure AI Search resource by inspecting connection metadata, extracting the corresponding connection_id.

• Tool Initialization – An AzureAISearchTool instance is created using the connection ID and a predefined index name, enabling integration of search capabilities into the agent.

• Agent Creation – A new agent is provisioned with the GPT-4.1-mini model, custom instructions for Retrieval-Augmented Generation (RAG) queries, and the AI Search tool attached as a resource.

• Thread Creation – A dedicated conversation thread is established to maintain the dialogue state with the agent.

• Interactive Execution Loop –

  • User inputs are continuously read from the console.

  • Inputs are posted as messages to the agent.

  • The agent is executed (create_and_process), responses are retrieved, and results are displayed.

  • The loop terminates when the user enters "end", after which the agent is deleted for cleanup.

import os
from azure.ai.projects import AIProjectClient
from azure.identity import ClientSecretCredential
from azure.ai.agents.models import AzureAISearchTool

# Entry point for the script
if __name__ == "__main__":
    # Load Azure credentials from environment variables
    credential = ClientSecretCredential(
        tenant_id=os.getenv("TENANT_ID"),
        client_id=os.getenv("CLIENT_ID"),
        client_secret=os.getenv("CLIENT_SECRET")
    )

    # Create a client instance for interacting with the Azure AI Project
    project_client = AIProjectClient(
        credential=credential,
        endpoint=os.environ["PROJECT_ENDPOINT"]  # Project endpoint reference
    )

    # Retrieve available project connections to locate the AI Search resource
    conn_list = project_client.connections.list()
    conn_id = ""

    # Identify the Azure AI Search connection by checking metadata fields
    # Assumes only one AI Search connection is configured in the project
    for conn in conn_list:
        properties = conn.get("properties", {})
        metadata = properties.get("metadata", {})
        if metadata.get("type", "").upper() == "AZURE AI SEARCH":
            conn_id = conn["id"]
            break

    print(conn_id)

    # Reassign the valid connection ID for AI Search
    conn_id = conn["id"]

    # Configure the Azure AI Search tool with the discovered connection and index name
    ai_search = AzureAISearchTool(index_connection_id=conn_id, index_name="rag-1754502262882")

    # Provision an AI Agent and attach the AI Search tool for RAG-style queries
    agent = project_client.agents.create_agent(
        model="gpt-4.1-mini",
        name="my-agent-aisearch",
        instructions="You are a helpful agent to perform RAG Queries using AI Search",
        tools=ai_search.definitions,
        tool_resources=ai_search.resources,
    )
    print(f"Created agent, ID: {agent.id}")

    # Initialize a new conversation thread for maintaining dialog state
    thread = project_client.agents.threads.create()
    print(f"Created thread, ID: {thread.id}")

    # Continuous user interaction loop
    while True:
        user_input = input("User: ")
        if user_input.lower() == "end":
            project_client.agents.delete_agent(agent.id)
            print("Ending the conversation.")
            break

        # Post user input as a message to the agent
        message = project_client.agents.messages.create(
            thread_id=thread.id,
            role="user",
            content=user_input,
        )

        # Run the agent and process the response
        run = project_client.agents.runs.create_and_process(thread_id=thread.id, agent_id=agent.id)
        print(f"Run finished with status: {run.status}")

        if run.status == "failed":
            print(f"Run failed: {run.last_error}")
            break

        # Fetch and print the agent’s most recent reply
        messages = project_client.agents.messages.list(thread_id=thread.id)
        for agent_response in messages:
            if agent_response.text_messages:
                last_text = agent_response.text_messages[-1]
                print(f"{agent_response.role}: {last_text.text.value}")   

    # Final cleanup of the agent instance
    project_client.agents.delete_agent(agent.id)
    print("Conversation ended")

Showcase - [ 02 ] with Visual Studio code in locally

Showcase - [ 03 ] using Streamlit Frontend

Streamlit is an open-source Python framework that makes it easy to build interactive web apps for data and machine learning projects.

📤I deployed my project on Streamlit Community Cloud and adapted the code to work seamlessly with the Streamlit frontend. This enables me to access the app from anywhere through a customized, user-friendly interface.

By the way, what key information are you looking to extract from that Microsoft whitepaper? 📄🤔

I’ve made it public 🌍, so feel free to give it a try 🚀.

[ Sample Q’s you can try out for:

• What specific governance controls do we need to implement for each type of agent creator (End Users, Makers, and Developers) in our organization?

• How can we effectively use Microsoft Purview's Data Loss Prevention policies to prevent our agents from accessing highly confidential SharePoint content?

• What's the best approach for setting up Power Platform environments and pipelines to ensure secure agent development and deployment across our teams? etc ]

Link : https://ragaifoundry.streamlit.app/
If you’re unable to access the working site, it’s probably because my Azure free credits ran out 💳⚡. You can still find all my project assets in the References section below 📂

Key Takeaways

Building enterprise RAG solutions with Azure AI Foundry reveals several crucial insights: prioritize data quality over complexity, leverage hybrid search for optimal results, and choose between low-code UI approaches for rapid prototyping or code-first methods for customization. Most importantly, start small and scale gradually while continuously monitoring performance.

Conclusion

Azure AI Foundry transforms enterprise RAG implementation from complex to accessible. Whether you prefer UI-based configuration, programmatic development, or web deployment, the platform provides enterprise-grade security and scalability without sacrificing ease of use.

The three approaches demonstrated Agent playground, Azure AI Foundry SDK, and Streamlit web app showcase the platform's flexibility for different team needs and technical requirements. As organizations increasingly need AI solutions that integrate seamlessly with existing Microsoft infrastructure, Azure AI Foundry offers both immediate value and long-term strategic advantages.

References

• Azure AI Search - Retrieval Augmented Generation Overview

• Azure AI Foundry - RAG Concepts

• Microsoft AI Agents Governance Whitepaper

• Project Assets

Thanks for reading! If you found this guide helpful, consider sharing it with others who might benefit from implementing RAG solutions in their organizations with Microsoft AI Foundry.

If you've been wrestling with this challenge, Microsoft Fabric's GraphQL API might just be the solution you've been looking for.

Why GraphQL Changes the Game

Think of traditional REST APIs as that over eager friend who tells you their entire life story when you just asked how their day went. You get everything-all the fields, all the data whether you need it or not. Then you're stuck filtering through the noise to find what you actually wanted.

GraphQL, on the other hand, is like having a conversation with someone who listens. You ask for exactly what you need, and that's precisely what you get. Need just customer names and sales amounts? That's all you receive. Want to combine data from multiple related tables? One elegant query handles it all.

Here's what makes GraphQL particularly powerful for Fabric:

• Precision: Request only the columns you need, nothing more

• Efficiency: Combine multiple data requests into a single call

• Flexibility: Query related data in nested structures that mirror real relationships

• Performance: Reduce network overhead and eliminate over-fetching

Data Path for GraphQL

A GraphQL API operates as a server that sits a top an HTTP server, making an API available for code interaction.

• Centralized Data Access: The GraphQL server contains code capable of communicating with various data sources.

• JSON-Based Communication: Send a JSON document to the GraphQL server, containing security tokens and data requests. The server processes this, retrieves data from the underlying sources, and returns another JSON document containing the requested data, messages, and error codes all at once.

• Schema Definition: GraphQL employs a schema, very similar to a SQL schema with the exception it's over all kinds of data,which defines the structure of available fields and data types, ensuring consistent data access across disparate sources.

• Operations: GraphQL utilizes specific operation types:

  • Queries: For retrieving data, akin to SQL SELECT statements.

  • Mutations: For changing data (insert, update, delete).

  • Subscriptions: To "subscribe to data as it changes," enabling real-time updates.

Query the data with API for GraphQL

First we will focus on creating GraphQL resource and GraphQL endpoint inside Microsoft Fabric.

• Go to the Microsoft Fabric workspace and select “New”, then choose “GraphQL API” to open the creation wizard.Enter a name and description for the GraphQL.

• Click “Add a data source”, choose Warehouse as the source type, and select the target warehouse from the list.Click Connect to proceed to the screen for selecting tables .

• Select the required tables or views and click Load to include them in the GraphQL API.

Then GraphQL query was created which fetches the first 10 sales records from the fact_sales table and retrieves customer details by traversing the relationship to the dimension_customer table.(This is a sample query to display the capability of nested structure of ‘json’ like format in GraphQL syntax.Fabric supports auto complete when coding inside the query pane which makes life much easy for the user.

In the results section it retrieves 10 sales transactions,and for each transaction, include related customer information in nested format.

So far so good.Now, let’s assume we need to retrieve the above results externally from an application. How can we achieve that? Let’s start the journey.

The easiest starting point is to capture the code generated through the toolbar.As of today, the generated code comes in two flavours. In this demo I am using ‘Python’ as my code flavour. Set the code aside for now; we'll return to it shortly.

Bringing It to External Applications

Now comes the interesting part: accessing this data from external applications. This is where many implementations fall short, but with proper setup, it becomes surprisingly straightforward.

An ‘App registration’ is needed first.I would not into details of creating ‘App registration’ as it is more straight forward.

We need give required permissions to ‘App Registration’ as below.

Then under Microsoft API’s select ‘PowerBI service’ and select as follows.

Then In the Manage > Authentication section, click on Redirect URI configuration

Here in the below ‘Redirect URI’ first I tested this in localhost:5500 and later I hosted the project in the link ‘https://nalakan.github.io/ms-fabric-graphql-spa/’ with additional components.

Base code which I copied from GraphQL ‘Generated code’ displays below.

Here, you can observe that the GraphQL endpoint on line 23, along with the query(Line 24 -46) used in the Fabric interface, is being passed here.

🚀 Project : GraphQL Playground

Rather than stopping at basic functionality, I decided to build something more comprehensive. The result is a full-featured GraphQL playground that extends the Fabric experience for external users.

Key Features

• Enhanced UI Integration: The interface feels familiar to anyone who's used the Fabric GraphQL explorer, but with additional functionality tailored for external access.

• Complete Schema Exploration: Users can browse the full schema and understand table relationships without needing access to the Fabric workspace itself.

• Direct Data Export: One of the most requested features-the ability to download query results directly to local files in various formats.

• Interactive Query Building: The playground provides real-time query validation and auto-completion, making it accessible to users with varying GraphQL experience levels.

• Comprehensive Documentation: Built-in docs and schema visualization help users understand data structures and build effective queries.

Application Components

The application structure follows clean architecture principles, with the ‘App registration’ details from our earlier setup included in the app.js file. This security model leverages the ‘App Registration’ we created earlier, ensuring that only authorized users can access your Fabric data while maintaining proper security standards.

The project is fully functional and available at: https://nalakan.github.io/ms-fabric-graphql-spa/

The complete source code is available on GitHub: https://github.com/nalakan/ms-fabric-graphql-spa

Here’s a quick sneak peek 👀.

Click on the image/gif for zoom

Looking Forward

GraphQL's integration with Microsoft Fabric opens up exciting possibilities for data access patterns. As organizations increasingly need to share data across diverse applications and teams, having a flexible, secure, and performant API layer becomes crucial.

The playground project I've shared represents just the beginning. The same patterns can be extended to build custom dashboards, integrate with external analytics tools, or power mobile applications—all while maintaining the security and governance that enterprise data requires.

Further Reading

For extra details about Fabric's GraphQL, I recommend checking out this nice guide:

https://data-mozart.com/microsoft-fabric-api-for-graphql-everything-you-need-to-know/

Thanks for reading ! I'm always exploring new ways to work with Microsoft Fabric and AI .Stay tuned for more technical walkthroughs and practical implementations.

Say you shared a report in May 2025, with slicers set to Year = 2025 and Month = May. Fast-forward to July or August, and unless someone manually updates the report or changes the default slicer values, it’ll still load with May selected. The report appears "stuck in time.".

This article introduces a clean, automated way to solve that. Using a PowerBI Button Slicer and a bit of simpler DAX, we’ll dynamically highlight the latest available month -without requiring manual updates or breaking best practices.

Data Model

Here's a simplified look at the data model I'm using. Not every table is relevant for this walk-through, but the focus will be on the Calendar table (highlighted below in my model) and optionally Customer,Orders tables.

Step 1: Create a Calculated Column -Latest Month PH

We start by adding a calculated column to the Calendar table called Latest Month PH. This column helps us identify the latest month dynamically and assign it a numeric value of 100.(Assign a preferred number greater than 12 ,which is the total number of months) . Other months will keep their actual month number.

Latest Month PH = 
VAR _maxym= CALCULATE(MAX('Calendar'[YearMonth]),all())
RETURN
IF('Calendar'[YearMonth]=_maxym,100, 'Calendar'[Month Number])

Step 2: Create a Second Calculated Column -Latest Month

Next, let’s create another calculated column(‘Latest Month’ Column) to show the month name. This will help us display a more user-friendly label in the visual.

Latest Month = 
VAR _maxym= CALCULATE(MAX('Calendar'[YearMonth]),all())
RETURN
IF(
    'Calendar'[YearMonth]=_maxym,'Calendar'[Month Name] & "", 'Calendar'[Month Name])

Step 3: Setting Up the ‘Button Slicer’

1. Add a Button Slicer to the canvas.

2. Set the field to Latest Month PH.

3. Under ‘Callout values’, toggle on the "Values" option / Turn off Values. The Magic will happen after this.

4. Still within ‘Callout values’, enable the "Label" and drag in the Latest Month column.Make sure you apply this setting to both the Default and Selected states if it doesn’t display in ‘Selected’ state.

Step 4 : Testing it in action

To test this setup without waiting for a new calendar month to roll around, I created a pseudo column using Month Number& propagated using parameter This lets me simulate a new month and verify that the button updates accordingly.

Lets change Month to Latest Month using a Parameter.

At the time of writing, the latest month is June, and sure enough, the slicer now shows “June” as expected -automatically and without manual input.

With this new approach, we achieve a dynamic and user-friendly experience by combining the flexibility of the button slicer with two lightweight calculated columns in the ‘Calendar’ table. Importantly, these additions do not significantly impact the model size, allowing us to maintain performance and follow best practices without unnecessarily bloating in the ‘Calendar’ dimension or Facts.

Feel free to share how you have tackled similar scenarios in the comments below

Thank You for Reading !!

Why Microsoft Fabric RTI for YouTube Stream Analytics?

RTI isn't just another event streaming platform—it's built for scenarios where milliseconds matter. Unlike traditional ETL pipelines that batch-process data every few hours, RTI creates a continuous data pipeline that can ingest, transform, and analyze YouTube API responses as they arrive.

What makes this particularly interesting is RTI's native integration with KQL (Kusto Query Language) databases, which can handle semi-structured JSON payloads from YouTube APIs without complex schema transformations. Plus, the EventStream connectors eliminate the need for custom message brokers.

For comprehensive fundamentals, Microsoft's official documentation provides excellent coverage: Microsoft Fabric Real-Time Intelligence Documentation.

Project Scope : Sri Lankan News Ecosystem

I targeted my country - Sri Lanka's major news broadcasters during their prime-time slots : Lunch (11:45-13:00), evening (18:30-19:45), and night (21:00-22:00). The hypothesis? YouTube concurrent viewers could serve as a leading indicator for overall channel engagement trends.

Here's the kicker : While YouTube Live represents maybe 10-20% of total TV or post streaming of Youtube Live viewership, it often shows engagement spikes 30-60 minutes before they appear in traditional viewership metrics and this leads a very crucial information of actual overall TV channel viewers preference and taste. This makes real-time YouTube analytics surprisingly valuable for media monitoring and competitive intelligence.

Architecture :

Here's how the pieces fit together :

The architecture leverages MS Fabric's strength : Seamless integration between compute ( PySpark notebooks), messaging (EventStreams), and analytics (KQL databases).

Step 01 : Understanding YouTube API Economics and Workarounds

YouTube Data API v3 (Free Tier) pricing creates interesting constraints. It gives you 10,000 quota units daily, but here's the catch:

Additional info : https://developers.google.com/youtube/v3/getting-started

Monitor with quota console: https://console.cloud.google.com/apis/api/youtube.googleapis.com/quotas

For my use case (checking 8 channels every 60 seconds for 300 minutes daily), I'd need ~2,400 search operations. That's 240,000 units—24x over the free limit.

Solution: I bypassed the expensive search.list by scraping YouTube's /streams endpoint directly:

This drops the cost from 100 units to ~1 unit per channel check. The trade-off? You're now dependent on YouTube's HTML structure, which could break with UI updates.

Tip: Use the hybrid approach—scrape for discovery, then use official APIs for metadata. This gives you the best of both worlds: cost efficiency and data reliability.

Note down API key in somewhere safe,we need this later.

Step 02 : Eventhouse Setup - KQL Database Configuration

We create a new Eventhouse to hold the KQL Database and subsequent tables. (refer my previous blog posts which explains several different use cases with custom endpoints)

I created an Eventhouse named ISS_DB_001 with a KQL database of the same name.In production, use descriptive meaningful names. Trust me, future-you will appreciate the clarity when managing multiple data sources.

Step 03 - Setting up Eventstream | YT_Live_ES

Creating a new Eventstream is very straightforward ; and after that we have to add a source. Since I am planning to retrieve Youtube API data, I would use custom endpoint as my source.

The EventStream (YT_Live_ES) acts as the ingestion gateway. When you create a Custom Endpoint source, Fabric generates a Service Bus connection string—this becomes your programmatic entry point for streaming data.

Once Custom endpoint added as a source we obtain the Connection string - primary key as below. (Protocol : Event Hub ),Note down this.

Step 3 - PySpark Notebook Implementation :

Okay, now we need to create a notebook and start writing the code to retrieve data from the API and push the payload to the ‘YT_Live_ES’ Eventstream .

Let me break down the core PySpark notebook implementation, focusing on what each key section accomplishes:

Configuration Setup (Lines 13-17)

Line 13 (100 minutes): Since I have many channels to cover across different time belts (lunch, evening, night), I set 100 minutes to ensure complete coverage of any prime-time news slot.

Line 14 (SAS Connection): This is the EventStream connection string we got from the custom endpoint setup.

Line 15 (API Key): The YouTube API key we saved earlier for authentication.

Line 17 (Channel Handles): These are the Sri Lankan news channels I want to track.

Line 20 (Skip IDs): Some channels transmit more than one live stream simultaneously, so I skip non-news streaming IDs to focus only on news channels.

Fetching Stream Details via API

This takes a list of video IDs, retrieves detailed information from the YouTube API, and formats it into a clean, structured dataset that includes:

• Stream metadata (title, channel, URL)

• Live viewer count

• Publishing and retrieval timestamps

• Stream status (live or ended)

Sending Data to Fabric EventStream

This function handles the critical task of streaming our processed data to Microsoft Fabric's EventStream service. It:

1. Extracts the entity path from the connection string

2. Creates a Service Bus client connection

3. Converts each message to JSON

4. Sends the batch of messages to the EventStream

5. Properly closes the connection to prevent resource leaks

Core Tracking Loop

This block includes:

1. Sets a 100-minute runtime limit

2. Checks each channel for live streams

3. Combines currently discovered streams with previously tracked ones

4. Fetches detailed metadata for all tracked streams

5. Filters for news-related content only (using both English "news" and Sinhala "ප්‍රවෘත්ති" keywords)

6. Updates the tracking state (adding new streams, removing ended ones)

7. Sends the filtered data to Fabric EventStream

8. Waits 60 seconds before the next cycle

Full code is in this github gist.

Summarized Flow of Entire NoteBook :

• Defines a list of Sri Lankan news channels to monitor (@HiruNewsOfficial, @AdaDeranaNews, etc.).

• Getting Live Video IDs:

  • Fetches HTML content from the /streams page of each channel using requests.

  • Extracts the current or upcoming live video ID using regex.

  • Skips any videos explicitly marked in a SKIP_VIDEO_IDS set.

• Fetching Stream Metadata:

  • Calls the videos.list endpoint of the YouTube API for all found video IDs.

  • Collects structured metadata such as title, channel name, viewer count, and live status.

• Filtering Relevant Streams:

  • Filters only streams that are:

    • Live (not ended).

    • Containing “news” or “ප්‍රවෘත්ති” in the video title.

    • Not explicitly skipped.

• Sending to Microsoft Fabric EventStream:

  • Converts the filtered stream data into JSON payload.

  • Sends them to a configured Fabric EventStream.

• Continuous Monitoring:

  • Runs for a configured time duration (default: 100 minutes).

  • Repeats checks every 60 seconds.

  • Uses in-memory tracking (active_streams) to handle ongoing live streams.

When executed the NB, payload is being pushed to Eventstream as we can verify below..

Step 04 : Inside EventHouse & KQL DB - Creating Visualizations

Once our YouTube live stream data is flowing into the Fabric Eventhouse and the KQL database, it's time for the fun part – turning this real-time data into actionable visualizations. Microsoft Fabric provides two compelling approaches for creating insights: KQL Dashboards and Power BI integration. Let me walk you through how I implemented both, and why you might choose one over the other.

Setting the Table : Our Data Landing Zone

After our EventStream delivers data to the KQL database, it populates a table named YTubeLive0001TBL. This table contains everything we need:

Approach #1 : Real-time Dashboards using KQL querysets

For real-time operational monitoring where every second counts, I built a Real-time Dashboard directly in the Fabric interface. Here's how:

1. From the KQL Database interface, I clicked ‘Real Time Dashboard and named it "Daily News Tracker"

2. For my first tile, I created a KQL query to show which channels were currently broadcasting in different time belts:

3. Next, I added a visualization showing real-time viewer competition across channels:

During breaking news events, I could literally watch the numbers climb in real-time as viewers flocked to different channels.What makes Real-time Dashboards special is their simplicity and speed. With just a few clicks, I created a monitoring system that provides actionable insights with minimal latency. When news breaks and every minute counts, this approach shines.

In my case, I ended up using both approaches complementarily: Real-time Dashboards during live broadcasts for operational monitoring, and Power BI for deeper analysis and broader sharing of insights afterward.

Real-time Dashboard

Below is the KQL queryset I used to derive the above information—for example, the evening news belt

YTubeLive0001TBL
| where retrieval_date == startofday(now() )
  and retrieval_time_only between (time(18:00:00) .. time(19:45:00))
  and (tolower(Title) has "news" or Title has "ප්‍රවෘත්ති") 
  and status =='Live'
| project platform, stream_id, Title, Channel_Title, published_at, video_url, Live_Viewers, Retrieval_DT, retrieval_date, retrieval_time_only, channel_id, status, actual_end_time
| sort by Retrieval_DT desc nulls first

Approach # 2: Power BI Desktop Integration - The Analysis Powerhouse

While Real-time Dashboards are perfect for real-time monitoring, I needed more sophisticated visualizations and analytical capabilities for deeper insights. This is where Power BI Desktop comes in , i used Direct Query connection mode as I need minimal latency and set page auto refresh in ever 30 seconds.

• Lunch Time news - Viewer trends by channel ,time on 21-05-2025

• Evening Time news - Viewer trends by channel,time on 21-05-2025

• Night Time news - Viewer trends by channel,time on 21-05-2025

• Image showing report interactivity and slicing/filtering options using PowerBI.

Wrapping Up : From Promise to Production

Completed relevant Project Assets in my Fabric Workspace :

Building this real-time YouTube analytics system taught me that the most interesting technical challenges often come from working within constraints—API limits, HTML parsing fragility, and cost optimization. Microsoft Fabric RTI provided the infrastructure backbone that let me focus on the data logic rather than complex ETL pipelines.

The hybrid scraping approach proved more resilient than expected, maintaining 99%+ uptime over a month of continuous operation. For anyone building similar streaming analytics systems, I'd recommend starting with official APIs and falling back to intelligent scraping when quota limits become prohibitive.(If paid plan is not a concern)

The complete NoteBook code is available in this GitHub link and I'd love to see what variations you build. Whether you're tracking crypto livestreams, monitoring gaming tournaments, or analyzing educational content, the core pattern—extract, stream, analyze, visualize—remains mostly consistent.

Is this a complete representation of the total TV audience? May be not. But gives important hints.YouTube live viewers represent only a small subset. However, the real value lies in how this data acts as a real-time barometer—often signaling shifts in public sentiment or news consumption trends well before they appear in traditional ratings.

Spikes in live viewership can indicate increased public interest during major events such as elections, social unrest, or breaking news. By analyzing patterns over specific time periods, we can uncover early signs of audience movement and changing channel preferences—offering a window into what’s happening across society in real time.

References:

• Microsoft Fabric Real-Time Intelligence Documentation

• YouTube Data API v3 Documentation

• Google Cloud API Quotas Console

• KQL Query Language Reference

Connect with me to discuss real-time analytics, Microsoft Fabric implementations, or media technology innovations. I'd love to hear about your own projects in this space !

Have you implemented similar real-time analytics solutions? What challenges did you encounter, and how did you solve them? Share your experiences in the comments below.

As always Thanks for Reading - BIDiaries 😊!!!

All good. But what if you need to do the reverse—sync existing GitHub repositories back into Fabric? In this blog, as a sequel to Sandeep’s post, I am explaining how I achieved that.

💡

This approach is particularly beneficial when an organization undergoes changes, such as onboarding new users or setting up a new capacity. Instead of manually identifying and syncing each repository one by one, this automated solution ensures that all relevant Fabric artifacts from GitHub are seamlessly restored.

Thanks to the power of Michael Kovalsky’s Semantic Link Labs and Sempy, we can automate this process and make it efficient. In this blog, we’ll walk through how to accomplish this in a structured and automated way.

What I have done previously

In my case, I previously synced my existing Fabric Workspaces to GitHub with the ‘Bkp_' prefix (Refer to Sandeep’s previous article )

Now, I can easily filter out these repositories and sync them into Fabric without the prefix (I created the new Workspaces with the same name in Github repos without the ‘Bkp_’ prefix programmatically). If a workspace already exists, I simply skip creating it and proceed with Git initialization.

Prerequisites

Before we dive into the process, ensure you have:

1. GitHub Personal Access Token (PAT): Required for authentication with the GitHub API. Create a classic token with appropriate scopes.

2. Microsoft Fabric Connection ID: This is generated when setting up a GitHub connection in Fabric.

3. Administrative Rights: You must have admin access to Fabric workspaces to sync repositories.

    pip install PyGithub semantic-link-labs
   

Key Functions used in Semantic Link Labs and Sempy

To effectively sync Fabric workspaces with GitHub, we rely on several key functions from Semantic Link Labs and Sempy:

Semantic Link Labs Functions

1. labs.admin.list_git_connections()

   • Fetches all existing GitHub connections linked to Fabric workspaces.

2. labs.connect_workspace_to_github(owner_name, repository_name, branch_name, directory_name, connection_id, workspace)

   • Establishes a Git connection between a Fabric workspace and a GitHub repository.

3. labs.initialize_git_connection(workspace)

   • Initializes the GitHub connection for a Fabric workspace and returns the latest commit hash.

4. labs.update_from_git(workspace, remote_commit_hash, conflict_resolution_policy='PreferRemote')

   • Synchronizes the Fabric workspace with the latest version from GitHub.

Sempy Functions

1. fabric.list_workspaces()

   • Retrieves a list of existing workspaces in Microsoft Fabric.

2. fabric.create_workspace(display_name, description, capacity_id)

   • Creates a new Fabric workspace with the specified name, description, and capacity.

These functions form the backbone of our automation, enabling us to efficiently manage workspaces and maintain synchronization between GitHub and Fabric.

Step-by-Step Guide

1. Initialize GitHub and Fabric Connections

First, we set up our connections to GitHub and Microsoft Fabric:

from github import Github
import sempy.fabric as fabric
import sempy_labs as labs
import time

# GitHub Personal Access Token and Connection ID
g = Github("your_github_pat")
conn_id = "your_connection_id"
capacity_id = "your_capacity_id"

2. Fetch Existing GitHub Repositories

To retrieve all repositories associated with your GitHub account:

user = g.get_user()
repos = list(user.get_repos())  # Convert iterator to list
print(f"ℹ️ Total repositories found: {len(repos)}")

3. List Existing Workspaces and Git Connections

Before creating new Fabric workspaces, check which ones already exist:

existing_workspaces_df = fabric.list_workspaces()
git_connections_df = labs.admin.list_git_connections()

4. Automate Workspace Creation and connect WS to Git

For each repository (With “Bkp_”), we check if a corresponding workspace exists. If not, we create it and sync it with GitHub.

# Initialize counters
total_repos_processed = 0
workspaces_created = 0
workspaces_skipped = 0
workspaces_already_exist = 0
git_connections_created = 0
git_connections_skipped = 0
workspaces_synced_from_github = 0

for repo in repos:
    total_repos_processed += 1
    try:
        if not repo.name.startswith("Bkp_"):
            print(f"⚠️ Skipping '{repo.name}', does not match 'Bkp_' prefix.")
            workspaces_skipped += 1
            continue

        workspace_name = repo.name.replace("Bkp_", "")
        print(f"Processing repo: {repo.name} -> Workspace name: {workspace_name}")

        existing_workspace = existing_workspaces_df[existing_workspaces_df['Name'] == workspace_name]

        if not existing_workspace.empty:
            workspace_id = existing_workspace.iloc[0]['Id']
            print(f"ℹ️ Workspace '{workspace_name}' already exists with ID: {workspace_id}.")
            workspaces_already_exist += 1

            if workspace_id in git_connections_df['Workspace Id'].values:
                print(f"⚠️ Workspace '{workspace_name}' already has a Git connection. Skipping.")
                git_connections_skipped += 1
                continue
        else:
            workspace_id = fabric.create_workspace(
                display_name=workspace_name,
                description=f"Workspace for {repo.name} synced from GitHub",
                capacity_id=capacity_id
            )
            print(f"🟢 Workspace '{workspace_name}' created with ID: {workspace_id}.")
            workspaces_created += 1

        labs.connect_workspace_to_github(
            owner_name=user.login,
            repository_name=repo.name,
            branch_name="main",
            directory_name="/",
            connection_id=conn_id,
            workspace=workspace_id
        )
        print(f"🟢 Connected workspace '{workspace_name}' to GitHub repository '{repo.name}'.")
        git_connections_created += 1
    except Exception as e:
        print(f"⚠️ Unexpected error processing '{repo.name}': {e}")

4. Initialize Git Connection and Git pull

        # Wait for Git connection to initialize
        time.sleep(10)  # Adjust as necessary
        try:
            commit_hash = labs.initialize_git_connection(workspace=workspace_id)
            print(f"🟢 Git connection initialized for '{workspace_name}'. Commit: {commit_hash}")
        except Exception as e:
            print(f"⚠️ Error initializing Git for '{workspace_name}': {e}")
            continue  # Move to the next repo

        # Automatically update workspace from Git
        try:
            update_status = labs.update_from_git(workspace=workspace_id, remote_commit_hash=commit_hash, conflict_resolution_policy='PreferRemote')
            print(f"🟢 Updated workspace '{workspace_name}' from Git. Status: {update_status}")
            workspaces_synced_from_github += 1
        except Exception as e:
            print(f"⚠️ Error updating workspace '{workspace_name}' from Git: {e}")

        # Sync all content from GitHub repo
        print(f"🟢 Workspace '{workspace_name}' setup completed. Moving to next repo.")

    except Exception as e:
        print(f"⚠️ Unexpected error processing '{repo.name}': {e}"

Output of the code

Workspaces are now automatically created and content is synced in Fabric.

Following Workspace pulled the content from Github Repo and now it is connected with Github automatically.

Conclusion

With these steps, you can now integrate previously synced Github artifacts into Microsoft Fabric with minimal effort.(No manual configuration). For Azure DevOps repos the process would be the same.

Full code is available in this GitHub Gist.

Happy syncing - Thanks for Reading!

Did you know that astronauts aboard the International Space Station (ISS) witness 16 sunrises and sunsets every day?

Travelling at 28,000 km/h (17,500 mph), the ISS completes 15.5 orbits around Earth daily, making it one of the most fascinating engineering marvels in human history. As of February 2025, the station has hosted over 270 astronauts from 19 countries, symbolizing a remarkable achievement in human ingenuity and cooperation.

This blog explores how real-time streaming intelligence can be achieved using Microsoft Fabric Python/PySpark notebooks, event streams, and KQL databases and my ever loving tool PowerBI. Rather than just building another ISS tracking report/dashboard, the focus is on showcasing the power of real-time data processing and end-to-end analytics within Fabric.

This work is inspired by many prior implementations, including those by Anshul Sharma and Bradley Ball @ Microsoft, who demonstrated ISS tracking with Azure Logic Apps. Kudos also to Vahid DM (Microsoft MVP ) for his article, which provided valuable insights on the same topic for API integration with Notebooks and Eventstream. Expanding on these foundations, this approach introduces new techniques that enhance flexibility and efficiency using custom endpoints and event-streaming without Azure Logic Apps.

Let’s dive in and see how we can track the ISS and its astronauts in real-time / near real-time 😊

System Architecture

To acquire data on the International Space Station (ISS), we utilize two APIs: the ISS Location API and the Astronauts Data API. We process this data using separate Python or PySpark notebooks, which send the payload to an event stream via custom endpoints. Subsequently, the streamed data is stored as tables in the Eventhouse KQL Database, serving as the foundation for our Power BI reports and dashboards.

Step 01

First, we create a new Eventhouse to hold the KQL Database and subsequent tables.

In my setup, the Eventhouse is named 'ISS_Eventhouse,' and the KQL Database is named 'ISS_DB_001.' Adopting a meaningful naming convention is always a best practice that enhances clarity and maintainability.

Since we are working with two distinct data streams—one for ‘ISS Geo location’ and the other for ‘ISS Astronauts’—we need to create two separate tables to store this data. For now, we'll focus on setting up the Eventstream and leave the table creation on hold. As you progress, you'll understand why it's essential to first establish the Eventhouse and KQL Database before moving forward with the Eventstream setup.

Step 02 - Setting up Eventstream 01 | ISS_Geo_Location_ES

Creating a new Eventstream is very straightforward ; and after that we have to add a source. In this exercisE since I am planning to get ‘ISS Geo’ API data I would use custom endpoint as my source (Refer to my previous blog post on comprehensive use case with Custom endpoint)

Once Custom endpoint added as a source we can get the SAS Key (Connection string - primary key) as below.

Step 3 - Code execution via Pyspark/ Python Notebook

Okay, now we need to switch to our notebook and start writing the code to retrieve data from the API and push the payload to the ‘ISS_Geo_Location_ES’ Eventstream . First, we need to install the following libraries .

• requests: Used to send HTTP requests to the API and fetch live data.

• pytz: Helps manage time zones and convert timestamps. In this blog, we’ll use Sri Lanka,Colombo Time.

• azure-servicebus: Allows us to send messages from the Notebook to Fabric’s EventStream.

pip install requests pytz azure-servicebus

import requests
import json
import time
from datetime import datetime
from azure.servicebus import ServiceBusClient, ServiceBusMessage
import pytz

# Replace with your Fabric EventStream connection string
myconnectionstring = "YOUR SAS KEY"

# API URL - Change this to any API you want to use
API_URL = "http://api.open-notify.org/iss-now.json"  
# API_URL = "https://api.wheretheiss.at/v1/satellites/25544"  # Example: ISS API

# Function to fetch data from any API
def fetch_api_data():
    try:
        response = requests.get(API_URL)
        response.raise_for_status()  # Raises an error if the request fails
        data = response.json()

        # Convert single object response to a list for consistency
        if isinstance(data, dict):
            return [data]  # Wrap single dictionary in a list

        return data  # Return list as-is
    except requests.exceptions.RequestException as e:
        print(f"Error fetching data: {e}")
        return None

# Function to add timestamps in Sri Lanka (Colombo) time with explicit UTC +5:30
def add_timestamps(data):
    colombo_tz = pytz.timezone("Asia/Colombo")
    now_colombo = datetime.now(colombo_tz)

    formatted_datetime = now_colombo.strftime("%m/%d/%Y %I:%M:%S %p") + " UTC +5:30"  # MM/DD/YYYY HH:MM:SS AM/PM UTC +5:30
    date_column = now_colombo.strftime("%d-%m-%Y")  # DD-MM-YYYY
    time_column = now_colombo.strftime("%H:%M:%S")  # HH:MM:SS
    utc_offset = "UTC +5:30"  # Explicitly set the offset

    for record in data:
        record["datetime"] = formatted_datetime  # Full timestamp with explicit UTC +5:30
        record["date"] = date_column  # Just the date
        record["time"] = time_column  # Just the time
        record["utc_offset"] = utc_offset  # Separate UTC offset for reference

    return data


# Function to send processed data to Microsoft Fabric EventStream
def send_to_eventstream(messages, connection_string):
    # Extract EntityPath from connection string
    entity_path = None
    for param in connection_string.split(';'):
        if param.startswith('EntityPath='):
            entity_path = param.split('=')[1]
            break

    if not entity_path:
        raise ValueError("EntityPath not found in connection string. Please check your connection details.")

    # Ensure data is always a list before sending
    if isinstance(messages, dict):
        messages = [messages]  # Convert single object to a list

    # Establish connection to Fabric EventStream
    servicebus_client = ServiceBusClient.from_connection_string(connection_string)
    try:
        with servicebus_client.get_queue_sender(entity_path) as sender:
            # Convert messages to JSON format
            batch_message = [ServiceBusMessage(json.dumps(msg)) for msg in messages]
            sender.send_messages(batch_message)
            print(f"Successfully sent {len(messages)} records to EventStream.")
    except Exception as e:
        print(f"Error sending messages: {e}")
    finally:
        servicebus_client.close()

# Infinite loop to fetch and send data every 2 seconds
print(f"Starting real-time data streaming from {API_URL} to Fabric...")
while True:
    data = fetch_api_data()  # Fetch data from API
    if data:
        processed_data = add_timestamps(data)  # Add date and time
        send_to_eventstream(processed_data, myconnectionstring)  # Send data to EventStream
        print(f"Sent {len(processed_data)} records at {processed_data[0]['datetime']}")
    time.sleep(2)  # Wait for 2 seconds before fetching new data

Here, we are using the following API to retrieve the current location of the ISS : "http://api.open-notify.org/iss-now.json".

This code block continuously fetches data from an above API (tracking the International Space Station's location), adds timestamps in Sri Lanka's time zone (UTC +5:30) (My current Location), and sends the processed data to Microsoft Fabric’s EventStream every 2 seconds.

Key Functions:

1. fetch_api_data() – Retrieves data from the specified API.

2. add_timestamps() – Adds the current date and time in Colombo (Sri Lanka) time zone.

3. send_to_eventstream() – Sends the processed data to Microsoft Fabric EventStream using Azure Service Bus.

The script runs indefinitely, pulling live data every 2 seconds and streaming it to Fabric.

After running the code above, we should receive a confirmation message indicating that the payload has been successfully ingested into the EventStream.I tested this with both Pyspark and Python notebooks in Fabric ,both worked well,Since the payload is quite small for a tumbling window I used Python Notebooks for the rest of the demo.Below figure shows the successful ingestion to EventStream.

If we go back to EventStream, we can see the incoming data stream for ‘ISS Geo Location’ as below.

Step 04 -Creating tables in Eventhouse KQL DB.

As you remember in Step 01 ,we created a KQL Database named ‘ISS_DB_001’ . In order to create a table to save the data receiving from Eventstream we select ‘Get data’ option and select ‘Existing Eventstream’ as below.

Once select that, we can choose to save a table name , in this case I chosen as ‘ISSGeoLocation’ and configured the rest as it is.

Once we finish this step and confirm that everything is working as expected, we can verify that the following flow in Eventstream has been correctly mapped in the Eventstream(ISS_Geo_Location_ES) canvas.

Step 05 - Setting up Eventstream 02 | ISS_Geo_Astronauts_ES

Similar pattern continues for Eventstream 02 and rest as above .

Locating the SAS Key for use of python Notebook as below

This time we use the API for Astronauts data “API_URL = "http://api.open-notify.org/astros.json"

Step 05 - Code execution via Pyspark/ Python Notebook

Only change Aare the API URL and SAS KEY ,others are same.

import requests
import json
import time
from datetime import datetime
from azure.servicebus import ServiceBusClient, ServiceBusMessage
import pytz

# Replace with your Fabric EventStream connection string
myconnectionstring = "YOUR SAS KEY"

# API URL - Change this to any API you want to use
API_URL = "http://api.open-notify.org/astros.json"  
# API_URL = "https://api.wheretheiss.at/v1/satellites/25544"  # Example: ISS API

# Function to fetch data from any API
def fetch_api_data():
    try:
        response = requests.get(API_URL)
        response.raise_for_status()  # Raises an error if the request fails
        data = response.json()

        # Convert single object response to a list for consistency
        if isinstance(data, dict):
            return [data]  # Wrap single dictionary in a list

        return data  # Return list as-is
    except requests.exceptions.RequestException as e:
        print(f"Error fetching data: {e}")
        return None

# Function to add timestamps in Sri Lanka (Colombo) time
def add_timestamps(data):
    colombo_tz = pytz.timezone("Asia/Colombo")
    now_colombo = datetime.now(colombo_tz)

    formatted_datetime = now_colombo.strftime("%m/%d/%Y %I:%M:%S %p")  # MM/DD/YYYY HH:MM:SS AM/PM
    date_column = now_colombo.strftime("%d-%m-%Y")  # DD-MM-YYYY
    time_column = now_colombo.strftime("%H:%M:%S")  # HH:MM:SS

    for record in data:
        record["datetime"] = formatted_datetime
        record["date"] = date_column
        record["time"] = time_column

    return data

# Function to send processed data to Microsoft Fabric EventStream
def send_to_eventstream(messages, connection_string):
    # Extract EntityPath from connection string
    entity_path = None
    for param in connection_string.split(';'):
        if param.startswith('EntityPath='):
            entity_path = param.split('=')[1]
            break

    if not entity_path:
        raise ValueError("EntityPath not found in connection string. Please check your connection details.")

    # Ensure data is always a list before sending
    if isinstance(messages, dict):
        messages = [messages]  # Convert single object to a list

    # Establish connection to Fabric EventStream
    servicebus_client = ServiceBusClient.from_connection_string(connection_string)
    try:
        with servicebus_client.get_queue_sender(entity_path) as sender:
            # Convert messages to JSON format
            batch_message = [ServiceBusMessage(json.dumps(msg)) for msg in messages]
            sender.send_messages(batch_message)
            print(f"Successfully sent {len(messages)} records to EventStream.")
    except Exception as e:
        print(f"Error sending messages: {e}")
    finally:
        servicebus_client.close()

# Infinite loop to fetch and send data every 2 seconds
print(f"Starting real-time data streaming from {API_URL} to Fabric...")
while True:
    data = fetch_api_data()  # Fetch data from API
    if data:
        processed_data = add_timestamps(data)  # Add date and time
        send_to_eventstream(processed_data, myconnectionstring)  # Send data to EventStream
        print(f"Sent {len(processed_data)} records at {processed_data[0]['datetime']}")
    time.sleep(2)  # Wait for 2 seconds before fetching new data

Eventstream 02 get the payload of Astronauts data as below.

Similarly, for astronauts, we create a new table in the KQL database for astronaut data as ‘ISSAstronauts’.

With this step completed, our work with Eventstream and Notebooks are concluded (Notebooks needs to be scheduled). The remaining tasks involve the KQL database, KQL Magics, and Power BI.

Step 06 - Design KQL query sets and apply transformations.

The KQL queries that have been used I have attached follows.These queries were first tested in KQL DB and then used inside the PowerBI and setting up the storage mode as Direct query.

// Code for the ISS latest tragectory -100 Time Points

ISSGeoLocation
| top 100 by timestamp
| project iss_position_longitude, iss_position_latitude, timestamp
| render scatterchart with ( kind=map )

// ISS Geo location in Past 90 Minutes

ISSGeoLocation
| extend Timestamp=todatetime(timestamp)
| where Timestamp > ago(90m)
| project iss_position_longitude, iss_position_latitude, Timestamp

// Details of ISSAstronauts

ISSAstronauts
| top 1 by ['datetime']
| mv-expand people
| project Name= people.name, Craft=people.craft

Obtain the Ingestion URI by copying as above in KQL DB.This will be our cluster URL for ADX / Kusto source.

Step 07 - Connecting the dots in PowerBI

We then use the same KQL queries and use as a Directquery storage mode in PowerBI in order to see the latest and greatest data from KQL DB tables.

ISS Location Table -Current Live Location of ISS

GetAstronauts Table -Astronauts Data

ISSOrbit Table - ISS Trajectory in Past 90 minutes

Step 08 - Show Time in PowerBI

I have enabled page-level refresh every 3 seconds in Power BI (Desktop/Service), ensuring that the latest location and details are updated every 3 seconds, providing a near real-time experience.

At the time of writing this blog, the ISS is traveling across the Atlantic Ocean at an astonishing speed of 27,584 km/h.

You may notice that I have included the altitude and speed (Km/H) of the ISS. To achieve this, I used an additional API (https://api.wheretheiss.at/v1/satellites/25544), following the same process as above. The retrieved data was stored in a new table and integrated into the Power BI report.

• altitude represents the ISS's altitude above Earth's surface in kilometers.

• velocity/speed indicates the ISS's speed in kilometers per hour.(See the speed 😮😮😮)

ISS Live

Orbit Trajectory

Conclusion

This blog post showcased a real-time streaming solution for tracking the ISS and astronauts using Microsoft Fabric, KQL, and Power BI—all without relying on Azure Event Hub. The use of custom endpoints and Python/Pyspark notebooks makes this approach flexible and scalable.

Key Takeaways

✅ No Need for Azure Logic Apps – This method simplifies the architecture while maintaining real-time streaming capabilities.
✅ Scheduled Refresh & Auto Page Refresh – Notebooks are scheduled for refresh, and Power BI auto-refreshes every 3 seconds for a near real-time experience.
✅ Lightweight & Efficient – Using Python notebooks, the resource usage remains minimal (max 2vCores/16GB), avoiding the need for a heavy cluster.
✅ Fully Housed in Microsoft Fabric – The entire solution is built within Microsoft Fabric, leveraging Notebooks, Eventstream, KQL DB, and Power BI for a seamless experience.

🔗 Resources – Check out my GitHub repo for PBIX files, KQL queries, and other references.

As usual Thanks for Reading !!!

Navigating the world of real-time data processing can be complex, but Microsoft Fabric's Real-Time Intelligence with Eventstream makes it surprisingly intuitive. This guide walks you through a practical scenario where we'll transform randomized locally generated bogus humidity sensor data using custom endpoints as both input and output sources.

System Architecture

The architecture consists of the following stages:

1. Custom Endpoint as input Data Source: Sensor data is ingested via a local custom endpoint(Generated in the local system via VS Code)

2. Microsoft Fabric Eventstream: The data flows through transformations to refine and enhance its usability.

3. Derived Streams as output Data Source: Processed data is sent to another stream and acts as an endpoint to send data.

4. Eventhouse KQL Database: Data is stored for further analytics and historical tracking.

Prerequisites

• Fabric Eventstream configured in Microsoft Fabric (Because we need to get the Eventhub Name and Primary Connection String to be included in the producer code)

Step 1: Simulating Humidity Sensor Data

A producer script simulates sensor data and sends it to a custom endpoint. This will be executed in VS Code. As a best practice, the Event Hub name and primary connection strings should be stored securely, such as in Azure Key Vault, and should not be exposed. However, for demonstration purposes, I will explain them as they are.

# INSTALL azure-eventhub before running!
# pip command:
# pip install azure-eventhub
# Primary Stream

from azure.eventhub import EventHubProducerClient, EventData
from datetime import datetime, timezone
import hashlib
import random
import time
import json

# Replace the placeholders with your Event Hubs connection string and event hub name
EVENTHUB_NAME = 'es_d8ecd8c2-45c2-41e8-8f67-2d37e0c8112b'
CONNECTION_STR = 'Endpoint=sb:/'
# Configuration variables
MIN_HUMIDITY = 30.0          # Minimum humidity value
MAX_HUMIDITY = 60.0          # Maximum humidity value
COUNTRY = "Canada"           # Country
CITY = "Toronto"             # City
SLEEP_TIME = 5               # Sleep time before sending next event

# Example message
'''
{
    "country": "Canada",
    "city": "Toronto",
    "timestamp": "2025-01-25T12:36:03.826769+00:00",
    "humidity_readings": [
        {
            "sensor": "sensor_1",
            "humidity": "45.34"
        },
        {
            "sensor": "sensor_2",
            "humidity": "50.21"
        },
        {
            "sensor": "sensor_3",
            "humidity": "55.69"
        }
    ],
    "event_id": "88709af29e138d8d906e009a800ebaddacd46d89d20ec91151e2ab91557170c5"
}
'''

# Create a producer client to send messages to the event hub
producer = EventHubProducerClient.from_connection_string(conn_str=CONNECTION_STR, eventhub_name=EVENTHUB_NAME)

def generate_fake_humidity(min_humidity, max_humidity):
    """Simulate a fake humidity reading within the specified range or generate null."""
    return str(round(random.uniform(min_humidity, max_humidity), 2))

def get_random_sensor_readings(min_humidity, max_humidity):
    """Generate a list of humidity readings for random sensors."""
    sensors = ["sensor_1", "sensor_2", "sensor_3"]
    selected_sensors = random.sample(sensors, random.randint(1, len(sensors)))
    return [{"sensor": sensor, "humidity": generate_fake_humidity(min_humidity, max_humidity)}
            for sensor in selected_sensors]

def generate_event_id(payload):
    """Generate a SHA256 hash as a unique event ID."""
    hash_object = hashlib.sha256(json.dumps(payload, sort_keys=True).encode('utf-8'))
    return hash_object.hexdigest()

def get_current_timestamp():
    """Return the current timestamp in ISO 8601 format."""
    return datetime.now(timezone.utc).isoformat()

try:
    # Continuously generate and send fake humidity readings
    while True:
        # Create a batch.
        event_data_batch = producer.create_batch()

        # Generate random sensor readings
        humidity_readings = get_random_sensor_readings(MIN_HUMIDITY, MAX_HUMIDITY)

        # Create the payload
        payload = {
            "country": COUNTRY,
            "city": CITY,
            "timestamp": get_current_timestamp(),
            "humidity_readings": humidity_readings
        }

        # Generate an event_id
        payload["event_id"] = generate_event_id(payload)

        # Format the message as JSON
        message = json.dumps(payload)

        # Add the JSON-formatted message to the batch
        event_data_batch.add(EventData(message))

        # Send the batch of events to the event hub
        producer.send_batch(event_data_batch)

        print(json.dumps(json.loads(message), indent=4))
        print(event_data_batch)

        # Wait for a bit before sending the next reading
        time.sleep(SLEEP_TIME)
except KeyboardInterrupt:
    print("Stopped by the user")
except Exception as e:
    print(f"Error: {e}")
finally:
    # Close the producer
    producer.close()

Here’s a brief breakdown of the code snippet:

• It generates random humidity readings between 30% and 60% for three sensors.

• The data includes location (Canada, Toronto) and a timestamp.

• The payload is serialized into JSON and assigned a unique event ID using SHA-256 hashing.

• The script continuously generates and prints this data every 5 seconds, simulating real-time sensor readings.

• It runs in an infinite loop until manually stopped (KeyboardInterrupt).

• Event hub name and Connection String-Primary key should be obtained from folloiwng (That’s why we need to first create an Eventstream and then select the source as CustomEndpoint )

• 

Following are the payload output in VS Code

Step 2: Transforming Data in Microsoft Fabric Eventstream

In these steps, i will explain different transformations done to the generated Eventstream output.

2.1 Expanding Humidity Readings

This transformation expands the humidity_readings array into individual events using Expand transformation function,what the Expand function does is ‘Create a new row for each value within an array’ like below.

{
    "sensor": "sensor_1",
    "humidity": 45.5,
    "country": "Canada",
    "timestamp": "2025-02-07T12:00:00Z",
    "event_id": "<unique_hash>"
}

2.2 Managed Field Transformations

Renaming column names and data type changes are done in this step

{
    "sensor": "sensor_1",
    "humidity": 45.5,
    "country": "Canada",
    "event_id": "<unique_hash>"
}

2.3 Group by Transformation

We will add a new column based on the average humidity for each given Event_id. To achieve this, first, perform a group-by operation and then Left join the result with the original table stream.

2.3 Left Join and Managed Fields Transformations

(New AVG_humidity column has been added)

These Processed data are then routed to:

1. A derived stream /secondary stream which acts as an input for another custom endpoint.

2. An Eventhouse/KQL DB (Will not explain the usage of this as the use case in this post is custom endpoints)

Step 3: Consuming derived stream /secondary stream via custom endpoint

The following code block, when executed in VS Code, retrieves grouped and transformed data which pushes through the secondary stream. Similarly, a new EventHub name and connection string need to be added.

# Secondary Stream
#  INSTALL azure-eventhub before running!
# pip install azure-eventhub

from azure.eventhub import EventHubConsumerClient
import json

# Replace the placeholders with your Event Hubs connection string and event hub name
EVENTHUB_NAME = 'des_cee26d55-449b-4c4e-a0dc-ed481eaed80c'
CONNECTION_STR = 'Endpoint=sb://.windows.net/;'
CONSUMER_GROUP = '$Default'  # Change if using a different consumer group

def on_event(partition_context, event):
    """Callback function to process received events."""
    try:
        # Decode event data
        event_data = json.loads(event.body_as_str())

        # Print received event
        print(json.dumps(event_data, indent=4))

        # Update checkpoint
        partition_context.update_checkpoint(event)
    except Exception as e:
        print(f"Error processing event: {e}")

# Create a consumer client
consumer = EventHubConsumerClient.from_connection_string(
    conn_str=CONNECTION_STR,
    consumer_group=CONSUMER_GROUP,
    eventhub_name=EVENTHUB_NAME
)

try:
    print("Listening for events...")
    with consumer:
        consumer.receive(
            on_event=on_event,
            starting_position="-1"  # Read from the beginning
        )
except KeyboardInterrupt:
    print("Stopped by user.")
except Exception as e:
    print(f"Error: {e}")
finally:
    consumer.close()

Conclusion

Adding custom endpoints as inputs and outputs in Microsoft Fabric Eventstream provides greater flexibility for real-time data ingestion and distribution, making it easier to integrate with various systems and applications.

As an Input Source

• Bring in real-time data from external systems, IoT devices, or third-party applications.

• Connect to proprietary or industry-specific data sources that aren’t natively supported.

• Ingest data from APIs, webhooks, or other event-driven sources.

As an Output Destination

• Route transformed event data to external applications, analytics platforms, or databases.

• Send data to business systems, dashboards, or machine learning models for further processing.

• Integrate with third-party APIs or messaging services for real-time automation.

By using custom endpoints, you can seamlessly extend Eventstream’s capabilities to fit your specific needs, ensuring your data flows where it’s needed in real-time.

Thanks for Reading !!!

In today’s fast-evolving data landscape, managing shortcuts efficiently is critical for ensuring smooth access to your data, when you are working with Microsoft Fabric. As your data ecosystem grows, automating processes like MS Fabric Shortcuts creation becomes essential. This guide will explore how you can manage these tasks using Pyspark Notebooks and Semantic Link Labs.

Whether you're moving shortcuts between environments or setting up shortcuts in bulk, you're in the right place. Let’s get started! 🚀

Shoutout to Semantic Link Labs 🙌

Before we dive into the scenarios, I want to give a big shoutout to Michael Kovalsky & Contributing team for creating Semantic Link Labs. Among many other plethora of functions, this has made things much easier to manage. Thanks, Michael! 👏

I'll skip the basics of what shortcuts are and it’s inner workings ,will jump straight to the point i.e. “Bulk creation of Shortcuts”. If you need a fundamental overview of Shortcuts, you can find it here.

Parameters and Configuration

To ensure a smooth shortcut migration or bulk creation process, you'll need the following parameters:

• source_lakehouse: The name of the source lakehouse (or environment) where the tables reside.

• source_workspace: The workspace associated with the source lakehouse (or environment).

• destination_lakehouse: The name of the destination lakehouse where shortcuts will be created.

• destination_workspace: The workspace associated with the destination.

• prefix: (Optional) A prefix to prepend to shortcut names for better organization and clarity.

These parameters ensure the scripts know exactly where to retrieve the data from and where to create the shortcuts.

Why Shortcuts? 🔄

Shortcuts are an incredibly useful tool when managing data environments like lakehouses. They are essentially metadata pointers that provide access to tables without duplicating data. The advantages of using shortcuts include:

• Data Sharing: Shortcuts allow you to share and access data across environments, whether it's lakehouses, workspaces, or other systems.

• Organizational Clarity: Shortcuts help you organize your tables better, making large datasets more manageable.

• Reducing Redundancy: They prevent unnecessary duplication of data, saving valuable storage resources.

By efficiently managing shortcuts, you can significantly streamline your workflows, especially in complex or large-scale migrations.

Scenario 1 : Automated Shortcut Migration with Validation ✅

Overview

In this scenario, we're migrating shortcuts from a source environment (lakehouse/workspace) to a destination one. It involves making sure that:

• No Duplicate Shortcuts: Shortcuts that already exist in the destination are not recreated.

• Naming Conventions: You can optionally prepend or append prefixes or suffixes to ensure naming consistency.

• Error Handling: The script gracefully handles any errors, ensuring the migration process doesn’t fail if something goes wrong.

In my case, the tables in my source lakehouse had different prefixes, which required me to modify them with a different prefix or suffix when creating the shortcuts in the destination. This ensures consistency across different environments.

Semantic Link Labs Functions Used:

• sempy_labs.list_shortcuts

• sempy_labs.lakehouse.create_shortcut_onelake

• Other functions and code snippets are mainly to handle the failures and code sanitization.

Code Implementation (First install Semantic Link Labs - %pip install semantic-link-labs)

 # Author Nalaka
import sempy_labs
import pandas as pd
from typing import Dict, Tuple
import time

def safe_api_call(func, retries=3, delay=2, **kwargs):
    """
    Safely calls a function with retry logic.
    """
    for attempt in range(retries):
        try:
            return func(**kwargs)
        except Exception as e:
            print(f"Attempt {attempt + 1} failed: {e}")
            if attempt < retries - 1:
                time.sleep(delay)
            else:
                raise

def create_shortcuts_batch(
    source_lakehouse: str,
    source_workspace: str,
    destination_lakehouse: str,
    destination_workspace: str,
    prefix: str = ""
) -> Dict[str, Tuple[bool, str]]:
    """
    Create shortcuts automatically, ensuring the correct table name is used from the OneLake Path column.
    """
    results = {}

    print("Fetching shortcuts from source workspace...")
    shortcuts_df = safe_api_call(
        sempy_labs.list_shortcuts,
        lakehouse=source_lakehouse,
        workspace=source_workspace
    )

    # Ensuring necessary columns
    if shortcuts_df.empty or 'Shortcut Name' not in shortcuts_df.columns or 'OneLake Path' not in shortcuts_df.columns:
        print("Missing required data columns ('Shortcut Name' or 'OneLake Path').")
        return results

    # Fetch existing shortcuts in the destination workspace
    print("Fetching existing shortcuts from destination workspace...")
    destination_shortcuts_df = safe_api_call(
        sempy_labs.list_shortcuts,
        lakehouse=destination_lakehouse,
        workspace=destination_workspace
    )
    existing_shortcuts = set(destination_shortcuts_df['Shortcut Name']) if not destination_shortcuts_df.empty else set()

    # Iterate through shortcut names
    print("Processing shortcuts...")
    for _, row in shortcuts_df.iterrows():
        try:
            # Extract actual table name from OneLake Path (removing 'Tables/' prefix)
            actual_table_name = row['OneLake Path'].removeprefix('Tables/') #Change accordingly

            # Prepare shortcut name with optional prefix
            new_shortcut_name = f"{prefix}{row['Shortcut Name']}"

            # Skip if the shortcut already exists
            if new_shortcut_name in existing_shortcuts:
                results[row['Shortcut Name']] = (True, f"Shortcut '{new_shortcut_name}' already exists. Skipped.")
                continue

            # Create shortcut using details from the row
            safe_api_call(
                sempy_labs.lakehouse.create_shortcut_onelake,
                table_name=actual_table_name, 
                source_lakehouse=row['Source Item Name'],  # Source lakehouse from the row
                source_workspace=row['Source Workspace Name'],  # Source workspace from the row
                destination_lakehouse=destination_lakehouse,
                destination_workspace=destination_workspace,
                shortcut_name=new_shortcut_name
            )

            success_msg = (
                f"Successfully created shortcut: {new_shortcut_name}\n"
                f"Source: {row['Source Workspace Name']}/{row['Source Item Name']}\n"
                f"Destination: {destination_workspace}/{destination_lakehouse}"
            )
            results[row['Shortcut Name']] = (True, success_msg)

        except Exception as e:
            # Log the error with detailed information
            error_details = f"Shortcut: {row['Shortcut Name']}\n" \
                            f"Source Lakehouse: {row['Source Item Name']}\n" \
                            f"Source Workspace: {row['Source Workspace Name']}\n" \
                            f"Error: {str(e)}"
            results[row['Shortcut Name']] = (False, error_details)
            print(f"Failed to process: {error_details}")

    return results

def print_results_summary(results: Dict[str, Tuple[bool, str]]) -> None:
    """
    Print detailed summary of shortcut creation results.
    """
    print("\n=== EXECUTION SUMMARY ===")
    print(f"Total shortcuts processed: {len(results)}")

    successful = sum(1 for status, _ in results.values() if status)
    failed = len(results) - successful

    print(f"Successful: {successful}")
    print(f"Failed: {failed}")

    if failed > 0:
        print("\n=== FAILED OPERATIONS ===")
        for table, (status, message) in results.items():
            if not status:
                print(f"\nTable: {table}")
                print(message)

    if successful > 0:
        print("\n=== SUCCESSFUL OPERATIONS ===")
        for table, (status, message) in results.items():
            if status:
                print(f"\nTable: {table}")
                print(message)

if __name__ == "__main__":
    config = {
        # Source location to list shortcuts
        "source_lakehouse": "SRC_LH_00",
        "source_workspace": "SRC_WS_00",

        # Destination location for new shortcuts
        "destination_lakehouse": "TGT_LH_00",
        "destination_workspace": "TGT_WS_00",

        "prefix": ""  # Optional prefix for destination shortcut name
    }

    # Measure execution time
    start_time = time.time()

    print("Starting shortcut creation process...")
    results = create_shortcuts_batch(**config)

    end_time = time.time()
    print_results_summary(results)

    print(f"\nExecution time: {end_time - start_time:.2f} seconds")

Explanation

This script helps:

• Fetch Shortcuts: Retrieves the shortcuts from the source and checks for any that already exist in the destination.

• Validation and Renaming: It ensures that no duplicates are created, with the option to add prefixes for easy organization. (Removed prefix named “Tables”)

• Error Handling: Uses retry logic with the safe_api_call function to handle temporary issues, ensuring smooth migration.

• Create Shortcuts: Finally, it loops through the list of source shortcuts and creates them at the destination.

Use Case

• Ideal for migrating shortcuts between environments (e.g., from Dev/Test to production).

• Helpful when maintaining naming consistency during the migration process.

What Does This Scenario Solve?

This process is perfect for environments where shortcuts need to be moved between different workspaces while maintaining data integrity and ensuring that shortcuts are not duplicated. It ensures that:

• The right shortcuts are created with the right names.

• Any existing shortcuts that meet the same criteria are not duplicated.

• Errors are handled through retry mechanisms to prevent interruptions.

It’s a flexible way to migrate shortcuts without unnecessary manual intervention.

Sample output of the above code

Scenario 2 : Bulk Shortcut Creation from existing source tables🏋️‍♂️

Overview

Bulk shortcut creation allows you to create shortcuts for all tables in a source environment in one go, with the option to replace existing shortcuts if necessary. This is perfect for situations where you're setting up a new environment or refreshing shortcuts to reflect updated data sources.

If you don’t want to check and delete existing shortcuts, you can skip the code blocks that handle the deletion of shortcuts. This step is optional and can be bypassed if the existing shortcuts aren’t a concern for your scenario.

Semantic Link Labs Functions Used:

• sempy_labs.lakehouse.get_lakehouse_tables

• sempy_labs.lakehouse.delete_shortcut

• sempy_labs.lakehouse.create_shortcut_onelake

Code Implementation (First install Semantic Link Labs - %pip install semantic-link-labs)

#Author Nalaka
import sempy_labs

# Parameters for source and destination
source_lakehouse = "SRC_LH_01"
source_workspace = "SRC_WS_01"
destination_lakehouse = "TGT_LH_01"
destination_workspace = "TGT_WS_01"

# Fetch the list of tables from the source lakehouse
tables = sempy_labs.lakehouse.get_lakehouse_tables(
    lakehouse=source_lakehouse,
    workspace=source_workspace
)

# Function to check for existing shortcuts
def shortcut_exists(destination_lakehouse, destination_workspace, shortcut_name):
    existing_shortcuts = sempy_labs.lakehouse.get_lakehouse_tables(
        lakehouse=destination_lakehouse,
        workspace=destination_workspace
    )
    return shortcut_name in existing_shortcuts['Table Name'].values

# Iterate through each row in the DataFrame to process the 'Table Name'
for _, row in tables.iterrows():
    shortcut_name = row['Table Name']  # Extract 'Table Name' column as shortcut name
    print(f"Processing shortcut: '{shortcut_name}'")

    # Check if the shortcut already exists
    if shortcut_exists(destination_lakehouse, destination_workspace, shortcut_name):
        try:
            # Delete the existing shortcut
            sempy_labs.lakehouse.delete_shortcut(
                shortcut_name=shortcut_name,
                lakehouse=destination_lakehouse,
                workspace=destination_workspace
            )
            print(f"🟢 The shortcut '{shortcut_name}' in '{destination_workspace}' has been deleted.")
        except Exception as e:
            print(f"Failed to delete existing shortcut '{shortcut_name}'. Error: {e}")
            continue  # Skip to the next shortcut if delete fails

    # Create a new shortcut
    try:
        sempy_labs.lakehouse.create_shortcut_onelake(
            table_name=shortcut_name,
            source_lakehouse=source_lakehouse,
            source_workspace=source_workspace,
            destination_lakehouse=destination_lakehouse,
            destination_workspace=destination_workspace
        )
        print(f"🟢 Shortcut created successfully for: '{shortcut_name}'")
    except Exception as e:
        print(f"🔴 Failed to create shortcut for '{shortcut_name}'. Error: {e}")

print("All shortcuts processed successfully.")

If you don't want to check for and delete existing shortcuts, you can skip Code Block 3 (shortcut existence check) and Code Block 4 (shortcut deletion).

What Does This Code Block Do?

• List Tables: Fetches all table names from the source lakehouse.

• Check Existing Shortcuts: Verifies whether shortcuts already exist in the destination.

• Delete & Replace: Deletes existing shortcuts in the destination and recreates them.

• Log Progress: Prints status messages for each shortcut processed.

What Does This Scenario Solve?

This scenario simplifies the creation of shortcuts when you're dealing with:

• Initial Setup: If you're setting up a new environment and need to create shortcuts for all tables, this method will automate that process.

• Shortcut Refresh: If your data source has been updated, this process will replace the old shortcuts with new ones, ensuring the shortcuts are always accurate.

It’s a straightforward, automated way to ensure your shortcuts are set up or refreshed efficiently.

Sample output of the above code

Finishing Touches ✨

Managing shortcut creation and migration can greatly enhance your data operations, making them more structured and efficient. With the help of the Semantic Link Labs, you can automate these processes, significantly reducing manual effort and minimizing errors. Whether you're migrating shortcuts between environments or creating them in bulk, the methods and strategies discussed here will help you streamline your workflows and ensure your data stays organized.

Happy shortcutting ! Thanks for Reading !! 👍

🎙️ This podcast dives into how Fabric capacity works, including the evolution of bursting & smoothing, the differences between RI vs PAYG, and the billing impact of pausing, restarting, scaling up/down Fabric capacities. 💼✨ 🧾⚙️ Several helpful resources were used, including LinkedIn Fabric Billing Article Series by Matthew Farrow from Microsoft. 🚄 Feel free to adjust the playback speed to suit your preference.

What's New in Microsoft Fabric Direct Lake (April 2025 Onwards)

Direct Lake on OneLake is a game-changer for working with large-scale data in OneLake, offering faster, simpler, and more flexible semantic modeling in Power BI. What is Direct Lake on OneLake? It's a new storage mode for Power BI semantic models that allows direct connection to Delta tables stored in Microsoft Fabric's OneLake. This bypasses the SQL analytics endpoint of Lakehouses and Warehouses.

• Real-Time Intelligence with Microsoft Fabric: Tracking the International Space Station

  Link :

• https://www.youtube.com/watch?v=K09Z1PBf8YY

• Journey through Incremental Refresh & Hybrid Tables in PowerBI.

  Link :

  %[https://youtu.be/aA99-oVZPFk]

In-House Sessions

Conducted brainstorming and knowledge-sharing sessions during "Wiley(Ex Employer) Innovation Week" and "Wiley Tech Talks".

Consistently secured the First place with numerous upvotes for all Innovation Week Speaking sessions as listed below.

• June 2022: Loading Datasets to Power BI Using Partitions

• November 2022: PowerBI Perspectives

• March 2023: ChatGPT Integration with PowerBI for Better Business Value

• March 2023: Deeper Dive on ChatGPT and Power BI

• May 2023: PowerBI Version Control for Humans

• July 2023: Microsoft Fabric Deep Dive

1. 📦 Artifact Comparison vs Runtime Behavior

• Pipelines compare model files (datasets/reports) — structure, not runtime behavior.

• Connection settings at runtime (e.g., database server) are NOT part of artifact comparison.

• "No difference" means file is same, not that connection string is right!

2. 🚀 Deployment Rules Behavior

• Deployment rules are only applied during promotion.

• Saving deployment rules does NOT change already deployed artifacts.

• You must promote again after setting deployment rules to apply them!

3. ⚡ How Differences Appear

4. 🔥 Environment Drift Trap

• If you manually change rules after promotion ➔ Environment drift detected ➔ Pipeline shows Different.

• If rules applied during promotion ➔ No drift ➔ Pipeline shows Same.

5. 🎯 Key Behavior

🧠 Quick Pitfalls to Remember

Steps

pgsqlCopyEdit[ Dev Model ]
     |
     | (Promote)
     ↓
[ Test Model (initially same as Dev) ]
     |
     | (Edit Deployment Rules? Save? Must Promote Again!)
     ↓
[ Test Model (now runtime behavior changed to Test server) ]

✅ Artifact stays same inside.
✅ Runtime settings change when promoting with deployment rules.

✅ Super Important :

Promotion applies rules. Saving rules alone does nothing. "No difference" checks files, not servers. Manual changes cause drift. Always check connections!

Thanks for Reading ..

https://app.powerbi.com/links/BmPRvBLLKh?ctid=eb5fac66-36d7-4571-b8ca-f7ee9578d45c&pbi_source=linkShare

Lead Generation Dashboard

Data source -Dynamics 365

Multi Slicer Panel Implementation

R & D -Nike store (Interactive link)

This blog post draws insights from James Serra’s book, Deciphering Data Architectures and reflects his perspectives on the various data architectures discussed.

Relational Data Warehouse (RDW) 📊

What is it?

A relational data warehouse serves as a centralized hub for consolidating data from multiple sources. It’s designed for historical analysis and provides the “single version of truth.” Unlike operational databases, it’s not intended for transactional (OLTP) purposes.

Why use it?

• Consolidates data for unified insights.(schema-on-write)

• Reduces the load on production systems.

• Provides reliable historical trend analysis.

• Ensures enhanced security and data quality.

🔍 Questions to consider:

• Does your organization depend on historical reporting?

• Are production systems overburdened with analytical queries?

RDWs have both a compute engine and storage. The compute engine is the processing power used to query the data. The storage is relational storage, which holds data that is structured via tables, rows, and columns. The RDW’s compute power can be used only on its relational storage—they are tied together.

Data Lake 🌊

What is it?

A data lake stores raw, unprocessed data in its native format and uses a schema-on-read approach. It’s an excellent option for exploration and experimentation.

Why use it?

• Cost-effective storage for vast data volumes.

• Flexible access for data scientists and power users.

• Frees up enterprise data warehouse resources.

• Retains complete historical data in one place.

🔍 Things to think about:

• How do you plan to manage semi-structured or unstructured data?

• Do you prioritize flexibility and experimentation for data users?

Modern Data Warehouse (MDW) ⚙️

How does it work?

The modern data warehouse combines the strengths of RDWs and data lakes, offering:

• Low-latency, high-performance analytics.

• Self-service business intelligence (BI) capabilities.

• Interactive ad-hoc querying for business users.

Benefits:

• Real-time data processing.

• Compatibility with diverse data sources.

• Enhanced compliance and security measures.

🔍 Ask yourself

• Are your current analytics tools meeting business demands?

• How effectively does your infrastructure support real-time data?

Data Fabric 🌐

What is it?

Data fabric weaves together disparate data systems to create a unified, accessible layer. Think of it as a modern evolution of the traditional data warehouse, with added features like metadata cataloguing and data virtualization.

Key Features:

• Streamlined data access policies.

• Support for real-time data handling.

• Integration through APIs and microservices.

🔍 Questions to ponder:

• How well do your systems integrate diverse data sources?

• Is real-time data access critical for your organization?

Data Lakehouse 🏠

What is it?

The data lakehouse bridges the gap between data lakes and RDWs, combining the scalability of a data lake with the transactional capabilities of a data warehouse.

Key Features:

• ACID transactions for data integrity.

• Unified batch and streaming data processing.

• Schema enforcement and evolution.

Who should use it?

Organizations dealing with:

• Reliability issues between data lakes and warehouses.

• Governance challenges for large-scale data.

🔍 Considerations:

• How important are transactional guarantees for your workflows?

• Are you facing persistent challenges with siloed data?

Data Mesh 🥅

What is it?

Data mesh decentralizes data ownership, giving individual teams responsibility for their data while treating it as a product. This approach fosters scalability, agility, and collaboration.

Key Principles:

1. Domain Ownership: Teams closest to the data take responsibility.

2. Data as a Product: Prioritize accessibility, quality, and usability.

3. Self-Serve Infrastructure: Equip teams with tools to build and manage their pipelines.

4. Federated Governance: Maintain consistency with centralized standards.

Why is it hard to implement?

1. Cultural and Organizational Barriers:

   • Shifting responsibilities requires a mindset overhaul.

   • Teams must adopt a product-oriented view of data.

   • Resistance is common from teams used to centralized ownership.

2. Governance Complexity:

   • Federated governance is difficult to enforce across domains.

   • Maintaining interoperability and data quality is a significant challenge.

   • Without coordination, data silos or duplication may emerge.

3. Technical Hurdles:

   • Self-serve infrastructure tools are still evolving.

   • Performance issues arise when aggregating data from domains.

   • Requires highly skilled engineers within each domain.

Why isn’t it more popular?

• High Cost: Organizational change and technical overhauls demand significant investment.

• Uncertain ROI: Benefits may take years to materialize, making it harder for companies to justify.

• Standardization Gaps: Lack of established tools and practices can result in inconsistent implementations.

Debates and Common Questions:

• Concept or Tool Agnostic? Data mesh is a conceptual framework that relies on principles rather than specific tools, sparking debates about standardization.

• Performance Concerns: Real-time insights can be delayed when aggregating data from multiple domains.

📈 Pros:

• Encourages collaboration and accountability.

• Scales effectively by leveraging domain expertise.

• Improves overall data quality and usability.

• Enhances agility by decentralizing ownership.

🔇 Cons:

• High implementation and organizational costs.

• Potential for data silos and duplication.

• Requires skilled engineers and cultural buy-in.

• Performance and interoperability challenges.

🔍 Is it right for you?

• Does your organization have the resources and commitment for such a transformation?

• Are your domain teams equipped with the necessary skills and tools?

• How will you ensure governance across all domains?

When to Use Each Architecture ✅

Choosing the right data architecture is highly context-dependent and influenced by various factors such as organizational size, data complexity, team expertise, and business goals. The following examples are not exhaustive but serve as a general guide to help you consider potential directions:

Modern Data Warehouse: Ideal for organizations with smaller datasets and traditional business intelligence (BI) needs. It’s best suited for scenarios requiring low latency and familiar relational database tools.

Data Fabric: Perfect for businesses needing to integrate diverse data sources. With its focus on real-time accessibility and governance, it’s a strong choice for enterprises managing complex systems.

Data Lakehouse: A great fit for organizations prioritizing scalability, cost-effective storage, and advanced analytics. It offers a hybrid solution that balances flexibility and governance.

Data Mesh: Best for large, domain-oriented companies struggling with scalability and data ownership issues. It’s ideal for organizations ready to invest in a cultural and technical transformation.

• Does your organization have the resources and commitment for such a transformation?

• Are your domain teams equipped with the necessary skills and tools?

• How will you ensure governance across all domains?

Comparison of Data Architectures

Most companies will use pieces of each architecture to build a solution adapted to their specific needs.

Final Thoughts 🌟

Most organizations will find success in adopting a hybrid approach, blending aspects of these architectures to suit their unique needs. Each framework offers distinct benefits and challenges — the key is to evaluate your goals, resources, and scalability needs carefully. Which path will your organization choose?

1. Big Data

2. Types of Data Architectures

3. The Architecture Design Session

4. The Relational Data Warehouse

5. Data Lake

6. Data Storage Solutions and Processes

7. Approaches to Design

8. Approaches to Data Modeling

9. Approaches to Data Ingestion

10. The Modern Data Warehouse

11. Data Fabric

12. Data Lakehouse

13. Data Mesh Foundation

14. Should You Adopt Data Mesh? Myths, Concerns, and the Future

15. People and Processes

16. Technologies

Each chapter delves into key concepts, offering valuable insights and practical guidance. For anyone navigating the world of data architecture, this book is a must-read.

Thanks for reading! 😊

Migrating terabytes of data from on-premises servers to the cloud is no small feat. The success of such an operation heavily depends on choosing the right infrastructure. This is where Azure Virtual Machines (VMs) come in, providing the flexibility, scalability, and performance needed for data migrations and cloud publishing tasks.

But how do you pick the best Azure VM for your migration needs? 🤔 With numerous VM types available, understanding their strengths and use cases is critical. Let’s dive into the details of Azure VM series and explore how to optimize your data migration process—whether you're pulling data from on-premise sources or moving it into Microsoft Fabric or other cloud solutions. 🌐

🔄 Azure VM Categories for Data Migration

Azure VMs are categorized based on their ability to handle specific workloads. For data migration, your choice should consider factors like compute power, memory capacity, and disk performance. Below, we analyze the most relevant VM series and their application in data migration tasks.

1. Dv5 and Dsv5 Series: General Purpose

The Dv5/Dsv5 series provides a balanced CPU-to-memory ratio, making it suitable for a wide variety of workloads. These VMs are ideal for small to medium data migrations where cost efficiency and performance balance are key.

Use Case:

• Small to Medium Data Migrations: Handle datasets under 1TB with moderate compute and memory needs.

• Cost Efficiency: Ideal for running basic SQL queries or data integrations.

VM Sizes:

• D16s_v5: 16 vCPUs, 64 GB RAM – suited for medium workloads.

• D32s_v5: 32 vCPUs, 128 GB RAM – handles larger datasets efficiently.

2. Ev5 and Esv5 Series: Memory Optimized

The Ev5/Esv5 series is designed for memory-intensive tasks, making it a great choice for handling large datasets and complex transformations. These VMs offer a higher memory-to-vCPU ratio.

Use Case:

• Large Data Transformations: Suitable for operations like joining datasets, cleaning data, or handling big XML/JSON files.

• In-Memory Operations: Use tools that require significant RAM for processing.

VM Sizes:

• E16s_v5: 16 vCPUs, 128 GB RAM – perfect for memory-heavy workloads.

• E64s_v5: 64 vCPUs, 512 GB RAM – excellent for larger datasets and in-memory computations.

3. Fsv2 Series: Compute Optimized

The Fsv2 series focuses on compute-heavy workloads, with a high vCPU-to-memory ratio. This makes it ideal for data compression, encryption, and other CPU-intensive tasks.

Use Case:

• Compute-Intensive Tasks: Accelerate operations like bulk copy (BCP) and parallel data extraction.

• Batch Data Processing: Process large datasets quickly with minimal memory overhead.

VM Sizes:

• F32s_v2: 32 vCPUs, 64 GB RAM – excellent for compute-heavy extraction and transformations.

• F64s_v2: 64 vCPUs, 128 GB RAM – ideal for demanding parallel processing.

4. Lsv2 Series: Storage Optimized

The Lsv2 series is tailored for storage-heavy tasks, offering NVMe drives with high throughput and low latency. These VMs are perfect for data staging, temporary storage, and IO-intensive operations.

Use Case:

• Large Data Transfers: Staging multi-terabyte datasets before cloud transfer.

• High IO Workloads: Optimized for fast, temporary file handling during migration.

VM Sizes:

• L32s_v2: 32 vCPUs, 256 GB RAM, 4x2TB NVMe – for large-scale migration prep.

• L80s_v2: 80 vCPUs, 640 GB RAM, 10x2TB NVMe – suitable for extreme IO tasks.

5. HBv4 and HC Series: HPC Optimized

The HBv4 and HC series are built for high-performance computing (HPC). They are ideal for mission-critical migrations requiring massive parallelism and real-time data transformations.

Use Case:

• Ultra-Large Datasets: Migrate multi-petabyte datasets with high-speed throughput.

• Real-Time Processing: Execute analytics and transformations during migration.

VM Sizes:

• HB120rs_v4: 120 vCPUs, 480 GB RAM – for parallelized data migration.

• HC44rs: 44 vCPUs, 352 GB RAM – handles extreme compute tasks.

🔍 Azure VM Series Comparison: Cost vs. Performance for Data Migration

🚀 Final Considerations

The right Azure VM series depends on your dataset size, complexity of operations, and cost constraints. To summarize:

• Cost-Conscious? Start with Dv5/Dsv5 for balanced performance.

• Memory-Heavy? Choose Ev5/Esv5 for complex transformations.

• Compute-Intensive? Opt for Fsv2.

• Storage-Heavy? Go with Lsv2 for IO-heavy workloads.

• Mission-Critical? Use HBv4/HC for extreme-scale, high-performance needs.

🌐 Why Choose Cloud over On-Prem?

While on-premises servers might seem like an alternative, they struggle to match Azure’s scalability, cost-efficiency, and flexibility. Azure VMs offer pay-as-you-go pricing, quick provisioning, and built-in scalability, making them a superior choice for most data migration scenarios.

By carefully choosing the right Azure VM, you can migrate your data faster, more efficiently, and at the right cost. Whether it’s 2TB or 20TB, Azure’s flexibility ensures your migration is a success.

Further Read

Thanks For Reading !!! 👍

Databricks isn't tied to one cloud—it works seamlessly on Azure, AWS, and Google Cloud (GCP). Let's dive into how Databricks plugs and plays across these clouds and what each cloud vendor offers to make it feel like their own. ✨

📦 What Does "Cloud-Agnostic" Mean?

The term cloud-agnostic means Databricks is not locked into any single cloud provider. Instead, it can run smoothly across multiple clouds while keeping the same core technology, UI, and user experience.

Imagine this: Databricks is like a universal app that can run on any smartphone—whether it's iOS (Azure), Android (AWS), or Google Pixel (GCP). The app works the same, but each phone adds a little flavor to the experience.

For Databricks, these cloud-specific flavours come from integrations like storage, compute, and security services. Now, let's see how Azure, AWS, and GCP make Databricks their own.

🛠️ Databricks on Azure (Azure Databricks)

If you're deep into the Microsoft ecosystem, Azure Databricks is the perfect fit. Azure Databricks is a fully managed service that combines Databricks' innovation with Azure's capabilities.

Why Azure Databricks Stands Out:

• Deep Integration with Azure Services: Works natively with Azure Data Lake Storage (ADLS), Azure Synapse, Power BI, and Azure Machine Learning.

• Azure Active Directory (AAD): Enterprise-grade security with single sign-on (SSO) and role-based access control (RBAC).

• Optimized for Azure Compute: Databricks clusters run on Azure VMs (virtual machines), giving you a streamlined setup.

• Unified Analytics: Perfect for companies using Power BI as the reporting layer on top of their Databricks-powered data lake.

Cost Segregation: Azure Databricks costs include compute (VMs), storage (ADLS), and Databricks service charges, all billed under the Azure portal.

Think of Azure Databricks as a tailor-made suit for Microsoft shops—it just fits perfectly.

🚀 Databricks on AWS (AWS Databricks)

For companies already invested in the Amazon Web Services (AWS) ecosystem, AWS Databricks is the go-to choice. AWS was the first cloud provider to partner with Databricks!

Why AWS Databricks Stands Out:

• Storage Integration: Works seamlessly with Amazon S3 (Simple Storage Service), which is AWS's backbone for cloud storage.

• Compute Power: Databricks clusters leverage EC2 instances, which are easy to scale up and down.

• Security and IAM: Databricks integrates with AWS Identity and Access Management (IAM) for fine-grained security.

• Native AWS Tools: Connect easily with Redshift, Glue, and SageMaker for a complete data and AI pipeline.

Cost Segregation: AWS Databricks costs are split between S3 for storage, EC2 for compute, and Databricks service charges managed within the AWS billing dashboard.

AWS Databricks feels like a high-performance sports car running on Amazon's robust infrastructure—fast, flexible, and reliable.

📑 Databricks on Google Cloud (GCP Databricks)

Google Cloud has entered the Databricks game more recently, but it brings some serious strengths to the table. GCP Databricks integrates nicely with Google's analytics and AI/ML offerings.

Why GCP Databricks Stands Out:

• Google Cloud Storage (GCS): Acts as the primary storage layer for Databricks clusters.

• BigQuery Integration: Connect Databricks with BigQuery for data warehousing and analytics.

• Vertex AI: Combine Databricks with Vertex AI for end-to-end machine learning workflows.

• Scalable Compute: Databricks clusters run on GCP Compute Engine, which offers flexibility and performance.

• Data-Driven Organizations: Ideal for companies already invested in Google's AI and big data tools.

Cost Segregation: GCP Databricks involves costs for GCS (storage), Compute Engine (clusters), and Databricks service charges, all billed under the Google Cloud console.

GCP Databricks feels like a cutting-edge tech lab—it thrives in environments where AI/ML innovation is the focus.

🌎 Why Do Databricks Work Across All Clouds?

So how does Databricks pull this off? Here’s the secret sauce:

1. Standardized Architecture: Databricks uses containerization (like Docker) and Kubernetes to make its platform portable across different clouds.

2. Unified User Experience: Whether on Azure, AWS, or GCP, the Databricks interface, APIs, and notebooks remain the same.

3. Decoupled Storage and Compute: Databricks can connect to any cloud storage (S3, ADLS, GCS) while managing compute clusters natively.

4. Partnerships: Databricks partners closely with Microsoft, AWS, and Google Cloud to provide deep integrations and managed services.

📊 Which Cloud Should You Choose for Databricks?

The best cloud for Databricks depends on your existing investments and business needs:

• Azure Databricks: Ideal for organizations deep into Microsoft Azure and Power BI.

• AWS Databricks: Perfect for AWS-heavy environments with S3 and EC2 at the core.

• GCP Databricks: Best for organizations focused on AI/ML innovation using Google tools like BigQuery and Vertex AI.

At the end of the day, Databricks gives you freedom of choice while delivering the same powerful platform wherever you go. ✅

📢 The Bottom Line

Databricks is like the ultimate team player—it works on Azure, AWS, and GCP without skipping a beat. Each cloud vendor adds its own flavor to Databricks with integrations like storage, compute, and security tools.

The result? You get a consistent, cloud-agnostic experience for big data, machine learning, and analytics—no matter where your data lives. ✨

Cost Segregation Summary:

• Azure: Compute (VMs), ADLS (storage), and Databricks fees.

• AWS: EC2 (compute), S3 (storage), and Databricks fees.

• GCP: Compute Engine (compute), GCS (storage), and Databricks fees.

So, whether you’re an Azure loyalist, an AWS powerhouse, or a GCP innovator, Databricks has you covered.

Thanks For Reading !!! 👍

🤖 What is a Managed Identity?

A Managed Identity (MI) is a feature provided by Azure that allows an Azure resource (like ADF, Virtual Machines, or App Services) to authenticate securely with other Azure services without requiring credentials such as usernames, passwords, or client secrets.

Think of a Managed Identity as a special type of account in Azure Active Directory (Azure AD). It is:

• 🔄 Automatically managed by Azure

• 🔗 Tied directly to the Azure resource

• 🎫 Used to acquire tokens for secure authentication with Azure services

🚚 How Does It Work in Azure Data Factory (ADF)?

When you enable Managed Identity for an ADF instance, Azure assigns it a unique identity in Azure AD. This identity allows ADF to securely access other Azure services, such as Azure Key Vault, Azure SQL Database, or Azure Blob Storage, without requiring hardcoded credentials.

Here's a step-by-step flow of how Managed Identity works in ADF:

1. Enable Managed Identity 🟢

   • ADF is given a unique Managed Identity by Azure

2. Assign Permissions 🔑

   • Grant this identity the necessary permissions to access resources like Key Vault or Azure SQL

   • This is done either via Access Policies or Azure RBAC

3. Token-Based Authentication 🎫

   • When ADF needs to access a resource, it requests an authentication token from Azure AD

   • The resource (e.g., Key Vault) validates the token to ensure the identity has the necessary permissions

📦 An Analogy to Simplify

Problem: Securely Transferring Sensitive Customer Data 🚨

Imagine the below analogy.
You're running a data logistics company facing a critical challenge: securely transferring sensitive customer records from your primary database to a new, advanced data warehouse, without exposing critical connection details or risking credential leaks. 🔒

Your specific mission:

• 📊 Move customer data from Azure SQL Database to Microsoft Fabric

• 🛡️ Ensure zero credential exposure

• 🔐 Maintain end-to-end security throughout the transfer

• 🤖 Automate the process without manual intervention

The solution involves a specialized secure transport system:

• ADF (Armored Transport Vehicle): A secure data movement mechanism with a special security badge (Managed Identity) 🚚

• Key Vault (Secure Key Vault): Stores encrypted connection credentials and access keys 🔑

• Fabric (Secure Vault): High-security destination that accepts both special badges (Managed Identity) and traditional security passes (Service Principal) depending on your setup 🏦

Data Transfer Security Flow:

1. 🛡️ The secure transport (ADF) uses its official badge (Managed Identity) to access the secure key locker (Azure Key Vault)

2. 🔓 The key locker verifies the badge and provides temporary, encrypted access credentials

3. 📦 Using these secure credentials, the transport safely extracts customer records from the source database (Azure SQL Database)

4. 🚚 The transport approaches the advanced data vault (Microsoft Fabric)

5. 🔐 The vault verifies authentication using either:

   • The transport's special badge (Managed Identity) for Azure-integrated scenarios

   • A traditional security pass (Service Principal) for specific configurations or hybrid scenarios

Key Security Objectives Achieved:

• 🚫 No hardcoded credentials

• 🛡️ Automated, secure authentication

• 💡 Dynamic, temporary access tokens

• 🔒 Elimination of credential management overhead

This approach transforms a potentially risky manual data transfer into a seamless, secure, and automated process. 🚀

🛡️ Why Use Managed Identity?

Managed Identity eliminates the need to manage credentials manually. Instead, Azure takes care of securely handling authentication, which reduces the risk of credential leakage or expiration. Key benefits include:

1. No Hardcoded Credentials 🚫

   • Passwords, client secrets, or keys are no longer needed

2. Lifecycle Management 🔄

   • Managed Identity is tied to the lifecycle of the Azure resource

   • Deleting the resource also deletes the associated identity

3. Enhanced Security 🔒

   • Tokens are automatically rotated and managed by Azure

🔑 How ADF Uses Managed Identity with Azure Key Vault

Scenario:

You want ADF to securely fetch secrets (e.g., database connection strings) from Azure Key Vault without hardcoding any credentials.

Steps:

1. Enable System-Assigned Managed Identity for ADF 🟢

   • In ADF's configuration, enable the System-Assigned Managed Identity

   • This assigns a unique identity to the ADF instance

2. Grant Access to Key Vault 🔓

   • In Key Vault, add an Access Policy or use Azure RBAC to grant permissions by searching for your Azure Data Factory instance

   • 💡 Important Note: When you select "Azure Data Factory" in the access policies or RBAC assignments, you're actually selecting its Managed Identity! Azure shows you the ADF name for simplicity, but behind the scenes, it's granting permissions to the Managed Identity associated with that ADF instance

   • Typically, grant Get and List permissions for secrets

3. Use Key Vault in Linked Services 🔗

   • When creating a Linked Service in ADF, choose Key Vault as the authentication method

   • ADF uses its Managed Identity to fetch the secrets at runtime

4. Token-Based Authentication 🎫

   • At runtime, ADF uses its Managed Identity to request an access token from Azure AD

   • Azure AD validates the identity and issues a token

   • Key Vault uses the token to verify permissions and grants access to the requested secrets

Azure Data Factory Managed Identity to Key Vault Authentication Flow:

🆚 Managed Identity vs. Service Principal

Service Principal ID sometimes referred as Application(client) ID.

❓ Common Questions About Managed Identity

1. Is Managed Identity a User or Account? 🤨

No, Managed Identity is not a user. It is a type of Azure AD identity assigned to an Azure resource (like ADF) to enable secure authentication.

2. When I Grant Key Vault Access to ADF, What Happens? 🔓

When you grant Key Vault access to ADF, you're actually granting access to the Managed Identity tied to that specific ADF instance. This allows ADF to authenticate securely without requiring credentials.

3. Can I Use Managed Identity Instead of Service Principal? 🔄

Yes, in most Azure-native scenarios, Managed Identity is preferred because it eliminates credential management and improves security. However, Service Principals may still be needed for hybrid setups.

4. How Does Microsoft Fabric Handle Authentication? 🎯

Microsoft Fabric supports both Managed Identity and Service Principal authentication. For Azure-native workflows, Managed Identity provides seamless integration, while Service Principal offers flexibility for specific scenarios like hybrid deployments or custom applications. Choose based on your specific integration needs and security requirements.

5. What's the Difference Between System-assigned and User-assigned Managed Identities? 🤔

• System-assigned: Automatically created and managed by Azure, tied directly to one resource's lifecycle (like your ADF instance)

• User-assigned: Created as standalone Azure resources, can be assigned to multiple resources, and managed independently

🏁 Conclusion

Managed Identity simplifies secure authentication in Azure by eliminating the need for manual credential management. When used with Azure Data Factory, it ensures a seamless and secure flow of data between services like Azure SQL Database, Key Vault, and Microsoft Fabric.

By understanding and leveraging Managed Identity, you can build secure, scalable, and efficient cloud workflows while reducing operational overhead. 💡

Have questions? Drop a comment below! 💬

Thanks for Reading !!!👍

As businesses grow, so do their reporting needs, and managing Microsoft Fabric capacity alongside Power BI licensing can become quite complex. With the increasing demand for robust data models, seamless report generation, and broad access across user bases, it’s essential to make the right choices around capacity and licensing to ensure performance, scalability, and cost efficiency.

In this blog post, we’ll dive into some common questions surrounding Microsoft Fabric capacity and Power BI Pro licenses. We’ll explore the best practices for managing capacities, the intricacies of semantic model access, and the most effective ways to handle a large user base without overspending on individual licenses.

Let’s break this down through a series of questions and answers to provide a clear roadmap for navigating your Power BI and Fabric capacity needs.

Q1: I have an F2/F4/F8/F16/F32(any one of these) Fabric capacity used for running pipelines, and notebooks etc and creating semantic models in OneLake. Can Power BI Pro users access reports and connect to these models even if they are not assigned to a Fabric capacity?

A: Yes, with the right permissions in place.

Your F2 or any which is less than F64 SKU, Fabric capacity is doing the heavy lifting by handling pipelines, data storage, and creating semantic models in OneLake. Power BI Pro users can still create, refresh, and share reports without the need for a specific Fabric capacity to be assigned to their workspaces because they are operating in shared capacity by default.

However, even though these users are not directly connected to a Fabric capacity, Power BI Pro users can still access semantic models created in the F2-F32 Fabric capacity, provided a few conditions are met:

• Same tenant: The semantic models and reports must be within the same tenant. Tenants serve as the overarching boundaries within which data and resources are shared.

• Permissions: Ensure the Pro users have Build or Read permissions on the dataset hosting the semantic model. This gives them the ability to connect to and use the models in their reports.

• Workspace settings: The workspace containing the semantic models must allow access from shared or external workspaces for cross-capacity interaction.

This setup allows organizations to maximize their Fabric capacity while enabling Pro users to work efficiently. However, keep in mind that free users (those without Power BI Pro licenses) won’t be able to refresh or interact with the data unless the reports are hosted in a Premium Fabric capacity, which supports free-user access. In shared capacity, they would be restricted to viewing content only if it’s published to them through other means.

Q2: With a user base of 1000+ employees, what is the best way to manage Power BI licenses? Should I assign Power BI Pro licenses to each user, or would it make more sense to use a larger Fabric capacity like F64?

A: For large-scale environments, using a Premium Fabric capacity like F64 is a more scalable and cost-effective solution.

Managing Power BI Pro licenses for over 1000 users is not only expensive but also introduces administrative complexities. Let’s explore why transitioning to an F64 capacity makes more sense in this scenario:

1. Cost-efficiency: Assigning individual Power BI Pro licenses for 1000+ users can lead to significant costs over time. Additionally, with each Pro user relying on shared capacity for refreshing and hosting reports, you may encounter performance bottlenecks as the system struggles to handle the load of a large user base. Free users, in particular, are heavily limited under shared capacity as they won’t be able to refresh or interact with data unless it’s published in a Premium workspace.

2. Scalability with F64: By shifting to an F64 Premium Fabric capacity, you gain the advantage of:

   • Unlimited free-user access: Free users can view reports in workspaces tied to Premium capacity without needing individual licenses, drastically cutting down licensing costs.

   • Performance: Dedicated resources in the F64 capacity ensure reports are generated and refreshed efficiently, even under heavy workloads. This is critical when you’re dealing with complex semantic models or high user traffic.

   • Centralized governance: Hosting all reports in a dedicated Premium workspace simplifies report management and governance. You gain better control over refresh schedules, security, and performance.

3. Ideal for enterprise environments: For large user bases like yours, transitioning to F64 or higher capacity supports enterprise-scale operations, making it easier to manage thousands of users while ensuring smooth performance across the board.

For smaller teams (eg: fewer than 100 users) or less complex use cases, assigning Power BI Pro licenses might be a viable, cost-effective option. However, as user numbers increase or if workloads become more demanding, moving to a Premium Fabric capacity is the logical step for both performance and cost management.

Q3: What exactly is Power BI shared capacity, and can any user view reports hosted in shared capacity?

A: Shared capacity refers to the default resource pool that Power BI provides to Pro users who are not assigned to a dedicated Premium capacity.

Let’s unpack this further:

• Shared capacity: This is a multi-tenant environment where resources (such as CPU, memory, and storage) are pooled and shared across multiple users and organizations. It’s the default mode of operation for Power BI workspaces that don’t have dedicated Fabric or Premium capacity assigned to them.

• Access limitations: Only Power BI Pro users can create, view, and interact with reports hosted in shared capacity. These users must be explicitly granted permissions (such as workspace member or app access) to access reports. Free users cannot view or refresh reports in shared capacity unless the reports are published in a workspace backed by a Premium Fabric capacity.

• Performance considerations: While shared capacity works well for smaller teams or simpler workloads, it has its limitations:

  • Resource constraints: Since resources are shared across many users, performance can degrade during peak usage periods, especially when handling large datasets or complex queries.

  • Limited refreshes: Shared capacity allows up to 8 dataset refreshes per day, which may not be sufficient for real-time or frequent data updates.

To unlock free-user access, better performance, and more refreshes, transitioning to Premium Fabric capacity is a smart move.

Q4: Is it true that Power BI Pro users can access semantic models created in another Fabric capacity within the same tenant, even though they aren’t assigned to that specific capacity? How does this cross-capacity access work?

A: Yes, Power BI Pro users can access semantic models in a different Fabric capacity within the same tenant, as long as the right permissions are set.

Here’s how cross-capacity access works:

• Tenant structure: A tenant serves as the boundary that governs all Power BI resources within an organization. Different Fabric capacities, like F2 or F64, can exist within the same tenant, and resources such as datasets or semantic models can be shared across them, provided permissions are correctly configured.

• Permissions: To access a semantic model hosted in another Fabric capacity, Pro users need Build or Read permissions on that dataset. These permissions allow users to connect to the model via DirectQuery or Live Connection, using the model in reports hosted in their own workspaces.

• Capacity isolation with tenant interaction: While each Fabric capacity provides isolated computational resources, they are not entirely “separate territories.” Data remains within the tenant boundary, and cross-capacity access is permitted through tenant-level security and permissions. The capacity hosting the semantic model handles the computation, while the user’s report runs in their own capacity (shared or Premium).

This flexibility allows organizations to leverage resources across multiple capacities while maintaining security and performance standards.

Q5: Should I continue using shared capacity, or is it time to upgrade to a Premium Fabric capacity for my reporting needs?

A: It depends on your scale and performance requirements. Here’s a comparison to help you decide:

Shared capacity is suitable for small teams or simpler reporting needs where only Power BI Pro users need to access reports. However, as your organization scales and you require more performance and access flexibility, moving to a Premium Fabric capacity (like F64) is a wise investment. It ensures smoother operations, supports more refreshes, and grants free user access, reducing the need for costly individual Pro licenses.

Conclusion:

Managing your Microsoft Fabric capacity and Power BI licensing strategy is crucial for optimizing performance, ensuring scalability, and managing costs as your organization grows. For organizations with large user bases or complex reporting needs, moving to a Premium Fabric capacity like F64 is the best choice(yet it depends on the work load 😌), as it provides dedicated resources, supports free-user access, and simplifies report management.

By carefully considering your organization’s needs—whether you’re a small team working in shared capacity or a large enterprise needing the scalability of Premium Fabric Capacity—you can ensure that your Power BI environment is future-proofed and ready to handle the demands of modern data management.

Thanks For Reading !!! 😊

When designing pipelines in Microsoft Fabric or Azure Data Factory (ADF), understanding the execution flow of activities is critical. One setting that often causes confusion is "Wait on Completion"—especially in how it differs from parallel execution.

In this post, I’ll explore these concepts, highlight their differences with examples and visuals, and explain how to choose the right execution approach for your pipeline.

What is "Wait on Completion"?

The "Wait on Completion" setting controls whether a pipeline waits for an activity to finish before triggering the next activity.

• Enabled (Default): The pipeline triggers Activity A and waits for it to complete before moving on to Activity B.

• Disabled: The pipeline triggers Activity A and then immediately moves to Activity B without waiting for Activity A to complete.

Here’s a quick summary of how this works:

With "Wait on Completion" Enabled

• Activity B begins only after Activity A has completed.

• Ideal for tasks where Activity B relies on the result of Activity A.

With "Wait on Completion" Disabled

• Activity B starts as soon as Activity A is triggered, even if Activity A is still running.

• Useful for tasks where Activity B does not depend on the outcome of Activity A.

Is It the Same as Parallel Execution?

At first glance, disabling "Wait on Completion" may seem like creating parallel execution. However, it’s not the same. Let’s clarify the difference:

Asynchronous Execution (Wait Disabled)

• Activity B starts as soon as Activity A is triggered.

• There is still a sequential dependency: Activity B won’t start until Activity A is triggered.

• Their execution overlaps, but they are not truly independent.

Parallel Execution

• Activities are configured as independent branches in the pipeline.

• Both activities start at the same time, with no dependency or sequence between them.

• True parallel execution must be explicitly defined in the pipeline design.

Visual Comparison

To illustrate this, let’s consider two activities:

• Activity A: Sending an email.

• Activity B: Logging the email status in a database.

With "Wait on Completion" Enabled

Activity B waits until Activity A is complete before starting.

|--- Activity A ---| (email sent)
                   |--- Activity B ---| (log written)

With "Wait on Completion" Disabled

Activity B starts as soon as Activity A is triggered, even while Activity A is still running.

|--- Activity A ---| (email in progress)
    |--- Activity B ---| (log written)

Parallel Execution

Both activities start at the same time and run independently.

|--- Activity A ---| (email sent)
|--- Activity B ---| (log written)

Use Cases for Each Approach

Why the Difference Matters

Disabling "Wait on Completion" introduces an execution overlap that speeds up sequential pipelines by eliminating unnecessary wait times. However, it still ensures a logical order: Activity A must trigger before Activity B starts. This contrasts with true parallel execution, where activities run independently and simultaneously, without triggering dependencies.

How to Decide Which to Use

1. Use "Wait on Completion" (Enabled):

   • When the next activity depends on the completion of the current activity.

   • Example: Loading data from a source and then transforming it.

2. Disable "Wait on Completion" (Asynchronous Execution):

   • When tasks can overlap but still require a sequence of triggers.

   • Example: Sending notifications and logging them simultaneously.

3. Use Parallel Execution:

   • When tasks are entirely independent and can run at the same time.

   • Example: Processing different datasets concurrently.

Tips

In a nutshell when "Wait on Completion" is disabled without explicitly setting up parallel activities, the triggers are still sequential (Activity A starts first, and then Activity B is triggered immediately after). However, the execution of the activities can happen in parallel (they can overlap), since Activity B doesn't wait for Activity A to finish before starting.

So, the activities are triggered sequentially but executed concurrently (parallel execution) as soon as they are triggered.

Can You Tell Just By Looking at the Pipeline Layout?

Here’s an important thing to note: Just by glancing at the layout of activities in the pipeline canvas, you can't always immediately tell if the execution is sequential or parallel. The visual layout might make it look like activities are in sequence or parallel, but the actual behaviour depends on configuration settings like "Wait on Completion".

• Sequential Triggering: Activities can appear sequential in a straight line. But if "Wait on Completion" is disabled, they can still run asynchronously (overlap in execution).

• Parallel Execution: Activities in multiple branches might seem like they run in parallel, but for them to truly execute in parallel, they must be set up as independent tasks with no dependencies or explicit parallel configuration.

So, the layout of activities alone doesn’t tell the whole story. It’s important to check how activities are configured in terms of triggers, dependencies, and the "Wait on Completion" setting to fully understand how they will execute.

Final Thoughts

Choosing between "Wait on Completion" and parallel execution depends on your pipeline’s requirements. By understanding their differences, you can design pipelines that are both efficient and reliable. Microsoft Fabric gives you the flexibility to optimize execution for various scenarios, so experiment with these settings to find what works best for your projects!

Have questions or your own tips to share? Leave a comment below—I’d love to hear your thoughts!

Thanks for Reading !!! ❤️