# Real-Time Intelligence with PySpark / Python Notebooks and Power BI –
The Perfect Trio for Tracking the International Space Station (ISS)

## Introduction

Did you know that astronauts aboard the **International Space Station (ISS)** witness **16 sunrises and sunsets** every day?

Travelling at **28,000 km/h (17,500 mph)**, the ISS completes **15.5 orbits around Earth daily**, making it one of the most fascinating engineering marvels in human history. As of February 2025, the station has hosted over 270 astronauts from 19 countries, symbolizing a remarkable achievement in human ingenuity and cooperation.

This blog explores how **real-time streaming intelligence** can be achieved using **Microsoft Fabric Python/PySpark notebooks, event streams, and KQL databases and my ever loving tool PowerBI**. Rather than just building another ISS tracking report/dashboard, the focus is on showcasing the power of **real-time data processing** and end-to-end analytics within Fabric.

This work is inspired by many prior implementations, including those by **Anshul Sharma and Bradley Ball @ Microsoft**, who demonstrated ISS tracking with **Azure Logic Apps**. Kudos also to **Vahid DM (Microsoft MVP )** for his [article](https://www.vahiddm.com/post/sending-api-data-to-fabric-real-time-intelligence-with-notebooks), which provided valuable insights on the same topic for API integration with Notebooks and Eventstream. Expanding on these foundations, this approach introduces **new techniques** that enhance flexibility and efficiency using **custom endpoints and event-streaming without Azure Logic Apps.**

Let’s dive in and see how we can track the ISS and its astronauts in real-time / near real-time 😊

## System Architecture

![Click to zoom](https://cdn.hashnode.com/res/hashnode/image/upload/v1739868196034/ed2a6bf6-cbc0-4153-9f6a-10dad8b29bb9.gif align="center")

To acquire data on the International Space Station (ISS), we utilize two APIs: the ISS Location API and the Astronauts Data API. We process this data using separate Python or PySpark notebooks, which send the payload to an event stream via custom endpoints. Subsequently, the streamed data is stored as tables in the Eventhouse KQL Database, serving as the foundation for our Power BI reports and dashboards.

### Step 01

First, we create a new Eventhouse to hold the KQL Database and subsequent tables.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739813267338/5b13e2bc-07bb-453b-b627-c246a872c1f9.png align="center")

In my setup, the Eventhouse is named 'ISS\_Eventhouse,' and the KQL Database is named 'ISS\_DB\_001.' Adopting a meaningful naming convention is always a best practice that enhances clarity and maintainability.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739813502483/db438cb9-ca45-4987-bccb-f65a4aca5b81.png align="center")

Since we are working with two distinct data streams—one for ‘ISS Geo location’ and the other for ‘ISS Astronauts’—we need to create two separate tables to store this data. For now, we'll focus on setting up the Eventstream and leave the table creation on hold. As you progress, you'll understand why it's essential to first establish the Eventhouse and KQL Database before moving forward with the Eventstream setup.

### Step 02 - Setting up Eventstream 01 | ISS\_Geo\_Location\_ES

Creating a new Eventstream is very straightforward ; and after that we have to add a source. In this exercisE since I am planning to get ‘ISS Geo’ API data I would use custom endpoint as my source (Refer to my previous blog post on comprehensive use case with Custom endpoint)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739813864157/11c8dd60-b3f4-454d-91d7-7df4991748a6.png align="center")

Once Custom endpoint added as a source we can get the SAS Key **(Connection string - primary key)** as below.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739814020516/5dfa3b26-94f9-477b-b736-2163e2396da6.png align="center")

### Step 3 - Code execution via Pyspark/ Python Notebook

Okay, now we need to switch to our notebook and start writing the code to retrieve data from the API and push the payload to the ‘ISS\_Geo\_Location\_ES’ Eventstream . First, we need to install the following libraries .

* **requests**: Used to send HTTP requests to the API and fetch live data.
    
* **pytz**: Helps manage time zones and convert timestamps. In this blog, we’ll use Sri Lanka,Colombo Time.
    
* **azure-servicebus**: Allows us to send messages from the Notebook to Fabric’s EventStream.
    

```python
pip install requests pytz azure-servicebus
```

```python
import requests
import json
import time
from datetime import datetime
from azure.servicebus import ServiceBusClient, ServiceBusMessage
import pytz

# Replace with your Fabric EventStream connection string
myconnectionstring = "YOUR SAS KEY"

# API URL - Change this to any API you want to use
API_URL = "http://api.open-notify.org/iss-now.json"  
# API_URL = "https://api.wheretheiss.at/v1/satellites/25544"  # Example: ISS API

# Function to fetch data from any API
def fetch_api_data():
    try:
        response = requests.get(API_URL)
        response.raise_for_status()  # Raises an error if the request fails
        data = response.json()

        # Convert single object response to a list for consistency
        if isinstance(data, dict):
            return [data]  # Wrap single dictionary in a list

        return data  # Return list as-is
    except requests.exceptions.RequestException as e:
        print(f"Error fetching data: {e}")
        return None

# Function to add timestamps in Sri Lanka (Colombo) time with explicit UTC +5:30
def add_timestamps(data):
    colombo_tz = pytz.timezone("Asia/Colombo")
    now_colombo = datetime.now(colombo_tz)

    formatted_datetime = now_colombo.strftime("%m/%d/%Y %I:%M:%S %p") + " UTC +5:30"  # MM/DD/YYYY HH:MM:SS AM/PM UTC +5:30
    date_column = now_colombo.strftime("%d-%m-%Y")  # DD-MM-YYYY
    time_column = now_colombo.strftime("%H:%M:%S")  # HH:MM:SS
    utc_offset = "UTC +5:30"  # Explicitly set the offset

    for record in data:
        record["datetime"] = formatted_datetime  # Full timestamp with explicit UTC +5:30
        record["date"] = date_column  # Just the date
        record["time"] = time_column  # Just the time
        record["utc_offset"] = utc_offset  # Separate UTC offset for reference

    return data


# Function to send processed data to Microsoft Fabric EventStream
def send_to_eventstream(messages, connection_string):
    # Extract EntityPath from connection string
    entity_path = None
    for param in connection_string.split(';'):
        if param.startswith('EntityPath='):
            entity_path = param.split('=')[1]
            break

    if not entity_path:
        raise ValueError("EntityPath not found in connection string. Please check your connection details.")

    # Ensure data is always a list before sending
    if isinstance(messages, dict):
        messages = [messages]  # Convert single object to a list

    # Establish connection to Fabric EventStream
    servicebus_client = ServiceBusClient.from_connection_string(connection_string)
    try:
        with servicebus_client.get_queue_sender(entity_path) as sender:
            # Convert messages to JSON format
            batch_message = [ServiceBusMessage(json.dumps(msg)) for msg in messages]
            sender.send_messages(batch_message)
            print(f"Successfully sent {len(messages)} records to EventStream.")
    except Exception as e:
        print(f"Error sending messages: {e}")
    finally:
        servicebus_client.close()

# Infinite loop to fetch and send data every 2 seconds
print(f"Starting real-time data streaming from {API_URL} to Fabric...")
while True:
    data = fetch_api_data()  # Fetch data from API
    if data:
        processed_data = add_timestamps(data)  # Add date and time
        send_to_eventstream(processed_data, myconnectionstring)  # Send data to EventStream
        print(f"Sent {len(processed_data)} records at {processed_data[0]['datetime']}")
    time.sleep(2)  # Wait for 2 seconds before fetching new data
```

Here, we are using the following API to retrieve the current location of the ISS : `"`[`http://api.open-notify.org/iss-now.json`](http://api.open-notify.org/iss-now.json)`"`.

> This code block continuously fetches data from an above API (tracking the International Space Station's location), adds timestamps in Sri Lanka's time zone (UTC +5:30) (My current Location), and sends the processed data to Microsoft Fabric’s EventStream every 2 seconds.
> 
> ### **Key Functions:**
> 
> 1. **fetch\_api\_data()** – Retrieves data from the specified API.
>     
> 2. **add\_timestamps()** – Adds the current date and time in Colombo (Sri Lanka) time zone.
>     
> 3. **send\_to\_eventstream()** – Sends the processed data to Microsoft Fabric EventStream using Azure Service Bus.
>     
> 
> The script runs indefinitely, pulling live data every 2 seconds and streaming it to Fabric.

After running the code above, we should receive a confirmation message indicating that the payload has been successfully ingested into the EventStream.I tested this with both Pyspark and Python notebooks in Fabric ,both worked well,Since the payload is quite small for a tumbling window I used Python Notebooks for the rest of the demo.Below figure shows the successful ingestion to EventStream.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739814383257/74a1fb19-437c-43c9-9a91-d2ecbc1a9bc2.png align="center")

If we go back to EventStream, we can see the incoming data stream for ‘<mark>ISS Geo Location</mark>’ as below.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739814487675/f211c2c1-70f2-4916-9c72-4490ff090a5c.png align="center")

### Step 04 -Creating tables in Eventhouse KQL DB.

As you remember in Step 01 ,we created a KQL Database named ‘ISS\_DB\_001’ . In order to create a table to save the data receiving from Eventstream we select ‘Get data’ option and select ‘Existing Eventstream’ as below.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739814694797/f6bf70ff-aede-4124-9f4a-77a8b8adba85.png align="center")

Once select that, we can choose to save a table name , in this case I chosen as ‘ISSGeoLocation’ and configured the rest as it is.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739814789224/e4873ce3-70ff-4032-a01a-ac58e4f20791.png align="center")

Once we finish this step and confirm that everything is working as expected, we can verify that the following flow in Eventstream has been correctly mapped in the Eventstream(ISS\_Geo\_Location\_ES) canvas.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739815263624/03f84aa8-cc5c-422a-b96f-ad56118f832b.png align="center")

### Step 05 - Setting up Eventstream 02 | ISS\_Geo\_Astronauts\_ES

Similar pattern continues for Eventstream 02 and rest as above .

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739815659781/6c116692-fcac-4088-82c7-4322771d716f.png align="center")

Locating the SAS Key for use of python Notebook as below

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739815723161/75bb5255-9847-4b1d-85fd-53c6a0b9660a.png align="center")

> This time we use the API for Astronauts data “API\_URL = "http://api.open-notify.org/astros.json"

### Step 05 - Code execution via Pyspark/ Python Notebook

> Only change Aare the API URL and SAS KEY ,others are same.

```python
import requests
import json
import time
from datetime import datetime
from azure.servicebus import ServiceBusClient, ServiceBusMessage
import pytz

# Replace with your Fabric EventStream connection string
myconnectionstring = "YOUR SAS KEY"

# API URL - Change this to any API you want to use
API_URL = "http://api.open-notify.org/astros.json"  
# API_URL = "https://api.wheretheiss.at/v1/satellites/25544"  # Example: ISS API

# Function to fetch data from any API
def fetch_api_data():
    try:
        response = requests.get(API_URL)
        response.raise_for_status()  # Raises an error if the request fails
        data = response.json()

        # Convert single object response to a list for consistency
        if isinstance(data, dict):
            return [data]  # Wrap single dictionary in a list

        return data  # Return list as-is
    except requests.exceptions.RequestException as e:
        print(f"Error fetching data: {e}")
        return None

# Function to add timestamps in Sri Lanka (Colombo) time
def add_timestamps(data):
    colombo_tz = pytz.timezone("Asia/Colombo")
    now_colombo = datetime.now(colombo_tz)

    formatted_datetime = now_colombo.strftime("%m/%d/%Y %I:%M:%S %p")  # MM/DD/YYYY HH:MM:SS AM/PM
    date_column = now_colombo.strftime("%d-%m-%Y")  # DD-MM-YYYY
    time_column = now_colombo.strftime("%H:%M:%S")  # HH:MM:SS

    for record in data:
        record["datetime"] = formatted_datetime
        record["date"] = date_column
        record["time"] = time_column

    return data

# Function to send processed data to Microsoft Fabric EventStream
def send_to_eventstream(messages, connection_string):
    # Extract EntityPath from connection string
    entity_path = None
    for param in connection_string.split(';'):
        if param.startswith('EntityPath='):
            entity_path = param.split('=')[1]
            break

    if not entity_path:
        raise ValueError("EntityPath not found in connection string. Please check your connection details.")

    # Ensure data is always a list before sending
    if isinstance(messages, dict):
        messages = [messages]  # Convert single object to a list

    # Establish connection to Fabric EventStream
    servicebus_client = ServiceBusClient.from_connection_string(connection_string)
    try:
        with servicebus_client.get_queue_sender(entity_path) as sender:
            # Convert messages to JSON format
            batch_message = [ServiceBusMessage(json.dumps(msg)) for msg in messages]
            sender.send_messages(batch_message)
            print(f"Successfully sent {len(messages)} records to EventStream.")
    except Exception as e:
        print(f"Error sending messages: {e}")
    finally:
        servicebus_client.close()

# Infinite loop to fetch and send data every 2 seconds
print(f"Starting real-time data streaming from {API_URL} to Fabric...")
while True:
    data = fetch_api_data()  # Fetch data from API
    if data:
        processed_data = add_timestamps(data)  # Add date and time
        send_to_eventstream(processed_data, myconnectionstring)  # Send data to EventStream
        print(f"Sent {len(processed_data)} records at {processed_data[0]['datetime']}")
    time.sleep(2)  # Wait for 2 seconds before fetching new data
```

Eventstream 02 get the payload of Astronauts data as below.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739816019396/49d5e2e2-5e78-4e94-b5c1-38c1093c4c0c.png align="center")

Similarly, for astronauts, we create a new table in the KQL database for astronaut data as ‘ISSAstronauts’.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739816134081/30927648-df9d-4850-acf4-8ab667b43889.png align="center")

> With this step completed, our work with Eventstream and Notebooks are concluded (Notebooks needs to be scheduled). The remaining tasks involve the KQL database, KQL Magics, and Power BI.

### Step 06 - Design KQL query sets and apply transformations.

The KQL queries that have been used I have attached follows.These queries were first tested in KQL DB and then used inside the PowerBI and setting up the storage mode as <mark>Direct query</mark>.

```plaintext
// Code for the ISS latest tragectory -100 Time Points

ISSGeoLocation
| top 100 by timestamp
| project iss_position_longitude, iss_position_latitude, timestamp
| render scatterchart with ( kind=map )
```

```plaintext
// ISS Geo location in Past 90 Minutes

ISSGeoLocation
| extend Timestamp=todatetime(timestamp)
| where Timestamp > ago(90m)
| project iss_position_longitude, iss_position_latitude, Timestamp
```

```plaintext
// Details of ISSAstronauts

ISSAstronauts
| top 1 by ['datetime']
| mv-expand people
| project Name= people.name, Craft=people.craft
```

> ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742889482125/36d9ae27-5cc3-4f74-b4df-f530a1efbd78.png align="center")
> 
> Obtain the Ingestion URI by copying as above in KQL DB.This will be our cluster URL for ADX / Kusto source.

### Step 07 - Connecting the dots in PowerBI

We then use the same KQL queries and use as a <mark>Directquery</mark> storage mode in PowerBI in order to see the latest and greatest data from KQL DB tables.

> ISS Location Table -Current Live Location of ISS

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739816469139/8d573c44-2da7-4f89-8fb7-4e15a0e75ba6.png align="center")

> GetAstronauts Table -Astronauts Data

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739816666269/191c5c85-cbd0-47ef-858d-f654b74a2df3.png align="center")

> ISSOrbit Table - ISS Trajectory in Past 90 minutes

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739816756114/79a4bac0-9830-416f-9c95-ec376eec2481.png align="center")

### Step 08 - Show Time in PowerBI

I have enabled page-level refresh every 3 seconds in Power BI (Desktop/Service), ensuring that the latest location and details are updated every 3 seconds, providing a near real-time experience.

<mark>At the time of writing this blog, the ISS is traveling across the Atlantic Ocean at an astonishing speed of 27,584 km/h.</mark>

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739885640438/26c0578d-30b1-4cbe-9737-851db1fa670a.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739885676428/24b8108a-a574-40b2-9f0d-581d048ea0a4.png align="center")

> You may notice that I have included the <mark>altitude and speed</mark> (Km/H) of the ISS. To achieve this, I used an additional API ([https://api.wheretheiss.at/v1/satellites/25544](https://api.wheretheiss.at/v1/satellites/25544))[,](https://api.wheretheiss.at/v1/satellites/25544) following the same process as above. The retrieved data was stored in a new table and integrated into the Power BI report.
> 
> * `altitude` represents the ISS's altitude above Earth's surface in kilometers.
>     
> * `velocity`/speed indicates the ISS's speed in kilometers per hour.(See the speed 😮😮😮)
>     

## **ISS Live**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739890026455/01d05865-8594-4174-a9c2-85ca55381479.gif align="center")

## Orbit Trajectory

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1739890428744/c4f74cb2-c8e3-49b1-818f-7765724e9a9f.gif align="center")

## Conclusion

This blog post showcased a real-time streaming solution for tracking the ISS and astronauts using Microsoft Fabric, KQL, and Power BI—all without relying on Azure Event Hub. The use of custom endpoints and Python/Pyspark notebooks makes this approach flexible and scalable.

## Key Takeaways

✅ **No Need for Azure Logic Apps** – This method simplifies the architecture while maintaining real-time streaming capabilities.  
✅ **Scheduled Refresh & Auto Page Refresh** – Notebooks are scheduled for refresh, and Power BI auto-refreshes every 3 seconds for a near real-time experience.  
✅ **Lightweight & Efficient** – Using Python notebooks, the resource usage remains minimal (max **2vCores/16GB**), avoiding the need for a heavy cluster.  
✅ **Fully Housed in Microsoft Fabric** – The entire solution is built within **Microsoft Fabric**, leveraging **Notebooks, Eventstream, KQL DB, and Power BI** for a seamless experience.

🔗 **Resources** – Check out my [GitHub repo](https://github.com/nalakan/ISS-Demo) for PBIX files, KQL queries, and other references.

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text">Honestly, this has been the most time-consuming and effort-intensive blog post I've written so far. But it's all for a good purpose, and I'm genuinely happy about that!</div>
</div>

**As usual Thanks for Reading !!!**