Configure Data Flows

Overview

The BptDataFlow component provides a visual flow designer and execution engine for building automated data pipelines. It moves data between email, files, FTP/SFTP, databases, and REST APIs — with real-time status streaming back to your Blazor dashboard via SignalR.

Architecture

┌──────────────────┐     SignalR      ┌─────────────────┐
│  DataFlow        │ ───────────────► │  Blazor Server  │
│  Service (.exe)  │  push status     │  (your app)     │
│                  │                  │                  │
│  Runs on each    │ ◄─────────────── │  BptDataFlow     │
│  target machine  │  flow commands   │  component       │
└──────────────────┘                  └─────────────────┘
        │                                     │
        │ executes                            │ displays
   Email, File, FTP,                     Visual flow
   DB, API pipelines                     designer + status

Supported Connectors

Email (IMAP)

Read messages and attachments from mailboxes. Filter by subject, sender, or date range.

File System

Read/write files by pattern. Supports recursive scanning and delete-after-read.

FTP / SFTP

Transfer files via SFTP with password or private key authentication.

Database

PostgreSQL, SQL Server, MySQL, and SQLite. Execute queries and stream results.

REST API

Call HTTP endpoints with Bearer, API Key, or Basic auth.

Step 1: Install the DataFlow Service

Install the BPT DataFlow service on the machine that will execute your pipelines. The service connects back to your Blazor app to receive flow definitions and push execution status.

Windows

Download and run the MSI installer. The service registers as a Windows service and starts automatically.

After installation, configure the hub URL in:

C:\Program Files\BptDataFlow\appsettings.json

{
  "DataFlow": {
    "HubUrl": "https://your-blazor-host/bpt-dataflowhub",
    "FlowDirectory": "C:\\ProgramData\\BptDataFlow\\flows"
  }
}

Linux

Install the .deb package:

sudo dpkg -i bpt-dataflow_1.0.0_amd64.deb

Configure and start:

# Edit configuration
sudo nano /opt/bpt-dataflow/appsettings.json

# Start the service
sudo systemctl start bpt-dataflow

# Enable on boot
sudo systemctl enable bpt-dataflow

# Check status
sudo systemctl status bpt-dataflow

Step 2: Configure the Blazor Server Hub

In your Blazor server's Program.cs, map the SignalR hub that communicates with the DataFlow service:

var builder = WebApplication.CreateBuilder(args);

// Add BPT services
builder.Services.AddBlazorPowerTools();

var app = builder.Build();

// Map the data flow hub endpoint
app.MapBptDataFlowHub();

app.Run();

Sandbox Mode You can use the component without any service installation by enabling sandbox mode. This generates synthetic flow data for testing your dashboard layout.

Step 3: Add the DataFlow Component

Place BptDataFlow on your dashboard page:

<BptDataFlow Theme="DataFlowTheme.Dark"
              Height="600px"
              Services="@services"
              OnFlowStatusReceived="OnStatus" />

@code {
    private List<BptDataFlowConfig> services = new()
    {
        new() { FriendlyName = "Pipeline Host", Domain = "localhost" }
    };

    private void OnStatus(DataFlowCallbackData data)
    {
        // Handle execution status updates
    }
}

Component Parameters

Parameter	Type	Description
`Theme`	DataFlowTheme	Visual theme (Dark, Light, Blueprint, Nord)
`Height`	string	Component height (CSS value)
`Width`	string	Component width (CSS value, default 100%)
`HubUrl`	string	Custom SignalR hub URL (default: auto-detect)
`UseSandboxData`	bool	Use synthetic data without a service
`ReadOnly`	bool	Disable editing (view-only mode)
`Services`	List<BptDataFlowConfig>	DataFlow services to connect to
`ImportedFlow`	byte[]	Pre-load a flow definition from JSON bytes
`OnFlowStatusReceived`	EventCallback	Triggered on execution status updates
`OnFlowSaved`	EventCallback	Triggered when a flow is saved
`OnFlowExported`	EventCallback	Triggered when a flow is exported

Service Configuration Reference

Setting	Description	Default
`HubUrl`	The SignalR hub URL on your Blazor server	(required)
`FlowDirectory`	Directory to store flow definition JSON files	/opt/bpt-dataflow/flows

Firewall & Networking

The DataFlow service makes outbound HTTPS connections to your Blazor server. Ensure:

The service can reach the Blazor server's hostname on the configured port (typically 443)
No firewall rule blocks outbound HTTPS from the pipeline host
If using a reverse proxy (nginx, IIS), ensure WebSocket upgrade headers are forwarded
Connectors need network access to their targets (IMAP server, SFTP host, database, API endpoints)

Tip Test connectivity from the pipeline host before deploying:

curl -I https://your-blazor-host/bpt-dataflowhub