Designing a Workflow Builder
for Data Engineers
OVERVIEW
Modern data teams build workflows that combine data processing, machine learning, and AI into a single pipeline. These workflows are no longer simple scripts — they evolve and run at scale.
Agent-style workflow builders make this possible, but they also introduce new complexity. Without the right abstractions, engineers struggle to understand what’s happening and why.
MY ROLE
End-to-end Product Design including Problem framing & UX strategy, Interaction design & High-fidelity UI.
TIMELINE
April 2024- Jan 2025
DOMAIN
Data Engineering, AI & ML
Showcasing an ongoing execution of a flow.
CHALLENGE
How to help Data Engineers build, validate, and execute complex Data/AI workflows without slowing them down or overwhelming them?
PERSONA
Data Engineer
Data Engineers build and maintain the pipelines that move and transform data for analytics, machine learning, and AI.
They work with complex workflows where small mistakes can quietly break everything downstream.
PAIN POINTS
Operator discovery is slow and technical
Misconfigured nodes fail only after long runs
Debugging is slow and scattered
DESIGN RESPONSE
Clarity before composition
Minimize incorrect operator usage
Safety during configuration
Block invalid configurations
Observability during execution
Make every run traceable
Finding the Right Operator
Selecting the right operator is difficult in large, complex libraries. Unclear intent and requirements lead to guesswork and rework.
Design Solution
Replace guesswork with intent-driven operator selection
Operators are grouped by intent with quick previews, helping engineers understand usage and requirements before adding them to the workflow.
Meaningful grouping of 100+ data/AI operators.
Previewing each operator detail before adding to flow.
Operator States & Configuration
Enabling accurate operator configuration while making readiness and state visible at every step of the workflow.
Design Solution
Make operator state visible at every stage
Each operator node reflects its current state—loading, success, or error—directly on the canvas. This allows engineers to quickly understand readiness and execution progress without inspecting logs or running the workflow.
Each operator states before, after and during execution.
Treat configuration as a first-class workflow step
Operator nodes must be explicitly configured to define inputs, parameters, and behavior. Clear configuration boundaries ensure each operator is correctly set up before it participates in the workflow.
Configuration modal of Translate operator.
The realtime validation catches issues while building, not after execution.
Executing a Workflow
How might we help Data Engineers execute workflows confidently before running at scale?
Design Solution
Enable progressive execution with clear feedback
Execution is broken into safe sample runs and full executions with real-time visibility. Every run is traceable, allowing engineers to validate logic, monitor progress, and diagnose failures with confidence.
Run history of a workflow and run details.
Execution of a Translate workflow.
IMPACTS
The workflow builder served as a foundational platform, enabling multiple solutions to be built and scaled on top of it. This flexibility supported vertical expansion into Marketing and Sales and helped the MarkovML team pivot effectively during the rapid rise of AI-driven use cases.
LEARNINGS
The workflow builder is powerful and will continue to be. But for knowledge workers to truly benefit from an agentic builder, it needs to be more domain-specific. It can still be valuable—just within a clearly defined context.
READ THE NEXT CASE STUDY
Copyright © 2026 rohitnayak.design - All Rights Reserved.