Anonymous View
Skip to content

Support parallel sub-workflow launches with coordinated callbacks #186

@jakobklippel

Description

@jakobklippel

Motivation

Today a parent workflow can launch sub-workflows sequentially via this.someWorkflow.run(args, { callback: { transition: 'onComplete' } }), but launching multiple sub-workflows in parallel is undocumented and not safely supported by the framework.

Under the hood run() returns Promise<QueueResult> and each queue() call is independent (see loopstack/packages/core/src/workflow-processor/services/workflow-orchestration.service.ts lines 38–79), so it looks like Promise.all([w1.run(...), w2.run(...)]) should work. It doesn't:

  • A wait transition has a single from place. When the parent launches N sub-workflows in one transition, the parent enters a single waiting place (e.g. sub_started).
  • The first sub-workflow callback to arrive fires the wait transition and the parent leaves sub_started.
  • Subsequent callbacks have no matching transition (from: 'sub_started' no longer applies) and are dropped.

A naive parallel example silently loses results. This blocks fan-out workflow patterns (e.g. processing N items concurrently through a sub-workflow).

Context: deferred from doc-gap remediation pass on 2026-06-10 (improvement 232 in context/how-to-6-workflow-patterns.md, lines 157–158).

Proposal

  1. Investigate the workflow processor's actual behavior with multiple in-flight callbacks against the same waiting place. Confirm whether late callbacks are dropped or re-queued (entry point: executeTransition in loopstack/packages/core/src/workflow-processor/services/processors/workflow-processor.service.ts ~line 474).
  2. Decide on a coordination pattern. Candidates:
    • State-based counter — parent transition records launched count; callback transition stays in the same place (from: 'waiting', to: 'waiting') and only transitions out when the counter reaches zero. Requires confirming wait transitions can have from == to.
    • Task tool wrapping — wrap each sub-workflow as a BaseTool with pending: { workflowId }, dispatched via LlmDelegateToolCallsTool (or a non-LLM equivalent). This is the path LlmDelegateService already implements (loopstack/registry/features/llm-provider-module/src/services/llm-delegate.service.ts lines 22–68) — it aggregates pendingCount and fires allCompleted.
    • New orchestration primitive — add a parallel-aware runAll(workflows[], { callback }) helper to BaseWorkflow that handles coordination internally and exposes the cleanest authoring surface.
  3. Build a registry example (parallel-sub-workflows-example) demonstrating whichever pattern wins. Include tests. Model from existing sequential example: loopstack/registry/examples/run-sub-workflow-example/.
  4. Document a new "Parallel Sub-Workflows" section in loopstack/docs/build/patterns/sub-workflows.md showing the pattern (e.g. Promise.all() + coordination mechanism).
  5. Close out improvement 232 — update coverage tags in context/how-to-6-workflow-patterns.md (lines 157–158) and delete todo-parallel-sub-workflows.md.

Scope

  • In scope: investigation, coordination-pattern decision, framework changes (if pattern 3 is chosen), one registry example, one docs section.
  • Out of scope: generalizing to parallel tools across workflows (already covered by LlmDelegateService), reworking the wait-transition semantics beyond what's needed for this feature.

Affected area

  • Package(s): @loopstack/core, @loopstack/contracts (if runAll-style API is added)
  • File(s):
    • loopstack/packages/core/src/workflow-processor/base-workflow.ts (run() returns Promise<QueueResult>, lines 60–68)
    • loopstack/packages/core/src/workflow-processor/services/workflow-orchestration.service.ts (lines 38–79)
    • loopstack/packages/core/src/workflow-processor/services/processors/workflow-processor.service.ts (executeTransition, wait handling ~line 474)
    • loopstack/packages/common/src/decorators/transition.decorator.ts (@Transition({ wait: true }) semantics)
    • loopstack/docs/build/patterns/sub-workflows.md (new section)
    • loopstack/registry/examples/parallel-sub-workflows-example/ (new)

Acceptance criteria

  • Framework behavior with multiple in-flight callbacks against a single waiting place is confirmed and documented in the issue
  • Coordination pattern is chosen and rationale recorded
  • parallel-sub-workflows-example registry package exists and has tests covering: all callbacks delivered, no callbacks dropped, ordering of results, error propagation when one sub-workflow fails
  • "Parallel Sub-Workflows" section added to loopstack/docs/build/patterns/sub-workflows.md with a working code snippet
  • Coverage tags in context/how-to-6-workflow-patterns.md lines 157–158 updated
  • todo-parallel-sub-workflows.md deleted

Open questions

  • Can a wait transition declare from == to (state-based counter pattern)? If not, is that change in scope here?
  • Should the chosen pattern be exposed as a BaseWorkflow.runAll(...) helper, or kept as a documented authoring pattern using existing primitives?
  • How should partial failures be handled — fail-fast vs. collect-all-results-then-decide?

Additional context

  • Existing sequential sub-workflow example: loopstack/registry/examples/run-sub-workflow-example/
  • Existing parallel pending coordination (tools, not workflows): loopstack/registry/features/llm-provider-module/src/services/llm-delegate.service.ts lines 22–68
  • Deferred TODO file: todo-parallel-sub-workflows.md (root of project repo)

Metadata

Metadata

Assignees

Labels

No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions