Limitations

Overview

PyWorkflow includes built-in safeguards to prevent runaway workflows from consuming excessive resources. These limits help ensure system stability and predictable behavior.

Event History Limits

Since PyWorkflow uses event sourcing, every workflow action is recorded as an event. To prevent unbounded growth and memory issues, there are limits on the number of events a workflow can generate.

Soft Limit (Warning)

Setting	Default	Description
`event_soft_limit`	10,000	Logs a warning when reached
`event_warning_interval`	100	Logs warning every N events after soft limit

When a workflow reaches 10,000 events, PyWorkflow logs a warning:

WARNING - Workflow approaching event limit: 10000/50000

After the soft limit, warnings continue every 100 events (10,100, 10,200, etc.) to alert you that the workflow is growing large.

Hard Limit (Failure)

Setting	Default	Description
`event_hard_limit`	50,000	Terminates workflow with failure

When a workflow reaches 50,000 events, it is terminated with an EventLimitExceededError:

from pyworkflow.core.exceptions import EventLimitExceededError

# This error is raised when hard limit is reached
# EventLimitExceededError: Workflow run_abc123 exceeded maximum event limit: 50000 >= 50000

The hard limit is a safety mechanism. If your workflow is hitting this limit, it likely indicates a design issue such as an infinite loop or processing too many items in a single workflow run.

Why These Limits Exist

Memory Protection: Each event consumes memory. Unbounded event growth can exhaust system resources.
Replay Performance: When workflows resume, all events are replayed. Large event logs slow down resumption.
Storage Costs: Events are persisted to storage. Excessive events increase storage requirements.
Bug Detection: Hitting limits often indicates bugs like infinite loops or improper workflow design.

Best Practices

Design for bounded event counts:

# BAD: Processing millions of items in one workflow
@workflow()
async def process_all_orders():
    orders = await get_all_orders()  # Could be millions!
    for order in orders:
        await process_order(order)  # Each creates events

# GOOD: Process in batches with separate workflow runs
@workflow()
async def process_order_batch(batch_ids: list[str]):
    for order_id in batch_ids[:100]:  # Bounded batch size
        await process_order(order_id)

# Orchestrate batches externally
for batch in chunk_list(all_order_ids, 100):
    await start(process_order_batch, batch)

Configuring Limits

Modifying event limits is not recommended. The defaults are carefully chosen to balance flexibility with safety. Only change these if you fully understand the implications.

If you must change the limits:

import pyworkflow

# This will emit a UserWarning - proceed with caution
pyworkflow.configure(
    event_soft_limit=20_000,      # Warning at 20K events
    event_hard_limit=100_000,     # Fail at 100K events
    event_warning_interval=200,   # Warn every 200 events after soft limit
)

Configuration Options

Option	Type	Default	Description
`event_soft_limit`	`int`	10,000	Event count to start logging warnings
`event_hard_limit`	`int`	50,000	Event count to terminate workflow
`event_warning_interval`	`int`	100	Events between warnings after soft limit

Transient Mode

Event limits only apply to durable workflows. Transient workflows (with durable=False) do not record events and are not subject to these limits.

@workflow(durable=False)
async def quick_task():
    # No event limits - events aren't recorded
    for i in range(100_000):
        await some_step(i)

Transient workflows sacrifice durability for performance. They cannot be resumed after crashes or restarts.

Monitoring Event Counts

You can monitor event counts using the storage backend:

from pyworkflow import get_storage

storage = get_storage()
events = await storage.get_events(run_id)

print(f"Event count: {len(events)}")

# Check if approaching limits
if len(events) > 8000:
    print("Warning: Workflow has many events, consider redesigning")

Handling Limit Errors

When the hard limit is reached, an EventLimitExceededError is raised. This error inherits from FatalError, meaning it will not be retried.

from pyworkflow.core.exceptions import EventLimitExceededError, FatalError

try:
    await start(my_workflow, args)
except EventLimitExceededError as e:
    print(f"Workflow {e.run_id} exceeded {e.limit} events")
    print(f"Actual count: {e.event_count}")
    # Consider splitting the work into smaller workflows

Getting Started

Core Concepts

Guides

Overview

Event History Limits

Soft Limit (Warning)

Hard Limit (Failure)

Why These Limits Exist

Best Practices

Configuring Limits

Configuration Options

Transient Mode

Monitoring Event Counts

Handling Limit Errors

Next Steps

Events

Configuration

​Overview

​Event History Limits

​Soft Limit (Warning)

​Hard Limit (Failure)

​Why These Limits Exist

​Best Practices

​Configuring Limits

​Configuration Options

​Transient Mode

​Monitoring Event Counts

​Handling Limit Errors

​Next Steps

Events

Configuration

Overview

Event History Limits

Soft Limit (Warning)

Hard Limit (Failure)

Why These Limits Exist

Best Practices

Configuring Limits

Configuration Options

Transient Mode

Monitoring Event Counts

Handling Limit Errors

Next Steps