Overview
PyWorkflow includes built-in safeguards to prevent runaway workflows from consuming excessive resources. These limits help ensure system stability and predictable behavior.
Event History Limits
Since PyWorkflow uses event sourcing, every workflow action is recorded as an event. To prevent unbounded growth and memory issues, there are limits on the number of events a workflow can generate.
Soft Limit (Warning)
| Setting | Default | Description |
|---|
event_soft_limit | 10,000 | Logs a warning when reached |
event_warning_interval | 100 | Logs warning every N events after soft limit |
WARNING - Workflow approaching event limit: 10000/50000
Hard Limit (Failure)
| Setting | Default | Description |
|---|
event_hard_limit | 50,000 | Terminates workflow with failure |
EventLimitExceededError:
from pyworkflow.core.exceptions import EventLimitExceededError
# This error is raised when hard limit is reached
# EventLimitExceededError: Workflow run_abc123 exceeded maximum event limit: 50000 >= 50000
The hard limit is a safety mechanism. If your workflow is hitting this limit, it likely indicates a design issue such as an infinite loop or processing too many items in a single workflow run.
Why These Limits Exist
-
Memory Protection: Each event consumes memory. Unbounded event growth can exhaust system resources.
-
Replay Performance: When workflows resume, all events are replayed. Large event logs slow down resumption.
-
Storage Costs: Events are persisted to storage. Excessive events increase storage requirements.
-
Bug Detection: Hitting limits often indicates bugs like infinite loops or improper workflow design.
Best Practices
Design for bounded event counts:# BAD: Processing millions of items in one workflow
@workflow()
async def process_all_orders():
orders = await get_all_orders() # Could be millions!
for order in orders:
await process_order(order) # Each creates events
# GOOD: Process in batches with separate workflow runs
@workflow()
async def process_order_batch(batch_ids: list[str]):
for order_id in batch_ids[:100]: # Bounded batch size
await process_order(order_id)
# Orchestrate batches externally
for batch in chunk_list(all_order_ids, 100):
await start(process_order_batch, batch)
Configuring Limits
Modifying event limits is not recommended. The defaults are carefully chosen to balance flexibility with safety. Only change these if you fully understand the implications.
import pyworkflow
# This will emit a UserWarning - proceed with caution
pyworkflow.configure(
event_soft_limit=20_000, # Warning at 20K events
event_hard_limit=100_000, # Fail at 100K events
event_warning_interval=200, # Warn every 200 events after soft limit
)
Configuration Options
| Option | Type | Default | Description |
|---|
event_soft_limit | int | 10,000 | Event count to start logging warnings |
event_hard_limit | int | 50,000 | Event count to terminate workflow |
event_warning_interval | int | 100 | Events between warnings after soft limit |
Transient Mode
Event limits only apply to durable workflows. Transient workflows (with durable=False) do not record events and are not subject to these limits.
@workflow(durable=False)
async def quick_task():
# No event limits - events aren't recorded
for i in range(100_000):
await some_step(i)
Transient workflows sacrifice durability for performance. They cannot be resumed after crashes or restarts.
Monitoring Event Counts
You can monitor event counts using the storage backend:
from pyworkflow import get_storage
storage = get_storage()
events = await storage.get_events(run_id)
print(f"Event count: {len(events)}")
# Check if approaching limits
if len(events) > 8000:
print("Warning: Workflow has many events, consider redesigning")
Handling Limit Errors
When the hard limit is reached, an EventLimitExceededError is raised. This error inherits from FatalError, meaning it will not be retried.
from pyworkflow.core.exceptions import EventLimitExceededError, FatalError
try:
await start(my_workflow, args)
except EventLimitExceededError as e:
print(f"Workflow {e.run_id} exceeded {e.limit} events")
print(f"Actual count: {e.event_count}")
# Consider splitting the work into smaller workflows
Next Steps
