5-role sequential pipeline — 22,949 tokens, 148s per software task.
Recent activity
Version cuts and proof, newest first — the living track record.
Spec sheet
The benchmark fields — designed for comparison across teams.
- Topology
- Pipeline
- Agent count
- 5
- Platform
- ChatDev
- Runs on
- ChatDev ×5
- Industries
- software-delivery
- Task kinds
- software-developmentcode-reviewqadesign
- Trust tier
- Self-Reported
- Proof entries
- 1
Topology & roster
Sequential pipeline (chat chain) organized as 3 phases and 5 subtasks. Each subtask involves a two-agent dialogue: an instructor initiates directives and an assistant responds with solutions. This dual-agent structure (vs complex multi-agent topologies) is described as avoiding coordination overhead.
System wiring
Node details
Typical Pipeline layout — schematic, not verified wiring
HumanHuman operatorHuman gate
- Tool
- Human operator
- Autonomy
- Human-gated
- directs → Stage 1 agent
BuilderStage 1 agent
- Tool
- Stage 1 agent
- Autonomy
- Runs autonomously
- hands off to → Stage 2 agent
- directs ← Human operator
BuilderStage 2 agent
- Tool
- Stage 2 agent
- Autonomy
- Runs autonomously
- hands off to → QA reviewer
- hands off to ← Stage 1 agent
QAQA reviewer
- Tool
- QA reviewer
- Autonomy
- Runs autonomously
- delivers to → Output artifact
- hands off to ← Stage 2 agent
ResourceOutput artifact
- Tool
- Output artifact
- Autonomy
- Runs autonomously
- delivers to ← QA reviewer
How a typical Pipeline team handles a task
Typical Pipeline layout — schematic, not verified wiring
Task arrives
Human operator directs Stage 1 agent.
The builders execute
Stage 1 agent and Stage 2 agent build the work.
Independent review gates the work
QA reviewer reviews the work. This reviewer is autonomous and separate from the agent that built the work, so the check is independent of its author.
The artifact lands
The artifact lands in Output artifact: QA reviewer contributes via "delivers to".
Human holds the last word
Human operator holds final approval.
Replicate a typical Pipeline setup
Typical Pipeline layout — schematic, not verified wiring
Ingredients
- HumanHuman operator
- BuilderStage 1 agent
- BuilderStage 2 agent
- QAQA reviewer
- ResourceOutput artifact
Setup order
- 1.Provision the substrate: Output artifact.
- 2.Wire Stage 1 agent: it receives "directs" from Human operator and sends "hands off to" to Stage 2 agent. Wire Stage 2 agent: it receives "hands off to" from Stage 1 agent and sends "hands off to" to QA reviewer. Wire QA reviewer: it receives "hands off to" from Stage 2 agent.
- 3.Give QA reviewer an independent workspace/verdict channel: "delivers to" to Output artifact.
- 4.Declare the human gate: Human operator holds final approval.
Performance metrics
Windowed metrics with provenance. [unknown] means it was not tracked — an honest hole beats an invented figure.
148.2 seconds average per software development task. Source: arXiv 2307.07924 Table 3 [evidence_linked]
Average token usage per software task (Table 3). Files generated: 4.39; lines of code: 144.3. Source: arXiv 2307.07924 [evidence_linked]
vs GPT-Engineer 0.36, MetaGPT 0.41. Source: arXiv 2307.07924 [evidence_linked]
Human evaluation: 77% of ChatDev tasks rated better than GPT-Engineer. Source: arXiv 2307.07924 [evidence_linked]
Token economics
Cost transparency is part of the honesty architecture. [unknown] means it was not tracked — not that it is zero.
Blueprint
Operational DNA — why it works, how it was built, and how it is overseen. Not files for sale; knowledge of the design.
Dual-agent dialogue (instructor + assistant) at each subtask stage enforces review before proceeding. Natural language bridging design and debugging reduces format translation errors. Communicative dehallucination is built into the dialogue structure rather than requiring separate verification agents.
Chat chain organizes sequential phases and subtasks. Natural language used for design work; programming language for debugging. Executability: 0.88 vs 0.36 (GPT-Engineer) and 0.41 (MetaGPT). Quality score 0.3953 vs 0.1419 (GPT-Engineer) and 0.1523 (MetaGPT). Files generated per task: 4.39; lines of code: 144.3.
"Communicative dehallucination" built into the dialogue structure — the instructor role checks and redirects the assistant's outputs, reducing error propagation across phases.
Proof (1)
The team's shared track record — tasks, incidents, lessons, milestones. Per-entry provenance tags are always visible.
- ArtifactJul 14, 2023evidence-linked
ChatDev paper published (arXiv 2307.07924)
Five-role sequential pipeline. Avg 22,949 tokens and 148.2 seconds per software task. Executability 0.88 vs 0.36 (GPT-Engineer). Wins 77% of comparisons vs GPT-Engineer (GPT-4 evaluation).
https://arxiv.org/abs/2307.07924
Sign in to add a proof entry.
Sign inAttestations (0)
Named third-party statements from people with first-hand experience. Attestations are what separates Peer-Attested from Evidence-Linked.
No attestations yet. Worked with this configuration or agent? Attest to it using the form below — attestations are named third-party statements and are what separates Peer-Attested from Evidence-Linked.
Sign in to attest to this team.
Sign in