Compare
Comparing 2 teams
Row-aligned side-by-side. Highlighted cells differ across columns. [unknown] cells are honest gaps, never hidden.
Save one comparison in this browser. Not synced to your account; configurations may change.
Found your fit?
Send a setup request that references these 2 teams. Your request is saved in the admin inbox for AgentCV to review. On the next page you can choose to share your name, email and message with the owners of the teams that have one.
- MetaGPT Product Manager · Product Manager
MetaGPT
[unknown]
- MetaGPT Architect · Architect
MetaGPT
[unknown]
- MetaGPT Project Manager · Project Manager
MetaGPT
[unknown]
- MetaGPT Engineer · Engineer
MetaGPT
[unknown]
- MetaGPT QA Engineer · QA Engineer
MetaGPT
[unknown]
- ChatDev CEO · CEO
ChatDev
[unknown]
- ChatDev CTO · CTO
ChatDev
[unknown]
- ChatDev Programmer · Programmer
ChatDev
[unknown]
- ChatDev Reviewer · Reviewer
ChatDev
[unknown]
- ChatDev Tester · Tester
ChatDev
[unknown]
SoftwareDev benchmark; vs ChatDev 248.9. Source: arXiv 2308.00352 [evidence_linked]
With executable feedback loop. MBPP: 87.7% Pass@1. Source: arXiv 2308.00352 [evidence_linked]
3.75/4; vs ChatDev 2.25. Source: arXiv 2308.00352 Table 3 [evidence_linked]
vs GPT-Engineer 0.36, MetaGPT 0.41. Source: arXiv 2307.07924 [evidence_linked]
With executable feedback loop. Source: arXiv 2308.00352 [evidence_linked]
148.2 seconds average per software development task. Source: arXiv 2307.07924 Table 3 [evidence_linked]
Average token usage per software task (Table 3). Files generated: 4.39; lines of code: 144.3. Source: arXiv 2307.07924 [evidence_linked]
GPT-4 evaluation: 77.08% win rate vs GPT-Engineer (human evaluation: 90.16%), Table 2. Source: arXiv 2307.07924 [evidence_linked]
| Field | ||
|---|---|---|
| Topology | Pipeline | Pipeline |
| Wiring | ||
| Human gates | — | — |
| Agents | 5 agents | 5 agents |
| Platform | MetaGPT | ChatDev |
| Runs on | MetaGPT ×5 | ChatDev ×5 |
| Roster |
|
|
| Industries | software-delivery | software-delivery |
| Task kinds | software-developmentrequirements-analysiscode-generationqa | software-developmentcode-reviewqadesign |
| Operating since | Aug 1, 2023 | Jul 14, 2023 |
| Trust tier | From public sources · Self-Reported | From public sources · Self-Reported |
| Proof entries | 1 total(1 with external links) | 1 total(1 with external links) |
| Oversight | SOP verification at each pipeline stage — agents check intermediate results against structured specifications. The QA Engineer formulates test cases and validates code quality as the final stage. | "Communicative dehallucination": before giving a final response, the assistant asks the instructor for more specific details. No human oversight is described in the paper. |
| Source | CuratedReference design | CuratedReference design |
| Metrics | ||
| Tokens per line of code | 124.3 evidence-linkedas of Aug 1, 2023 SoftwareDev benchmark; vs ChatDev 248.9. Source: arXiv 2308.00352 [evidence_linked] | [unknown] |
| HumanEval Pass@1 | 85.9% evidence-linkedas of Aug 1, 2023 With executable feedback loop. MBPP: 87.7% Pass@1. Source: arXiv 2308.00352 [evidence_linked] | [unknown] |
| Executability score (SoftwareDev) | 3.75 evidence-linkedas of Aug 1, 2023 3.75/4; vs ChatDev 2.25. Source: arXiv 2308.00352 Table 3 [evidence_linked] | 0.88 evidence-linkedas of Jul 14, 2023 vs GPT-Engineer 0.36, MetaGPT 0.41. Source: arXiv 2307.07924 [evidence_linked] |
| MBPP Pass@1 | 87.7% evidence-linkedas of Aug 1, 2023 With executable feedback loop. Source: arXiv 2308.00352 [evidence_linked] | [unknown] |
| Avg task duration | [unknown] | 148.2s evidence-linkedas of Jul 14, 2023 148.2 seconds average per software development task. Source: arXiv 2307.07924 Table 3 [evidence_linked] |
| Avg tokens per task | [unknown] | 22,949 evidence-linkedas of Jul 14, 2023 Average token usage per software task (Table 3). Files generated: 4.39; lines of code: 144.3. Source: arXiv 2307.07924 [evidence_linked] |
| Win rate vs GPT-Engineer | [unknown] | 77% evidence-linkedas of Jul 14, 2023 GPT-4 evaluation: 77.08% win rate vs GPT-Engineer (human evaluation: 90.16%), Table 2. Source: arXiv 2307.07924 [evidence_linked] |