The test was not a benchmark suite or a collection of LeetCode problems. It was a real feature: a rate-limited async job queue with retry logic, dead letter handling, and a Redis backend, implemented from scratch in Python. Every assistant got the same spec. No hints, no scaffolding, no pre-written tests. Here is what each one actually produced.
Analysis Briefing
- Topic: AI coding assistant comparison on a real-world Python task
- Analyst: Mike D (@MrComputerScience)
- Context: A structured investigation kicked off by Claude Sonnet 4.6
- Source: Pithy Cyborg | AI News Made Simple
- Key Question: When you give five AI coding assistants the same real engineering task, which one actually ships working code?
The Task and Why It Was Chosen
The spec was deliberately non-trivial. An async job queue in Python using Redis with the following requirements: workers pull jobs via BLPOP with a configurable timeout, failed jobs retry up to a maximum attempt count with exponential backoff, jobs that exceed max retries move to a dead letter queue, the system must handle Redis connection failures gracefully without dropping jobs, and the whole thing needed type annotations and unit tests.
This is representative of real backend work. It requires knowing asyncio, Redis data structures, error handling patterns, and enough production awareness to think about what happens when Redis goes down. It is not a trick question or an obscure algorithm. It is the kind of task a mid-level Python engineer would be assigned on their first week.
The five tools tested: GitHub Copilot (in VS Code), Cursor (using Claude Sonnet backend), Codeium (free tier), Claude at claude.ai (direct, not IDE-integrated), and ChatGPT-4o (direct).
Results: What Each Tool Actually Produced
GitHub Copilot excelled at autocomplete within a file I was already writing. When I wrote the class skeleton and method signatures, Copilot completed the method bodies accurately about 70% of the time. When I asked it to generate the entire module from the spec via the chat panel, it produced code with the right structure but missed the dead letter queue entirely and used synchronous Redis calls in an async context, a subtle bug that would surface under load.
Cursor was the strongest overall performer on this task. Its multi-file awareness meant it could see that I had already written a config.py with Redis connection settings and referenced those correctly in generated code without being told. The initial generation included the dead letter queue, correct use of asyncio primitives, and a reasonable retry implementation. The tests it generated actually ran. Two of them had wrong assertions but the structure was sound.
# Cursor generated this correctly on the first attempt
async def process_job(self, job_data: dict) -> bool:
attempt = job_data.get('attempts', 0) + 1
try:
await self._execute(job_data['payload'])
return True
except Exception as e:
if attempt >= self.max_retries:
await self.redis.rpush(self.dead_letter_queue, json.dumps({
**job_data,
'failed_at': time.time(),
'error': str(e)
}))
return False
delay = self.base_delay * (2 ** attempt)
await asyncio.sleep(delay)
await self.redis.rpush(self.queue_name, json.dumps({
**job_data,
'attempts': attempt
}))
return False
Codeium on free tier produced the weakest result. The generated code used redis-py synchronously in an async context throughout, type annotations were absent despite being in the spec, and the retry logic was a flat retry count with no backoff. It is useful for autocomplete on well-trodden patterns. It struggled with the async requirements here.
Claude at claude.ai (direct, no IDE) produced the most technically correct implementation when prompted well. It caught an edge case the spec did not mention: a job that fails during deserialization should go directly to the dead letter queue rather than retrying, because retrying malformed JSON will always fail. No other tool flagged this. The limitation is the absence of file context: it cannot see your existing codebase without manual copying.
ChatGPT-4o (direct) was close to Claude in code quality. It correctly implemented exponential backoff and the dead letter queue. It over-explained in comments and added abstractions (a JobStatus enum, a RetryPolicy dataclass) that were not in the spec and that I would not have written myself. Whether that is good or bad depends on your preferences. The tests it generated were comprehensive but used unittest.mock.AsyncMock incorrectly in one case.
The Honest Verdict
For greenfield Python in a modern IDE, Cursor is the current leader because cross-file context awareness is the feature that produces the most real leverage on actual projects. Copilot is the safer enterprise choice because of SSO, audit logging, and the Anthropic/OpenAI model flexibility in the business tier, but it is behind Cursor on raw output quality for complex generation tasks.
Claude and ChatGPT direct are the right choice when you need to think through a design rather than generate code. The conversation format works well for architecture decisions. It works poorly for staying synchronized with a growing codebase.
Codeium free tier is fine for tab completion. It is not competitive on complex generation tasks at this writing.
The category is moving fast. Every tool in this list was measurably better six months ago than it was a year ago. The gap between first and fifth place is smaller than it appears on paper because all of them require skilled developers who can evaluate and correct generated code. None of them are a replacement for understanding what the code should do.
What This Means For You
- Use Cursor for complex multi-file Python work if you are an individual contributor or small team without enterprise procurement constraints, because cross-file context awareness is the feature that produces the most leverage on real projects.
- Keep Claude or ChatGPT open alongside your IDE for architecture discussions and spec clarification, because the conversational format handles design reasoning better than the inline chat in any IDE assistant.
- Always review generated async code for sync/async mismatches before running it, because every tool in this test produced at least one case of sync Redis calls in an async context, and this class of bug does not surface in unit tests but causes thread pool exhaustion under load.
- Write the spec before prompting, because the quality of AI-generated code correlates directly with the specificity of the prompt, and a three-sentence spec produces dramatically better output than asking for “a job queue.”
Enjoyed this deep dive? Join my inner circle:
- Pithy Cyborg | AI News Made Simple → AI news made simple without hype.