Autonomous bash — qwen3.5:2b as a worker · V1
Observation Suite 2B
A 2-billion-parameter local model is given 50 plain-English commands and asked to translate each into the correct bash command. The output is executed against a captured expected output. We measure exact match, valid alternative, partial match, and failure.
- accuracy (exact + valid)
- 56%
- strict exact
- 42%
- hard failure
- 1
- avg per command
- 20.97s
What 2B can and can't do
| Category | Accuracy | Notes |
|---|---|---|
| system_info | 80% | Highest accuracy — model handles ls / pwd / uname patterns well |
| network | 60% | Network introspection (ip / netstat / ss / ping). Some pipe-chain failures. |
| file_ops | 60% | File-system operations. Predictable patterns translate cleanly. |
| process_mgmt | 50% | ps / kill / nice / a detached background process. Errors in nested process queries. |
| security | 30% | Hardest category. iptables, auditd, sudo, file permissions, key handling — confused by syntax variants. |
What we learned
2B is viable for system_info
80% accuracy on info gathering
The model can be deployed as an orchestrator subordinate for safe read-only command translation. Use it for uname, uptime, df, free, basic ls / find / cat.
Security is dangerous
30% accuracy
Do not delegate security operations (iptables, sudo, key management) to a 2B model. The 70% wrong-answer rate at this scale crosses into actual risk territory.
Pipe chains break
Only failure was top -bn1 | head -5
The model emitted a valid command that produced empty output. Pipe-composed commands with non-trivial behavior are the hardest single class.
Production pattern
2B worker + large orchestrator
The pattern is to have a large model (Claude / Codex / GPT-5) plan, then dispatch atomic translate-to-bash commands to a 2B worker. Saves 95%+ of inference cost on the routine.