Open-Source LLMs for Coding and Security Audits: Capability Limits and a Deployment Checklist
Open-source large language models can already handle substantial coding, agent, and security-audit work, but their output still needs human review and their execution rights must stay tightly limited. This article covers what they can do in code engineering, which boundaries to set before agents run, how model-suggested security fixes can fall short, and why teams should evaluate models on their own tasks before automating any step.
Start with the conclusion
Open-source large language models can already do a meaningful amount of real work in code engineering, agent tasks, and security audits. Their output still needs human review, and their execution permissions need to be tightly constrained. Before an organization adopts them, it should evaluate them on its own tasks and only then decide which steps can be automated.
1. Code engineering: what models can do and how to call them
Models have improved noticeably at multi-file code generation, code review, and refactoring suggestions. Given interface definitions, table schemas, and project conventions, they can produce business code with relatively consistent style. They can also flag performance issues such as N+1 queries and poorly structured nested loops.
These results depend on input quality. Putting architectural constraints, coding standards, dependency versions, and the relevant files into the prompt usually works better than providing only a code snippet. For very large codebases, split the code first and retrieve relevant fragments as needed, rather than placing the whole repository in context.
The example below reads the endpoint and key from environment variables, sets a timeout, and retries a limited number of times with exponential backoff for rate limits and server errors:
import osimport timeimport requestsAPI_URL = os.environ["LLM_API_URL"]API_KEY = os.environ["LLM_API_KEY"]MODEL_NAME = os.environ["LLM_MODEL_NAME"]TIMEOUT_SECONDS = 60MAX_RETRIES = 3def review_code_with_llm(code_snippet: str, project_context: str) -> str:"""Ask an LLM to review code and return its text output. The output needs human review."""headers = {"Content-Type": "application/json","Authorization": f"Bearer {API_KEY}",}prompt = ("You are a senior architect. Review the following code snippet, ""identify security vulnerabilities and performance issues, and suggest refactors.\n"f"Project context: {project_context}\n"f"Code snippet:\n{code_snippet}")payload = {"model": MODEL_NAME,"messages": [{"role": "user", "content": prompt}],"temperature": 0.2,}last_error = "unknown error"for attempt in range(1, MAX_RETRIES + 1):try:resp = requests.post(API_URL, json=payload, headers=headers, timeout=TIMEOUT_SECONDS)if resp.status_code == 429 or resp.status_code >= 500:last_error = f"HTTP {resp.status_code}"else:resp.raise_for_status()return resp.json()["choices"][0]["message"]["content"]except requests.HTTPError:raise # Other 4xx errors are problems with the request itself; do not retryexcept requests.RequestException as exc:last_error = str(exc)if attempt < MAX_RETRIES:time.sleep(2 ** attempt)raise RuntimeError(f"LLM call failed after {MAX_RETRIES} retries: {last_error}")
Two points matter after the call. First, vulnerability findings and refactored code from the model can be wrong, so an engineer should review them before merging. Second, the example assumes an OpenAI-style response. When connecting another platform, check the response fields.
2. Agent tasks: set boundaries before automatic execution
An agent can adjust commands based on terminal errors. For example, when it hits a dependency conflict, it can try changing versions and rerunning the build. This sense–decide–act loop is useful for builds and troubleshooting, but without an upper bound it can create cost and damage.
Set at least the following limits:
- A maximum number of steps, such as no more than 20 steps per task.
- A timeout for each command, and a maximum number of retries for the same command.
- A command allowlist that blocks high-risk actions such as
rm -rfand direct connections to production databases. - Read-only by default. Write actions such as pushing code, deploying, and deleting files must go through human approval.
- A snapshot or a separate branch before changes, so a failure can be rolled back.
- A full record of each step’s command, output, and reason for the decision, so the run can be audited later.
In CI/CD or operations-alert workflows, an agent can pull logs, locate the relevant code, and submit a draft fix. A human should review the draft pull request before it is merged. The agent should not have permission to merge directly into the main branch.
3. Security audits: finding issues and completing the defenses
Models now go beyond simple SQL injection and XSS detection in security audits. They can trace how user input flows through a system, point out issues such as unauthorized access and business-logic flaws, and suggest fix code.
A suggested fix is not necessarily complete. For SSRF, an IP allowlist alone is not enough. You also need to:
- Restrict protocols to those the business actually needs, such as HTTPS.
- Restrict ports to those the business needs open.
- Resolve the domain name and check the final IP, blocking private, loopback, and link-local addresses.
- Guard against DNS rebinding. After validation, the connection should use the same IP that was checked.
- Re-validate the destination after a redirect, or disable automatic redirect following.
- Block cloud metadata addresses, such as 169.254.169.254, and enable the cloud provider’s authentication mode for the metadata service.
- Deny outbound traffic by default at the network layer, and allow only explicitly required destinations.
4. Enterprise rollout: selection, deployment, and data protection
1. Evaluate on your own tasks first
Public leaderboards are only a reference. A team should select real tasks from its own work.
