A code tool for an LLM, tested without a sandbox

An agent's code tool runs code and describes the Execution. That description, with errors and output limits, is testable with recorded sandbox messages.

Exampletool.py

from e2b_code_interpreter.models import Execution

MAX_CHARS = 2000


def describe(execution: Execution) -> str:
    """The text a model reads back after its code ran."""
    parts = []
    if execution.logs.stdout:
        parts.append("stdout:\n" + "".join(execution.logs.stdout))
    if execution.logs.stderr:
        parts.append("stderr:\n" + "".join(execution.logs.stderr))
    if execution.error:
        parts.append(f"error: {execution.error.name}: {execution.error.value}")
    elif execution.text is not None:
        parts.append("result: " + execution.text)
    return "\n".join(parts)[:MAX_CHARS] or "The code ran and printed nothing."


def run_python(sandbox, code: str) -> str:
    """The tool: run model-written code in the sandbox and describe what happened."""
    return describe(sandbox.run_code(code, timeout=30))

describe turns an Execution into short text: output, the error if any, otherwise the result.
Errors are described, not raised, so the model can read them and fix its code.
MAX_CHARS keeps a runaway print from filling the model's context.
run_python sets a 30 second limit on every call.

Examplerecorded.py

import json

from e2b_code_interpreter.models import Execution, parse_output


class RecordedSandbox:
    """Answers run_code from recorded sandbox messages.

    It runs nothing and isolates nothing: it replays what a sandbox sent, so code
    that handles an Execution can be tested without an E2B key.
    """

    def __init__(self, messages):
        self.messages = messages
        self.code = []

    def run_code(self, code, **kwargs):
        self.code.append(code)
        execution = Execution()
        for message in self.messages:
            parse_output(execution, json.dumps(message))
        return execution

RecordedSandbox has the one method the tool uses. It replays messages through the SDK's real parse_output, so the tool gets the same Execution objects a sandbox would produce. It is a test double, not a sandbox: it executes nothing.

Exampletest_tool.py

from recorded import RecordedSandbox
from tool import MAX_CHARS, run_python


def test_result_and_output_are_reported():
    sandbox = RecordedSandbox([
        {"type": "stdout", "text": "rows: 3\n", "timestamp": 1},
        {"type": "result", "text": "120.5", "is_main_result": True},
    ])
    assert run_python(sandbox, "df.total.sum()") == "stdout:\nrows: 3\n\nresult: 120.5"
    assert sandbox.code == ["df.total.sum()"]


def test_errors_are_reported_not_raised():
    sandbox = RecordedSandbox([{"type": "error", "name": "KeyError", "value": "'total'", "traceback": "..."}])
    assert run_python(sandbox, "df['total']") == "error: KeyError: 'total'"


def test_huge_output_is_cut():
    sandbox = RecordedSandbox([{"type": "stdout", "text": "x" * 50_000, "timestamp": 1}])
    assert len(run_python(sandbox, "print('x' * 50_000)")) == MAX_CHARS

Example

pytest -q

The tests fix three behaviours a model depends on: output and result are reported together, an error is described rather than raised, and huge output is cut. None of them needed a key.

Try it yourself

Add a test for a cell that prints nothing and returns nothing.
Report stderr after stdout and test it.
Add the traceback's last line to the error description.

PreviousSecurity settings: network, secrets and timeouts

Slow is fine. Stopping is the only problem.