LLMoxie RSE Sandbox
Designing a Reproducible Environment for AI-Assisted Science (Research Software Engineering)
Context
University of Washington eScience Institute (Scientific Software Engineering Center)
Winter–Spring 2026
Team
Interaction Researcher (Me), Research Software Engineer, Product Manager
My role
Project status
Researcher-facing experimental GitHub Codespaces environment
Product overview
Generic AI coding assistants are optimized for commercial software development. They frequently fail in scientific contexts where code is judged not just by whether it runs, but whether it can be cited, audited, and reproduced. Off-the-shelf agents ignore the conventions of scientific Python libraries, mishandle embargoed data, and leave behind untraceable decision trails.
To address this, the UW eScience Institute developed LLMoxie: an institutional AI platform and RSE (Research Software Engineering) plugin ecosystem. My role focused on the LLMoxie RSE Sandbox—the researcher-facing workspace where these capabilities actually meet the user.
Our initial goal was to build functional infrastructure. But as we integrated models, scientific tooling, and chat interfaces, the challenge shifted. Building the infrastructure revealed the workflow we actually needed to design.
The Baseline: A Constrained Starting Point
To create a reproducible environment, the baseline implementation combined GitHub Codespaces, a Pixi-managed scientific Python environment, and a secure connection to the LLMoxie API Gateway.
[IMAGE: Simplified Ecosystem Diagram — Visual redrawn from the presentation showing the Researcher connecting to GitHub Codespaces, routing to the LLMoxie Model Provider, and accessing RSE Agent Skills]
Getting these components to communicate revealed hidden dependencies. The sandbox and the model provider disagreed on configuration conventions, and acquiring remote extensions raised provenance and security questions. As an engineering task, we solved this by pinning artifacts and moving toward compatible prebuilt images. As a design task, this exposed a critical user problem: a connected model did not automatically equal a usable scientific workspace.
A researcher opening a containerized environment faces immediate ambiguity. They need to know if the setup succeeded, if the AI model is listening, and how to invoke specialized scientific skills.
ONBOARDING SEQUENCE
Enter → Ready → Configure → Select → Work
We structured the onboarding experience into a legible, five-step path: Enter: Open the provisioned Codespace. Ready: System signals successful startup and environment preparation. Configure: Verify secure model-provider connectivity. Select: Choose an available UW SSEC-governed model. Work: Invoke RSE workflows through Copilot Chat or the CLI. By treating researcher entry as an explicit system layer, we transformed a collection of technical capabilities into an accessible tool.
One of our most consequential discoveries was how deeply interface mechanisms dictate AI behavior.
Prompt wrapper
One-shot response
Native Agent Skill
Multi-turn collaborative dialogue
We initially introduced commands like /research and /plan using standard chat prompt wrappers. However, this interaction pattern forced the agent to generate flat, one-shot responses. When we integrated direct, native Agent Skills (e.g., /researching), the interaction changed entirely. Native commands successfully invoked the underlying interactive behavior, allowing the agent to ask clarifying questions, evaluate context, and collaborate with the researcher. The technical integration choice fundamentally altered the user’s conversational depth.
The Research Loop: Correcting Assumptions with Evidence
COLLABORATIVE · Research → Plan → Experiment · pause for input
DIRECT · Implement → Validate → Reproduce · execute with defined stops
Initially, our documentation assumed a uniform flow across all six phases. However, by reviewing recorded skill sessions, we observed two distinct human-AI interaction patterns: Collaborative Phases (Research, Plan, Experiment): The agent naturally paused to request scoping input, present design options, and wait for the researcher’s approval. Direct Phases (Implement, Validate, Reproduce): The agent proceeded autonomously through execution, hitting defined hard stops (like branch limits or verification pauses) without needing continuous conversational hand-holding. We revised our documentation and mental models to reflect this observed reality, proving that actual usage data must override speculative workflow design.
Securing the Validation Boundary
README.md
→ quarantine →
GROUND_TRUTH.md
During evaluation, we realized the agent could potentially scrape expected outcomes from standard dataset README.md files, giving the illusion of analytical success. We explicitly redesigned the repository architecture to quarantine this data into a separate GROUND_TRUTH.md file, excluding it from the agent’s intended path. This established a boundary that protected the integrity of the demonstration.
Outcomes & Reflections
The LLMoxie RSE Sandbox successfully provided a reproducible environment that made parts of the scientific AI workflow observable.
Key Takeaways: Access Capability: Providing a researcher with an API key and a chat window is insufficient. They require structured onboarding and explicit invocation paths to do meaningful work. Interaction Dictates Inference: The UI mechanism chosen to trigger an LLM (wrappers vs native skills) directly limits or expands the model’s ability to act as a collaborative partner. Research State Must Persist: While chat interactions are ephemeral, the scientific record—artifacts, code, tests, and git history—must be durable and inspectable. Building this sandbox reinforced that designing for agentic AI is not just about crafting better prompts; it is about architecting the socio-technical environment in which those prompts occur.