Known Limitations

The honest gaps - what is proven against a real running app and what is not.

Known Limitations

BxAgents is under active development. This page tracks the honest gaps - what's tested against a real running app, what still only runs against bx-ai's "mock" provider, and real upstream quirks this project ran into.

Most testing runs against the mock provider - one real LLM round trip now runs in CI

Every fast-lane spec (build pipeline, generators, CLI verbs) and the ColdBox integration suite exercise bx-ai's built-in "mock" provider - never a real network call to an LLM. This is deliberate (fast, free, deterministic CI). It means no automated test proves any of OpenAI/Anthropic/Gemini/etc.'s own provider-specific request/response shaping actually works against their real APIs - do at least one manual chat/serve run against the real provider you intend to use and a throwaway API key before depending on this in production.

One narrow but real gap this used to leave is now closed: tests/specs/integration/LiveLlmSpec.bx runs a genuine LLM round trip - a plain completion, and a real tool-call turn (the model actually deciding to invoke a BoxLang closure, not a scripted mock response) - against Novita AI's OpenAI-compatible endpoint, via bx-ai's own "openai-compatible" provider (OpenAICompatibleService.bx - a real, existing bx-ai provider this required no changes to). Gated on the NOVITA_API_KEY repository secret via each it()'s own skip argument (TestBox's real conditional-skip mechanism): CI (which has the secret) runs it for real; a local ./gradlew testBx/fork PR without it reports the two specs as cleanly "Skipped," never a failure. This proves bx-ai's HTTP call/response-parsing/tool-calling machinery is sound against a real model - it does not extend to any provider besides the OpenAI-compatible surface Novita implements.

Real ColdBox integration testing

./gradlew testColdBoxIntegration boots a real boxlang-miniserver process against a generated app's own Application.bx/Bootstrap, and makes a genuine HTTP request through a toAi()-registered route. This is the strongest proof point in the suite - it caught three real bugs in ColdBoxAppGenerator.bx during development (a missing WireBox Binder extends, wrong Bootstrap constructor argument order, and a bare getInstance() call that doesn't exist on Binder).

The toAi() first-request race

The very first HTTP request to a freshly booted app's toAi() route can transiently fail with "Function [getInstance] not found" - a genuine ColdBox/WireBox lazy-injection race on the Router's own getInstance delegate, not a BxAgents bug. It succeeds reliably on every request after something else has already forced WireBox to build the GeneratedAgent singleton once (a health check, a chat session, another route). Send a warm-up request before relying on a freshly deployed toAi() route under load.

Three integration checks - closed via a different route than originally planned

tests/specs/integration/RuntimeStartupSmokeSpec.bx still has three xit()s, since this file runs via the CLI runner (runTests.bxs/testBx), and BoxLang's CLI mode never gives a cgi scope - which ColdBox's RoutingService needs even to load the router at startup. That's a structural fact about CLI mode, not a gap: all three checks are now proven for real elsewhere, in tests/specs/integration/coldbox/ColdBoxRuntimeSpec.bx (which runs inside a real HTTP request served by a real boxlang-miniserver process):

• A real ColdBox-routed HTTP request reaching the generated agent end-to-end - proven by ColdBoxRuntimeSpec.bx plus runColdBoxIntegrationTests.bxs's own POST /api/chat/invoke assertion.
• schedules/* actually registering with a live ColdBox Scheduler - proven via SchedulerService.getSchedulers()["appScheduler@coldbox"].hasTask(...) against a real boot, without waiting out an actual cron fire (the coarsest supported granularity is 1 minute, which would tax every CI run for marginal extra proof once the task is confirmed live and registered).
• chat and a served HTTP route never diverging - corrected from its original wording (chat deliberately never boots WireBox at all, so it can never share WireBox's singleton object by design) to what actually matters: instantiating GeneratedAgentFactory outside of WireBox produces a behaviorally-equivalent agent to WireBox's own singleton.

Fixed: real OS-process CLI testing caught a real module-resolution bug

ModuleCliProcessTest.java spawns genuine java -jar <boxlang-jar> module:bxagents <verb> ... child processes against a copy of the actual installable module structure (build/modules/bxagents), pointed at a real modulesDirectory - exactly how a real BoxLang installation loads this module. Every other CLI spec calls ModuleConfig.main()/each verb's run() in-process instead, which is faster but never proves the module resolves correctly once genuinely installed. This test caught every CLI verb failing with "class not located" the moment it ran through an actual installed-module process, because internal cross-references used a bare bxagents.models.... dotted path that only resolved thanks to this repo's own dev/test boxlang.json (a hand-declared /bxagents mapping) - a genuinely installed module never gets that mapping. Fixed by switching every nested class's internal references to the Class@bxagents module-relative suffix form (see BuildPipeline.bx's init() docblock for the full explanation); build.gradle's module-structure output moved from build/module to build/modules/bxagents (folder name must equal the module name for modulesDirectory discovery to find it at all), and the dev/test boxlang.json now also loads it as a real module.

Fixed: a built .bxa was not actually portable

Every runtime path the generated agent/GeneratedAgentFactory.bx needed - its own tools/ directory, each agent node's instructions.md, and each class-based node's copied Agent.bx - was baked in as the absolute path it had on the machine that built it. Building on CI at /home/runner/work/x and unzipping the .bxa at /opt/x therefore threw on the first getInstance( "GeneratedAgent" ), and simply renaming a project directory broke a local build the same way - despite package being documented as producing "a portable .bxa". instructions.md was worse still: its baked path pointed back into the source project, a file package never even put in the archive.

Fixed in ColdBoxAppGenerator.bx, in two parts:

• Every agent node's instructions.md is now copied into the app, mirroring the source layout under agent/: the root project's at .build/app/agent/instructions.md, each subagent's at .build/app/agent/subagents/<name>/instructions.md (see Build Pipeline).
• The generated factory resolves all three paths at runtime through its own emitted resolveAppPath( relative, fallback ) helper, which locates them relative to the factory's own file via getCurrentTemplatePath() - the same self-location WebUiGenerator already relies on.

The build-time absolute path is still emitted, as that helper's fallback: self-location is proven for a module-loaded class but not independently verified for the DynamicClassLoader load path chat/invoke use, so if it ever fails to resolve, a built app behaves exactly as it did before - never worse.

One consequence worth knowing: a .bxaignore pattern broad enough to match *.md will strip the agent's own copied instructions.md out of the .bxa.

Fixed: a cluster of real bugs found building the testing framework

Building BaseAgentSpec's toHaveCalledTool matcher and the ColdBox integration harness surfaced - and fixed - a run of real, previously-undetected bugs. Recorded here as a changelog of lessons learned, not as open gaps:

• Agents never received their own tools. ColdBoxAppGenerator's generated aiAgent() call never passed a tools: argument at all - confirmed against real bx-ai source (AiAgent.bx never references aiToolRegistry() internally). No agent BxAgents ever built, in any context, actually received its declared tools; undetected because no test asserted on a real tool invocation, only a non-empty response. Fixed in ColdBoxAppGenerator.renderAgentFactory(): every generated GeneratedAgentFactory.bx now loads its own tools/ directory via ToolRegistryLoader.bx (a real filesystem path - aiToolRegistry().scan("tools")'s mapping-relative resolution silently fails from a DynamicClassLoader-loaded context like chat, and that rejection still stands), then passes tools: aiToolRegistry().getAll() to every aiAgent() call. That path started out as the absolute path of the machine that built the app; it is now resolved at runtime relative to the generated factory's own location, with the build-time absolute kept only as a fallback - see the portability entry below.
• var cannot be used at the top level of a .bxs script. It declares into the local scope, which only exists inside a function, so a top-level var x = ... throws Scope [local] is not available in this context at runtime, not parse time - it survives review and only fires on the code path that reaches it. Never write var outside a function in a .bxs, and wrap diagnostic output in its own try/catch so it can never mask the failure it's describing.
• The Application setting for a default datasource is this.datasource, NOT this.defaultDatasource. The latter is silently accepted and does nothing - queryExecute() still fails with No default datasource defined..., naming the registered datasource you're trying to select, which reads as though selection is broken rather than the setting being misspelled.
• qb's moduleSettings.qb.defaultOptions did not reach QueryBuilder in a real ColdBox boot, even though qb's own onLoad() wiring reads as correct - every query failed with "no default datasource" despite the datasource being registered. ChatDb.query() now calls .mergeDefaultOptions( { datasource : static.DATASOURCE } ) on every builder explicitly rather than depending on that plumbing.
• A generated app now also declares this.defaultDatasource, so anything that runs a query without naming one (ColdBox itself, another module) reaches the one datasource instead of failing. ColdBoxAppGeneratorSpec covers both the web-UI and no-web-UI cases.
• The generated models/ChatDb.bx did not compile, and no unit spec noticed. A BoxLang string-literal quoting bug (?: "" emitted as ?: ", opening an unterminated string) shipped in every generated web-UI project; every WebUiGeneratorSpec case only grepped generated text, never tried to compile it. There's now a case asserting every generated line has an even number of double quotes - the wider lesson being that a generator spec that only greps its own output doesn't test that the output is valid code.
• GET /chat/api/health proves far less than a stale comment claimed. The generated health() action renders a literal 200 and touches nothing else - it says nothing about WireBox, ChatDb, or SQLite. A comment in runColdBoxIntegrationTests.bxs used to claim otherwise; corrected in place.
• TestBox's JSONReporter StackOverflow'd on specs holding live framework singletons (real controller/WireBox/qb objects reflected recursively during serialization) - a passing suite reported as a failed run. tests/runner-coldbox.bxm now assembles its own report via a toScalar() helper instead, and records pass/fail counts before the report is built.
• In a BoxLang time mask, nn is NANOSECONDS, not minutes - use mm. (Note the inverse trap in a date mask, where mm is the month.)
• Two handlers/ChatUi.bx actions had an authorization hole despite deriving identity correctly elsewhere. /pending and /resume were addressed by threadId alone, so a visitor with someone else's threadId could read their pending tool calls and answer their approvals - deriving identity server-side buys attribution, not access control. Both now compare the caller against the userId the agent checkpointed in the run options.
• ProjectValidator's path check missed .. traversal, catching only a leading separator - ../../var/lib/chat.db passed cleanly. Now rejected on both, matching whole path segments so legitimate names like ..hidden still pass.
• chr() does not exist in BoxLang - the BIF is char(). Cost several CI cycles because the failure is a runtime one that survives review (tests/runner-coldbox.bxm called it in its very first progress marker, so every request 500'd on entry).
• Instrumentation that a retry can overwrite records nothing. A progress file was being truncated on every retry, hiding the real evidence and turning a plain 500 into an opaque "HTTP 408, never responded"; the retries themselves were also actively harmful, stacking concurrent full test runs onto a small worker pool. Fixed: one request only (the health probe ahead of it already proves the server is up), append-only progress log tagged per-request.
• request is a reserved built-in scope name. A loop variable named request silently shadows it with no error at all - fixed by renaming to recordedRequest in BaseAgentSpec.bx's matchers.

serve's miniserver lookup is PATH-only

serve looks for boxlang-miniserver only on PATH (MiniServerLauncher.findExecutable()). There's no fallback to a configured path or a bundled binary - if it isn't installed and on PATH, serve fails with a clear, actionable error, but there's no alternate way to point it at one. invoke --server reuses serve internally, so it inherits the same PATH-only lookup - its own InvokeSpec.bx test for the real HTTP round-trip checks MiniServerLauncher.findExecutable() first and skips (rather than fails) when no real binary is on PATH, the same idiom RuntimeStartupSmokeSpec.bx already uses for its own jar-presence check. Do at least one manual bxAgents invoke --message=... --server run on a machine with boxlang-miniserver installed before depending on the real HTTP path in production - the same honest framing already used elsewhere in this page for gaps this suite can't close itself in every environment.

The scoped BoxLang runtime home reaches serve/invoke --server unconditionally; the in-process verbs only if your BoxLang install loads .env

serve scopes the miniserver's own BoxLang runtime home to .build/runtime via serverHome (a real, confirmed MiniServerConfig field - boxlang-web's own MiniServer CLI help text: -s, --serverHome <PATH> BoxLang server home directory (default: ~/.boxlang)), so each project's compiled-class cache and any config overrides are isolated per-project rather than shared globally at ~/.boxlang. invoke --server inherits this since it reuses serve internally. This part is unconditional - we write the launch config for that process ourselves.

chat, build, test, and default (in-process) invoke are different: they run inside the already-started bxAgents process, whose own BoxLang runtime - and therefore its home - was resolved before any of our BoxLang code, including ModuleConfig.bx's own main(), ever got a chance to run (the engine has to exist to interpret that code at all). BoxRuntime is a JVM-wide singleton; its home is fixed at first initialization, so nothing a verb class does from inside that already-running process can change it retroactively.

There IS a real lever for this, though: BoxRunner (BoxLang's own core CLI entry point, not just the miniserver) reads a genuine BOXLANG_HOME environment variable before BoxRuntime initializes - confirmed directly against the real runtime jar (not just documentation): running it with BOXLANG_HOME=<path> set as a real OS environment variable (relative paths resolve against the CWD, same as an absolute path) populates the full home structure at that path instead of ~/.boxlang, every time. new scaffolds a .env declaring BOXLANG_HOME=.build/runtime - the same path serve uses - for exactly this reason (mirroring ortus-boxlang/bx-ai-intro's own .env/BOXLANG_HOME convention), at the project root - i.e. the directory a user actually runs bxAgents <verb> from for the in-process verbs, which is the right place for a CWD-based .env loader to find it.

What we confirmed, and what we couldn't. boxlang-miniserver (what serve launches) does have real, built-in .env auto-loading - confirmed by decompiling and then actually running ortus.boxlang.web.MiniServer directly: with no envFile configured, it resolves .env relative to the server's webRoot (not the project root) and, if found, loads it as Java Properties, applying each key via System.setProperty() - real values loaded this way ARE visible to BoxLang code via getSystemSetting(). The raw core runtime jar (BoxRunner, used by chat/build/test/default invoke) has no equivalent logic anywhere in it (confirmed by grepping every class in the jar for .env) - so those verbs only pick up .env if the actual boxlang CLI you have installed (a BVM-provided native launcher, not this raw jar) does its own .env loading before the JVM starts, the way ortus-boxlang/bx-ai-intro relies on. That's plausible and consistent with real-world usage of that project, but isn't something this sandbox has the real binary to verify directly.

One specific, confirmed gotcha even where .env loading does happen: BOXLANG_HOME itself does not take effect unless it's a real OS environment variable - not a JVM system property, and not a -D flag. Verified twice directly against the real jars: (1) running boxlang-miniserver against a webRoot whose .env declared BOXLANG_HOME=customhome loaded the file (confirmed via its own "Loaded environment variables from:" log line and via getSystemSetting() correctly returning other .env values), yet the server still logged Logs Directory: /root/.boxlang/logs - the default home, not customhome; (2) launching BoxRunner directly with -DBOXLANG_HOME=<path> (a JVM system property, no .env involved) also had no effect on the resolved home, even though getSystemSetting( "BOXLANG_HOME" ) happily returned the flag's value from BoxLang code. So BOXLANG_HOME resolution specifically reads only the true OS environment variable - unlike most settings, getSystemSetting() returning a value for it doesn't mean the runtime home actually moved. If your boxlang CLI's .env loader works the same way MiniServer's does internally (System.setProperty after the JVM has already started) rather than exporting a real env var before launching the JVM, BOXLANG_HOME in .env won't reach the runtime home even though .env itself loaded successfully - everything else in it will still work. If your install doesn't give you this at all, source it yourself before running commands (e.g. set -a; source .env; set +a) to get the same isolation serve already gets unconditionally.

new's box install convenience step isn't exercised by an automated test

new runs box install inside the scaffolded tests/ folder by default, so bxAgents test works immediately (see CLI Reference). This is a real network operation (CommandBox resolves testbox against ForgeBox) - confirmed to take ~25 seconds and fail with a certificate error in this development sandbox specifically, the same ForgeBox-unreachable constraint already noted elsewhere in this project's own tooling. NewSpec.bx only exercises the fast, deterministic --skipInstall path (and the OS-process ModuleCliProcessTest.java/in-process ModuleConfigCliSpec.bx invocations of new also pass --skipInstall for the same reason) - the default install-attempt path itself is verified by manual testing only, not CI.

new --repo's real clone path is covered by CI, network permitting

RepoCloner.bx clones via a bundled JGit dependency rather than shelling out to a git binary (see CLI Reference), so unlike the box install gap above there is no "is the binary installed" failure mode to skip around - only the network call itself can fail. RepoClonerSpec.bx's live end-to-end test attempts a real clone of a small, stable public GitHub repo and only self-skips if that specific attempt doesn't produce the expected "cloned, but no Agent.bx" outcome. Confirmed empirically in this development sandbox: a real git clone to github.com over HTTPS DOES succeed here (through the same proxy this sandbox's other outbound HTTPS calls use), so this test exercises the real path rather than skipping - a genuinely better position than box install's, whose own separate certificate/proxy configuration fails here.

new --forgebox's real install path is covered by CI, network permitting

ForgeBoxClient.bx talks to ForgeBox's own REST API (https://www.forgebox.io/api/v1) directly via bx:http, rather than shelling out to the box CLI the way new's own tests/ install step above does - confirmed empirically that this development sandbox CAN reach forgebox.io this way, even though the separate box CLI process fails here with a certificate error. ForgeBoxClientSpec.bx/ForgeBoxInstallerSpec.bx cover every not-found/download-failure/missing-Agent.bx code path deterministically via ForgeBoxClient.bx's injectable entryCaller/downloadCaller seams (the same test-seam idiom GitHubGateway.bx's apiCaller already uses) - a real improvement over the old box install-based design, which had no way to unit-test any of those paths at all. ForgeBoxInstallerSpec.bx's one live end-to-end test (installing a real, small ForgeBox package) only self-skips if that specific attempt doesn't produce the expected "installed, but no Agent.bx" outcome - confirmed to actually run, not skip, in this sandbox as of this writing. Real ForgeBox entries' exact JSON shape ({ data, error, pagination, messages }, an entry's latestVersion.downloadURL) was confirmed against the live API (GET .../entry/coldbox), not guessed at.

--repo strips the cloned .git folder - a deliberate choice, not a bug

new --repo always removes the freshly-cloned .git directory before returning, with no flag to keep it. This is intentional: a template repo's own git history and origin remote have no legitimate reason to survive into a fresh project, and leaving them in place is a real foot-gun - a new project's first git init/commit/push could otherwise interact with the template's own upstream repo instead of the user's. Someone who genuinely wants the template's git history can clone it manually instead of using bxAgents new.

chat needs a real TTY

chat uses BoxLang's own MiniConsole, which shells out to stty to set up raw terminal mode - it can only run against a genuine interactive terminal. It doesn't work piped, redirected, or from a non-interactive process (a CI job, a script). There's no non-interactive fallback mode.

chat's streaming has no cancel, and no history beyond the live session

chat --server/--connect stream the reply over the route's /stream (SSE) sub-route, but two things ColdBox's toAi() surface does not offer are therefore missing:

• No cancel. There is no way to interrupt a turn once it starts. toAi() registers no cancel endpoint, and the SSE stream is server-driven - a client can stop reading, but the agent keeps generating to completion on the server. Ctrl-C ends the whole chat process rather than the turn.
• No history. A session's threadId keeps one conversation coherent while chat is open, but nothing replays a prior conversation into a new session: reconnecting with --connect starts fresh. The agent's own memory may still give it recall server-side - what is missing is the client showing you what was said before.

Both are gaps in what the HTTP surface exposes, not in the client - closing either needs a route that does not exist yet.

SSE streaming depends on whitespaceCompressionEnabled being off

chat --server/--connect stream through BoxLang's native SSE: ColdBox's toAi() /stream sub-route emits with the SSE() BIF, and AgentClient.stream() consumes with http( url ).sse( true ).onChunk( ... ). There is no frame parser in this codebase - ortus.boxlang.runtime.net.SSEParser does that.

That works only when the web runtime's whitespace compression is off, and it is on by default. SSE frames are terminated by a blank line; compression collapses consecutive newlines and eats exactly those terminators. Measured against a live /__bxagents/stream:

It is not specific to ColdBox, to toAi(), or to this client. A bare two-line SSE() endpoint loses its terminators the same way (79 bytes, zero \n\n), and so does a handler that writes \n\n itself with writeOutput. A browser EventSource against any of them fires no events either.

Where it is set:

• Serve.bx writes whitespaceCompressionEnabled: false into the served project's own runtime config, under the .build/runtime server home it already scopes per project - so it never touches the machine's ~/.boxlang, and clean sweeps it.
• This repo's boxlang.json sets it for the test suite.

A project served some other way - a different container, a servlet deployment, someone else's boxlang.json - needs the same directive, or its SSE endpoints emit unterminated frames and every SSE client sees nothing. The ColdBox integration run asserts the delta count on a real stream, so a regression here fails the build rather than silently degrading to a non-streaming reply.

Fixed: chat and default (non---server) invoke used to fail for a class-based Agent.bx

Previously, both chat and default invoke threw The requested class [agent.classes.agentClass] has not been located in any class resolver. before ever reaching the agent. Root cause: both verbs load the generated GeneratedAgentFactory.bx in-process via DynamicClassLoader.instantiate() (no ColdBox container involved), and the generated factory used to instantiate a class-based Agent.bx via a relative dotted-path lookup, which only resolves once something has registered a mapping making the app root resolvable - and nothing did outside a real ColdBox boot. Registering one mid-script doesn't fix it either (confirmed empirically).

Fix: ColdBoxAppGenerator.copyAgentClass() now returns the copied class's own absolute file path, and renderClassBasedAgentStatement() instantiates it via DynamicClassLoader.instantiate( absolutePath, context ) - the same primitive chat/invoke already use to load GeneratedAgentFactory.bx itself - rather than a relative new "..."(). Confirmed against examples/class-based-agent/: chat, default invoke, invoke --server, and serve all build and run the same agent correctly.

No box.json executable install smoke test

box.json declares "boxlang": { "executable": "bxAgents" } so a real module install produces a native bxAgents command (see Installation). This wiring itself isn't exercised by an automated test - it relies on the BoxLang module installer's own documented behavior for generating executable wrappers, verified by reading its source, not by an install-and-run test in this repo's own CI.

schedules/Scheduler.bx isn't validated at build time

Since it's real, hand-written ColdBox code passed through untouched (see schedules/), build can't meaningfully check it the way { cron, action } config used to be checked - a syntax error, a typo in a getInstance( "..." ) call, or referencing an agent name that doesn't exist all pass build cleanly and only surface when the generated app actually boots (serve), same as any other real BoxLang class this project doesn't own the contents of. build does catch one narrower, adjacent mistake: two agents (root or subagent, any depth) declaring the same name - that's a config/WireBox.bx binding collision this project generates itself, so it's checked at validation time same as everything else it generates.

Fixed: a real bug in the test verb - resolving "the current BoxLang jar" was ambiguous

TestRunnerService.bx spawns a child process to run a project's tests/specs and needs to know which jar to launch it with. The original "which jar was this class loaded from" trick broke once boxlang-miniserver-*.jar (a fat jar that bundles its own copy of BoxRuntime) was also on the classpath - the classloader could resolve to the miniserver jar instead, silently launching MiniServer.main (which exits 1 immediately) instead of BoxRunner.main. Reproducible only through the full testBx suite, not in isolation, which is what made it easy to miss.

Fixed by resolving the jar from java.class.path instead - scanning for a boxlang-*.jar entry that does not contain miniserver, falling back to the old codeSource trick only if no such entry is found.

Fixed: no CLI verb's non-zero exit code ever reached the actual OS process - every failure looked like success to a shell script

A real, previously-shipped bug affecting every verb (build, clean, deploy, package, everything): ModuleConfig.bx's main( args ) computed the right exitCode internally and returned it, but confirmed directly against the official BoxLang_Docs MCP source, a module's main() return value is never mapped to the real OS process exit code by the runtime - only an explicit CLIExit( exitCode ) call (a genuine System.exit()) does that, and nothing here ever called it. So boxlang module:bxAgents build && boxlang module:bxAgents deploy ... in a CI/deploy script would run deploy immediately after a failed build, silently, because the shell always saw exit code 0 regardless of what actually happened. Every in-process test (ModuleConfigCliSpec.bx, each verb's own spec) exercises the computed exit code directly and would never catch this - only a real OS-process invocation exposes the gap, and nothing here was checking a real process's exit code for a failing verb before now (ModuleCliProcessTest.java only ever asserted the success path).

Fixed by splitting dispatch from exit: main() is now a thin wrapper - return CLIExit( dispatch( args ) ) - and all the actual argv-parsing/verb-dispatch logic moved into a new dispatch() method that still just returns an int, exactly as main() used to. CLIExit() can only live in the outermost main(), never in dispatch() itself: dispatch() is called directly, in-process, by every existing CLI test, and a real CLIExit() there would System.exit() the whole TestBox JVM on the first assertion. ModuleCliProcessTest.java gained a new real-OS-process test - package against a project with no prior build - asserting the child process's actual exit code is non-zero, the one check able to tell "genuinely exits non-zero" apart from "never calls System.exit() at all, which also happens to exit 0."

Fixed: .github/workflows/release.yml's prep_next_release job had no JDK, so it would have failed the moment a real release ran

prep_next_release runs ./gradlew bumpMinorVersion against a freshly-checked-out development - a plain actions/checkout@v7 with no Setup Java step anywhere in the job. Gradle itself needs a working JDK to run at all; the sibling build job (in the same workflow) has its own Setup Java step for exactly this reason, but prep_next_release never did. Never actually triggered a real failure yet - this repo has no main branch cut and no real release has run through this job - but it was an unconditional break waiting for the first one: gradlew would have failed immediately with no JDK on PATH/JAVA_HOME, before it ever reached bumpVersion(). Fixed by adding the identical actions/setup-java@v5.6.0 step (same temurin/${{ env.JDK }} as the build job) right after checkout. Verified locally: ./gradlew bumpMinorVersion --stacktrace --console=plain against a copy of gradle.properties correctly bumps 1.0.0 → 1.1.0 with a JDK present - the exact command this job runs.

deploy's ssh/docker/digitalocean targets aren't exercised by an automated real-process test

SshTargetSpec.bx/DockerTargetSpec.bx assert on the exact scp/ssh/docker command each target builds (a real ProcessBuilder argument array) without ever invoking the real binary - the same "capture, don't execute" approach used elsewhere in this suite for anything that needs a binary that might not be on PATH in CI (see MiniServerLauncherTest's assumeTrue skips). local's copy logic IS exercised for real (LocalTargetSpec.bx), including a regression test for a real latent bug this refactor fixed: the original Deploy.bx picked the "newest" .bxa via a lexical filename sort, which silently mis-picks once a project reaches double-digit versions (v9.0.0 sorts after v10.0.0) - fixed by sorting on actual file modification time (DistArtifactLocator).

DigitalOceanTargetSpec.bx similarly only unit-tests the pure buildAppSpec()/findExistingAppId() logic - the real GET/POST /v2/apps calls are never exercised in CI, since that needs a live DigitalOcean account and API token. Do at least one manual deploy --name=<ssh-entry> against a real disposable VM, and one deploy --name=<digitalocean-entry> against a real DO account with a throwaway app, before depending on either in production - the same honest framing already used above for the mock-provider-only testing gap.

deploy's ftp/sftp targets: real connection handling proven, a real successful upload isn't

Unlike ssh/docker (which shell out to external binaries and so can only have their command construction tested without a real server), ftp/sftp call the real bx-ftp module's bx:ftp component in-process - there's no "capture the command, don't execute" option. BaseFtpTargetSpec.bx instead makes a genuine connection attempt against 127.0.0.1 on a port nothing listens on, and asserts that the real refused-connection error is caught and re-thrown as a clear BxAgents.DeployFailed (confirmed via bx-ftp's own source that every action throws on failure rather than returning a soft succeeded: false). This proves the real connect/error-wrap/cleanup path end-to-end - what it can't prove is a real successful upload.

That gap is specifically because this development sandbox has no outbound raw TCP egress at all - only HTTPS through the environment's proxy (confirmed directly: curl ftp://test.rebex.net and a raw /dev/tcp connect to a public FTP host both hung and timed out, and docker info shows no daemon running, so bx-ftp's own bundled Docker FTP/SFTP test servers couldn't be started here either). This is a sandbox constraint, not a code limitation - do at least one manual deploy --name=<ftp-entry> and one deploy --name=<sftp-entry> against a real reachable server (or bx-ftp's own docker-compose up test servers, from a machine that has Docker/network access) before depending on either in production, the same honest framing already used above for ssh/digitalocean.

build doesn't roll back a partially-written .build/app on a mid-generation crash

BuildPipeline.build() deletes and recreates .build/app up front, then runs Phase 5's generators in sequence. Every input that Phase 3 (ProjectValidator) can check is checked before any of that happens, so a genuine mid-generation crash should be rare in practice - but if one still occurs (e.g. a models//schedules//mcp/ entry that fails to load, or an environment/filesystem problem), .build/app is left on disk in a partially-written state rather than restored to its prior contents or cleaned up. A subsequent successful build overwrites it cleanly, so this isn't sticky, but anything that inspects .build/app between a failed build and the next one (a CI step, a manual package retry) can see a broken half-generated app. There is no rollback/temp-directory-then-swap step yet.

An intermittent StackOverflowError was observed in testBx, unrelated to this project's own code

While investigating this milestone, ./gradlew testBx occasionally (not every run) crashed the whole JVM with a StackOverflowError inside the BoxLang engine's own generic-object JSON serialization (DynamicObjectSerializer/BoxStructSerializer calling each other, alternating, until the stack is exhausted - confirmed to still happen with -Xss16m, so it's a genuine cycle, not merely a deep-but-finite structure). Isolated via git stash (the crash reproduced identically with none of this session's changes applied, against the exact same commit already on development) and via bisecting tests/specs/** into every subdirectory both individually and combined (every subset, and even every subset-but-one, ran clean) - only the single full run occasionally reproduces it, and re-running the identical full suite immediately after a crash sometimes passes clean. This points to a timing-dependent race (plausibly interacting with ExampleScheduler's real background everySecond() task, which prints from its own thread pool concurrently with whichever spec happens to be running at that moment) rather than a bug in any one spec or in this project's own generated code. If testBx fails with a StackOverflowError and no other explanation, retry it before assuming a real regression - it is not currently reproducible on demand, so no automated regression test exists for it, and running it down further to a single root cause was out of scope for this round.

Push-style gateways (Telegram) are tested against a mocked API/scheduler seam only - no live platform integration runs in CI

TelegramGatewaySpec.bx exercises TelegramGateway's own logic (inbound normalization, outbound chunking at the 4096-char limit, HITL inline-keyboard building, scheduler task registration/removal) entirely against an injectable apiCaller/setScheduler() test seam - never a real Telegram Bot API call, and never a real ColdBox scheduler boot. This proves the gateway's own code is correct, not that it actually works end-to-end against Telegram's real API or a real running scheduler. Do at least one manual bxAgents serve against a project with a real botTokenEnvVar-backed Telegram bot before depending on it in production - the same honest framing already used above for the mock-provider-only/no-live-connection testing gaps elsewhere in this file. The same caveat will apply to every future push-style gateway (Slack, Discord, Email, WhatsApp) built the same way.

SlackGatewaySpec.bx carries the same gap, one level deeper: SlackGateway's persistent websocket connection is tested via an injectable setSocketOpener() seam (a fake object standing in for the real java.net.http.WebSocket), so every frame-handling/reconnect-logic assertion in the spec runs with zero real network I/O. What genuinely was verified directly (not mocked): a standalone smoke test instantiated the real SlackSocketListener(gateway) (which implements="java:java.net.http.WebSocket$Listener" directly - BoxLang compiles it as a genuine JVM implementer, no proxy needed) and called the real HttpClient.newWebSocketBuilder().buildAsync(...) against an unreachable address, confirming the BoxLang-to-Java interop itself works correctly all the way to the network boundary (it failed with a plain java.net.ConnectException, not a casting/interop error) - but no test here has ever completed a real Socket Mode handshake against Slack's actual servers. Do at least one manual bxAgents serve against a project with real botTokenEnvVar/appTokenEnvVar-backed Slack app credentials before depending on it in production.

DiscordGatewaySpec.bx carries the identical gap, for the identical reason: DiscordGateway's frame-handling/heartbeat/reconnect logic is exercised entirely against injectable setApiCaller()/setSocketOpener() seams, zero real network I/O. The same standalone smoke-test discipline was applied here too - gateway.onConnect() driven against a real HttpClient.newWebSocketBuilder().buildAsync(...) call to an unreachable address failed with a plain java.net.ConnectException, not a casting/interop error, confirming the interop chain works. What was NOT verified: a real Gateway handshake (Hello → Identify → READY) against Discord's actual servers, real heartbeat timing under Discord's own tolerance, or that the default intents value (GUILDS+GUILD_MESSAGES+DIRECT_MESSAGES+MESSAGE_CONTENT = 37377) is actually enough to receive message content once MESSAGE_CONTENT is enabled/approved for a real bot in the Discord Developer Portal. Do at least one manual bxAgents serve against a project with a real botTokenEnvVar-backed Discord bot (with MESSAGE_CONTENT enabled) before depending on it in production.

EmailGatewaySpec.bx carries a larger version of the same gap. Inbound IMAP is tested entirely via an injectable setImapPoller() seam (canned normalized message structs, no real mailbox), and outbound is tested entirely via an injectable setMailService() seam (FakeMailService/FakeMail, standing in for MailService@cbmailservices - EmailGateway never goes through real WireBox at all in these specs, since there's no real ColdBox boot). What genuinely was verified directly this session (not mocked, not assumed): the real jakarta.mail API surface fetchInboundMessages() depends on (Session.getDefaultInstance(), Flags/Flags.Flag/FlagTerm, Store.getStore("imaps"), Folder.READ_WRITE, MimeMultipart, InternetAddress) against the real jakarta.mail-api/Angus Mail jars (downloaded standalone for this - they aren't vendored in this repo's own test classpath), confirming every class/method name used actually exists and resolves; a real Store.connect() against an unreachable address, driven through EmailGateway.pollInbox() itself (not a bypassed helper), failed with a plain connection-timeout error, not an interop/casting error, confirming the interop chain reaches the real network boundary correctly, same discipline as Slack/Discord's websocket smoke tests. What was explicitly NOT verified, and is a strictly bigger gap than the chat-platform gateways: no real IMAP handshake against a real mailbox, no real cbmailservices/bx-mail module installed anywhere in this repo or its test harness (neither is vendored the way bx-ai/TestBox are), so the WireBox resolution path (MailService@cbmailservices really existing, BXMail's bx:mail call really sending) has never been exercised at all, mocked or real, in this codebase. Do at least one manual bxAgents serve against a project with real IMAP credentials AND a real cbmailservices/bx-mail install (confirm box install succeeded and moduleSettings.cbmailservices resolves) before depending on this gateway in production - this is the least-verified of the four push-style gateways shipped so far.

WhatsAppCloudGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: the signature-verification path is exercised with genuinely computed HMAC-SHA256 signatures (javax.crypto.Mac/SecretKeySpec, cross-checked independently this session against both openssl dgst -hmac and Python's own hmac module before trusting the BoxLang computation - a real reference-vector mismatch was caught and turned out to be a typo in the hand-copied expected value, not a bug, but only cross-verification caught that), the verify handshake, webhook dispatch/dedup, outbound send, and interactive button/list rendering are all driven through the gateway's real public methods with only the outbound Graph API HTTP call itself stubbed (setApiCaller()). What was NOT verified: the generated handlers/WhatsAppCloud.bx's own ColdBox request-context calls (event.getHTTPContent()/event.getHTTPHeader()/event.renderData(), rc's URL-scope-merged dotted-key query param access for the GET handshake) against a real ColdBox boot - these ARE the documented, standard ColdBox REST-handler idioms (confirmed against ColdBox's own "Building REST APIs" recipe docs, not guessed), a meaningfully more trustworthy starting point than the undocumented aiGatewayRegistry() key assumption that turned out to be wrong elsewhere in this file - but "documented" isn't the same as "proven to work in this generated context," and extending the project's own real runColdBoxIntegrationTests.bxs/miniserver harness to cover it would have required either passing fake env vars into a separately-spawned miniserver subprocess (no existing mechanism for that) or risking real config-related boot failures in the shared e2e-coldbox-route fixture other passing tests depend on - deferred rather than risking that blast radius for what's a lower-confidence-of-actually-being-wrong gap than the registry-key bug was. Do at least one real bxAgents serve + a real Meta webhook test (or curl) against /webhooks/whatsapp-cloud before depending on this route in production. No real Graph API call has ever been made either - deliver()/requestHumanInteraction()'s HTTP layer is exercised only via the apiCaller test seam.

TeamsGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: JWT verification is exercised against a genuinely generated 2048-bit RSA keypair (java.security.KeyPairGenerator) and hand-signed test JWTs built entirely inside the spec (no pre-computed fixtures, no external openssl dependency at test time) - a valid signature is accepted and dispatches, while a tampered signature, a wrong aud, a wrong iss, and an expired exp are each independently confirmed to reject with 401. The invoke-activity (Adaptive Card button click) path, message dispatch, personal-scope-only filtering, threading via replyToId, chunking, and Adaptive Card rendering are all driven through the gateway's real public methods with only the outbound Connector REST call itself stubbed (setApiCaller()) and the JWKS/OAuth2 token fetches stubbed (setJwksFetcher()/setTokenFetcher()). What was NOT verified: the generated handlers/Teams.bx's own ColdBox request-context calls against a real ColdBox boot (same category of gap as WhatsApp Cloud's own handler, and deferred for the identical reason - see that entry above); no real OAuth2 token fetch or Connector REST call has ever been made against Microsoft's actual endpoints; and the JWKS-cached-for-instance-lifetime tradeoff (see docs/conventions/gateways/teams.md) means a real key-rotation scenario has never been exercised either. Do at least one real bxAgents serve + a real Teams app registration (App ID/password from the Azure/Bot Framework portal) + a real Teams client sending a DM before depending on this gateway in production.

TwilioGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: the X-Twilio-Signature HMAC-SHA1/base64 verification path is exercised with a genuinely computed signature built inline in the spec, cross-checked independently this session against Python's own hmac/hashlib modules before trusting the BoxLang implementation (same discipline as WhatsApp Cloud's own HMAC-SHA256 cross-check) - a real reference signature was computed in Python for a known auth token/URL/params combination and confirmed to match BoxLang's output exactly. Form-body parsing (including a literal + correctly round-tripping through %2B percent-encoding), the publicUrl override for proxy/tunnel deployments, the dual-path TwiML-ack-then-async-REST-reply model, outbound chunking, and the phone-number-keyed HITL reply correlation are all driven through the gateway's real public methods with only the outbound Messages API HTTP call itself stubbed (setApiCaller()). What was NOT verified: the generated handlers/Twilio.bx's own event.getUrl() call against a real ColdBox boot (same category of gap as WhatsApp Cloud/Teams's own handlers, deferred for the identical reason) - event.getUrl() is a documented ColdBox Routable/Request Context method (confirmed via the ColdBox docs MCP), not a guess, but "documented" isn't "proven in this generated context." No real Twilio Messages API call has ever been made either. Do at least one real bxAgents serve + a real Twilio phone number webhook test before depending on this route in production - and note the phone-number-keyed HITL correlation (see docs/conventions/gateways/twilio.md) has a real, documented limitation: a second HITL request to the same phone number before the first is answered would overwrite the first's pendingApprovals entry, silently orphaning it. No allowlist/rate-limiting is built for inbound SMS either - unlike Eve, which documents (but doesn't enforce in code) that its own allowFrom config is "mandatory," this port has no equivalent gate at all; any phone number can message a deployed Twilio number and reach the agent.

GitHubGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked, and this gateway's core logic was additionally driven through a standalone real-BoxLang smoke test during development (not just the permanent spec) - which is how a genuine bug was caught before it ever reached the test suite: the mention-extraction helper's substring logic called left( body, 0 ) whenever the @mention occurred at the very start of a comment (a very common case), and BoxLang's left() throws "Count cannot be zero" rather than returning an empty string for a zero count - confirmed by triggering it with a real comment body, then fixed by explicitly branching around the zero-length case rather than assuming left()/mid() tolerate it. The X-Hub-Signature-256 verification, @mention regex-lookahead gating (confirmed via a dedicated smoke test that a bot named mybot does NOT fire on @mybot2), bot-loop guards, delivery-id dedup, issue-vs-review-thread conversation identity, deliver(), and the @mention-to-reply HITL correlation are all driven through the gateway's real public methods with only the outbound GitHub REST call itself stubbed (setApiCaller()). What was NOT verified: the generated handlers/GitHub.bx against a real ColdBox boot (same category of gap as every other webhook gateway's own handler in this project, deferred for the identical reason). No real GitHub API call has ever been made, and no real GitHub App/PAT has ever been used against a real repository - do at least one real bxAgents serve + a real GitHub webhook configured against a test repository before depending on this gateway in production.

SignalGatewaySpec.bx covers the gateway's own logic thoroughly and for real, not mocked: handleSseEvent()'s JSON-RPC/SSE parsing (blank-line/invalid-JSON handling, group-message filtering, quote-threading, sourceUuid fallback when a display name is absent), deliver()'s send shape and chunking, and HITL decision correlation/matching are all driven through the gateway's real public methods with only the outbound rpcCaller/connector I/O calls stubbed, the same seam-testing discipline as every other gateway in this project. Two BoxLang-level findings surfaced during development, both resolved and worth recording as general landmines rather than gateway-specific bugs: (1) a throwaway smoke-test script that named a local variable request was silently interacting with BoxLang's own reserved request scope instead of creating a plain variable, producing misleading "method not found"/"argument type mismatch" errors from HttpClient.send() that looked exactly like a genuine Java-interop limitation but disappeared entirely once the variable was renamed - SignalGateway.bx itself never had a bug; (2) a try/catch placed directly at the top level of a standalone .bxs smoke-test script (not inside a function) triggered a java.lang.VerifyError: Inconsistent stackmap frames, a real bytecode-verification limitation of BoxLang's top-level-script compiler, traced via the stack trace to the test script's own generated class, not SignalGateway.bx - fixed by wrapping the try/catch inside a named function instead. What was NOT verified, and is the largest gap among all the push-style gateways shipped so far: no real signal-cli daemon has ever been available in this environment, so the entire async SSE connection lifecycle - opening the stream via HttpClient.sendAsync()+BodyHandlers.ofLines(), the exponential reconnect-backoff loop against a genuinely flaky connection, the 30s/120s idle watchdog forcing a reconnect, and a live JSON-RPC round trip - has never been exercised end-to-end, only smoke-tested at the interop-plumbing level (a standalone test reached a real java.net.ConnectException against an unreachable address, proving the chain is sound, not that it works against a live daemon). Do at least one manual bxAgents serve against a project with a real, running signal-cli daemon and a real linked Signal account before depending on this gateway in production - this is a genuinely new transport architecture in this codebase (the only SSE-based gateway among Telegram/Slack/Discord/Email/Signal), not just a new platform on an already-proven transport shape.

WhatsApp Personal (unofficial personal-account bridge) - researched, not built

The original plan (matching Hermes Agent's own architecture) called for a WhatsAppPersonalGateway built by spawning a Node.js subprocess running @whiskeysockets/baileys (the multi-device WhatsApp Web protocol client Hermes itself uses - MIT licensed, its full bridge.js was read directly from Hermes's real source, not summarized). That approach was set aside mid-session on direct instruction to prefer a native BoxLang/JVM integration over a subprocess bridge, and to use a native Java library only if it's open source and neither GPL nor LGPL.

That search found Cobalt (com.github.auties00:cobalt, formerly WhatsappWeb4j) - a real, MIT-licensed, actively maintained (900+ stars) Java implementation of WhatsApp's multi-device "Linked client" protocol, with a documented fluent API (WhatsAppClient.builder().linkedApi().webClient()..., addNewMessageListener(), sendMessage()) that would coerce cleanly from a BoxLang closure via BoxLang's own documented Java-SAM coercion (confirmed via the BoxLang docs MCP - no createDynamicProxy() needed for a single-abstract-method listener). Two real blockers surfaced during verification, not guessed:

1. The first pom.xml read (Cobalt's master branch, an in-progress multi-module rewrite) requires Java 25 - two majors ahead of BoxLang's own documented baseline (Java 21+, confirmed via the BoxLang docs MCP and matching this project's own 21.0.10 JDK). Re-checking against the artifact actually published to Maven Central (cobalt:0.0.10, the real thing a <dependency> would resolve today, not the unreleased rewrite) showed <java.version>21</java.version> - so the Java-25 finding was a false alarm caused by reading the wrong branch, not a real blocker. Worth recording as a caution: a GitHub repo's default-branch pom.xml is not necessarily what's on Maven Central.
2. The real, published cobalt:0.0.10 pulls in com.aspose:aspose-words as a hard compile-time dependency (used internally for generating link-preview thumbnails from Word documents) - Aspose.Words for Java is commercial/proprietary-licensed, not open source, so bundling it would violate the same license constraint Cobalt itself was chosen to satisfy. The full dependency graph (~15 jars: zxing, qr-terminal, curve25519, protobuf-base, jackson or fastjson2 depending on version, libphonenumber, dd-plist, apk-parser, link-preview, jaffree, ez-vcard, slf4j, plus Aspose) would all need to be manually downloaded and bundled under this module's libs/ folder - BoxLang modules have no Maven-style dependency resolution of their own (confirmed via the BoxLang docs MCP: third-party jars are bundled directly into a module's libs/ folder, loaded by a per-module classloader - there's no javaLibraries key in box.json that would resolve a dependency tree automatically).

Given the license contamination via Aspose and the manual-fat-jar-assembly effort with no dependency-resolution tooling available to verify the result actually loads, WhatsApp Personal was descoped rather than shipped as either a real gateway or a stub. ProjectValidator's validGatewayTypes and GatewayGenerator's TYPE_CLASS_MAP do not include a whatsapp-personal entry - a project that tries to declare one gets the existing "unknown gateway type" validation error, same as any other unsupported type, rather than a misleading half-built stub. Revisiting this is reasonable if Cobalt ever drops the Aspose dependency (it's a narrow feature - link-preview thumbnailing of Word docs - not core to messaging), or if a future session decides the Node/Baileys subprocess-bridge approach (rejected this round on architecture preference, not on a technical blocker) is preferable after all.

GatewaySession is project-wide and root-agent-only (v1)

A project with at least one push-style gateway entry gets exactly one generated GatewaySession, bundling every push-style gateway and always bound to the project's root agent - matching the existing precedent that exposes: "agent" HTTP exposure is also always root-agent-only (see gateways/).

This is the architecture, not a gap. A project has one root agent, and subagents/ are delegated to by that root - the root decides what goes to whom. Every entry point (gateways, the HTTP agent route, the web UI, schedules) therefore lands on the root by design. Binding a gateway straight to a subagent would route around the agent that is supposed to be orchestrating it, which is why there is no per-gateway targetAgent key: to make "Telegram talks to support, Slack talks to research" work, give the root that routing logic and let it delegate, rather than splitting the front door.

The docs site's nav group headings stay English in a translated locale

Every nav entry backed by a real page takes its label from that page's own frontmatter title, so a translated tree gets translated nav labels for free - Instalación, Inicio rápido, Introducción a BxAgents and so on all render correctly under /es/.

The four synthetic grouping nodes in docs/nav.json (Getting Started, Conventions, Guides, Reference, plus the Example Agents sub-group) have no page behind them, so there is nothing to read a translated title from. bx-sites v1 also hands every locale tree the default tree's own navConfig verbatim - by design, so a locale's nav always mirrors its default tree's structure - which means there's no per-locale nav.json to override them in either. They therefore render in English on every locale.

Measured on a real build: the /es/ sidebar renders ["Getting Started", "Instalación", "Inicio rápido", "Introducción a BxAgents", "Conventions", "Agent.bx", ...]. Page labels localized, group headings not.

Closing this needs an upstream bx-sites change (a per-locale nav override, or localizable group titles), not a change here.

Fixed: config/ColdBox.bx's own interceptors = [...] array never actually registered anything on a real boot - the web UI's login gate and API-key gate were both silently wide open

A real, serious, previously-undetected bug, found only by driving the built web UI over genuine HTTP with real cbauth/ColdBox module dependencies for the first time (see the bx-ai 3.4.0 entry below - retest everything after that update is what surfaced it). config/ColdBox.bx's interceptors = [{ class: "interceptors.X" }] array - the documented ColdBox convention this project used to emit, and the ONLY place it registered WebUiLoginGate/WebUiAuthGate/GatewaySessionBootstrap/any project-declared interceptor - never reached ColdBox's InterceptorService at all on this stack. Confirmed directly: ModuleService.cfc's own module-settings merge (a separate but related bug, see the cbauth entry below) and ApplicationLoader.cfc's interceptor parsing both read the app's config CFC instance via getPropertyMixin(), a reflection helper that reaches into a CFC's own variables scope from outside it - and a direct test this session (new config.ColdBox(); cfgClass.configure(); cfgClass.moduleSettings immediately after) proved that scope is simply not reachable that way on this BoxLang/ColdBox combination, variables.-prefixed or not. interceptors = [...] in config/ColdBox.bx is dead code on this stack, full stop - not a web-UI-specific bug.

The practical consequence: a webui entry configuring users or apiKeyEnvVar documented (see docs/conventions/web-ui.md) that doing so "turns on a real sign-in gate." It did not. Every preProcess interceptor silently never ran, so every /chat/api/* route answered normally with zero session and zero API key, on every project that had ever configured either gate - the exact opposite of the documented behavior, and undetected because nothing in the test suite drove a real login/API-key-gated request over actual HTTP; WebUiRuntimeSpec.bx only ever called UserService.isValidCredentials() directly (bypassing authenticate(), the interceptor, and the whole registration path), and ColdBoxAppGeneratorSpec.bx/WebUiGeneratorSpec.bx only ever asserted on the generated TEXT of the (dead) array, never that it worked. The same root cause meant the web UI's afterConfigurationLoad-based schema warm-up (WebUiSchema.bx) and the push-gateway starter (GatewaySessionBootstrap.bx, see the entry above) were ALSO dead - the former was harmless (ChatDb.onDIComplete() migrates on first use regardless, which is exactly what hid it), the latter meant no push-style gateway (Telegram, Slack, Discord, Teams, WhatsApp Cloud, Twilio, GitHub, Signal) ever actually started in a real boot, a second, independently serious consequence of the identical bug.

Fixed by abandoning the declarative array entirely, following the precedent BuildPipeline.bx had already set for the account seeder (see its own comment: "as an interceptor it silently never fired"):

• WebUiLoginGate/WebUiAuthGate/every project-declared interceptor are now registered for REAL from Application.bx's onApplicationStart(), via interceptorService.register( target:, name: ) - a plain function call against already-built objects, using getMetadata() reflection (which works), never getPropertyMixin() (which doesn't). See ColdBoxAppGenerator.renderInterceptorRegistration().
• WebUiSchema's one-shot afterConfigurationLoad action and GatewaySessionBootstrap's one-shot gateway-starting action are no longer interceptor files at all - both are now bare onApplicationStart() statements, the same category as the seed statement (WebUiGenerator's webUiWarmStatement, GatewayGenerator's renderSessionStartupStatement()).

Verified for real, not just re-generated: rebuilt a throwaway project declaring users: through the actual (fixed) build pipeline, booted it with a real boxlang-miniserver, and drove it over genuine HTTP end to end - GET /chat/api/health unauthenticated → 200 (the health probe stays exempt, see below), GET /chat/api/history unauthenticated → 401 {"error":true,"message":"Not signed in"}, a wrong password → a clean 302 to /chat/login.html?failed=1, the correct password → a clean 302 to /chat/ with a session cookie, and that cookie → 200 with real data on the same previously-open route. WebUiRuntimeSpec.bx now also calls authenticate() itself (not just isValidCredentials()) against both a wrong and a correct real password, and confirms WebUiLoginGate is genuinely registered as a preProcess listener via InterceptorService.getStateContainer( "preProcess" ).exists( "WebUiLoginGate" ) - not getInterceptor(), which only resolves interceptors registered by class path and doesn't apply to register()'s object form used here, even though that form genuinely registers and fires (confirmed separately, over real HTTP).

One more, smaller bug surfaced by the same real-HTTP pass, fixed alongside it: event.noExecution() alone does not stop ColdBox from still trying to render a view/layout - confirmed against real ColdBox 8.1.0 source, Bootstrap.cfc checks getIsNoExecution() (skips the handler) and isNoRender() (skips rendering) as two entirely separate flags. ChatAuth.bx's hand-rolled redirectTo() and WebUiLoginGate's own raw-redirect branch called only the former, so a plain redirect (no renderData() set) 500'd trying to render a default view that was never asked for, clobbering the intended 302 - invisible before this fix because the interceptor calling it never ran, and ChatAuth.doLogin()'s own noExecution()-only bug was masked by authenticate() throwing first for an unrelated reason (see below). Both now call event.noRender() too.

Fixed: moduleSettings.cbauth.userServiceClass in config/ColdBox.bx never reached cbauth either - and the test harness's own pinned cbauth version was separately broken against real ColdBox 8.1.0

Two distinct, compounding bugs, both found only by calling authenticationService@cbauth.authenticate() for the first time against a real boot (previously, coverage stopped at UserService.isValidCredentials() - see the interceptor-registration entry above for why that leaves a real gap).

First bug (a version pin): tests/box.json pinned cbauth: "^5.0.3" (resolving to 5.0.7), whose AuthenticationService.cfc calls interceptorService.processState( ... ) - a method ColdBox removed; real ColdBox 8.1.0's InterceptorService exposes only announce(). Every call to authenticate()/login()/logout() threw Method 'processState' not found before ever reaching credential checking. Confirmed directly against cbauth's real published source: cbauth 7.0.0 (ForgeBox's real "latest", confirmed reachable and resolvable) already calls announce() throughout - the fix is version currency, not a code change. tests/box.json now pins cbauth: "^7.0.0" (with its own cbstorages: "^3.0.0" dependency, 3.5.0). IUserService/IAuthUser - the two interfaces this project's generated UserService.bx/User.bx implement - are byte-for-byte identical between cbauth 5.0.7 and 7.0.0, confirmed directly, so no generator changes were needed for this half. A real end user running a bare box install cbauth today gets 7.0.0 and was never exposed to this specific bug; it was a test-harness-only gap, but a real one, since it meant the harness's own authenticate() path had never genuinely run end-to-end even once.

Second bug (config, not a version issue - see the interceptor entry above for the same root cause): even with announce() working, authenticate() still threw IncompleteConfiguration - No [userServiceClass] provided, because moduleSettings.cbauth in config/ColdBox.bx - the ColdBox-documented way to override a module's settings, and what this generator used to emit - never reaches the module on this stack, for the identical getPropertyMixin() reflection gap described above. Fixed by using ColdBox's OTHER, genuinely-working per-module settings mechanism instead: a config/modules/cbauth.bx file, which ModuleService.cfc's loadModuleSettingsOverride() invokes with a plain function call (oConfig.configure( mConfig.settings ), returning the merged struct) rather than reflection. See ColdBoxAppGenerator.renderCbauthModuleOverride(). Confirmed for real: settings.modules.cbauth.settings.userServiceClass resolves to "UserService" in a live boot, and a real sign-in against a real seeded account succeeds end-to-end (see the interceptor-registration entry's own real-HTTP verification, which exercises both fixes together).

qb's own moduleSettings.qb.defaultOptions/defaultGrammar carries the exact same root cause and is already documented, separately, below (ChatDb.query() was already worked around by never depending on it). bxai's own moduleSettings.bxai (skillsDirectory, the security passthrough) was NOT specifically re-verified against a real boot this session - bx-ai is a BoxLang-native module, not a ColdBox one, so it is plausible (not confirmed either way) that it reads its settings through an entirely different, unaffected mechanism. Treat it as unverified, not as known-working, until someone checks.

Fixed: GatewaySessionBootstrap.bx looked gateways up by the wrong aiGatewayRegistry() key

A real, previously-shipped bug: the generated interceptor looked gateways up by the discovered gateways/* entry's own filename (e.g. "telegramChannel"), but bx-ai's GatewayRegistry.register() always keys by the gateway class's own fixed getName() (e.g. "telegram") - never anything caller-supplied. This meant GatewaySession construction threw at ColdBox boot for every generated project with a push-style gateway; it shipped undetected because the only prior coverage asserted on generated file text, never against a live registry. Fixed in GatewayGenerator.generate(): the interceptor now looks gateways up by their TYPE string, deduplicated, with a permanent regression test (GatewayGeneratorSpec.bx) that registers a real gateway and proves the emitted key resolves it via a live aiGatewayRegistry().

A real, permanent consequence this fix surfaces (still true today, not a bug): because the registry is keyed by type, not by entry, two gateways/* entries of the same push-style type collide on the same registry slot project-wide - e.g. two type: "telegram" entries (two different bot tokens) would silently have the second registration overwrite the first, and GatewaySession would only ever see one of them. There's no per-entry alias/registration-name override today. One instance per push-style type, per project, is the actual v1 ceiling.

Fixed: bx-ai's published snapshot used to lag its own development branch

While bx-ai's aiGatewayRegistry()/GatewaySession/BaseGateway API and its userId/conversationId-scoped IAiMemory.summarize() (needed by /compact) were still unreleased, this project pinned bx-ai@3.4.0-snapshot - a continuously-republished ForgeBox artifact that repeatedly lagged bx-ai's real development HEAD by some amount of time. Three separate times, a fresh downloadModules fetched a snapshot predating the API this project's own generated code already called, breaking gateway boot (Function 'aiGatewayRegistry' not found), the whole gateway layer (missing BaseGateway.bx), or /compact (silently summarizing the wrong scope). CI worked around it by cloning and building bx-ai's development branch directly and overlaying it over whatever downloadModules fetched.

bx-ai has since cut a real, stable 3.4.0 release carrying all of it - confirmed directly against the published ForgeBox artifact (BaseGateway.bx, GatewaySession.bx, aiGatewayRegistry/aiGatewaySession BIFs, MockService.bx, and summarize( config, userId, conversationId ) are all present). build.gradle now pins that stable version instead of the snapshot, and the CI overlay step is removed - downloadModules alone is enough.

The v1 web chat UI (exposes: "webui") - what's real, and what could only be checked against docs, not a live server

WebUiGeneratorSpec.bx and a BuildPipelineSpec.bx end-to-end test both drive the real WebUiGenerator/BuildPipeline classes against real fixtures: the static <path>/index.html shell is confirmed to be written and correctly templated (__API_BASE__/__APP_TITLE__ placeholders substituted, never left in the output), and the optional interceptors/WebUiAuthGate.bx is confirmed to be generated only when apiKeyEnvVar is configured, to gate exactly <path>/api/* and never the bare <path> shell itself. Registration is no longer a config/ColdBox.bx interceptors:[...] entry at all - that mechanism never reached ColdBox's InterceptorService on a real boot (see the dedicated "Fixed" entry below; this file used to claim the opposite here, which was itself part of the bug going undetected). WebUiAuthGate/WebUiLoginGate are now registered for real from Application.bx's onApplicationStart(), and this was re-verified end-to-end over genuine HTTP this session (see that entry) - not just re-generated and re-asserted on as text.

What was NOT verified - and couldn't be, in this dev environment: a real bxAgents serve + real browser test of the page actually loading, streaming a reply, and the X-API-Key gate actually rejecting/accepting requests over real HTTP. This project's own runColdBoxIntegrationTests.bxs/tests/coldbox harness (the same one that proved toAi()'s /invoke route works end-to-end for http-gateway-agent) requires tests/coldbox to be present via a real box install inside tests/ - CommandBox and ForgeBox network access were both unavailable in this session's sandbox, so that harness could not be exercised for the web UI (or re-confirmed for anything else) this session. Two direct consequences:

• The exact /invoke JSON response shape used by this project's own generated pages/docs ({"input": "..."} in, a response containing "success": true) is empirically confirmed only via runColdBoxIntegrationTests.bxs's own prior, already-passing assertion (a substring check, not a full shape assertion) - not re-verified this session.
• The /stream SSE wire format the web UI's own JS parses (data: {"token":"..."} lines, terminated by data: [DONE]) is taken directly from ColdBox's own official "AI Routing" documentation, not independently re-confirmed against a live server this session - unlike almost every other wire-format claim in this project's docs, which were cross-checked against real running code wherever possible (see e.g. the HMAC-SHA256/SHA1 signature schemes' independent Python/openssl cross-checks elsewhere in this file).

Do at least one real bxAgents serve + a real browser test (message sent, reply streams in, the X-API-Key gate actually 401s a request missing the key) before depending on this feature in production - the same standing advice already given for every generated webhook handler's own untested-against-a-real-boot gap elsewhere in this file.

The web UI's SQLite store - verified at the library level, not through a real ColdBox boot

The qb + bx-sqlite stack underneath models/ChatDb.bx was verified directly this session against the real jars, not inferred from docs: qb 13.1.0's .cfc sources compile and run natively on BoxLang 1.16 with no bx-compat-cfml; SQLiteGrammar + SchemaBuilder really do create the v1 tables and indexes against a real SQLite file; a second migration pass is a clean no-op; QueryBuilder round-trips inserts, filtered/ordered reads and deletes; and the preferences composite primary key really does reject a duplicate (userId, prefKey). Two constraints were found this way rather than assumed - qb requires a named datasource (its own appendSqlComments() types that argument as string, so an inline struct throws before any SQL runs), and SchemaBuilder@qb is mapped with its grammar argument only, never receiving moduleSettings.qb.defaultOptions - and the generated code is shaped around both.

The three wiring assumptions this depends on - that getInstance( "ChatDb" ) resolves through the generated app's own WireBox, that SchemaBuilder@qb/QueryBuilder@qb resolve once qb is installed as a real ColdBox module (qb is a box.json dependency, not vendored here - the same honest gap cbmailservices already carries), and that this.datasources in the generated Application.bx is picked up as expected - are now exercised end to end for the shared-preferences path: runColdBoxIntegrationTests.bxs writes and reads a preference through POST/GET /chat/api/preferences from outside the process (see below), which only works if all three hold. What is not yet exercised the same way is conversation CRUD, the cross-user guard, and the migration pass - those are proven at the library level and via WebUiRuntimeSpec.bx reading real WireBox objects inside a boot, but not driven from a second, independent HTTP caller the way the preferences path now is. Do one bxAgents serve against a webui project with qb and bx-sqlite genuinely installed before relying on the store in production; the failure mode if any of the three wiring assumptions is wrong is loud at boot (an unresolvable WireBox mapping or an unknown JDBC driver), not silent.

The web UI over real HTTP: proven for /health and the shared-workspace preference routes

runColdBoxIntegrationTests.bxs fetches GET /chat/api/health from outside the server, against the generated fixture app booted by a real boxlang-miniserver, and fails the build if it is not a 200. On CI it returns:

+ App probe GET /chat/api/health -> status=200
  body: {"success":true,"status":"ok"}

That request is the baseline end-to-end proof for the web UI's server side: ColdBox routing reaching handlers/ChatUi.bx, WireBox resolving the handler and ChatDb, this.datasources giving bx-sqlite a usable SQLite file, and qb being a genuinely activated ColdBox module - none of which any source-text assertion can establish. The same runner script goes further and drives two more routes over real HTTP against that same boot: POST /chat/api/preferences/set and GET /chat/api/preferences, called as two separate, cookie-less HTTP requests specifically to prove the accountless web UI's "one shared workspace" behavior - that a preference written by one caller is read back by a completely independent second caller, through ChatDb/SQLite, not just echoed from a session. WebUiRuntimeSpec.bx then covers the rest of the store's behaviour (migrations, scoped conversation CRUD, the cross-user guard) by reading real objects out of WireBox inside that same boot.

What is still not covered over real HTTP: conversation list/history and SSE streaming. Driving those from WebUiRuntimeSpec is not possible as structured, because that spec runs inside a request served by the same single miniserver, so a loopback call would compete for the worker pool the runner itself occupies. Covering them needs either a second server process or a larger miniserver thread pool, and is a natural follow-up rather than something to fake.

The full /chat/login round trip was ALSO driven over genuine HTTP this session (see the "interceptor registration" and "cbauth" Fixed: entries above for what it caught) - a wrong password redirecting cleanly to /chat/login.html?failed=1, a correct one to /chat/ with a working session cookie, and that cookie then authorizing a previously-open route while a cookie-less request to the same route still 401s. That pass was manual and ad hoc (a throwaway project outside this repo, not e2e-coldbox-route), because the fixture's own secureChat.bx account uses a passwordHash with no known plaintext (deliberately, so CI needs no secret) - there is nothing to log in WITH over real HTTP without adding one. WebUiRuntimeSpec.bx now covers the same ground a different way instead: it calls authenticationService@cbauth.authenticate() directly (both a wrong and a real, freshly-seeded correct password) and confirms WebUiLoginGate is genuinely registered with InterceptorService - real coverage, permanently in the suite, just not over the wire. Extending runColdBoxIntegrationTests.bxs itself to drive /chat/login over HTTP (an env-var-backed account on the fixture, the same pattern UserSeeder already uses) is a reasonable follow-up, not done this session.

Note that this paragraph previously cited runner-coldbox.bxm never responded: HTTP 408 as evidence of that starvation. That attribution was wrong, and is worth recording because it misled this project for several CI cycles: the 408 had nothing to do with loopback calls. The runner page called chr( 10 ) - a BIF that does not exist - so it returned 500 on entry to every request, and the orchestrator's retry loop ground through 90 seconds of those 500s before reporting the final timeout. Both are fixed. The loopback-starvation concern above is a genuine structural reason not to drive more routes from inside that spec, but it remains a well-founded expectation rather than something that has actually been observed failing here.

The web UI's own front-end: driven in a real browser, but against a mocked API

The shipped page's JavaScript was exercised for real this session, not just asserted against as source text: the generated index.html was loaded in headless Chromium and driven end to end with every <path>/api/* route intercepted and answered with realistic payloads. Confirmed working that way - the conversation sidebar rendering from GET /conversations and switching/renaming/deleting through its own routes; GET /info shaping the toolbar (Compact appearing only when capabilities.compact is true, the model name landing in the header); the theme arriving from server-side preferences and applying; /history rehydrating the transcript; a real SSE turn streaming content, reasoning and tool-call chips out of the bx-ai envelope; markdown rendering; and New chat creating a conversation server-side and opening it. Zero JavaScript errors, zero unreplaced __TOKEN__ placeholders, and the narrow-screen layout was screenshotted at 390px.

What that does not prove: the API was a Playwright mock, not the generated handlers/ChatUi.bx running under a real ColdBox boot against a real SQLite store. The request and response shapes were taken from the generated handler's own source, so a drift between the two would not be caught by this. Everything in the known-limitations entry above about the store applies here too - one manual bxAgents serve against a webui project with qb and bx-sqlite genuinely installed remains the honest gate before production use.