# Full-program parsing and generation extension

The user requested manual parsing problems and clarified the target: given an unfamiliar grammar, which structures can an AI correctly parse and generate? This extends the experiment with full-program structure tasks. The already frozen reading main remains a separate run with its original estimand and source; it is not relabelled as a whole-program parsing experiment.

The current primary hypothesis concerns **generation under a supplied unfamiliar miniature grammar**: whether kind-labelled closers (`end func`) and kind-and-name closers (`end func n_abcdefgh`) improve complete-source generation over generic `end`. Complete-source parsing is a paired adjunct. This fixed P2 pilot remains exploratory; improvement, no effect and deterioration are all valid outcomes.

## Tasks and grading

- `source_to_ast`: give the complete balanced source, grammar description and eight fixed solved examples. Ask for its complete neutral JSON AST. The TEST section contains no target AST, observation, probe, repair instruction or omitted suffix.
- `ast_to_source`: retain the existing shuffled neutral AST table and require complete source in the supplied grammar. Use the same structural families, names, values, ordered statements, example ASTs, lexical mappings and five styles as the parsing direction.
- Both directions have `view=complete` and output cap 16,384. Parse/generate are reported separately; table input and JSON output are different representations, so their absolute accuracies do not isolate direction alone.

Neutral JSON consists of a root `{"body":[...]}`. Statements are `{"type":"scope","kind":"unit|func|area","name":"n_abcdefgh","body":[...]}`, `{"type":"let","variable":"x_a","value":1739}` or `{"type":"nop","payload":"p_abcdefgh"}`. Kind values name semantic categories regardless of the supplied vocabulary. JSON object-key order and whitespace are immaterial; body-array order is significant. Unknown/missing fields, unsupported tags, invalid identifier/value types, duplicate JSON keys and malformed ASTs fail strict grading. Decode into `Program` and compare every node, parent-child relation, kind, name, variable, value, payload and statement order with separately saved gold. A schema-valid wrong tree remains a model failure. Preserve refusal/incomplete, infrastructure missingness and unexecuted denominators.

## Fixed exploratory P2 run

New phase `p2` permits explicitly selected `ast_to_source`/`source_to_ast` tasks in smoke/pilot. Legacy P0/P1 task selection and counts remain unchanged. P2 main/freeze is rejected until a distinct confirmatory plan and matching capabilities are designed.

The fixed pilot is 48 structural families: depths `[2,4,8,16]`, filler statements `[0,8,32]`, four families per cell. Two directions × natural/nonce × five styles × one replicate gives **960 real generations**, or 96 per direction/style. Master and bootstrap seeds are `2026100602`; eight examples; model `gpt-5.6-terra`, effort `low`. Report task/style/depth/filler/lexical rates, exact failure classes, usage, missingness and family-cluster contrasts. Never pool with historical pilot/main, retry wrong answers, remove failures or stop for significance. Mock runs are labelled synthetic.

The grammar is explicitly explained in the prompt. This measures the use of supplied unfamiliar miniature grammars, not inference from examples alone or proven absence from training. Covered structures are generated nested scopes, mixed scope kinds, ordered sibling/leaf statements and closing sequences. The finite grid does not cover arbitrary grammars or all trees.

Depth and deepest-body filler count are controlled factors. Each family currently contains a nested chain and two shallow sibling scopes; branching width, sibling location and homogeneous versus mixed-kind nesting are not independently varied. Depth also changes total node count and output length. Report performance for these sampled structures without treating it as a general taxonomy of parseable grammars. Exact AST correctness includes copying identifiers, values and payloads; the first mismatch diagnostic is not a separate structural-only or partial-node score.

Implementation and tests run in the isolated `feat/full-program-parse-generate` checkout. The documented interface is native sbt on JDK 21; see the [public build and execution instructions](https://github.com/kmizu/syntax-locality-experiments#readme). Run `sbt test` before new source-bound artifacts, audit and offline reproduction. `scripts/sbt.ps1` is an optional Windows launcher for the same JDK/sbt toolchain. Live execution needs explicit numerical CLI limits and retains requests, responses, attempts, IDs, reservations and usage.

## Current verification state

As of 2026-10-06, the post-change implementation passed the required `./scripts/sbt.ps1 test` in saved [CI run 37419923104, job 112126775863](https://github.com/kmizu/syntax-locality-experiments/actions/runs/37419923104/job/112126775863): 28 core tests, 19 bench suites, 12 named preflight checks and 13 parsing cases.

Fresh local checks also passed on JDK 21.0.3, Scala 3.3.8 and sbt 1.10.7: the required full test command completed with exit 0, with the same 28 core tests, 19 bench suites, 12 named preflight checks and 13 parsing cases. The local artifact `verification/p2-live-preparation-local-sbt-test-20261006.log` records the actual run. A later local artifact `verification/p2-live-preparation-helper-audit-full-test-20261006.log` records six synthetic verification-helper cases, the saved P2 dataset/prompt/schedule audit and a successful full-suite rerun; those synthetic cases do not measure model accuracy. These local transcripts are not included in the public source tree.

The distinct standalone [P2 mock run 37371856651, attempt 3](https://github.com/kmizu/syntax-locality-experiments/actions/runs/37371856651/attempts/3), [job 112109491633](https://github.com/kmizu/syntax-locality-experiments/actions/runs/37371856651/job/112109491633), completed 960 synthetic trials across 48 structural families with `synthetic_mock=true`. All five saved reports were byte-identical on offline reproduction; resume added 0 logical trials and 0 adapter attempts. This verifies the mock workflow, not model performance.

The distinct live plan is `runs/full-program-pilot-live-v1`: 48 structural families, 960 planned generations, 480 per direction and a 16,384 output cap in every saved request. Its source hash is `41ffc2647b8f3c3838a17c2425b0be97007fe611f5d0a68d3dd9d1c79e1b0b96`; protocol hash is `2ac04cc60928800d0c1fdd37986546a6fee23bd1955ccc011c76099cbde60471`. Actual OpenAI count plus generation verified matching `gpt-5.6-terra` / `low` / 16,384 settings on 2026-10-06: two HTTP attempts, 29 known tokens, zero unresolved reservations, `synthetic_mock=false`. The [saved public capability evidence](https://github.com/kmizu/syntax-locality-experiments/blob/main/experiments/data/supporting/p2-live-capabilities.json) records this probe. Its scope is a tiny capability probe, not whole-program task performance; the production preflight alone tests 8,192.

On 2026-10-06 the provider returned credit_balance_exhausted and P2 stopped at 300 terminal dispositions. That historical interruption record remains unchanged. On 2026-10-07 UTC the user reported charged; a fresh matching actual capability probe passed and the same fixed plan resumed. The provider returned credit_balance_exhausted again at 13:56 UTC, and both exact native model processes stopped. The new released checkpoint preserves 521 terminal dispositions: 512 completed/model-evaluable, seven credit-refusal terminals and two expired count-only not-dispatched outcomes. Distinct generation dispatches are 519; 439 trials remain nonterminal, comprising one count-only operation and 438 untouched. The strict saved result is 348/512 overall, separated into generation 168/268 and whole-program parsing 180/244. Generation D (kind + name) is 52/53; C (kind) 34/54; B (generic end) 25/54; A (braces) 33/54; E (padding) 24/53. These are partial exploratory observations, not completion of the 960-trial pilot. Every paired CI remains not_computed; there are no incomplete/refusal responses in this snapshot. Audit/score/report used saved data only with zero model HTTP. The source, 968 plan/request/mode artifacts, old 300 terminals and raw backing, and old event/ledger prefixes were verified. Two reused derived token-count cache durations became 0 while their exact token counts/request identities and original raw count HTTP timing bytes were preserved; 0 denotes cache reuse, not physical HTTP latency. Seven unresolved reservations retain 166,039 tokens, including all old 119,976 and two new holds totaling 46,063 tokens. No terminal retry, fallback, best-of-N, automatic charge or account-setting change was performed. See the [saved P2 report](https://kmizu.github.io/syntax-locality-experiments/data/p2/report.md), [execution snapshot](https://github.com/kmizu/syntax-locality-experiments/blob/main/experiments/data/supporting/p2-live-current-status.json) and [current execution history](https://kmizu.github.io/syntax-locality-experiments/data/supporting/live-recovery-20261007.txt). The independent frozen reading Main also remains partial and keeps its original source/protocol.

Historical local limitation: an earlier required-test attempt failed during project loading because the offline cache lacked Scala 2.12.20 metadata and the Scala 3.3.8 compiler cache was inaccessible under the restricted account (`verification/restricted-local-cache-sbt-test.log`). That earlier restricted-account attempt remains failed; the new local success above records the subsequently available JDK, cache and network environment.