Results
The planner’s motions on the step scene: the end-to-end task overnight (floor to x = 25 cm on the platform, 15 of 16 with the goal push aimed at the window’s centre, the default since Nov 10; 13 aimed at its edge; both with a temporary ±20 cm push range, without which the task fails), the learned ranker against uniform (40 vs 3 of 40), lower friction; the November 9 change to elliptic contacts, the redesigned search’s teacher results (lift 37/40, lower 40/40), the paired comparison with CMGMP, and the earlier runs on pyramidal contacts.
The newest results come first: the night of November 9–10, when the search did the whole task end to end, then November 9, on elliptic friction contacts. The sections after it are the November 8–9 runs on MuJoCo’s pyramidal contacts, kept for the record and marked as superseded.
Each clip replays calls that the planner actually made in its teacher runs (branch phase1/planar). No motion here was staged for the video. The search chose the edits (the contact changes), and each edit was executed by the local solver the run used: the quasi-static realizer Λ_qs or a learned skill Λ_RL (see The local solver). Each call was re-run in MJX from the logged start state, recorded, and drawn. The re-run was checked against the log: the same success, the same end formation, and the end pose within a few millimetres. One clip that did not match is labelled DIVERGED below.
- View. From the side, with +x to the left, as in the paper. The green surface is the platform, the grey one the floor, the blue ball the disk. A red stripe marks the face the box rests on at the start, so you can follow its rotation.
- Top-left box. The task and instance; the current segment k/n with its edit and formations (σ → σ′); the solver that ran it; the formation the recognizer reads right now, with [TIE] when two formations can’t be told apart; and whether the re-run matched the log.
- Speed. Stated in the overlay at every moment. Λ_qs calls are about 21 s long and mostly at rest, so they play at 3× while the box rests and in real time while it moves. The learned skills act in a fraction of a second, so their first 0.6 s plays at 0.25× (slow motion).
- Cards. A short hold after each segment, then the strict test’s verdict (SOLVED, or FAILED with the clause that failed).
Night of November 9–10: end to end
The end-to-end, lower-friction and quasi-static runs in this section ran with the transport’s push range widened from the stock ±5 cm to ±20 cm, a temporary setting until the parametric push certificate (below) is built. A single push could then move the box up to 200 mm. This was not stated here before (it was set outside the recorded run options). The teacher runs, the learned-ranker comparison and the November 9 results below use the stock range. Without the wider range the end-to-end task fails: see the push range decides the task.
Overnight the search did the whole task: lift the box onto the platform and push it to a set place there. The task (lift_to) asks for the box’s centre at x = 0.25 ± 0.01 m on the platform, at rest. On 16 instances the redesign solves 13 (Wilson 95%: 57–93%), and 15 with the last push aimed at the window’s centre (described below; the default since November 10), both with the ±20 cm push range. Every solve takes the same route:
push to the riser → make s → pivot → release onto {t} → disk to the box’s side face → push → push to the window.
lift_to, instance 0: SOLVED in 180 calls, 7 on the path, every call a success; the box ends at x = 0.243 m. The replay reproduces all 7 calls to 0.0 mm.
lift_to, instance 2: SOLVED in 196 calls. One call on the path (a release the realizer did not finish) still pushed the box 108 mm along the platform; the search kept that outcome and finished with a push to x = 0.240 m.
What made the platform pushes work
All evening, every push along the platform was refused before the disk moved: “the target is not in the disk’s run along the contacted feature”. The cause was the disk’s placement, not the mechanics. The relaxed gate put the disk 0.8 mm past the end of the stretch of face the controller recognises. The fix (decided by Aykut): for a push, the disk goes to the most robust of 5 placements inside that stretch, as it already did for contact changes. In the replays, two pushes that had moved the box 0 mm then moved it 48.6 mm and 47.2 mm.
Where the misses stop
The solved instances end at x = 0.240–0.245 m, the near edge of the window. A push toward the goal aims at the nearest of the goal poses, which are spread across the window, so it aims at the edge, x = 0.24, and the realizer stops a few millimetres short of its aim. Once the box is within 10 mm of a goal pose no further push counts as progress, so one instance stalled at x = 0.2374, just outside. Aiming the last push at the window’s centre (the default since Nov 10) solves that instance (ends at x = 0.247) and keeps the others solved. Paired on all 16 instances, the flag solves 15 against 13, loses none, and never needs more calls (median 148 against 180). The one instance it misses fails for another reason (its long pushes came too late in the budget).
The push range decides the task
At the stock range (±5 cm per push) the same task fails: 0 of 3 instances on November 9 and 0 of 7 on the morning of November 10 (the second run with the push feasibility check on), 0 of 10 in all. Each 200 mm slide along the platform becomes four or five 50 mm pushes, each needing its own approach, and the search runs out of its 256 calls with the box about 5 cm short of the window. The ±20 cm range works because it checks a 200 mm push as one call. Doing that properly is the job of the next build, a parametric push certificate: certify that the disk can push the box along the whole path, as a function of the position along it, instead of on a fixed 5 cm grid. These runs are the evidence for why it comes next.
A quasi-static lift, and a caveat
With releases switched off, the search must keep the box held at every instant (these runs also used the ±20 cm push range). On the evening’s code it solved lift instance 2 that way in 52 calls, 5 on the path: the pivot, a held tilt-up from 3° to 73°, and a push over the step’s corner that leaves the box upright on the platform. The replay reproduces all 5 calls to 0.0 mm.
Rerun overnight with the robust disk placement, instance 1 ran out of budget (184 calls), and instance 0 was solved through the pivot and two held tilt-up pushes (5° → 63°), with the box finally resting on the step’s corner. But its last call, a transfer that the realizer did not complete, ended with the box toppling onto the platform and being dragged about 0.8 m by the disk, which was flung off. That end is at rest and on the platform, so the search accepted it. It was a solve by the rules then, not a quasi-static one. Decided Nov 10: such outcomes no longer count. An end with the disk outside its workspace is invalid in every run, and quasi-static runs reject ends reached through a dynamic event (a topple). Under these rules this instance 0 “solve” is rejected; the video stays as the example of what the rule removes.
Lower friction
The same lift task with the disk–box friction lowered in both the planner and the simulator (with the ±20 cm push range):
40 lifting instances each, budget 192 calls, the same code throughout:
| Disk–box friction μ | Lifts solved | Wilson 95% | Median calls |
|---|---|---|---|
| 0.35 (the default) | 39 of 40 | 87–100% | 109 |
| 0.2 | 36 of 40 | 77–96% | 92 |
| 0.1 | 25 of 40 | 47–76% | 100 |
Paired on the same starts (exact McNemar): 0.35 against 0.1, 15 instances are solved only at 0.35 and 1 only at 0.1 (p ≈ 0.0005); 0.35 against 0.2, 4 against 1 (p ≈ 0.38). Lowering the friction to 0.2 costs little; at 0.1 the lift often fails. (At μ = 0.35 this run lifts 39 of 40, where the evening’s run lifted 37; that run used the stock push range and earlier code, so the difference is not due to the fixes alone.)
At μ = 0.1 the failures are not budget: the search runs out of moves, most of them at the pivot, where the release onto the platform needs more friction than there is.
Where to put the disk on a face
The 5-placement rule is a stopgap. For a disk on a box face, the moment couples to the disk’s position s only through s·F_n, and with τ = s·F_n (exact, since F_n ≥ 0) the set of positions where the box is held is an interval that two linear programs find. The margin is quasi-concave in s, so the best placement is found exactly. This rule is now in the code (off by default). Two findings came out of building it:
- When the box needs the disk to stand, the realizer keeps the disk where it already is on the box, so no placement rule, sampled or exact, is carried out there (errors up to 34 mm).
- The same lifting applied to pushing (support friction at its limit, no tipping, the disk sticking) gives the interval of positions from which a push can move the box. On 63 logged pushes it predicts failure with precision 0.88 and recall 0.79, and it flags pushes the holding test offers but that cannot move the box at all (a disk on the top face trying to drag it 182 mm). Since November 10 this check is on by default (Aykut’s decision), with a paired run against the same instances under way.
The literature (from sampling and scoring, as in Posa’s group, to exact contact regions for grasps, to convex relaxations) is collected in the parametric certification review.
The learned ranker, against a uniform ranker on the same starts
The ranker π trained yesterday evening, on 40 lifting instances from a seed the training never saw, at budget 64, paired with a uniform ranker on the same 40 starts and the same code:
| Ranker | Lifts solved | Wilson 95% | Median calls |
|---|---|---|---|
| Learned π | 40 of 40 | 91–100% | 47.5 |
| Uniform | 3 of 40 | 3–20% | 52 |
The 37 instances solved only by the learned ranker against none the other way give an exact McNemar p ≈ 1.5 × 10⁻¹¹. One caveat: π was trained with the older, looser lift goal (part of the box over the platform); every solve above is graded with the full rule, and the training data is now built with it.
November 9: elliptic contacts, the redesign, and CMGMP
Four things happened on November 9. The simulator’s friction model turned out to be wrong for a box at rest, and was changed. The CMGMP baseline was re-run on the new contacts. The lift’s pivot was fixed. And the redesigned search (Search) ran on 40 lifting and 40 lowering instances: it lifts 37 of 40 and lowers 40 of 40, and, paired with CMGMP on the same starts, lifts more within 30 minutes while CMGMP is faster at short budgets (results below).
The simulator, not the gate, was wrong
MuJoCo’s default pyramidal friction cones let a box creep off a rest that the statics say is held. Elliptic cones (impedance ratio 10), at the same step and contact parameters, remove the creep. On the box leaning on the step corner with the disk pulled away:
| pyramidal (old default) | elliptic, impratio 10 (adopted) | |
|---|---|---|
| creep, °/s | 0.40 | 0.008 |
| rests held for 10 s | 0 of 84 | 84 of 84 |
| simulator agrees with the gate (poses away from the tipping boundary) | 87 of 453 | 452 of 453 |
So the gate’s statics were right; the simulator let held boxes slide. Every scene now uses elliptic cones and full float32 matrix products. The redesigned search and CMGMP are run on them; the earlier pipeline is superseded by the redesign and is not re-run (Aykut’s decision, Nov 9). The results further down this page, below the November 9 section, used the pyramidal cones.
CMGMP on elliptic contacts
CMGMP (the contact-mode-guided planner of Cheng et al., the authors’ C++ code, patched for our disk’s 1 N motors) ran on the 50 shared test starts, 25 per task. Each start gets up to 1,800 s; CMGMP is restarted with fresh seeds until an executed plan passes. Every plan is executed in our MJX simulator on elliptic contacts and graded by our strict test. A lift counts only if the box ends fully on the platform, at rest.
| Task (25 starts) | within 60 s | within 300 s | within 1,800 s | plans claimed |
|---|---|---|---|---|
| lift | 1 (1–20%) | 5 (9–39%) | 15 (41–77%) | 22 |
| lower | 14 (37–73%) | 21 (65–94%) | 24 (80–99%) | 25 |
95% Wilson intervals in brackets. Seeds run 0, 1, 2, …, so each budget’s column is a prefix of one run. The lift needs many restarts (median 141 planning attempts per start). An earlier grading of the same lift run used a looser rule (the box only partly over the platform: 3 / 14 / 20) and is superseded.
The paired comparison with our planner on these same starts is below.
How CMGMP lifts
Every CMGMP lift we rendered takes the same route, read by our formation recognizer:
{g} → {m_x, g} → {m_x, s, g} → {m_x, g_v, s_v} → {m_x, s_v} → {m_x, t}The disk pushes the box against the riser and tips it into our pivot formation {m_x, g_v, s_v}. The floor corner lets go and the box hangs on the step corner, {m_x, s_v}. The disk keeps pushing and rolls the box a quarter turn onto its +x face on the platform, {m_x, t}, with the disk still on the face it pushed. Our planned route shares the pivot, then moves the disk round the box’s corner onto its top instead.
CMGMP, lift instance 6, seed 1 (23 s, real time). Passes under the full rule. The box drops off the pivot once for 0.15 s and regains it before the roll.
CMGMP, lift instance 21, seed 3 (15 s). Passes. The disk leaves the box for a moment after the roll.
CMGMP, lift instance 18, seed 63 (15 s). Passes, on the 64th seed: the same route, found late.
The redesigned search: lift 37 of 40, lower 40 of 40
The redesigned search ran tonight on 40 random lifting and 40 random lowering instances, on elliptic contacts, with everything as decided: the gate at the controller’s friction, the most robust disk placement (5 candidates), box transport and release edits, a search ordered by the value estimate V₀ (η = 1 + h/g), no learned ranker (every admissible edit equally likely), and no retries. A call budget caps the search: the number of times the local solver Λ may be run.
| Task (40 instances) | budget | solved | 95% CI | Λ calls per solve, median (range) | wall time per solve, median |
|---|---|---|---|---|---|
| lower | 64 | 40 | 91–100% | 10 (6–13) | 396 s |
| lift | 192 | 37 | 80–97% | 88 (52–148) | 572 s |
| lift | 64 | 6 | 7–29% | 52 (52–64) | 390 s |
On every solve the per-run check of the call bound (the paper’s Theorem C) holds. The three unsolved lifts at budget 192 ran out of things to try (the search tree was exhausted), not out of budget.
How the search lifts. Every one of the 37 solutions has the same core:
{m_x, g, s} ⇒ {m_x, g_v, s_v} pivot: the box tips onto its floor corner against the step corner
{m_x, s_v, g_v} → {t} release: the box tips onto its +x face on the platform
{t} → {m_z, t} make: the disk goes to the box's −z face
{m_z, t} → box on its +z face release: the box rolls once more and rests fully on the platformBefore the core, most solutions first push the box flush against the riser. This is not the route we had planned (the disk going round the box’s corner onto its top, a transfer that still fails); the search found a way that lets the box go twice instead. CMGMP’s lifts also roll the box onto its +x face; ours adds a second roll. Transport was tried 27 times and is in no solution.
Redesign, lift instance 2: SOLVED (52 calls in the search, 4 on the solution; 19 s). Pivot, release onto the platform, make on −z, release. The replay reproduces all 4 logged calls exactly (0.0 mm).
Redesign, lift instance 16: SOLVED (88 calls, 7 on the solution; 31 s). The disk first pushes the box to the riser (make, release, make), then the same four-edit core.
Why budget 64 lifts only 6. The run at 64 calls makes exactly the same calls as the run at 192, just cut off: 33 of its 34 unsolved instances had already reached {t}, one make and one release from the finish. Without a learned ranker every edit is equally likely, so a node’s priority falls by a factor of about 7 per level, and the value estimate shifts it by at most about 2.5. The search therefore tries every shallower branch before the platform node, four edits deep. That is a ranking problem, which is what the learned ranker is for.
How the search lowers. Every solution is one release off the platform, half of them after making the disk’s contact on the box’s +x face. No transports. The release still throws the box (known issue, below).
Paired with CMGMP
The same 25 test starts per task for both planners, the same goal test (the box at rest; for the lift, fully on the platform), one CPU core each, wall clock. A start counts within a budget if it was solved within that time. Exact McNemar test on the starts where the planners disagree.
| 60 s | 300 s | 1,800 s (30 min) | |
|---|---|---|---|
| lift, ours (budget 192) | 0 | 0 | 24 of 25 |
| lift, CMGMP | 1 | 5 | 15 of 25 |
| lift, McNemar p | 1.0 | 0.06 | 0.012 (ours only 10, CMGMP only 1) |
| lower, ours | 0 | 12 | 25 of 25 |
| lower, CMGMP | 14 | 21 | 24 of 25 |
| lower, McNemar p | < 0.001 | 0.004 | 1.0 |
Plainly: CMGMP is faster at short budgets; ours solves more lifts at 30 minutes. Our wall time includes loading compiled code on one pinned core and is dominated by simulating each call (median 396 s per lowering, 572 s per lift); neither planner was tuned for speed. One scene, 25 starts per task.
A first learned ranker
A value network V₁ and a ranker π were trained on tonight’s lowering run and the 64-call lifting run (the solutions relabeled in hindsight). On 11 held-out instances V₁ predicts the remaining Λ calls with a mean error of 0.19 calls (the graph’s hop count: 0.63; V₀: 0.77), and π puts its top choice on the solution’s edit 97% of the time (uniform: 24%).
Used in the search (95% learned, 5% uniform, so no edit is ever ruled out), on 3 lifting instances from a seed the training never saw, at budget 64: the learned ranker solves 3 of 3 (in 44, 53 and 43 calls), the uniform ranker none. Three instances are a small sample; it shows the mechanism, not a rate. V₁ itself does not yet order the search. (Update, Nov 10: on 40 unseen instances, paired with uniform, 40 against 3; see above.)
Learned ranker, unseen lift instance (seed 1, instance 2): SOLVED in 43 calls (26 s). The uniform ranker does not solve it within 64. The replay reproduces the log exactly.
The redesign’s first lowering solutions, with a known issue
In the redesign’s smoke test (three instances, a check of the code, before the full run above) the search solved both lowering instances in 13 calls each. Each solution is two calls: make the disk’s contact on the box’s +x face, then a release: the disk pushes the box off the platform and lets it fall to the floor. Both replays reproduce the logged calls exactly.
The release’s push does not set the box down; it throws it. In instance 14 the box leaves at 1.46 m/s and lands 0.54 m past the predicted landing; in instance 3, 0.19 m past. Both still end at rest inside the goal, so both count. Releases also end in the same pose whatever formation they were asked for. This is a property of the release’s scripted push, not yet fixed.
Redesign, lower instance 14 (2 calls, 12 s). make, then the release. Watch the throw: the box lands well past the step.
Redesign, lower instance 3 (2 calls, 9 s). The same two calls; the box lands 0.19 m past the prediction.
Tonight’s work: the lift’s pivot
On elliptic contacts the lift stalled at its first hard step, the pivot {m_x, g, s} ⇒ {m_x, g_v, s_v} (box flat against the riser tipping onto its floor corner). The cause was a friction mismatch. The gate assumed the simulator’s disk–box friction, 0.99, and placed the disk 45 mm below the middle of the box’s face. The controller only uses 0.35, and from there it could not tip the box: 0.01° over 680 samples. On pyramidal cones the same pivot had passed only because the disk slipped up the face.
Two fixes, both decided by Aykut tonight:
- The gate uses the controller’s friction, 0.35, at the disk. The hard break that follows the pivot stays legal.
- The disk goes to the most robust placement. The gate is re-checked exactly at 5 placements along the face, and the one with the largest margin wins. This is an interim rule: in the long run a learned pivot skill chooses its own placement (a first one is trained; compared below).
With both, lift instance 2’s pivot passes into exactly {m_x, g_v, s_v}, with the disk 44 mm above the face’s middle, and the search went on to a release onto the platform. A side finding: the exact gate passes at every disk height on that face, but the controller succeeds only from about 10 mm below the middle upward, so low on the face the gate is no test of what the controller can do. Checking 5 placements costs about 5.6× the gate at a node where the disk touches the box.
The full run above uses this fix: every lift solution passes the pivot. On 353 held-out pivot states the quasi-static realizer with the robust placement passes 348, a learned pivot skill 337, and the two together all 353.
Since then: Aykut’s quasi-static route test has run (releases off; see the night of November 9–10). V₁ and the ranker were retrained on the full lift goal on Nov 10 (V₁’s error on lift-goal pairs 1.03 → 0.33 calls; the ranker’s top-1 on lift steps 0.93 → 0.98); V₁ enters the order behind a flag, and a paired rerun decides it.
Earlier runs on pyramidal contacts (superseded)
Everything below was run on MuJoCo’s pyramidal friction cones, under which held boxes creep (see above), and with the search as it was before the redesign. These runs are not being continued; they are kept as the record of what the old pipeline did.
The tie rule (transfer_ties)
Shelved on November 9 (decision). The rule is off and kept only as a variant of the local solver. The redesign (Decision 23) replaced the final roll with a release, judged on coming to rest in the target formation or the goal, so the tie problem below no longer arises. None of the November 9 lift solutions uses it. A code review also found the implementation broader than described: it counted every sample of a transfer call as untied, not only those near the hand-over. This section is kept as a record of the pyramidal-contact runs.
The strict success test asks that an edit go from σ to σ′ with at most 40 “in-between” samples. In a transfer (one contact handing over to a related one, such as the box’s edge on the step corner becoming its face on the platform), both contacts sit inside the recognizer’s tolerance band for a moment. Those samples are ties, and the current test counts them as “in-between”. The rule (proposed then, shelved since) read a tied sample during a transfer as σ or σ′. The final dwell must still be tie-free.
Clips that depend on the rule carry an orange TIE RULE badge. That dependence is fragile. Both lift runs made the same final rolls, bit for bit. On the cluster, the run without the rule rejected them and the run with it accepted them. On this workstation’s CPU, the re-runs of two of those rolls pass even without the rule. Whether a roll needs the rule depends on where one or two tied samples fall around the 40-sample limit. So the badge says TIE RULE (cluster) when it rests on the cluster’s verdicts.
Lift: floor to platform
The box starts on the floor and has to end on the platform. The plan is the same every time:
- Λ_qs pushes the box against the riser and tips it onto the step corner, reaching the pivot.
- The learned break skill lifts the box off the floor.
- The learned roll skill sets it down on the platform.
The lift’s goal test asks for the box to be flat, at platform height, and partly over the platform (cmg/teacher/scenes.py, goal_test). Every solved lift ends with the box straddling the step edge: its centre is within about 1 cm of the corner, mostly over the floor side, and the disk presses on its top. It probably would not stay there without the disk. This is an open issue: a stricter goal, or a push-into-place (transport) edit at the end.
Solved without the tie rule
Lift, instance 2 (3 calls, 24 s). Λ_qs tips the box onto the step corner. The learned break skill flips it about 70° in 0.2 s onto its side against the corner, and the learned roll skill sets it down. Every call passes the strict test without the tie rule.
Lift, instance 7 (6 calls, 44 s). The disk first moves around the box’s top corner and back to its −x face. Then come the same tip, break and roll.
Solved only with the tie rule
Lift, instance 3 (4 calls, 33 s) · TIE RULE (cluster). On the cluster, the run without the rule rejected the final roll on the sequence clause, and the run with it accepted the same roll. The re-run here passes either way.
Lift, instance 8 (4 calls, 33 s) · TIE RULE (cluster). The same situation as instance 3.
The roll rejected without the tie rule (lift instance 9, call 71; 9 s) · TIE RULE. This one needs the rule on both machines. The box lands within 0.1 s. Then a tied sample 0.46 s later, while its face still touches the step corner, stretches the transition window to 44 samples against a limit of 40. Without the rule the strict test rejects the roll; with it, the test accepts it.
The learned break skill against the quasi-static realizer
break(g_v) from one start (lift instance 2, call 9; 9 s). Top: the learned skill Λ_RL lifts the floor corner off and holds the box on the step corner. The strict test passes. Bottom: Λ_qs from the same state only tilts the box further on its floor corner and never lets go of the floor. The departure, dwell and sequence clauses fail. On the 84 break(g_v) states the teacher’s search reached, Λ_qs succeeds on none and the learned skill on all 84. The bottom panel is a comparison run made for this video, not a logged call.
Sub-goals
The three short tasks the teacher run solves almost always (119 of 120).
Disk on the box’s −x face (1 call, 9 s). make(m_x) brings the disk down onto the face it will push.
Box flush against the riser (2 calls, 17 s). make(m_x), then make(s) pushes the box against the riser.
Pivot (2 calls, 17 s). make(m_x), then one compound edit tips the box onto its far bottom corner with its +x face leaning on the step corner. That is the pivot formation {g_v, s_v}.
Open: lowering the box off the platform
The planner solves 0 of 40 lowering tasks. The CMGMP baseline lowered 19 of 25 (pyramidal contacts, 60 s per start; now 24 of 25 in 1,800 s on elliptic contacts). The only way off the platform in our graph is a knife-edge transfer: the box’s bottom moves from the platform onto the step corner. It was tried in the 11 instances whose box started 0.6–32.8 mm from the edge; farther away nothing brings the box to the edge. Λ_qs fails it every time (52 of 52 calls). These are the two ways it fails.
Lower, instance 14, call 13 (9 s). The disk slides across the box’s top while the box barely moves (0.4 mm). It never reaches the step corner, so the dwell and sequence clauses fail.
Lower, instance 3, call 7 (12 s) · DIVERGED. Pushed from the side, the box tips over the step edge and falls to the floor. On the cluster, 6 of the 52 such calls ended with the box down, all six inside the lowering goal, in 3 of the 40 instances (3, 14 and 27). The task’s goal is met, but the edit asked for a different formation, so the strict test rejects it. The redesign decided on Nov 9 keeps such outcomes (Search). The re-run’s end pose matches the log within 7.3 mm, but a different clause fails first (persistence here, dwell in the log), so the clip is labelled as diverged.
Where these come from
- Script:
cmg/scripts/step4/videos/render.pyinc-gnn(commit4b5194b).replay off|onre-runs the logged calls, andrenderdraws the clips. Itsmanifest.jsonlists every segment, whether its re-run matched the log, and the strict test’s verdict with and without the tie rule. - Runs: the teacher runs
t-full(sub-goals and lowering),t-lift-roll(lift, tie rule off) andt-lift-roll-ties(lift, tie rule on). The tasks and numbers are on Search and ranking.