Lee Sun-woo

Give AI a test that can reveal the failure

When deleting a comment failed in Pulse, the alert said that posting had failed. Elsewhere in the same flow, crossing the 500-character limit made the submit action disappear without explaining why. A successful submission would reveal neither problem.

The October 24 correction gave the alert an action-specific title and added a count near the length limit. For AI-assisted work, the case illustrates why “test the interface” needs a more discriminating question: which action would expose the behavior being corrected?

Make the test distinguish the old behavior from the new

The length feedback begins at 450 characters and shows the remaining or exceeded count. It uses the same trimmed-text count as the submission rule, so leading or trailing spaces do not create a disagreement between the hint and the action.

A focused review can exercise the boundary directly:

  1. Cause a deletion to fail and read the alert title. It should describe deletion.
  2. Enter 450 characters of non-whitespace text and inspect the remaining count.
  3. Compare 500 and 501 characters, checking both the explanation and submission availability.
  4. Add leading and trailing spaces and confirm that the count still agrees with the submission rule.
Illustrative boundary. One character changes the count from allowed to over the submission limit.

These cases specify what to observe. A code review can confirm where the title and count come from, while an interaction check can confirm what the person actually sees. One does not automatically establish the other.

Give the agent access to the relevant feedback

In a January 2026 post, Claude Code creator Boris Cherny emphasizes giving the agent a way to verify its work, with examples ranging from a command or test suite to a browser or phone simulator. The important design choice is which feedback can reveal the failure at hand.

For this comment flow, a resting screenshot is insufficient. The useful surface must allow a failed action and text around the length boundary. The resulting report should name the case, the visible behavior and anything that could not be exercised.

Andrej Karpathy’s MenuGen retrospective describes a quick local interface followed by substantial integration and deployment work. It is a useful reminder that evidence from one environment establishes a limited milestone.

A local check can support “this interaction worked in the development build.” Claiming that readers have the fix requires checking the deployed version. Giving AI a feedback loop helps only when the observation and the conclusion describe the same thing.