Smoke Tests & Harnesses in Vibe Coding

Smoke Testing

A smoke test is the simplest check you can write — and the most important one. Does the thing start? Does the main path run without immediately blowing up? That's it.

The name comes from hardware: power on a new circuit board and watch for literal smoke. Smoke means stop everything. No smoke means keep going.

In vibe coding, Claude generates 200 lines in 30 seconds. You are not reading all of it carefully. A smoke test tells you whether the output is real before you build anything on top of it.

# Does it even start?
def test_smoke():
    from my_app import run
    result = run(limit=1)
    assert result is not None   # smoke cleared

Test Harness

A test harness is the scaffolding that lets you run code safely — without hitting real databases, real APIs, or real production. Mocked credentials, in-memory databases, fake responses. Build it once at the start of a project and it pays off forever.

Without a harness, testing means running the real thing — which means you can't run tests freely, can't run them in CI, and definitely can't run them without real credentials in the codebase.

# conftest.py — runs before anything else is collected
from unittest.mock import MagicMock
import bq_helpers

# BQ client fires at module level — patch it before any test loads
bq_helpers.get_bq_client = lambda: MagicMock()

# Every test now runs without GCP credentials or a live database

# Route write_dataframe to a local file via env var — no DOMO account needed
os.environ['DOMO_OUTPUT_NAME'] = 'My Output Dataset'
exec(compile(open('t1_prep.py').read(), 't1_prep.py', 'exec'), {})
del os.environ['DOMO_OUTPUT_NAME']

# Tile runs exactly as in production — but locally, instantly

Stop Whack-a-Mole.
Reproduce the Error.

When something breaks in vibe coding, the instinct is to describe the symptom to Claude, accept a fix, and move on. This works about 60% of the time. The other 40% the same bug comes back in a different form — because neither you nor Claude understood what actually caused it.

Before you ask Claude to fix anything, reproduce the error in a test first. Now you have something concrete. Claude fixes against a failing test rather than a vague description — and you know the fix worked because the test goes green.

# Bug: "some deals are being scored wrong" — too vague

# Step 1: reproduce it
def test_clean_deal_approves():
    bgcheck = MagicMock()              # looks fine. it's not.
    result = score_deal(bgcheck)
    assert result.decision == "approve"  # FAILS — gets "review"

# Step 2: now we understand it
# MagicMock().ftc_action_found is truthy — it's a MagicMock, not False
# The scorer treats it as a red flag and downgrades every deal

# Step 3: fix — explicit safe defaults on every field
bgcheck.ftc_action_found = False
bgcheck.ofac_hit         = False
bgcheck.risk_flags       = []

# Test passes. Bug documented. Locked in as a regression test.

Security Prompts

AI-generated code is fast and usually functional. It is not written with security in mind — it's written to pass your prompt. Claude won't volunteer that it just wrote a SQL injection vector. You have to ask.

Use these prompts at the end of every session, before you push anything.

# Hardcoded secrets
grep -r "api_key\s*=\s*['\"]" --include="*.py" .

# f-string SQL (injection risk)
grep -r 'f"SELECT' --include="*.py" .

# Known package vulnerabilities
npm audit
pip audit