How 84 Secret Versions Cost Me More Than My Entire Cloud Run App

The $5 Budget Alert

When I deployed a small Flask app on Cloud Run a few months ago, I did one thing right: I set a GCP budget alert at $5/month. It's free. Takes 30 seconds. Then I forgot about it entirely.

The app itself is modest — a Python service that processes inbound emails every 10 minutes via Cloud Scheduler. It reads a few secrets, calls an external API, writes some data back. Cloud Run bills per request, and at ~4,300 invocations per month it comfortably sits inside the free tier. The entire thing costs $0.00 to run.

So when the budget alert email arrived — “Your project has exceeded 100% of its $5.00 budget” — I assumed it was BigQuery or maybe Artifact Registry. I pulled the billing CSV. It was Secret Manager. Fifty-five percent of the entire bill was a service I thought cost nothing.

The Bill Breakdown

Here's the actual billing CSV, line by line. The percentages are month-over-month change.

Cloud Run and BigQuery — the two services I actually think about — cost nothing. The bill was entirely support infrastructure I'd configured once and never revisited.

Root Cause

Secret Manager pricing has two dimensions: API access operations ($0.03 per 10K) and active secret versions ($0.06 per version per month). I was hitting both.

Problem 1: No client caching

The app accesses 9 secrets at startup for each run. It runs every 10 minutes. That's 9 × 6 × 24 × 30 = roughly 39,000 API calls per month. But the real issue was worse than the math: the code created a brand new SecretManagerServiceClient() on every single call. No singleton. No caching. Every invocation spun up a fresh gRPC channel, authenticated, fetched the secret, and threw the client away.

Problem 2: Secret version accumulation

One of the 9 secrets is an OAuth refresh token for an accounting API. The token rotates roughly twice a day. Every rotation wrote a new secret version via add_secret_version(). Nobody ever called destroy_secret_version() on the old ones.

Over seven weeks, that produced 84 active secret versions of the same secret. At $0.06 per version per month, that single secret was costing $5.04/month on its own trajectory — more than the budget for the entire project. Secret Manager doesn't have a built-in retention policy. If you don't clean up old versions, they sit there billing forever.

Problem 3: Artifact Registry bloat

Every git push to main triggered a Cloud Build that pushed a new Docker image to Artifact Registry. No cleanup policy. Over a few months, 26 GB of old images accumulated at $0.10/GB/month. Not dramatic on its own, but it only goes in one direction.

The 30-Minute Fix

Five changes, all straightforward. None required architectural rethinking.

1. Singleton client

Move client instantiation to module level. One gRPC channel for the lifetime of the container.

# Before: new client on every call
def get_secret(name):
    client = SecretManagerServiceClient()
    return client.access_secret_version(...)

# After: module-level singleton
_client = SecretManagerServiceClient()

def get_secret(name):
    return _client.access_secret_version(...)

2. In-memory cache with TTL

Secrets don't change between invocations. A simple dict cache with a 5-minute TTL eliminates redundant API calls entirely.

import time

_cache: dict[str, tuple[str, float]] = {}
CACHE_TTL = 300  # 5 minutes

def get_secret(name: str) -> str:
    now = time.time()
    if name in _cache and now - _cache[name][1] < CACHE_TTL:
        return _cache[name][0]
    value = _client.access_secret_version(
        request={"name": f"{name}/versions/latest"}
    ).payload.data.decode("utf-8")
    _cache[name] = (value, now)
    return value

3. Auto-destroy old versions on rotation

After writing a new token version, list all versions and destroy everything except the latest three. This caps version accumulation permanently.

def rotate_secret(secret_path: str, new_value: str):
    _client.add_secret_version(
        request={"parent": secret_path,
                 "payload": {"data": new_value.encode("utf-8")}}
    )
    # Destroy all but latest 3 versions
    versions = _client.list_secret_versions(
        request={"parent": secret_path,
                 "filter": "state:ENABLED"}
    )
    enabled = sorted(versions, key=lambda v: v.create_time,
                     reverse=True)
    for old in enabled[3:]:
        _client.destroy_secret_version(
            request={"name": old.name}
        )

4. Artifact Registry cleanup policy

One gcloud command. Keep the 5 most recent images, delete everything else automatically.

gcloud artifacts repositories set-cleanup-policies REPO \
  --project=PROJECT_ID \
  --location=us-central1 \
  --policy=policy.json

# policy.json
[{
  "name": "keep-latest-5",
  "action": { "type": "Keep" },
  "mostRecentVersions": {
    "keepCount": 5
  }
}]

5. Path filters on the deploy workflow

The Cloud Build trigger fired on every push to main — including README edits and config changes that don't affect the container. A path filter in the workflow stops unnecessary builds and the Artifact Registry bloat they create.

# cloudbuild trigger or GitHub Actions
on:
  push:
    branches: [main]
    paths:
      - "src/**"
      - "requirements.txt"
      - "Dockerfile"

The Results

Projected monthly cost after the changes:

The biggest win is Secret Manager: from $3.15 to roughly $0.09. That's a 97% reduction from two changes — caching the client and cleaning up old versions. The version cleanup alone was responsible for most of the cost. Eighty-four versions at $0.06 each is $5.04/month for a single secret that only ever needs one active value.

GCP Cost Hygiene Checklist

Everything I wish I'd done on day one. Screenshot this or bookmark it — it takes 20 minutes to implement and will save you from the same surprise.

None of this is complex. The total fix took 30 minutes. But the default behavior of most GCP services is to accumulate silently — versions, images, build minutes — and the billing console doesn't surface it until you go looking. The budget alert is what made me look. Everything else followed from reading a CSV file and asking “why is this line item here?”