Designing a False-Positive Verification Layer

The critical finding is in <redacted>.js from cdn.<redacted>. Let me fetch that file to verify what's actually there.

The <redacted>.js file itself has no hardcoded credentials — that's expected. But the scanner found the apiKey somewhere in it. The real question is whether this is a subscription/agent key (which is intentionally public) or something else. Let me check how it's initialized on the actual site.

This is a false positive. Here's the full picture:

The "exposed credential" is the <redacted> subscription key (agent key). It's called in the site's JS as:

})('<redacted-uuid>');

This is intentionally public by design:

1. It's not a secret — the <redacted> agent/subscription key is analogous to a Google Analytics tracking ID. It identifies which account to send behavioral analytics to. The vendor's own docs expect it to live in client-side JS.
2. It's already in the CDN URL — the same GUID appears in the script's URL path, so it's doubly public just from the <script> tag.
3. It can't access the management API — the REST API (which does expose sensitive analytics data) requires a separate Integration API key generated in the vendor UI. The agent key alone grants no read/write access to the account.

The [REDACTED] in the report is PRISM Scout masking the value for display, not something the site is hiding. The value is fully visible on the page.

Good context gathered. I now understand the full pipeline well.

The Generic Secret Var pattern is the culprit — it's a broad catch-all (apiKey = "...") that inevitably hits public/intentional keys. The more specific patterns (AWS AKIA prefix, OpenAI sk-, Stripe sk_live_) are rarely false positives.

Should false-positive verification apply broadly to all secret patterns, or only to the noisy catch-all (Generic Secret Var)?

My instinct is to scope it narrowly — patterns like AWS Access Key, OpenAI API Key, GitHub Token are almost never wrong, while Generic Secret Var is inherently fuzzy. Targeting just the fuzzy pattern keeps the change focused.

Should false-positive verification work even when no LLM is configured?

Right now, if LLM_BACKEND isn't set, Phase 5 scoring is skipped entirely. We could build a purely deterministic layer (CDN allowlists, UUID detection, variable name analysis) that always runs, and optionally layer LLM verification on top. Or we could make it LLM-only and just skip verification when no LLM is available.

Where in the pipeline should verification happen? There are two natural spots:

• At detection time (inside secret_detector.run()) — each fuzzy match gets verified before it becomes a Finding. The scanner itself becomes "smart" and self-filters.
• As a new Phase between analysis and scoring — secret findings accumulate normally, then a dedicated verification step reviews fuzzy ones before Phase 5 scoring sees them.

What should happen to a finding the LLM determines is a false positive — should it be dropped entirely from the report, or kept but downgraded/flagged?

Should the LLM verify fuzzy findings one at a time, or in a single batch call?

Batching is faster and cheaper, but per-finding calls give the LLM more focused context and makes it easier to add per-finding reasoning to the output.

Perfect. I have everything I need. Here are three approaches:

My recommendation is Option A. It keeps the LLM agent layer cohesive, the verification logic is isolated and testable, and FUZZY_PATTERNS gives you a clean extension point as you find more patterns that need this treatment.

Design — Section 1: Architecture & Data Flow

The pipeline gains a new Phase 4.5: Secret Verification between subscan deduplication and LLM scoring:

secret_detector.py exports a FUZZY_PATTERNS set — initially just {"Generic Secret Var"}. Any finding whose title contains one of these pattern names is a candidate for verification.

The new prism_scout/agent/verifier.py exposes one public coroutine:

async def verify_secret_finding(finding: Finding, llm: LLMClient) -> Finding

It sends a focused prompt to the LLM with the finding's pattern name, masked value, file URL, and context snippet. The LLM returns structured JSON:

{
  "is_intentional_public_key": true,
  "service_name": "<redacted> Analytics",
  "reasoning": "..."
}

If is_intentional_public_key is true, the verifier reclassifies the finding in-place: category → TECH, severity → INFO, title becomes "Third-Party Tracking: <service>". If false, the finding is returned unchanged (remains CRITICAL).

The orchestrator loops through fuzzy findings sequentially, skips verification if llm.is_available is False, and logs each decision.

Design — Section 2: Verification Prompt

A new SECRET_VERIFICATION_PROMPT in prompts.py. The LLM receives all the context needed to make a confident judgment:

You are reviewing a potential credential exposure found in
client-side JavaScript.

Finding details:
- Pattern matched: {pattern_name}
- Code context: {context}
- File URL: {file_path}
- Masked value: {masked_value}

Determine whether this is an intentionally public client-side
key or a genuinely sensitive credential that should not be
exposed.

Intentionally public keys are designed to be in client-side
code — examples: [vendor] agent keys, Google Analytics IDs,
Mixpanel project tokens, Segment write keys, Intercom app IDs.

Genuinely sensitive credentials include: secret API keys,
service account tokens, database passwords, private keys.

Output JSON only, no prose:
{
  "is_intentional_public_key": <true|false>,
  "service_name": "<service name, or null>",
  "reasoning": "<one sentence>"
}

The context field passes the code snippet that secret_detector already captures — giving the LLM the variable name and surrounding code.

Noted — and since it's a local LLM we can pass the unmasked context snippet for better accuracy. The prompt just needs to be clear and directive, not defensive about exposure.

Design — Section 3: Extension Point

FUZZY_PATTERNS in secret_detector.py is the single control surface for routing findings to verification:

# Any pattern name listed here is routed through Phase 4.5.
FUZZY_PATTERNS: set[str] = {
    "Generic Secret Var",
}

Any future pattern added here automatically gets the verification treatment in Phase 4.5 — no orchestrator changes needed.

That covers the full design. Ready to write the spec doc and move to implementation planning?

Spec written and committed to docs/superpowers/specs/2026-03-26-secret-verification-phase-design.md. Please review it and let me know if you want to make any changes before we start writing out the implementation plan.