Incomex AI Portal
KB-7C24 rev 5

GCP-COST-AUDIT Report — 2026-03-05

7 min read Revision 5
reportgcpcost-auditartifact-registryapp-enginebudget

GCP-COST-AUDIT Report — 2026-03-05

Mission Scope

Audit and reduce Google Cloud cost spikes after VPS migration, with focus on App Engine and Artifact Registry.

Input billing snapshot (provided):

  • Total (first 4 days of 2026-03): ₫159,806
  • App Engine: ₫144,064 (+324%)
  • Artifact Registry: ₫15,719
  • Cloud Storage: ₫23
  • Cloud Run: ₫35

1) App Engine Investigation

Findings

Project with App Engine app:

  • github-chatgpt-ggcloud only

App status:

  • Service: default
  • Version: 20241023t221130
  • Runtime: python39 (Standard, F1)
  • Deploy time: 2024-10-23T15:12:36Z
  • Version URL served: "Hello, World! This is a test deployment with ChatGPT and Google Cloud."

Traffic/logs check:

  • Query: gcloud logging read "resource.type=gae_app" ...
  • request_log since 2026-03-01T00:00:00Z: only 1 request, generated by this audit (curl/8.7.1, timestamp 2026-03-05T13:59:19Z).

What App Engine is serving

An old test Flask deployment (Hello World), not production workload.

Why cost spiked (inference)

  • App Engine default service was still SERVING and publicly reachable before mitigation.
  • This can produce runtime instance-hours when hit.
  • Retained logs in the checked window did not show sustained external traffic; cost root signal remains "orphaned App Engine runtime still active".

Platform constraints discovered

  • gcloud app versions stop ... failed: automatic scaling versions cannot change serving status.
  • gcloud app versions delete ... failed: default service must keep at least one version.

2) Full Running-Services Audit

Projects discovered

  1. gen-lang-client-0894449808 (billing disabled)
  2. chatgpt-db-project (DELETE_REQUESTED, billing disabled after decommission on 2026-03-06)
  3. github-chatgpt-ggcloud (billing enabled)

Billable resources by project

gen-lang-client-0894449808

  • Billing: disabled
  • Enabled API: generativelanguage.googleapis.com
  • No App Engine / Run / Functions / SQL / Artifact Registry / buckets observed

chatgpt-db-project

  • App Engine: none
  • Compute VM: none
  • Cloud Run: none
  • Cloud Functions: none
  • Cloud SQL: none
  • Firestore DBs: ("(default)", "test-default")
  • Secret Manager: 0 secrets
  • Artifact Registry: us/gcr.io (was large)

github-chatgpt-ggcloud

  • App Engine: default service + 1 version (legacy test app)
  • Compute VM: none
  • Cloud Run: agent-data-test (internal ingress)
  • Cloud Functions: none
  • Cloud SQL: none
  • Firestore DB: (default)
  • Secret Manager: 14 secrets
  • Cloud Storage: multiple buckets (including run/function/build/staging buckets)
  • Artifact Registry: 7 repositories

3) Project Retention Decision (Constitution: move to 1 primary project)

Recommended keep:

  • Keep github-chatgpt-ggcloud (contains active Firebase-related APIs, Secret Manager, Cloud Run service, storage assets, App Engine app binding).

Decommission executed:

  • chatgpt-db-project was deleted on 2026-03-06 by explicit user request (no backup).
  • Current state: lifecycleState = DELETE_REQUESTED
  • Billing link removed: billingEnabled = false.

Single billed project now:

  • github-chatgpt-ggcloud remains the only billing-enabled project on billing account 01ABB6-1FF01E-16010C.

4) Cost-Cutting Actions Executed

App Engine (priority 1)

  1. Set service ingress to internal-only:
    • gcloud app services update default --ingress=internal-only --project github-chatgpt-ggcloud
  2. Removed running runtime instance:
    • gcloud app instances delete ...
  3. Verified:
    • networkSettings.ingressTrafficAllowed = INGRESS_TRAFFIC_ALLOWED_INTERNAL_ONLY
    • current gcloud app instances list = no instances

Artifact Registry cleanup

Initial scan:

  • Delete candidates: 807 versions across 10 packages

Bulk cleanup result:

  • Deleted: 804 versions
  • Failed: 2 versions (manifest referenced by parent manifest; expected OCI dependency behavior)

Post-cleanup scan:

  • Remaining delete candidates: 2 (chatgpt-db-project/us/gcr.io/agent-data-mcp-gateway)
  • Reason: child manifests referenced by kept parent manifest indexes

Artifact Lifecycle Policy (automatic)

Applied to all active Docker repos in both billed projects:

  • Policy A: delete-older-than-30d (tagState=ANY)
  • Policy B: keep-most-recent-3
  • Dry-run: disabled (active enforcement)

Budget alert

Created new budget alert:

  • Budget ID: 75bae06d-e978-4d7e-af8c-d21cd8666aa3
  • Name: ₫130,000 (~$5) Monthly Alert
  • Thresholds: 50%, 80%, 100% (current spend)
  • Calendar period: MONTH
  • Email: default billing IAM recipients enabled

5) Size/Cost Impact (Artifact Registry)

Repository size change

Before cleanup:

  • Total Artifact size: 53,743.353 MB

After cleanup:

  • Total Artifact size: 5,136.247 MB

Reduction:

  • 48,607.106 MB (~90.44%)

Inference on Artifact Registry cost:

  • With ~90% storage reduction, Artifact Registry monthly storage charge should drop sharply from prior baseline.

6) Expected Post-Cleanup Cost State

  • App Engine: expected to drop to near-zero runtime cost if no internal callers invoke the legacy service.
  • Artifact Registry: strong reduction expected due 90%+ storage cut + continuous cleanup policy.
  • Cloud Run: retained (single internal service, low cost profile).

7) Remaining Items / Risks

  1. App Engine default service cannot be fully removed by stop/delete command path because of default-service constraints.
    • Current mitigation is active (internal-only + no running instances).
  2. chatgpt-db-project is DELETE_REQUESTED; final hard-delete is asynchronous by GCP and may take time.
  3. Residual Artifact manifest dependency in chatgpt-db-project is no longer material to cost because billing is disabled and project is decommissioning.
  1. Monitor chatgpt-db-project until GCP finalizes deletion beyond DELETE_REQUESTED.
  2. Recheck billing in 48-72 hours for confirmed cost slope after App Engine/Artifact cleanup + project decommission.
  3. (Optional) Decide whether to delete unbilled gen-lang-client-0894449808 for governance hygiene.