Prod Credentials Leaked Into a Dev Environment
During an incident review, you discover that three weeks ago an
engineer, debugging a discrepancy between dev and prod data, copied the
production Postgres connection string into their local .env file "to
compare the two quickly," then later copied that same .env file into
a shared dev-environment Airflow deployment to unblock a teammate who
hit a connection error — without realizing which credential was in it.
Since then, a dev-environment DAG that runs on every commit to the dev
branch has been silently reading from (and, in one buggy run, briefly
writing a test column to) the production database. Nobody noticed
until a downstream analyst flagged an unexpected column in a prod
table.
- Identify every point in this chain where a structural control — not a process reminder — should have made this impossible or contained it, and name the specific control for each point.
- The dev DAG both read from and, in one run, wrote to prod. Explain why "read-only prod access from dev" is not actually a safe middle-ground fix, using this incident as the example.
- Design the actual remediation plan: what you'd do in the first hour, and what you'd change structurally in the following week.
Share this question