Correcting a Stakeholder's Reading of a p-Value

An experiment reports p = 0.03 for a lift in sign-up rate. In the review meeting three claims are made:

• PM: "There is a 97% chance the new flow is better."
• Engineer: "p = 0.03 means only a 3% chance the result is due to random chance."
• Analyst: "Since the segment test on returning users gave p = 0.20, we've shown the change does nothing for returning users."

1. Explain what p = 0.03 actually means and what is wrong with each claim.
2. The returning-user segment had 400 users versus 9,000 in the overall test. What concept is missing from the analyst's claim, and what would you compute or report instead?
3. Give the one-sentence phrasing you would put in the write-up.