“Customers are getting eight copies of this report in their inbox…”

Have you ever pushed send on an email then realized too late that there was a mistake?

One time at work, I made this happen in like an automated way that was kind of a mistake. So like the require- —  I work as a software developer. And so, we had a feature that we wanted this certifications report to go out every Monday morning to customers. And so it’d be like a list of their people and their certifications that were expiring soon. And it was just — they just wanted this report to come to their email inbox every Monday morning.

And so I built that, looked good locally, looked good in the staging environment, merged it, deployed it.

But then Monday morning rolls around and customers are getting eight copies of this report in their inbox. I’m like, “Oh man. Did not mean to do that.” So we were like looking into it, looking into it. We were like, “Okay, I think this is gonna work.” And then the next Monday rolls around, customers still getting eight copies of this message in their inbox. And you know, we’re — our Customer Success Team is getting messages about this. They’re like, “Hey, I’m getting eight copies, bluh-bluh-bluh-bluh-bluh.”Yeah, it lasted for like, three or four weeks before we finally figured out, “Oh, we’re running a redundant deployment in production.” We have eight copies of our application server running for like a load balancing strategy, sort of. And, uh, yeah. Yeah, each one was sending this email message on a schedule whereas in our staging environment we just have one application server so it sent one email. So yeah, took us like almost a month to realize that. But then once we realized that, you know, we patched it, and it was fine, and it’s fine. But, uh, but yeah. Definitely, definitely understandable to send a email that you didn’t totally mean to send.

Recent Stories

Archive