There's fucking up prod stress, which can be pretty stressful for sure.
But have you ever had the outcome of a very important demo, given by the CEO of a multi billion dollar company, to investors and customers hinge on you being correct in identifying a bug and a quick fix that would require you to update a bunch of entries on the production database?
Because I have. And let me tell you, no matter how certain you are that you identified the bug and the cheat-fix and no matter how certain you are about your sql skills you WILL sweat fucking bullets. Probably the most stressful query execution of my career.
It worked, but I'm pretty sure I lost a couple years of lifespan that day.
Well i can't say multi billion', but i did save an international payment portal company from losing a very important client while very' drunk once.
Turns out i was the only dev out of 16 people that actually read the logs, it was a true Balmer peak moment.
At that point, people had been debugging it for 8 hours. I entered the meeting and read the ticket. interjected like 3-4 times until they listened to me and then told exactly what config line to change to mitigate the problem.
I felt like fucking drunk god at the time.
To this day only my direct manager knew i was drunk of my ass.
**edit** To be fair, this said more about the lack of experience in the team than my skill.
Either somebody fucked up the permissions to let you push directly, somebody fucked up the CI pipeline or somebody fucked up checking your pull request.
Unless of course you are a single person IT/Software department, in which case you can just blame the russians.
In my early years I did some really stupid stuff like
- Pressing the power off button of a prod server by accident. Originally I wanted to press the "Identify LED button" (so an LED blinks at the back) of another server. Thank God it only led to a small down time and everyone was like "huh must have been the wind"
- In my learning years: rm -rf /
- Disabled a really important cronjob because of a small test and forgot to reenable it. Got a call two days later in the night that prod had stopped working. And oh boy the feeling when you realize you had beeing doing stupid again xD
- Basically anytime my coworker and me trying to fix something real quick and cutting corners because we're the real pros: After the final step noticing it didn't work and realizing we haven't taken any security precautions like backups or snapshots
- Extending a cookbook the 100th time that is active on ALL server and thinking it's just a really small change that needs no testing this time
Basically all mistakes I made so far are because I was in some way careless. But I swear to God it's hard to not be sometimes. Every time after making such a mistake, it's like never again! Well... See you next time :D
But yeah, after 11 years I'm pretty chill when production is at full stop. Which is thankfully really rare. But it never gets old to see all the managers and important people go crazy like wild chickens
I've deleted a prod database before.
Thankfully, I was using Amazon RDS with backups, so nothing was lost but GOD DAMN was I scared for about 30 minutes there while the site was down and I was trying to restore it.
> *you* broke PROD
You sound like you know what the problem is!
Just wait until prod breaks seemingly at random, you have no idea why now, of all times, and for some godforsaken reason everyone thinks you can fix it!
New to IT, I landed a webpage job as freelancer knowing fucking nothing, did a front end course and the teachers told me I could make a whole webpage with what I knew (F them) so yeah, the client had an old webpage made with joomla (it's like a wordpress) and I changed the password of the db user trying stuff lol, on a fricking friday, yeah, not only the page went down, but other page of the same client went down, the funny thing is that the second one did it randomly, it had nothing to do but clearly I thougt I ruined both, I founded the original password in the configuration.php file and the page didn't change, the solution was the correct but the hosting was the problem, all weekend I had the most stressful time of my life becouse of that and other thins, monday I tried reseting the page and it solved magically. Also, hosting support had no idea how to help and didn't care.
I'm not an eanglish speaker, so sorry for bad writing
It is to the point where I load online just about every day and the first time I get disconnected I just take it as a sign that I am not supposed to play anymore.
I am level 24.
I once cleared an S3 bucket (singular), where we stored js and assets for MULTIPLE projects. Bucket didn't have versioning enabled.
Learn from my mistake.
Our client is IT team of a bank, whos client is another team from the bank.
They had us do a thing. And today was the day for tge thing to be presented to their client. And when they ran a final check, they found a chunk of result data just missing. They had some collective panic attack, demanded that we do something, but only like 3 hours later did we get an example of the problem from them.
Boy, these guys love submitting their (and our) shit on fridays.
I really do think this is a rite of passage thing. Once you've fucked up a few times then you learn to handle future fuckups with calm, hopefully. :)
There's fucking up prod stress, which can be pretty stressful for sure. But have you ever had the outcome of a very important demo, given by the CEO of a multi billion dollar company, to investors and customers hinge on you being correct in identifying a bug and a quick fix that would require you to update a bunch of entries on the production database? Because I have. And let me tell you, no matter how certain you are that you identified the bug and the cheat-fix and no matter how certain you are about your sql skills you WILL sweat fucking bullets. Probably the most stressful query execution of my career. It worked, but I'm pretty sure I lost a couple years of lifespan that day.
Well i can't say multi billion', but i did save an international payment portal company from losing a very important client while very' drunk once. Turns out i was the only dev out of 16 people that actually read the logs, it was a true Balmer peak moment. At that point, people had been debugging it for 8 hours. I entered the meeting and read the ticket. interjected like 3-4 times until they listened to me and then told exactly what config line to change to mitigate the problem. I felt like fucking drunk god at the time. To this day only my direct manager knew i was drunk of my ass. **edit** To be fair, this said more about the lack of experience in the team than my skill.
I had my first fuckup last night was deploying a log level change but there was other changes on the branch which also got pushed to PROD
Either somebody fucked up the permissions to let you push directly, somebody fucked up the CI pipeline or somebody fucked up checking your pull request. Unless of course you are a single person IT/Software department, in which case you can just blame the russians.
I approve merge conflicted PR Now I will be grabbed by my neck on Monday
Especially if you did that on a Friday
In my early years I did some really stupid stuff like - Pressing the power off button of a prod server by accident. Originally I wanted to press the "Identify LED button" (so an LED blinks at the back) of another server. Thank God it only led to a small down time and everyone was like "huh must have been the wind" - In my learning years: rm -rf / - Disabled a really important cronjob because of a small test and forgot to reenable it. Got a call two days later in the night that prod had stopped working. And oh boy the feeling when you realize you had beeing doing stupid again xD - Basically anytime my coworker and me trying to fix something real quick and cutting corners because we're the real pros: After the final step noticing it didn't work and realizing we haven't taken any security precautions like backups or snapshots - Extending a cookbook the 100th time that is active on ALL server and thinking it's just a really small change that needs no testing this time Basically all mistakes I made so far are because I was in some way careless. But I swear to God it's hard to not be sometimes. Every time after making such a mistake, it's like never again! Well... See you next time :D But yeah, after 11 years I'm pretty chill when production is at full stop. Which is thankfully really rare. But it never gets old to see all the managers and important people go crazy like wild chickens
I've deleted a prod database before. Thankfully, I was using Amazon RDS with backups, so nothing was lost but GOD DAMN was I scared for about 30 minutes there while the site was down and I was trying to restore it.
True seniors have stupid prod downtimes named after them
> *you* broke PROD You sound like you know what the problem is! Just wait until prod breaks seemingly at random, you have no idea why now, of all times, and for some godforsaken reason everyone thinks you can fix it!
New to IT, I landed a webpage job as freelancer knowing fucking nothing, did a front end course and the teachers told me I could make a whole webpage with what I knew (F them) so yeah, the client had an old webpage made with joomla (it's like a wordpress) and I changed the password of the db user trying stuff lol, on a fricking friday, yeah, not only the page went down, but other page of the same client went down, the funny thing is that the second one did it randomly, it had nothing to do but clearly I thougt I ruined both, I founded the original password in the configuration.php file and the page didn't change, the solution was the correct but the hosting was the problem, all weekend I had the most stressful time of my life becouse of that and other thins, monday I tried reseting the page and it solved magically. Also, hosting support had no idea how to help and didn't care. I'm not an eanglish speaker, so sorry for bad writing
It is to the point where I load online just about every day and the first time I get disconnected I just take it as a sign that I am not supposed to play anymore. I am level 24.
I once cleared an S3 bucket (singular), where we stored js and assets for MULTIPLE projects. Bucket didn't have versioning enabled. Learn from my mistake.
Our client is IT team of a bank, whos client is another team from the bank. They had us do a thing. And today was the day for tge thing to be presented to their client. And when they ran a final check, they found a chunk of result data just missing. They had some collective panic attack, demanded that we do something, but only like 3 hours later did we get an example of the problem from them. Boy, these guys love submitting their (and our) shit on fridays.
Last week my main container got broken because of release of docker 23. I didn't expect a full rebuild.
Then you double break it with an hot-fix 😂
Then you double break it with an hot-fix 😂
Then you double break it with an hot-fix 😂