It is 3:07 a.m. Your online store’s database has run out of disk space, and the checkout page has started returning errors. You are asleep. Your customers, in another time zone, are not.
This story is made up. But every step of it is what really happens when a product is in our custody.
3:07 — someone is watching
Every product we put live is watched around the clock, from the outside and from the inside. From the outside, a check visits the important pages every minute, the way a customer would. From the inside, error logs and disk, memory and database usage are tracked continuously.
At 3:08, the outside check fails twice in a row on the payment page. That fires an alarm — not an email nobody reads, but an alert that demands a response.
3:10 — diagnosis starts from facts
The first rule of an incident is not to guess. The error log shows the database refused to write. The dashboard shows the disk at 100%. The cause is on the screen, not in anyone’s gut feeling.
The second rule is to separate what is cheap to undo from what isn’t:
- freeing space by deleting old debug logs is safe and reversible;
- restoring a backup over the live database is not — that only happens with explicit authorization.
Tonight, the first step is enough. The disk has room again, the database writes again, and at 3:19 the payment page responds.
3:20 — back up is not the end
The site is back, but the important question is still open: why did the disk fill up? A process had been writing overly detailed logs since the last update. If nobody fixes it, the same thing happens again next week.
The incident becomes two records in Kore: an account of what happened, with times, and an issue to fix the cause. The fix follows the normal path — standards, review, pipeline — not a rushed patch.
Fixing things at three in the morning is reacting. Making sure it doesn’t happen again is the real work.
In the morning, the message
When you wake up, the message is already there: what stopped, for how long, what was done and what is still going to be done. Twelve minutes of downtime in the middle of the night, resolved before it became a business problem.
The copies nobody wants to use
All of this depends on something you only appreciate the day you need it: backups. Every product has backups stored away from the machine it runs on, and — most importantly — restoring them is tested. A backup that has never been restored is just hope stored on a disk.
This night didn’t need them. But it is because they exist that anyone can sleep soundly.