That's of course, a false statement. Recent events around the big "Crowdstrike" crisis have demonstrated how cutting corners can occur even at the highest levels of the "IT food chain," i.e. in places where such "basics" should never happen.
Creating a perfect software release process is expensive and time consuming. This leads businesses to cut corners to save money and meet deadlines. The problem starts when you "don't have enough" of that process or "cut too much from it." It might look similar, but it's different.
When you don't have enough process, you burn more time on manual labor and are more prone to human errors. Your false sense of security is somewhat limited.
In the latter case, you might come from a comfortable situation where things have always worked. Your sense of security is higher because of streamlined processes. However, when you remove elements from that chain, your sense of security becomes false.
My battle-tested processes
Below is a list of my own battle-tested processes that I have relied on for the past 8+ years.
Test environments
You really need them. I made a whole video about the problem of not having one. Being able to test things properly outside of production is crucial. This can be achieved with Docker or similar technologies. Even if you don't know much about these tools, you can get an environment up and running within 15 minutes using resources like ChatGPT.
It's rare, but some companies still do development on production. Such "holy grails" still exist. Have it separated completely.
If you want to use production data - do it, but remember about GDPR. There are also projects like anonimatron that can aid that process while the changed dataset retains its production properties and connections.
Zero-Downtime deployments with DB rollbacks
You really want this. Everything around your deployment should be automated. It's a huge time saver and makes releases easier, allowing you to push code more often to production.
A bit of initial investment is required, but once you have it, things work like a breeze. Database rollbacks are easy if you're disciplined. During deployment with a DB migration, never make destructive/in-place changes. Always make additive changes. Tear down unused columns/tables a few days later, once everything is confirmed stable.
This rollback capability has saved my customers countless times.
Canary and Clandestine deployments
In the old days, miners used canaries to check for methane. Similarly, you deploy to a small number of customers first and check whether everything is okay.
Canary deploys work especially well in startups. You can get even more if you couple them with clandestine deployments. This means you release a feature, but the majority of your user base has no idea it's there. It's hidden, and those who should know can access it. It's similar to "feature toggles," but much cheaper to implement, which is a factor in startups.
Real-time in-app logging
You want to know when something bad happens. Immediately. Uptime monitoring is important, but you get an alert when it's already too late.
Products like Rollbar, Sentry, Raygun, Datadog, Instabug, or Dynatrace specialize in capturing errors and exceptions in your code and presenting them nicely. The number of issues and fires we have detected and prevented from escalating in the past 8+ years is staggering.
Testing
Testing is necessary, whether done by a human or as part of CI/CD. Your testing should involve at least two stages:
- Developers who made the feature should test it.
- People who know the requirements but weren't involved in coding should test it too, as they aren't biased.
Automated testing is optional and often skipped due to expense, especially with dynamic codebases. Use your intuition to find critical elements in your software that can take down the "house of cards." Maybe they should be tested automatically?
Conclusions
The points I wrote above have worked and still work for me and many of my customers. Each requires some customization and adaptation, as almost every business case is different. However, they form a great foundation to minimize the chances of doing the wrong thing.
Most importantly, all the things I mentioned are available within a reasonable budget.
