DevOps experience from realPad
Follows a list of DevOps-style problems we had in realPad, along with solutions to each of them.
- Problem: we need to test new functionality before we deploy it to our customers.
- Solution: 2 almost identical environments, production and testing, created together at the beginning.
- Problem: developers need to know when something crashes on the server.
- Problem: BitBucket is down.
- Solution: nevermind, Git is a distributed source control system, meaning every developer has a “full” copy of the repository. It’s enough if the build system can be run locally.
- Problem: we forgot to renew a domain. Yes this happened It was not directly realpad.eu, but for some historic reason, the DNS was directed through this other domain.
- Solution: keep a clear DNS routing scheme, every expiration needs to be in a team calendar.
- Problem: sysadmin goes on vacation.
- Solution: on a vanilla Linux system I can do 80% of the configuration necessary to get realPad running (once I got it running on Raspberry Pi!). With the help of my colleagues, this goes up to 95%. Devs should be DevOps!
- Problem: a new version of the system broke several API endpoints, iOS, Android and web clients could not work correctly.
- Solution: just like key parts of the code should have their automated tests, the whole API should be checked (at least superficially) by a tool like Runscope.
- Problem: hosting goes down. As in, we lose our virtual servers and cannot access them anymore.
- Solution: after the necessary recovery (we lost a few days worth of data), we started taking hourly database backups, and we now rsync all the files uploaded to our system to a NAS in… well, the same city, but you get the point.