Lately I've been thinking a lot about the process that I follow when I'm creating a new piece of software or designing a system with many interacting components. As my career advanced gradually, the requirements and the tasks I'm confronted with daily changed dramatically.
From creating simple web sites, over the years I found myself doing more and more backend development and designing the underlying architecture of services that are used daily by tens (sometimes hundreds) of thousands of users, under high performance and reliability requirements.
The way I approach my work now is different and in this post I wanted to summarize a set of rules for myself to follow when I'm working on such systems. I hope they can also be useful to others who find themselves in a similar situation.
Rule 1. No shortcuts
You will pay for every shortcut you make. Every time you see something that can potentially become a problem – it most likely will. Do things right from the beginning, when you're still in design / implementation phase. When the code is in production, even small problem can have big consequences and small problems also tend to add up, creating very unpredictable and mystical behavior, which is often very hard to debug in production.
Rule 2. Scalability from the beginning
Do not design system which you are not sure how to scale. During design phase when the canvas is empty, in most cases it's not hard at all to find a way to create a system which can scale, even if in the beginning you won't enable this capability.
Rule 3. Build with failure in mind
Everything fails. Define the boundaries of your system and understand the external dependencies that may fail you. Don't assume things will always go well, because they most certainly won't and you'll be better off if your system takes that into account. Make a hard decision of how the code should fail because of the dependency problem at the time when you're writing it, otherwise the behavior is going to be hard to predict when failure happens and starts propagating through the system unrestricted.
Rule 4. Performance is important
Performance equals money, it's that simple. Make it work, then make it fast is the right approach, but make sure to not build things that are slow by design. If the code is inefficient or runs on an inefficient platform/language/framework, you will be paying more for your hosting services. In my experience, a very simple rewrite of a critical piece in a more efficient manner can easily cut your server needs in half. Even on medium scale deployments, it's very noticeable.
Rule 5. Protect yourself and your users
Security is hard, but absolutely essential. Right from the start, encrypt whatever is necessary, don't use weak authentication methods, don't store user's data that is not necessary, anonymize data when you can, filter sensitive information in logs, rely on accepted industry standards for security, don't invent your own security techniques unless you know very well what you're doing.
These things will be much harder to update once the system is running in production, so you will save yourself a lot of time by thinking about this in advance. And if (or when) your security is breached, you'll thank yourself for taking all the precautions you could and protecting as much data as you could.
These rules are not final, as is my understanding and my ability to follow them in my day to day work. Yet I think they represent a solid set of base principles to keep in mind when you're building something that should last.