Things I learned doing long-term projects

In my 7 years of professional web development I've been fortunate enough to experience a few long-term (6+ months) projects from start to finish (does it ever though?). And during these projects I've always stood on the shoulder of giants; amazing people I've been lucky enough to have worked with. Interesting minds, approaches and different ideas all aiming for the same goal: "Lead the project down a successful path". With that said, I'd like to pass on the knowledge I've collected through this blog post. This blog post will be a bit chaotic since these are just different points of attention throughout the entire lifetime of a project, but I hope it can be of use for you.

Who am I working with?

You can learn a lot from how people communicate and act in certain situations and there are lots of models that you could refer to. Using those models might be a good idea but I can never remember how they work and don't have the interest and time to get out my notebook and score and label them. For me it always comes down to what the Dutch call mensenkennis ("judgement of character"). Identify those who are logical thinkers, analytical thinkers, concrete thinkers, etc. Use that knowledge to involve the right people at the right time and cater to their needs when necessary. Knowing this is key to navigating and communicating with your team. You will then know who to involve in working through the domain's complexity, tackling user interaction problems or deciding on certain priorities.

Workshops

When starting a project, you need to figure out what you're building. When you're not familiar with the product, this is like looking at a world map covered in Vantablack. There's a world of knowledge beneath that layer but revealing that all at once is like trying to read an encyclopedia in one go reading each page in random order. The knowledge won't stick and there's probably no structure to the information you're receiving from the product enthusiasts (domain experts). But then how do you start?

Well, it depends (I'm allowed to say that now). But my go to method is doing some fun ol' exploration sessions. You probably do have a general idea of the product to be build, so pick any process you think exists and organize a workshop around that. The whole point of the workshop is to do some domain exploration. How you do this is entirely up to you, whether you simply start drawing out the process, you organize an event storming session, or do some user story mapping. The only advice I can give here is that you should base that decision on the vibe you get from the domain experts. If they're experienced with scrum, they might prefer user story mapping. If they have technical affinity you might choose to do some event storming. Identifying your team members as explained in the previous chapter will greatly help you make this decision.

The goal is to explore the domain, so try to identify the key concepts you hear and put them on separate stickies so they start to become a more prominent part of your language. I like to focus on language early on, it usually exposes some existing problems and you might start to see some struggle with their domain. This is also the point where your team realises that the purpose of this workshop is not only for you to gain knowledge but also for them to structure theirs and work out some of the kinks. Remember, discussions are healthy, it's ok to have them but try to judge whether this is a discussion worth having now rather than later.

The more you go through this process of exploration, the more you start to understand whether the complexity is at. Try to focus on that but don't overwhelm yourself, start easy and work your way around it. But it's essential to keep everyone actively involved, knowledge it's not something you document easily. It should become a story everyone can tell thanks to this shared understanding and concise language you develop. Don't feel bad to question any terms coined, just because it's always been called that doesn't mean it's necessarily correct.

Prioritization and decision-making

Prioritization is hard, but it's not magic. When you find yourself having a hard time prioritizing your backlog, try to take a step back and lay down a better foundation for your product. I tend to start with a product vision board as it's quite effective at making you question everything about your own product. This product vision board should give you a decent starting point in defining your MVP (Minimum Viable Product) or MLP (Minimum Lovable Product). If it's still tricky you can also try rating your different components using the MoSCoW method. Get creative! But focus on laying down a good foundation, this is what helps your team make autonomous decisions.

Going full circle at the start of your project

The amount of times I've regretted dealing with something later rather than earlier is painful. Here's what I learned doing long-term projects

Your infrastructure should be setup in sprint 0

What that infrastructure is depends on your vision, but having a infrastructure for all your environments early on helps you tackle the usual complexities like load balancing, zero-downtime deployments and failovers. Being able to test and use your infrastructure during the development will give you a huge advantage later on (especially when using new tech). Sometimes setting this up depends on a third party, this inherently means there's risk involved. Eliminating that risk by tackling this early on will help clear the way to a more predictable roadmap.

Monitoring is always relevant

How do you know your application is running as it should? I can assure you that clicking through the checkout process is not the answer. Automated tests? Sure! But those can't detect whether your session storage has suddenly died after running flawlessly for 143 days. And how's that queue doing? It seems to be up, but are the messages being handled properly or is it spitting out errors in STDOUT or /var/log/errors.log? Creating a proper dashboard and setting up some log aggregation goes a long way in detecting problems before they become a bigger problem. There's a plethora of tooling out there that can deal with monitoring (ELK stack, Prometheus, Sentry, etc.), but it requires some discipline to make it a priority now rather than after the problems start arising.

Plan B

Your application will fail, at some point. And that's fine, anyone with a sane mind knows that's inevitable. But what is plan B? How do you plan on recovering? Now this might be hard to predict, but it's not a crazy idea to have some way of intercepting a process as a developer without having to risk your data integrity by manually editing the database. One way of introducing this flexibility is to have some CLI tooling available to manually kick a process that didn't fire for some reason. In one of the projects I've been a part of we even developed entire web-based interfaces that would give certain people a bit more power in controlling the data that is normally hidden. Sometimes you don't control all the variables, so account for that appropriately.

Shared technical vision

When working in a large team it can be hard to maintain a certain quality or vision. As I mentioned before it's incredibly important to lay down a good foundation for your team to make decisions on. The technical vision of a project is part of this foundation. I don't suggest this is shaped into a bible-like document, rather it should be a story that lives on and develops itself as you progress. This requires attention and care, but can greatly reduce the amount of discussion you have during code reviews and other time-pressing points in your development workflow.

I once stepped into a project as a lead and what hit me the hardest was the never ending stream of questions and overall confusion about how to deal with some of the architectural decisions that had to be made. After being overwhelmed by this I decided to schedule what we then called an Architectural Exploration (not purposely based on any existing methodology). The goal was to have everyone raise some of the pain points they came across during development and we would discuss them with the goal of finding a vision that would eventually eliminate these pain points. Key to making this a shared vision was to invite everyone (we even invited the tester) to take part in this process, fully utilizing the collective knowledge and experience. This also helps everyone understand why decisions are made rather than having those decisions thrown at them as a take it or leave it ordeal.

Final thoughts

There's a lot more to cover, some topics deserve their own blog post or maybe even an entire talk. But I feel like I've covered some of the basics that are now a part of my handbook to starting a new project. I'd be lying if I said that I follow the above to the dot... you can't account for everything.

I invite you to tweet or otherwise share your feedback regarding the above, I'd love to hear what your experience is and what heuristics you've come up with.