The Software Battlefield - Vol. I
Don't Trust Your Software Architect ! if you do, then trust & verify
image credit: smartai.studio - “Warriors in The Battlefield”
Greetings and welcome to the inaugural installment of what promises to be a radically honest and revelatory exploration into the innards of software architecture, process, and leadership. With over a quarter-century in the trenches of software development, we will cover the overlooked topics that could make or break your project
First up, my favorite topic. The Architect.
Architects - Idiots, Charlatans & S/Heroes
The Pretenders: Narcissists & Silver-Tongued Pundits
Ahhhh, the charlatans dressed as architects—these are the individuals whose flamboyance is only rivaled by their lack of substance. Armed with a vocabulary chock-full of trendy jargon, they sliver into meetings with stakeholders, dazzling them into confusion while ensuring their job security.
As they slip away when it's time for implementation, their façades crumble, leaving development teams in a maelstrom of broken code and unworkable structures. Sound familiar? If you've been in the industry long enough, you have definitely come across one, and good chance they’ve caused a ton of damage to a project, team or organization
The Unassuming Titans: Architects as S/Heroes
Then we have the architects who are the unsung s/heroes of software development. They exhibit quiet fortitude, listening more than speaking. They absorb the multifaceted needs and requirements of both technical and non-technical stakeholders, synthesize and distill the problem and craft functional and effective architectures, if not truly impressive architectures, aligning business needs with technical brilliance.
Be Wary, Choose Wisely:
Over and over again, I’ve seen architects make or break a software project, a software organization and even a company, specifically startups. I’ve seen architects string projects along for months, if not years with half-truths, hand-waving, potions and shiny objects - usually to the tune of millions of dollars, only to find that once development actually starts, it was all a sham. The most important decision you’ll make in your software efforts, is choosing the right architect/s. Have a system of check and balances within your engineering org, ensure that no architect can run wild.
image credit: smartai.studio - “Abandoned Architecture”
NEVER, EVER BLINDLY TRUST THE ARCHITECT
The chasm between a project's success or failure, especially for fledgling teams or companies, hinges on selecting the right architect. Don't let that weight intimidate you; instead, embrace methods to validate your choices.
The Lost Art of Architectural Justification
Trust but verify; the old adage has never been more crucial. The practice of architectural justification—a review process involving other seasoned architects and senior developers—offers a gate through which all architectural decisions must pass.
Incubations, Proof of Concepts and Stress Testing
Proof-of-concept (POC) can serve as your litmus test. A POC can help validate the assumptions and unseen constraints in a proposed architecture. A well thought out incubation process, can help mitigate many downstream risks that are inherent in software architecture and design.
The architect should be allowed to employ developers to help with developing the POC, but they must be held accountable and responsible for the outcomes. I’ve seen many instances where architects would use a young POC developer as a human shield, when an incubation goes wrong - as a leader, don’t allow this behavior, it can negatively impact an impressionable developer while enabling bad behavior in your organization, which leads to resentment from the team. Don’t mistake silence for acceptance, they may just not feel safe to shine a light on it.
Stress testing your architecture under simulated or real user loads can unveil invaluable insights. Nothing, absolutely nothing, shines a spotlight on your architecture's limitations and possibilities like putting it under genuine strain.
I personally recommend you utilize a robust sampling of your most important use cases as your stress test cases. Each stress test case should be tested at various user loads, for consumer facing, cloud-based software systems, I tend to get the best results using the following user load factors: 1, 10, 100, 250, 500, 1K, 5K, 10K, 25K, 50K, 100K concurrent users, of course this depends on the nature of the system.
Stress testing will definitely ‘break’ your architecture, but its ok, it’s literally designed to find out where your architecture and system scalability limits exist, so the architects can make informed decisions on mitigations.
image credit: smartai.studio - “Multiverse, Intersecting Domains”
Software Domains and Runtime Dynamics
Many architects believe they fully understand their architectures and how their architectures will drive system behavior when developed, an often misguided and costly sentiment. The multiverse nature of distributed systems, especially cloud-based systems, introduces a slew of variables. From the back-end nuances like API behavior, authentication mechanisms, to database choices—each architectural decision injects unique dynamics into your software ecosystem
The Architecture Domain - Chatty versus non-chatty API interfaces, session dependent service layers, sync/async workflows and/or transactional designs, authentication and authorization mechanisms, storage and database choices. Each architectural decision injects unique dynamics into your software ecosystem.
The Development Domain - a great architecture design doesn’t necessarily guarantee the development team was able to successfully adopt the architecture, or doesn’t guarantee they didn’t take shortcuts - inadvertently injecting anomalies into the system.
The Configuration Domain - nearly any system with a backend will have a set of configurations at the coding/compiling layer, the application/process layer, the API layer and the data layer that can all impact system behavior
The Cloud / Infrastructure Domain - the choice of infrastructure, either on the cloud or on-prem introduces an entire landscape of configurations across networking, compute, system accounts, security, key management, hardware that in itself guarantee unpredictable dynamics
The User-Driven & Utilization Domain - lastly, users, with their preferences, habits, and geographical variances, breathe life into the system, often in unpredictable ways,
System Monitoring & Mitigations
Monitoring, a robust monitoring framework is indispensable for dissecting the enigmatic behavior your system exhibits under different loads. Understanding these dynamics is the prelude to finetuning the different domains effectively to manage system scalability
Mitigations, in an ideal scenario, you can effectively mitigate scalability issues by making alterations in the Infrastructure and/or Configuration Domains defined above. This may entail bumping up the compute capacity or optimizing the elastic scaling properties of a cloud VM or cloud compute capability. It becomes more costly if you need to make alterations in the Development domain, usually by opening up bug reports on the code, or introducing technical spikes to bring the code up to architectural standards, usually performed in post-production sprint cycles, requiring a developer to change already tested and delivered code, introducing significant risk
High-Stakes Game of Architectural Redress - by far the costliest set of alterations exist in the Architecture Domain. Suboptimal architecture isn't just a minor hiccup—it's often a financial sinkhole and an operational quagmire. Redressing fundamental architectural issues can trigger a cascade of changes across all domains, requiring a herculean effort, both financially and developmentally.
I’ve seen these situations over and over in my career, during the first decade of my career, I specialized in ‘hired gun’ IT contracts to go in and ‘fix’ these architecture debacles, these were extremely difficult engagements. Angry stakeholders, resentment and erosion of goodwill between business/product and IT teams and groups. More often than not, I could fix the architectural problems, but not the loss of confidence and goodwill between the teams.
image credit: smartai.studio - “Robots in Love”
A Love Letter to Architecture
Yes, I've offered a critical view of architects, but let's not forget the splendor that good architecture—and by extension, great architects—can bring to the table. There’s something beautiful in simplified architectures, distilling complex problems into elegant solutions is both an art and a science. When an architecture functions as designed, when it gracefully scales under stress, when it serves the business without straining resources, you realize it's not just about Snippets & Bits (see what I did there) —it's about creating something transcendent
Architecture, my friends, is where engineering romances art, where theory meets practice, and where ideas materialize into tangible solutions. That's not just effective software development; that's a love story for the ages.
Until our next deep dive, code safely, architect wisely and build, build, build.
~Bert