It’s early 2024 and S3 continues its ubiquity in computing. The often touted adjectives “serverless” or “cloud-native” usually mean—”we’re using S3 for something.” Amazon continues to push the boundaries of scale and further solidifies itself as the foundation for countless applications, platforms, and companies.

Recently, Amazon released their S3 Express One Zone Storage Class, advertised as the “fastest cloud object storage for performance-critical application.” In this post, I’ll highlight three key features of Express One and share some guesses regarding their implications on S3’s internal architecture.

S3 Express One Zone Highlights

Single Availability Zone

Up until now, S3 has always been scoped to an AWS region. With S3 Express One, your data is now scoped to an Availability Zone (AZ). By enabling this, you sacrifice a bit of durability (no longer tolerant to AZ failures), but gain the performance benefits of co-located storage and compute.

Crazy Default Throughput

Classically, S3 enables up to 5.5K Requests-Per-Second (RPS) GETs (3.5K RPS for updates) for a given prefix (path leading up to an object’s key). Customers can easily bump up their throughput by manually adjusting their prefixes. However, if you’re on S3 Express One, you’ll get hundreds of thousands of RPS out of the box.

Support For New Medias (SSDs?)

Single-digit millisecond access times for persistent storage sounds excellent. The S3 team has never explicitly mentioned SSDs, but I believe it’s highly likely that this kind of performance is backed by new storage medias. Behind Amazon’s “Storage Classes”, there are strong signs that they are investing in new hardware types alongside new and novel software techniques.

The Vision

S3 is becoming the infinitely durable, infinitely scalable hard drive for the Internet. Let’s dive into each of these areas in a little more detail.

Regional Scope to AZ Scope

S3 is a regional service where the availability of one region (us-east-1) can never affect the availability of another region (eu-west-2).

The first implication of AZ-scope is that latency improves for both writing and reading.

Whenever you upload an object, S3 will break it into many pieces, or shards. The important part of this procedure is not the sharding itself, but how the shards are distributed. The distribution of the shards—and the healthy upkeep of this distribution—is what gives S3 its 11 9s of durability.

Even though the algorithm for the shard distribution is proprietary, we can guess at what properties must be maintained at all times:

1. Given some amount of faulty hardware in an AZ (disks, rack, etc.), your data must still be available.

2. Given the loss or partial loss of an AZ, your data must still be available.

Before your PUT request returns a 200, your object has been already been broken up and distributed in a way that meets these requirements. S3 would also employ heavy amounts of background workers to ensure these properties stay true at all times.

With S3 Express One, one improvement with write latency is that your object will only need to be sharded and distributed to a single AZ instead of a full region. This is important for latency-sensitive applications because even ~60 miles between AZs makes a difference.

For read latency, access time for data are improved if you’re able to co-locate compute with storage in the same AZ. Latency is further improved by new media types which we will discuss soon.

An interesting side-effect of Express One’s AZ-scope is that it may influence how companies choose to deploy compute. In order to be available in the face of AZ failure, you may have set up K8S clusters to deploy regionally (across many AZs) by default. However, if you want to get the performance benefits of S3 Express One, you will have to start being explicit with where your compute is deployed to and subsequently more careful with handling AZ failures.

100s of 1000s of Requests Per Second!

With classic storage classes, S3 gradually scales with you as your application accesses your objects more frequently. For a given prefix (/my-bucket/analytics/2023/jan_1.dump) the limit is 5.5K RPS on GETs and a similar order of magnitude for updates. Customers are empowered to scale by adjusting prefixes (/my-bucket/{some-distribution-of-values}/analytics/2023/jan_1.dump) in order to gain more throughput, e.g allocate 10 prefixes and get an aggregated 55K RPS.

The one drawback of this is that it does not support bursts of new traffic well. S3 doesn’t start by giving you 5.5K RPS; it needs time and an smooth upward slope of traffic to scale your prefix up. Given that S3 is marketing itself as the premiere storage for latency-sensitive and AI/ML applications, can’t they just give us more throughput from the get-go?

It turns out that Amazon S3 can easily give you this throughput without breaking a sweat. For many multi-tenant software services, there is a common partitioning scheme that software engineers follow:

1. Your application is getting off the ground. You make the reasonable choice to partition by some tenant_id or customer_id.

2. Your application is successful and now some tenants are orders of magnitude larger than others. Hot spots start popping up. Instead of re-architecting everything, you opt for some kind of selective partitioning, e.g. Kim Kardashian’s X account gets special treatment in the backend.

S3 has actually reached a third level of scale where #2 is no longer an issue for them. With Amazon’s ultra-scale, the aggregate of all of their workloads (no matter how sporadic and bursty they may be) will be smooth—and subsequently the partitioning scheme may be very simplistic.

S3 may be one of the few software services in this world that can do this.

My guess is that for classic buckets, S3’s partitioning schema might have looked something like {customer_id}_{bucket}_{object_prefix}_{hash_object_key_and_other_metadata}. If you imagine this being the partitioning scheme for a consistent hash ring, then it would be convenient for the architecture to slowly scale up for a given customer, bucket, or object prefix.

With S3 Express One, you are forced to use the new “Directory Bucket.” Under the hood, I believe this is actually just a way for the S3 architecture to utilize a different, probably more uniform, partitioning scheme, e.g {one_hash_of_many_fields} specifically for Express One objects.

Given Amazon’s trillion-object scale, they’ve grown beyond hotspots and have no problem distributing heavy amounts of your application’s load to their fleets. This is a scenario where insane scale actually makes things easier.

And if that wasn’t enough throughput for you, S3 has even more ways to increase parallelism such as multi-value DNS entries and multi-part upload requests.

Single-digit Millisecond Access with ShardStore

The release of S3 Express One represents Amazon’s commitment to support high-performance, latency-sensitive applications. Up until now, S3 has scaled to amazing levels with commodity HDDs. They have focused on durability and availability, but raw performance has never quite made it to the top of their requirements list—until now.

Assuming that the mechanics of HDDs are not sufficient for high-performance applications, S3 has shown signs of evolving to support new hardware, or in marketing terms “Storage Classes.” Managing persistence of brand new media types would necesitate significant changes in the software and data structures used. Said another way, it would not be realistic for S3 to keep their existing software and simply swap out the hardware.

For the last few years, Amazon S3 has been developing and gradually deploying “ShardStore“, a new Rust-based service dedicated to managing the storage of shards (pieces of your S3 objects) which I believe may be purposefully designed for SSDs. The linked paper focuses on verification techniques of ShardStore, but they include a short section on its design which we’ll review.

Introducing a New Style of LSM Trees

If S3 wants to consistently deliver the first bytes of data to you within single-digit milliseconds, they will need hardware that is exceptional at random access. SSDs have their own trade-offs (e.g. expensive), but the one area where they shine is with random access.

One challenge of LSM Tree databases is that there can be a high amount of write and read amplification. For a given write, layers of SSTables may be triggered for compaction. Similarly for a given read, layers of segments may need to be seeked to and scanned.

The paper “Internal Parallelism of Flash Memory-Based Solid-State Drives” produces some important test results regarding SSDs:

1. Concurrent read and write operations can cause “interference” that can negatively impact performance.

2. In good conditions, when random reads are issued concurrently, the aggregate throughput can match sequential throughput for some workloads.

Another other important property of SSDs worth mentioning is that their health and longevity is correlated with how often you write data due to the physics of the “Program/Erase” cycle.

Bringing all these points together, we come to the conclusion that classic LSM Tree data structures, with their heavy amounts of I/O amplification, may work effectively for HDDs, but would not be able to bring out the full potential of SSDs.

Value-Less LSM Trees

The only official publication I could find about ShardStore was in the paper “Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3.” An important note is that the design of ShardStore is heavily influenced by WiscKey (2016).

In a sense, these new LSM Trees are “value-less”—instead of holding the KV value pairs, the tree maintains pointers to disk offsets where the actual values (data bits of your S3 object) reside. The primary advantage here is that the size of the tree is significantly reduced and subsequently write amplification may be significantly lower. Of course this new setup introduces its own challenges and complexities, but there must be a compelling reason why the S3 team has chosen this architecture!

Complexities and Their Potential Solutions

Conceptualizing the Complexity—Managing Two Things Instead of One

In a classic LSM Tree, we conceptually think of one very large tree. The tree contains all KV pairs and is the data. This data might exist in a memtable or in various layers of SSTable segments.

Now take each KV pair in the tree, move the values into special regions on persistent storage, and replace those values with pointers to those regions. Conceptually, the database is no longer a tree, but the combination of a special LSM Tree and various regions of storage.

The following diagram (taken from Amazon’s paper) is a high-level overview of ShardStore’s architecture. An LSM tree still exists, but is now significantly smaller because it only containers pointers. This technique also enables a lot of flexibility with where the actual data can be placed, e.g. ShardStore might manage multiple physical SSDs drives and selectively “spread out” the writes on the SSD to promote hardware longevity.

To repeat the previous key point again—our conceptual “database” now consists of two essential components: the value-less LSM Tree and all the persistent storage it references.

When considering this approach, the software has to keep both the tree and the extents synchronized at all times. The tree, alongside all its pointers, is consistent with the chunks of data being persisted in extents.

If anything in the tree changes, a corresponding extent would need to be updated. Similarly, if a background process were to clean up an extent, then something in the original LSM Tree would need to be updated.

If you compare this to the classic LSM Tree, you may notice the delta in complexity. When a classic LSM Tree goes through background compaction, it’s just the merging of SSTables segments; no memory or special pointers need to be touched. When you write data, you edit the tree in the memtable and that is the only way an SSTable can be updated.

Now think about how merging and compaction might work with a ShardStore LSM Tree—it’s not as simple as just combining a few immutable files on disk.

Range Queries Still Work

One of the defining characteristics of classic LSM Trees is that all keys and values are in sorted order. To read a value, or range of values, we locate a particular SSTable (usually with some sparse index) and can perform a sequential scan (potentially across many layers of files) to get our content. This is not extremely fast, but it’s also not extremely slow as HDDs are quite good at sequential scans; most of the time spent on reads is for the seek.

As we mentioned earlier, ShardStore’s LSM tree contains just pointers; the actual data isn’t there anymore. Getting a range of key/values is no longer a simple sequential scan of one file; it now becomes potentially a bunch of random access queries into persistent storage. This would perform extremely poorly on HDDs.

The WiscKey paper mentions that when random access reads are issued concurrently on an SSD, the aggregate throughput can actually match sequential throughput. Said another way—SSDs are really good at random access reads.

This is one of the main reasons why it’s likely ShardStore’s value-less LSM Trees are purpose-built for SSDs. The internal parallelism built into SSD hardware can support this data structure’s access patterns whereas this setup would be completely untenable on spinning disks.

Reclaiming Space

In classic LSM Trees, the process of reclaiming space is achieved via background tasks which merge and compact SSTables. This process is functionally simple—it’s a pseudo merge-sort between files. The challenge with these tasks is that they may cause heavy amounts of write amplification and even get in the way of the database serving requests.

With ShardStore LSM Trees, we have one small LSM Tree that still goes through normal compaction and merge procedures. However, the actual bulk of space to be reclaimed will be in extents. ShardStore introduces a lightweight background “garbage collection” task that handles this.

In ShardStore, the extents are append-only. There is no way to immediately “delete” or “free-up” a chunk in an extent after a shard is deleted. However, we do have a way to mark a chunk as invalid. Consider a DELETE request for a given shardID, the only thing that has to happen is that the shardID’s entry in the LSM Tree is removed.

In the diagram above, the gray box in extent 18 is invalid, i.e. the shard that was referencing this chunk was deleted.

A reclamation task running on extent 18 will:

1. Iterate through every single chunk from the beginning of the extent

2. If the chunk is still referenced by the LSM Tree, it is still VALID and we append it to another extent

3. If the chunk is not referenced by the LSM Tree, it is INVALID and we continue on to the next chunk

4. Once all chunks are evaluated, the write pointer for that extent is reset and the space in extent 18 is reclaimed.

As you can see, this process is quite different than a classic compaction/merging process for an LSM Tree!

Crash Consistency

Finally, another major difference between these two designs is crash consistency. How is everything left in a consistent state after unexpected failures?

In classic LSM Trees, this responsibility was reserved for the Write Ahead Log (WAL) that would be initially updated at the start of every single operation. ShardStore and WiscKey state that they do not bother with a WAL and still manage to ensure crash consistency. I won’t review the strategies here, but please check out the papers if you’re curious.

Conclusion

S3 Express One Zone is a huge step forward for Amazon. S3 is no longer a big bucket full of image and video files, but is becoming the insanely reliable, durable, and performant hard drive for the Internet. Every day a new company pops up with a “serverless” platform that leverages S3 under the hood. Amazon S3—alongside YouTube, Netflix, WhatsApp, etc.—continues to strengthen its presence as one of the modern wonders of the computing world.

If you care enough about the work, you probably have these feelings often.

Maybe there’s context you’re missing and you’re being too sensitive. Maybe the implementation is fine and you’re being too pessimistic. But—maybe you’re right and you have the opportunity to save the company years of toil and unmeasurable amounts of opportunity loss.

Realistically, you probably can’t affect the situation. Who are you to call off an important project? Who is another person to question your work? Everyone, from managers to executives, has a hard time with this. Nevertheless, your opinion matters; you might be able to nudge the ship and prevent a terrible outcome. At the very least you can try.

Why bother? When is this appropriate? How do you even communicate this?

Why

As a professional, you have the responsibility to voice concerns affecting the success of organization. Stopping what isn’t working is equally important as investing in what’s working.

If not for the sake of the organization, raise the alarm for the sake of yourself. Practice your seniority. Give a proper escalation with measurable impact, all while communicating effectively and respectfully to the right people. Few can do this.

Honestly, it doesn’t matter if you don’t get your preferred outcome; just knowing that you spoke up counts for something.

When

Consider voicing your opinion when stakes are high:

• Large change in architecture

  • e.g. Initial steps towards a monolith to microservice migration, first attempt at event sourcing.

• Large change in deployment

  • e.g. On-premise to “hybrid” to cloud, back to on-premise, and everything in-between.

• Large change in development

  • e.g. Splitting all the repos, merging them all back, adopting a tool, deprecating it later.

If your development environment is distributed and isolated between teams, changes are less likely to have broad impact. However, watch out for situations where you’re working in a highly-shared system. This is when any change easily becomes high-stakes. As with anything, the more people it affects, the more significant it becomes. Here are some other questions to ask:

• Is the project overly ambitious?

• Is the project taking forever?

• Is the lead engineer on the project too defensive?

• Are the goals of the project now irrelevant with respect to the company’s strategy?

If you’re having uneasy feelings, I guarantee that there are other engineers feeling similarly. It’s common for a group of people to question something but to collectively do nothing about it.

If you care enough and the stakes seem high enough, this might be the time to do something.

How

How is the trickiest.

If you’re a senior engineer, who are you to raise the alarm on projects that support the Principal Engineer’s 5-year strategy? High-stakes changes are accompanied by strongly-held opinions. Someone has spent a lot of time thinking about this. If you believe there is a fundamental flaw in this work, things will inevitably get personal. As much as we’d like to believe we are detached from our work, we are not.

99% of the time, you should not attempt a direct approach. It’s unlikely that expressing your concerns in a 30-minute call with that Principal Engineer leads to the company killing a project.

Throughout the process, keep an open mind. You don’t have all the information. You will learn more about the issue as you engage with it. If at any point in the process your worries are assuaged, feel free to stand down. If at any point in the process you feel like your concerns aren’t being properly comprehended, feel free to push on.

Find a Supporter

I recommend the “How” to be indirect. Raise your concerns with someone who is not involved with the project.

The default person is your manager. You need to communicate your concerns clearly and share the reasons why you believe this project will not succeed. It’s important to include an appropriate severity level—what is the actual impact? Too often engineers raise critical issues only to have a follow-up be placed at bottom of their manager’s TODO list.

I believe this {architecture change, adoption of new tool, migration, insert project here} is unlikely to succeed because we {don’t have the right expertise in place, don’t have the right staffing in place, have overlooked a fundamental design issue, insert concern here}. If we continue to pursue this, we may be at risk of {accumulating a significant amount of tech debt, risking a heavy blow to morale, severely destroying our velocity, insert other trigger phrase here}.

You

Communication goes both ways. You need to simultaneously be a good communicator and a good listener. For the managers, pay attention to subdued escalations—do you think the engineer is downplaying the situation to not upset someone? For the engineers, be clear about impact and get your point across—is your manager actually connecting with you on the severity level?

Hopefully, you deliver your concerns effectively and your manager decides to take action. Consider this effort a close collaboration; this is not a baton pass from you to the manager. Use the manager to assist with back-channel alignment at higher levels. Write a concise summary of your concerns (an “asset” if you will) that your manager can reference and disseminate. Work together on how to propose additional accountability for the project at different levels. Figure out a way to get the right people in a room with the right set of expectations.

In the situation where your manager does not perceive your warning to be particularly significant, you may want to take things into your own hands. This will be de-motivating, but if you still care enough to do something—good for you.

Your feelings of unease are shared. Locate others that have similar concerns and consolidate multiple voices into one. This will give your collective opinion more weight. You are now the voice for the group. The nice part about this is that you’ll be more motivated to keep pushing forward knowing others are with you.

With a stronger signal in hand, find an objective third party, as senior as possible, and have the same conversation as you tried to have with your manager. For example, approach a Principal Engineer who has nothing to do with the project and get their thoughts. The goal here is to connect with as much seniority as possible, convincing them that they need to care about this.

If the third party engages with you, use your discretion to get a feel for how they’d like to follow up. Are they taking things seriously and willing to follow up without you? Are they mildly interested and want to get more data points? Are they not interested whatsoever? You may need to approach multiple people. Most of the time, an objective third party doesn’t have a stake with your personal concerns because it’s not their job to worry about your concerns (like your manager’s is).

Whether or not it’s your manager or a random Principal Engineer, your goal here is to have multiple people care about this as you do. If you’re actually able to do this (the hard part), subsequent pieces will flow naturally.

The Outcomes

The good outcome is impossible to measure. If you escalated an issue and stopped a doomed project, you will never know how much potential time and money you saved.

The bad outcome is when you do nothing and the project continues. The work causes years of pain until finally an executive pulls the plug and you were proven right. In these situations, do not give a “I Told You So” speech.

The probable outcome is that you raise your concerns, but the project continues and eventually fails. Upon failure, your contribution may or may not be retroactively recognized. The unfortunate (but still positive) outcome is that you and the organization learn from the experience.

What is actually important?

The actual organizational outcome is not that important. You will rarely succeed at “raising the alarm” and stopping a project. Companies learn the hard way through wasted time and excessive tech debt.

What is actually important is the action you took.

The act of maturely and professionally raising a concern is what’s important. You practice your seniority by aligning the right people, communicating impact, and getting your opinions out there.

If you are able to do this—or even just make an attempt at it—then consider this whole effort a win.

What is actually hard?

The hard part in this whole process will be that you’re going up against another person. It will take courage to raise your concerns and in doing so you will inevitably upset people. Very few can truly detach from their work. This includes you as well; you will need maturity to concede when things don’t work out your way.

If you care, you can at least try.

Either way, you should not be attached to the outcome.

Software project planning is extremely difficult. In every step of the process—design, build, test, release—a multitude of things will go wrong.

For all the senior engineers, team leads, and engineering managers embarking on a challenging software project, here is a cheat sheet of activities to ensure your work runs as smoothly as possible. Every situation will be different, but common challenges frequently re-occur.

If you are leading such a project, keep these 3 themes in mind as they may easily run away from you:

1. Expectations—are the timelines and goals still reasonable?

2. Alignment—is leadership properly aligned and sponsoring this work?

3. Feasibility—given the resources, can this thing be built properly?

Best of luck.

The post Software Project Planning appeared first on David Xiang.

Art is subjective, soft, human, unique.

Science is objective, formulaic, algorithmic, process-heavy.

Think about a recent problem you faced and its surrounding context. Was your approach more artistic or more scientific? Did your perception change as you solved the problem? How did it compare with your colleagues?

As with anything difficult to define, let’s begin with principles:

1. Nothing is all art or all science; everything is on a spectrum.

2. Any positioning on this spectrum is subjective.

3. A common interpretation is that science leads to art. A scientist invents after decades of study. A dancer expresses after decades of practice. Said another way—science at a master level feels like art.

With respect to software development, which day-to-day activities lean one way or the other? What is art and what is science?

Art or Science?

Art

The Difficult Road To Consensus

“Alignment” is one of this year’s nominees for the Most-Used Corporate Buzzwords Award. Jokes aside, alignment as an outcome is critical. We flounder without it. While rough science can be drawn from alignment-building frameworks, it will always be more of an art. This is because it will always depend on people and how you influence them. This one dependency is why consensus can sometimes be so difficult to achieve.

Partnerships Not Contracts

Partnerships exist at every level of an organization. Individual contributors work across teams. A Team Lead collaborates with his or her Product and Design counterparts. A CTO matches with a CPO.

A common mistake (and subsequent growth opportunity) is when an analytical engineer approaches a partnership as a science. This is natural if you come from a world of APIs and interfaces. However, a contract of deliverables will never produce a healthy working relationship between you and a PM. There is a time and place for contracts, but a professional partnership is not one of those places.

Coaching Through Patterns

Coaching represents a healthy mix of art and science. If I had to pick one, I believe that art edges out science—but barely.

With respect to science, many underestimate psychology. Research and methodologies really do provide insights behind personalities and there’s a reason why Myers-Briggs type indicators are so popular. Patterns can—and should—be generalized across people. Coaching effectively is just pattern-matching after years of coaching repetitively. Will Larson probably recognized Staff Engineer archetypes after countless hours working alongside Staff Engineers. In theory, pattern-matching is how our brains work in general.

While pattern-matching sounds scientific, applying and extending it is an art. Effective pattern-matching depends on experience, personalities, constraints, and context. Patterns span across disparate domains; a lesson from your religion somehow makes its way into your software design philosophy. This kind of artistry is reserved for humans. This is also the scary stuff that we hope to not be seeing in robots any time soon.

Science

Coding

Coding, in the context of professional software development, is more science than art. There is a difference between programming to poetically express yourself versus programming to build things for people to use. Software must be extensible, maintainable, and readable. As a system scales, these attributes are only attainable if the science outweighs the art.

Operations

“Ops” is a broad term that encompasses all activities contributing to software delivery. This includes change management to project management and everything in-between.

Operations evolves from an art to a science as a group progresses through the classical stages of forming, storming, norming, and performing. When a group forms, artistic liberties are taken to establish the process itself. The group’s end-game is to operate as a well-oiled and predictable software delivery machine.

Feedback

When it comes to people, do not assume that everything is an art.

Feedback and performance review are examples of activities where managers should rely on science. Management’s goal is to set clear objectives and key results for employees. If the results are hit, compensation and promotions should follow. If the results are not hit, constructive feedback and coaching should be provided.

If you’re a manager, lean scientific with your approach to performance and accountability. Career ladders should be detailed. Performance reviews should be predictable. This is contrasted with coaching—something that remains perpetually ON in various ad-hoc and artistic capacities.

Conclusion

The challenge with art versus science remains in its subjectivity. Art for you is science for someone else. As long as you remember this principle and keep your mind open to compromise, you will find elegant solutions to your problems whether they be technical or not.

Do not seek fault in others while staying blissfully unaware of your own.

• When others code slowly, it’s because they’re inefficient and don’t manage their time well. When I code slowly, it’s because I’m being thorough and have to deal with a difficult system.

• When others hold their technical opinions strongly, it’s because they’re stubborn. When I hold my technical opinions strongly, it’s because I’ve done my homework and need to be resolute.

• When another engineer PRs into my codebase, they’re overstepping their bounds. When I PR into another codebase, I am exhibiting initiative.

#2 — Be flexible with others. Be strict with yourself.

This is the explicit reversal of the previous point.

Be flexible with others and offer them grace. Be strict with yourself and stay thoughtful in areas of self-improvement.

By default, examine your own weaknesses before looking for fault in others.

#3 — Do not presume you can teach others.

Do not presume you have a superior understanding. You have some amount of knowledge, wisdom, and experience. Be willing to share it.

This mindset helps avoid condescension and increases the overall effectiveness of criticism.

#4 – Follow through with your commitments.

Criticism and feedback is only heard when there is trust.

In any relationship, you contribute to trust by following through with your commitments.

Microservices are nothing new. After being deployed successfully by mega-companies like Netflix, many growing engineering organizations gravitate towards this architecture as an aspirational symbol of success. This journey is known to many as the “monolith to microservices” migration.

A few companies (Spotify) have the clairvoyance to adopt microservices from the get-go and may never have to endure such arduous re-architecting. Other companies (Etsy) double down with their monoliths (a.k.a founding architecture) and scale into multi-billion dollar organizations. Many others opt for the long journey from monolith to microservices with roughly the following scenario:

We’ve seen immense success with our monolithic application and we are ready turn up the dial on growth. We need to explore new product lines, hire more developers, and increase velocity. To enable all of these wonderful things, we’ll start transitioning our monolith to microservices. This architecture will enable autonomous workstreams, failure isolation, flexible scaling, and {insert other microservices benefits}.”

I am not a fan of the term “microservice architecture” because it has become increasingly difficult to define. In regards to software, your system is either distributed—or not distributed. With many of us working in the context of distributed computing, I would venture to guess that you’ll find it impossible to use a textbook definition (microservices vs. service-based vs. event-based) to describe your architecture. You may describe your system as a “hybrid” or “ball of mud.”

While microservices are wonderful in principle, many software engineers are intimately aware of their challenges in practice. Parodies like this are a little too close to reality.

When pursuing grand endeavors, engineering leaders often forget to explicitly address organizational concerns. These constraints have the potential to block progress more so than any other technical issue.

Monolith To Microservices Vs. Your Organization

#1 — Executive (Budget) Alignment

Supporting microservices is expensive.

This is expensive not only with respect to compute/storage/network costs, but also with respect to operations and head count. This is not as simple as instructing software engineers to pivot how they develop. For example, a dedicated team may need to be formed to support the microservice “framework” itself. What are common tools for deployment and monitoring? How will dependencies be consistently upgraded and patches be successfully installed if we plan to have 100s or 1000s of services?

This investment should be discussed early with leadership, specifically with the executives that control the budget. For budgets that are controlled by non-engineers, spend time developing the business case and getting firm commitments to support the effort. Leadership background matters. A worthwhile investment to engineers can easily be perceived as a poorly-defined cost to others.

#2 — Experience

Has anyone done something like this before?

Fundamental software architecture changes are not to be taken likely. During times of migration, seemingly trivial decisions influence the software for years and years into the future.

Before embarking on the journey, have you located an individual (or collective set of experiences) in your organization that has been through a monolith to microservices journey before? This means actually participating in similar migrations (even if it they were unsuccessful) and not just living pleasantly in a post-migrated world. These experiences are rare.

#3 — Philosophical Alignment

Software developers are opinionated. Experienced engineers hold philosophies molded over time by unique companies, projects, and teams.

Before embarking on this migration, key engineers need to be bought in and aligned with the vision. This alignment is deeper than sharing signatures on a document. It is a perpetual alignment and re-alignment of philosophies surrounding how to approach problems.

A common philosophical difference in software development is the spectrum between generalized and specific thought.

Generalized developers love interfaces and can’t get enough of C++ templates. These engineers constantly try to integrate their solutions into “frameworks” and always seem to find common patterns across wildly different domain use-cases.

Specific developers don’t care much for Java Generics and are annoyed with obfuscated code. Not all their classes need to be implementing an interface and duplicated code is not that big of a deal. Straight to the point is their motto.

Now imagine that two of your most-senior, most-experienced developers sit on opposite sides of this spectrum. Up until now, these engineers have balanced each other out and have designed the successful application you have today. However, each of these engineers would have a drastically different approach to problem solving as your software systems and headcount grow.

Organizationally, how will you resolve these philosophical differences? How can you be confident this migration will be executed in a consistent and enjoyable manner?

#4 — Ownership

Given your large and successful monolith, how would you rate it on clarity of ownership? Can the majority of modules be assigned to various teams or is there a kitchen sink of forgotten endpoints?

Ownership is not a technical issue. It’s an organizational issue and should be addressed before any attempt is made at a migration. If critical steps in the strategy depend on changes to an unowned part of the codebase, it will be a headache to get that work planned and executed.

Ownership is psychological. Teams and engineers go to great lengths to skirt around code that they do not own. “This is everyone’s code” only works at small levels of organizational scale. Lack of ownership especially hurts when you’re trying to make fundamental changes to your software.

#5 — Decomposition Vs. Morale

Many engineers would not describe migration work as glamorous. Usually, the first step in a monolith to microservices migration is decomposition. Before the system can feasibly be re-architectued, modules need to be split out and put behind interfaces so implementations can safely be manipulated. For developers, this may feel like they are picking up rocks just to put them down somewhere else—the rocks are still the same rocks!

How can this be broken down into manageable pieces? How will you set up your teams for success?

Poor strategies around decomposition work will rapidly demoralize teams. If you are not strategic with execution steps, this work will turn into a slog and you will burn through your engineers. How do you expect a team to carve out modules in the codebase when half of it is unowned?

Carefully plan and assign this work while paying attention to motivations and incentives.

Conclusion

The monolith to microservices migration is a long journey. You’ll experience gratifying wins along with unwelcome additions to your technical debt. These foundational changes are an organizational effort just as much as they are a technical lift. If the company does not fully support this work, no amount of software ingenuity or pool of 10x programmers will help.

1-1s are core to management. Ad-hoc feedback delivery, bi-weekly check-ins with your directs, monthly meetings with your skip-levels, annual performance reviews. Their use cases are well-defined and well-understood.

1-1-1s, on the other hand, can be much more interesting. Significant progress can be made if you’re able to get the three right people in a room together. 1-1-1s are versatile and streamline communications for a variety of situations.

While managers are usually responsible for 1-1s, any leader can call for a 1-1-1.

So, when should you host one?

Your intuition will serve you best. Here are a few questions that will help you exercise these muscles:

• A new role is being integrated into team—how’s it going?

• A new project has caused something to change—are expectations clear?

• A team is visibly disengaged—is there conflict?

In this post, we’ll review 5 effective 1-1-1 formats that deserve a spot in your leadership toolbox.

Use More 1-1-1s

#1 – Finalize A Reporting Change

A 1-1-1 can finalize a transition for an internal reporting change. This is held between you (the manager), your soon-to-be ex-direct, and their future manager. The goal of the meeting is to review past performance along with future expectations for the engineer. It’s imperative that this is done together with all three parties.

#2 – Temporary Project Resource Reallocation

An important project is gaining momentum and its proper staffing depends on temporary resource reallocations. Temporary only because no one enjoys a re-org for the sake of a project. Nevertheless, something has changed—are expectations clear?

Maybe not because we have new people in the mix. What is the time commitment for the new project members? Which milestones are expected to be finished before they can disengage? How is work going to be delegated?

If you’re leading such a endeavor, you’ll be grateful for the additional resources and stressed at the same time with additional project coordination. To increase the odds of success, an early 1-1-1 can firmly establish working expectations with new project members. This can be held between you (the project lead), the re-allocated engineer, and their manager.

How is this all going to go down? Work it out and get it in writing.

#3 – Staff Engineer Expectation Setting

Staff Engineers are held to a different set of expectations than other engineers. Companies expect Staff Engineers to have a large, organization-influencing, scope of impact.

Ironically, company-impacting problems are found inside local teams. To avoid the trap of the “Floating Engineer”, some Staff Engineers (with the support of their managers) choose to embed inside a team.

If this happens, how do we manage expectations?

A common scenario is when an Engineering Manager utilizes a Staff Engineer with strong mentorship skills to level up a team. Their impact will be measured by how much each individual progresses as an engineer; their impact will not be measured by lines of code written or projects shipped.

These expectation must be established between you (the manager), the resident team leader, and the Staff Engineer. The team leader should not expect the Staff Engineer to perform the same duties as his or her other engineers. A Staff Engineer on the team doesn’t mean we get 20 extra points of sprint velocity. It means that each IC will receive thoughtful technical reviews for their work.

Use the 1-1-1 to make sure everyone is on the same page.

#4 – General Role Expectation Setting

The previous idea can be leveraged more generally. As your responsibilities and personnel grow, you will have special roles that periodically need to be integrated into your teams.

These situations often arise when an individual—who is scoped at a group level—gets assigned to work with a specific team. Common examples of this include scarce pools of Technical Writers, Product Designers, or Technical Program Managers.

If the group you manage has just been assigned its first TPM, what should you expect out of them? Is this person focused on goal-setting processes or project delivery? Or are you not even sure?

There are a couple 1-1-1 formats that can help set expectations.

As a manager, you have your own ideas about how your group can utilize the new TPM. Perhaps you want him or her to focus on goal-setting and to produce a colocated list of opportunities across all your teams. Before pushing further, is this what the TPM is actually expected to do? You can host a quick 1-1-1 between yourself (the manager), the TPM, and the TPM’s manager. Here you’ll be able to set short and long terms goals for the TPM and ensure he or she is being evaluated properly.

If you decide that a TPM is best situated helping out a specific team with project delivery, you may ask one of your team leaders to integrate the TPM into their team. If you’re the team leader in this situation, you can hold a tactical 1-1-1 between yourself (the team leader), the to-be-integrated TPM, and your partner Product Manager. Here you’ll be able to discuss new team dynamics and brainstorm ideas for how the TPM can add value.

#5 – Pushing Decisions Forward

At a large organization, one of the most important things you can do is to make decisions.

One of the major obstacles standing in the way of decision making is two—exactly two—opposing opinions. Two high-ranking engineers have two competing views on how to design a system. Neither opinion is right or wrong; they are just different.

A 1-1-1 can be used in these situations to move a stalemate forward. Opposing opinions are usually held by peers. This is often the result of what Andy Grove has dubbed “peer-group syndrome”, when a meeting amongst peers goes around in circles.

Andy’s fix for this is the “peer-plus-one” method, when a senior person is injected into the conversation. More concretely, this can take the shape of a well-moderated 1-1-1. Example formats could be between you (a principal engineer) alongside two Staff Engineers or perhaps you (an engineering manager) alongside two senior engineers.

Conclusion

The 1-1-1 meeting is a versatile tool for engineering leaders. If utilized properly, these meetings create clarity in expectations, push decisions forward, and diplomatically establish working agreements.

What kind of 1-1-1 formats do you enjoy using?

• How comfortable am I developing code and debugging issues?

• Could I give a presentation about my team?

• Do I know who to contact if I’m not sure how to solve a problem (besides my mentor)?

• Did I release a project end-to-end that required me to do more than coding?

As a Senior IC

• Has another team around the company changed what they’re working on based on my actions?

• How strong is my opinion regarding my team’s product goals and technical investments?

• Do I have context around the challenges of my peers (even though it’s not my responsibility)?

• Has another engineer on the team surprised me by taking care of a tricky incident without my involvement?

As a Manager

• Did my reports demonstrate an expanded scope of impact? Is it easy to justify their promotions?

• Did my level of ownership and responsibilities increase in the last year? If so, was it situational or did I seek out those opportunities?

• Is there a clear choice for an individual who could take over my responsibilities?

• My group drove a lot of impact these past two quarters—is it because my teams have grown or is it because I’ve driven innovations to make my teams more effective?

As you progress in your career, your success largely depends on your personal brand as a professional.

Everything you do contributes to this brand.

Early on, you are focused on learning, adding value, and developing as a professional. During this time, it’s essential that you avoid behaviors that have the potential to become brand-damaging habits. Curating these behaviors is difficult for a few reasons:

1. These behaviors are subtle and are easily overlooked. Nevertheless, it’s better you notice them before your manager does.

2. It’s impossible to fully control how you’re perceived. A well-intentioned gesture always has the chance to be taken the wrong way. In life, this is inevitable.

3. While your personal brand is mostly in your control, its effectiveness depends heavily on your environment. The same you may be a great fit at a large corporation, but may fall flat at a start-up.

In this post, I share 5 mistakes that damage your personal brand as a young professional. Recognizing and avoiding these behaviors may save you a lot of time in your career.

5 Mistakes That Damage Your Personal Brand As A Young Professional

#1 — Optimizing The Individual Instead Of The Team

If you are working at a company, place the company first. This is a general attitude that lays the foundation for impactful behavior.

Do not repeatedly ask your boss for a checklist of activities to get you promoted; repeatedly ask your boss what’s the best thing you can do to ensure the team is successful. The latter strategy usually gets the promotion faster.

Place yourself in the shoes of a manager or CEO. What attitude would you prefer in an employee—individual-first or team-first?

You look out for yourself by looking out for the groups you’re apart of. If you’re able to internalize a team-player attitude and add value at the same time, it’s only natural for responsibility and compensation to follow.

#2 — Pushing Too Hard For Compensation / Promotion

Everyone must push for compensation. Money talks and people have responsibilities. Compensation will always be a critical factor with anything you pursue.

Raises and promotions are sensitive issues that must be dealt with tactfully. There is a time and place to discuss compensation—new job offers, performance review cycles, large changes in responsibility. Recognize these moments and take advantage of them.

When the time isn’t right, refrain from excessively pushing for compensation. If you’re unhappy with your performance scores and the subsequent raise you’ve received—how much time do you want to spend fighting for more money, or making a plan to increase your performance? Is your best move to switch employers? Perhaps—but what if this situation happens again?

Maintain goodwill and negotiate gracefully. Build a case and accumulate data to back yourself up. Do not resort to questionable tactics that modestly bump you in the short-term, but compromise you in the long-term.

#3 — Speak Ill Of The Company

Do not speak ill of the organization you work for.

Whether it’s over a casual lunch or a 1-1 with your team lead, bad-mouthing the company hurts your professional brand. You do not want to be known as the engineer who is constantly complaining. Having a discussion about the last all-hands meeting is one thing; labeling the company’s software as subpar compared to your previous employer is another thing.

Do not give off negative energy. People notice.

#4 — Being Defensive

Every day you will be tempted to be defensive. Your manager gives you feedback you don’t agree with, you receive a snide comment on a code review, a staff engineer makes a flippant remark about a technical design you’ve spent weeks thinking about. Whatever comes your way, humbly try to not be defensive.

Assume positive intent.

Even when someone else is rude, be courteous.

None of us have a perfect track record in this category; you just don’t want your default behavior to be defensiveness.

#5 — Visibly Moody At Work

Everyone has their lives to deal with.

Do not let your personal mood affect your professionalism. Being visibly grumpy and disengaged at work will be noticed—”Hey, Jerry seems to be in a funk this week. That meeting was a little awkward. What’s going on?“

Your energy should not affect others to the point it makes them uncomfortable. Work is work. As a professional, you are committed to being engaged and accountable. If something happens in your personal life that prevents this from happening (which is OK), acknowledge it quickly and take time off to rest and recover.

Conclusion

Mistakes are natural and unavoidable. Don’t be too hard on yourself if you make some, just be sure to learn something from them. Developing your personal brand is an active exercise. It includes both leaning into brand-building behavior and stepping away from brand-damaging behavior. You’ve worked for many years to develop this, don’t let a preventable mistake ruin it.

As a Staff Software Engineer, one of your hardest challenges is to influence without authority. If your position doesn’t hold authority by default, then your best bet is to align with it.

This topic was inspired by an insightful quote from Will Larson’s Staff Engineer: Leadership beyond the management track.

Stayed aligned with authority to remain an effective leader over time. Technical leadership roles rely on proxied authority from another (usually, managerial) leader, and continued access to that authority depends on staying aligned, trustworthy, and predictable.

Will Larson

Alignment is essential. As a Staff Engineer, internalized alignment re-orients your approach when persuading people to follow your ideas. This will manifest in subtle—but significant—differences.

In this post, I’ll share a few ideas on how to achieve the two core components behind Will’s idea: staying aligned and building trust.

Staff Software Engineer Responsibilities – Align With Authority

#1 — Staying Aligned

Figure Out Who To Align With

First, decide on who you’re going to align with. This is an individual, usually a manager or a director. Choose someone you respect and would enjoy working with. Factor in your interests, technical strengths, and where you believe you can make the most impact.

Aligning to a specific person means aligning to a specific scope.

• Align to an Engineering Manager

  • A large team of Individual Contributors needs technical leadership.

• Align to multiple distinct Engineering Managers

  • Same as above, but for two (maybe three) distinct teams.

• Align to an Engineering Director

  • A group of teams belonging to a specific domain needs technical leadership.

These ideas break down at higher levels. If you’re aligning to a VP of Engineering, you’re most likely a Principal+/CTO and are overseeing an entire department. You’re probably focused on training other Staff Engineers to align with their respective managers and directors.

Who you align with determines your home base. Aiming for the largest scope isn’t necessary the right move.

Avoid “Floating”

Floating is when your job description sounds something like, “Find the biggest technical risks at the company and mitigate them.”

Floating is a risk because it does not foster alignment with anyone. You’ll have difficulty truly understanding the problems that affect teams. Staff Software Engineers in these situations struggle to obtain technical influence and usually do not have a say in what work actually gets done.

You will also be personally frustrated.

The Manager’s Perspective

After identifying a managerial partner, you’ll need to gain their perspective on how their team(s) are doing. Get a bird’s-eye view of the situation.

• Which teams are doing well? Which ones not so much?

• What kind of engineering personalities are on the teams? Which senior engineers need to be pushed?

• Can you point me in the direction of the biggest technical risks?

Gaining this perspective is not just an onboarding task, but an ongoing activity that a Staff Software Engineer invests in.

Get On the Ground

You can’t stay high-level for too long. In addition to the understanding the manager’s perspective, you need to get on the ground to see how things are doing.

• What kind of goals and key results has the product manager set up?

• What are the day-to-day pain points with this group’s software development?

• What kind of engineering processes are unique to the team?

An effective way to do this is to temporarily embed in a team. However, being on a team as a Staff Engineer is not the same as being on a team as an Individual Contributor. Set clear expectations with a working agreement.

Write A Working Agreement

Establish a working agreement with the manager you’ve partnered with.

• What are your responsibilities and how exactly will you work?

• If you do “embed” on a specific engineering team, what does that dynamic look like?

• How should engineers interact with you? What value are you expected to bring?

This working agreement is the physical manifestation of your alignment.

One Voice

The end goal of alignment is for a Staff Engineer to be able to represent the group with one voice. You speak and make decisions with the group’s best interests in mind. When you attend the exclusive Staff-only architecture forum, you communicate as if your managerial counterpart and all their teams are there alongside you.

#2 — Trust

… continued access to that authority depends on staying aligned, trustworthy, and predictable.

Trust takes time to develop and continuous accountability to maintain. Trust can also be broken quickly and is sometimes impossible to repair if damaged. It is foundational to all relationships.

As a Staff Software Engineer, there must be trust between you and your managerial counterpart.

Meet Consistently

Even without an established reporting relationship, meet regularly 1-on-1.

Regular Feedback to the Manager

As you spend time on the ground, share relevant feedback with the manager. Are the team’s technical processes sustainable for future IC growth? How is the Tech Lead doing? What investments or debt repayments are most valuable?

Avoid feedback that is rooted in personal preferences. Build trust by sharing feedback that keeps the team’s best interest in mind.

Develop Relationships Within Your Scope

As a Staff Software Engineer, establish yourself as a team member. You are not a contract engineer for hire.

Develop relationships inside the team(s) within your scope. Help a Tech Lead organize their technical debt. Engage in code reviews to elevate ICs. Work closely with a senior engineer to help him or her move into a Staff role.

As a manager, observing that team members are leaning into their relationships with the resident Staff Engineer is a wonderful sight to see. Trust with the team creates trust with the manager.

Be Accountable

In many situations, you are not directly reporting into your authority counterpart. Perhaps you report to a VP of Engineering and you’ve decided to align with a specific Engineering Director. Regardless of the org chart, allow the Engineering Director to assign you work. Even if he or she is a peer on paper, operate as if you’re in service to them. Let them set deadlines for you. Meet them.

To reiterate, the purpose of this strategy is to build trust with an authority figure. This provides you with “proxied authority”, enabling you to have technical influence, which is the definition of your role.

Example of Misalignment

You are a tenured Staff Engineer. You have deep historical knowledge and are one of three people at the company who can articulate how the full tech stack works. You have strong opinions on the system and know in your heart which areas need the most attention.

As the team becomes successful, the company hires a new Engineering Manager to facilitate growth. The manager proposes investments that are at odds with what you think is best. The manager wants to prioritize local developer workflow improvements while you strongly believe the team needs to finish a data migration. You become defensive and do not put your full alignment or energy into the manager’s new initiatives.

This turns into a tricky situation for you two to reconcile.

Conclusion

Staff Software Engineer is a challenging position. Without aligning to an authority figure, many Staff Engineers find themselves frustrated with a pile of innovative ideas that sit in a Google document. If this sounds familiar, I encourage you to give this strategy a try.

Thank you again to Will Larson for his original idea. Please check out his work!

As a leader, you may have grown accustomed to a variety of conveniences when it comes to communication. You rarely get interrupted during meetings, your ad hoc Slack messages receive prompt responses, and your under-researched opinions influence people more than they really should. Unsurprisingly, not everyone enjoys these conveniences.

As a leader, you must be generous with your communication. Here are some ideas.

Don’t Push For A Decision Too Early

During discussions, do not prematurely push for a decision. To achieve genuine consensus, all thoughts need to be openly and thoroughly laid out first. Ideally, this is done without your influence. As a leader, your opinions carry extra weight. If you rush, you will inadvertently stifle your colleagues’ ability to fully share their thoughts.

Talk Less

Talk less. Actively listen.

Allow For Correction

Opinions are opinions. Create a culture that allows for correction.

• Preemptively give the OK for correction. Short phrases—”Please correct me if I’m wrong…“—make a difference.

• Facilitate feedback amongst individuals. One of your senior engineers feels strongly about X; you know another junior engineer disagrees. Use your position to allow the junior engineer to provide their feedback.

• Facilitate feedback for yourself. Directly engage individuals who have different views than you.

• Periodically remind people that every decision is a trade-off. Acknowledge and discuss these trade-offs.

Fix Interruptions With Callbacks

During a meeting, you notice that an engineer hasn’t been able to share their opinion as they are constantly being talked over by aggressive Staff engineers. Use your position and call back on them. Minimal effort for you creates a lot of space for them.

Set An Expectation For Response Times

For asynchronous communication, set an expectation for response time. Not all your messages need ASAP responses. A simple preface—”[Not Urgent]” or “[Please let me know by end of day]“—provides a much appreciated timing expectation for your requests.

Be Decisive, Offer Discussion

There will be times when a decision must be made before properly laying out everyone’s opinions. Your reports are going in circles trying to solidify a software design and you decide to step in to break a couple ties. When this happens, proactively engage in follow-up discussions—”We’re moving forward with {controversial decision} because of {valid reason}. I will invite our stakeholders to an office hours session next week to discuss any additional concerns.” Notice how this is different than, “Please reach out to me if you have any concerns.”

You can be decisive and thoughtful at the same time.

Share Your Context

As a leader, your job requires you to develop and maintain a vast amount of organizational context. You understand both political constraints and subtle technical challenges. This context has led you to your current role and it will remain an essential asset enabling you to function as a leader. Share this context (with discretion) with your colleagues.

Conclusion

As you grow in your career, be generous with your communication. A new hire experiences a meeting differently than how you experience a meeting. Use your position to build a culture where people have space to comfortably express their thoughts. Remind everyone that opinions are always opinions and that anyone can be corrected. Finally, when you have to be decisive, be thoughtful as well.

This post’s goal is to provide you with a deeper intuition behind two of the most common software messaging patterns—queues vs. pub/sub. This is a deep topic which deserves thoughtful attention when designing software. This article is structured in three parts:

1. Develop an intuition for messaging, queues, and pub-sub through analogies.

2. Share a short heuristic to help choose between queues or pub-sub.

3. Discuss software implications for each paradigm.

Software Messaging — Thinking Abstractly Before Concretely

Software messaging is a topic where discussions weave in and out of the abstract and the concrete. Logical paradigms like queues and pub/sub are discussed in the same breath as physical technologies like Kafka and RabbitMQ.

For technical topics, carefully delineating between the abstract and the concrete provides you with mental clarity. An interface is implemented with a class, a stack is implemented with a linked list, an instruction set architecture is implemented with a processor. Virtual, logical, abstract. Implementation, physical, concrete. There will be times when you need to draw boxes on a whiteboard; there will be times when you need to tune obscure Kafka consumer settings.

Let’s begin by thinking abstractly.

What is the essence of communication? Communication is the sharing of information between multiple entities for a shared purpose.

What is the essence of messaging? Messaging facilitates communication. A message is any information passed between multiple entities.

Queues vs. Publish/Subscribe

Two of the most common software messaging patterns are queues and pub/sub. In the context of software development, the word “pattern” can be substituted with “paradigm” or “architecture.”

Analogies can be an effective way to build intuition.

Queue <> Restaurant

A restaurant is built around a queue. Waiters produce orders. The kitchen consumes orders and produces meals. The orders are kept in a queue of messy hand-written tickets. First order in is the first meal out. The restaurant queue is purposeful—we want orders to be processed by the kitchen and only the kitchen. There is no need to prepare an order twice. If orders get backed up, adding a few chefs may help. The restaurant (waiters, tickets, chefs) all work together as a unit.

Publish/Subscribe <> Bulletin Board

A bulletin board is built around pub/sub. Looking to make some extra pocket money, you put up an ad for private programming lessons on the town bulletin board. You have no idea who will read it. You’re hoping a few students contact you, but you keep your expectations low—it’s OK even if no one reaches out. By default, the town takes down ads after 7 days. They tell you that if you pay more, they’ll build a bigger board to keep your ad up longer.

Which One Do You Need?

Based on your requirements, does your design need to be more like a restaurant or more like a bulletin board? Decide on the pattern (paradigm, architecture, etc.) before you decide on the implementing technology.

There is a shortcut to help you decide. What are you trying to communicate? There are two common message types—events vs. commands.

Events are customerNameChanged, orderPlaced, or jobPosted.

Commands are sendEmail, processLogs or cookFood.

Command-style messages lean towards towards queueing.

Event-style messages lean towards pub/sub.

This is only a heuristic.

Concrete Implementations

After arriving with your chosen messaging paradigm, it’s time to choose an implementing technology. Any single technology can achieve a variety of different messaging patterns. You can achieve queueing with either Rabbit or Kafka. You can achieve pub/sub with either Rabbit or Kafka. Give a programmer a database and a cache, and many, many software patterns can be achieved.

Despite the flexibility of various technologies, they are still built with specific use-cases in mind. Kafka is optimized for pub/sub while RabbitMQ is optimized for queueing. Both these technologies come with their own pros and cons. It’s impossible for any single messaging software to be optimized for every single messaging pattern.

Deeper discussions arise when evaluating a technology to be used for a paradigm that it is not optimized for. If your design calls for queueing, you may be debating whether or not Kafka is a sound solution—the answer may likely be yes and that is perfectly OK. Perhaps Kafka is a preferred technology based on your organization’s operational expertise. Perhaps the expected write throughput is so high that Kafka is the only feasible solution. Perhaps nothing off the shelf works and you’ll need to build it yourself! Implementation choice is always situational.

Software Design Implications

A choice in software paradigm will quickly be followed by a variety of software implications.

Extensibility & Decoupling

With pub/sub, producers and consumers are naturally decoupled, making separation of concerns easier to achieve. Introducing a new consumer is an isolated activity which does not affect message production.

With queueing, a new consumer represents a change to the overall system, both on the producer side, the consumer side, and everything in-between. In order to “activate” new consumer functionality, a new queue must be established with connections to a corresponding producer and consumer. Since a variety of components must be updated, the queueing paradigm is considered less extensible than the pub/sub pattern (in the context of new consumer behavior).

In the realm of queueing, technologies like AMQP can assist in making your life easier. When setting up a new consumer, a few changes to exchange and binding configurations may get most of the job done. However, regardless of AMQP or any home-grown queueing solution, adding a new consumer fundamentally requires a holistic change to the system.

Message Contracts

With pub/sub, the ease of adding new consumers comes with a potential cost—a tight contract for messages. The convenience of introducing consumers assumes messages are produced under the same contract (payload, format, etc.) However, what happens if a consumer wants to consume an existing message type, but needs a little bit of additional data? Is this new functionality worth it for us to change how we fundamentally publish messages? How would this affect other consumers of the topic? Perhaps pub/sub leads to more coupling than we initially expected.

For queueing, message contracts are flexible. Restaurants and queueing systems behave as a unit—we know exactly what we are producing and exactly how it’s going to be consumed. We can change how we write tickets at our pizza shop without affecting the shop across town. Message contract flexibility also helps optimize consumers; we provide them exactly what they need.

Security

Security must be top of mind when designing a pub/sub system. If a thief is snooping through the town bulletin board, you might think twice before putting your phone number on your ad. In a highly distributed setting, it may be difficult to keep track of all consumers for any given Kafka topic. Even when everything is internal, a consumer may be misbehaving somewhere.

Observability & Monitoring

One of the benefits of queueing is that you know exactly how your messages will be consumed. There is clear, expected behavior for both production and consumption. These expectations yield strong observability and monitoring. Systems which lean into pub/sub patterns require additional effort to properly observe and monitor.

Performance

Popular pub/sub solutions like Kafka allow extremely high throughputs for message production. Though I have never seen it myself, Kafka advertises that you can write up to 1M messages/sec to a topic! RabbitMQ gets no where close to this. If your system requirements call for this level of scale, there are only a few technologies that are viable.

More Responsibility For The Consumers

The pub/sub pattern puts extra onus on the consumer. A producer’s job is easy, but that doesn’t mean the overall work is easy—it just gets transferred to a different location.

Imagine a software team that is responsible for emitting important event-style messages when customer data changes. If that team is optimizing for the least amount of work possible, they may propose a pub/sub strategy. Fire and forget to the “CustomerEvents” topic is low effort and appears extremely extensible.

However, the customer analytics team isn’t a big fan of this approach. They are left with the burden of consuming a firehose of customer events, many of which are meaningless for them. Furthermore, important data is missing in the message payloads, forcing the analytics team to jump through hoops to resolve additional data, putting burden on other systems.

The customer analytics team makes a case for an architecture which promotes a tighter collaboration utilizing queueing paradigms rather than pub/sub. For large distributed organizations, it can be a risk when so much responsibility falls upon one side of the equation. Consuming a Kafka topic is easier said than done.

Conclusion

Software messaging is a huge topic. We’ve only scratched the surface here, but I hope this post has left you with a deeper intuition between two of the most popular messaging patterns.

Queuing is like a restaurant.

Pub/sub is like a bulletin board.

When making technical decisions, merge your thinking between abstract paradigms and concrete technologies. It’s not helpful to memorize how AMQP/Kafka/etc. work if you’re not aware of the software patterns they are designed to solve.

“The 25 Micro-Habits of High-Impact Managers” is a wonderful article highlighting wisdom from various managers across various industries. The article also reinforces an important universal idea—the details always matter. I subscribe to this idea and believe it can be easily re-applied to individual contribution.

At any given company, the criteria for promotion can be bland and frustratingly opaque—consistently deliver value turns into create impact for your team, which quickly evolves into create impact for multiple teams, and somehow leads to create impact for the entire company and drive its future. These large expectations are difficult to break down. It’s easier to start with the details (“micro-habits” if you will) and see where that takes you.

Before continuing, I highly recommend reading, “On Being A Senior Engineer“, which provides a holistic set of expectations on what it means to be “senior.” With these expectations in mind, the following is an opinionated list of micro-habits exhibited by high-impact software engineers. I’ve broken them down into 4 themes, each with 5 habits.

1. Leadership & Culture

2. Execution

3. Scope & Complexity

4. Communication

20 Micro-Habits Of High-Impact Software Engineers

Leadership & Culture

#1 — Incident Root Cause Analysis

You proactively dive into incident root cause analysis. You value software quality, measure problems in terms of customer impact, and make long-term improvements to engineering operations. Incident root cause analysis is often valued above incident response itself. It is also a clear sign of technical leadership.

#2 — Steps Up For Project Duty

You proactively step up for project management duties. Not all companies have technical project managers to help coordinate success. These responsibilities fall upon gracious product managers, designers, and engineers willing to fill in the gaps. Organizing meetings and tracking action items sit alongside coding as essential work.

#3 — Vocal About Tech Debt

You are vocal about tech debt and are opinionated about its relative importance compared to day-to-day project work. Technical debt is pervasive. Product managers will inevitably push back and question its value; you must persist and pay it down. What’s important is not only the debt itself, but also how you showcase your values—quality, follow-through, engineering excellence.

#4 — Elevate Teammates

You make time for your colleagues and voluntarily lend your expertise. You give thoughtful code review comments, proactively ask for pair-programming sessions, and tactfully deliver feedback. You do not assume your colleagues will naturally come to you for help.

#5 — Vision For The Future

No matter the scope of work, you have a technical vision for the future. Things can always be better! This might be a long-term strategy for how the engineering department does hiring or it could be a set of best practices for how your team writes tests. Similar to technical debt, this is important not just as a vision, but as a symbol of how you value the evolution of your work.

Execution

#6 — Clean Pull Requests

You create clean pull requests that are focused and tell a story. You avoid large changes and provide as much context as possible in your descriptions. Simplicity takes effort.

#7 — Release Plans

You acknowledge how tricky releases can be and spend time with your product manager to create release plans. How will your team safely release this feature? How can you safely roll back if something goes wrong? Are all stakeholders informed? Creating a release plan signals that you value thoroughness, safety, and collaboration.

#8 — Follows Through With TODOs

You are a good citizen of the source code and follow through with your TODOs. Sift through any codebase and you will find thousands of “delete this section after migration” and “temporary fallback behavior” littered around the source. Lingering TODOs quickly cause cognitive overload for other developers.

#9 — Observability

You consider observability by default. How will you and your team know your code is actually working? You install a level of observability that matches the context of the work. If a feature is in alpha, there is no need to add 5 new alerts that might wake up your colleagues at 3am. When a feature moves into general availability, you ensure a well-thought-out set of on-call responsibilities is created.

#10 — Work Auto-Scheduler

Within your scope, you do your best to prioritize and schedule incoming work. When a customer support specialist brings up a fresh bug, you evaluate if the issue is critical and needs to be front-loaded. Prioritization is crucial at every level of the company and it starts with how you choose to spend your time. Any effort spent on prioritization will be greatly appreciated by your manager.

Scope & Complexity

#11 — Best-Effort Estimates

Despite knowing that estimates are always wrong, you still put in the time to establish them. You break down tasks, create milestones, and enable your non-engineering counterparts to plan their timelines. As you build, you maintain a pulse on new requirements and ongoing progress, quickly updating your stakeholders if any timings significantly change.

#12 — Constantly De-Risking, In And Out Of Software

While working on a complex and long-running project, you ingest all input and constantly de-risk the project. You are keen to hear about new requirements—a design tweak, a new auditing standard, or a nuance in revenue recognition may all influence how the technology is built. Armed with as much context possible, you are able to evaluate any new information (not just software-related) for potential risks.

#13 — Works Through Unknowns

You’re able to make forward progress on a project despite have 20% of the information you need. As a beginner, you’re expected to handle a stream of well-scoped tasks with clear acceptance criteria. As you grow, you will be sent in a general direction where the unknown heavily outweighs the known. To work through these unknowns, you begin to build relationships, explore technologies outside your immediate domain, and continuously developer your Debugger’s Mindset.

#14 — Leans Into Dependencies

You lean into dependencies and constantly grow your understanding of the entire software ecosystem. High-impact engineers are sent in general directions to deal with complex problems. Complexity comes hand-in-hand with dependencies. When you’re handling integrations with multiple teams and vendors, you refrain from siloing your work and never throw things over walls. It’s easy to say, “Hey, we’ll emit an event when this {important_operation} happens; the rest is on you.” It’s harder to lean into dependencies and fully grok the integration. This has the wonderful side-effect of strengthening your personal relationships with different teams.

#15 — Writes Often

You understand that writing is not just a medium for communication, but is also an opportunity to organize your thoughts. You look for opportunities to write beyond the technical planning during the onset of a project. Don’t understand how multiple software components fit alongside an abstract architecture diagram a Staff Engineer just shared with you? Try writing about it.

Communication

#16 — Provides “The Why”

When sharing, you remember to accompany your content with “The Why.” Why is this even important? You may know why something is important for a colleague before they even know why.

#17 — Avoids Loose Language

You avoid loose language. “I’m not sure why that’s happening; it might because of a recent commit” is completely unhelpful. If you’re unsure, communicate explicitly that you are unsure and that you will follow-up with an answer. Loose language is cumbersome to sift through, introduces red herrings, and is potentially disastrous during incident response. Ensure your communication is precise.

#18 — Bookmarks Conversations

You remember the context of conversation threads as you schedule between them. At any moment, you will be interacting with a variety of different people. You’ll catch up with a designer over a new prototype then immediately jump into a white-boarding session for a new software architecture. For each conversation thread, save the context and create a “bookmark” of where you left off. This streamlines communication and signals that you value 1-on-1 communication. Constantly starting conversations with “Could you please remind me again why…” is a waste of time.

#19 — Correct Context, Correct Medium

You thoughtfully choose the correct communication medium based on context. You’re aware of the pros and cons of Slacks, emails, and meetings and transition between them based on communication requirements. The level of information you share is fluid; creating a document for peer engineering review is different than creating a document for executive review.

#20 — Shares When Appropriate

You find opportunities to share your work despite your packed schedule. Sharing does not have to be delivering a full-blown presentation with slide animations. When a coworker is starting to integrate with a new cloud service, you proactively schedule 30 minutes with them because you went through the same integration last quarter. Effective sharing is the culmination of tiny sharing activities.

Conclusion

These “micro-habits” are more than habits. Habits affect your actions, actions affect how you operate, and how you operate culminates in how much value (not only lines of code) you add as an engineering professional.

Software engineering decision-making is a stressful and time-consuming process. When the stakes are high, you will be met with opinionated programmers, insurmountable organizational constraints, and unfortunate baggage from old projects. Consensus is difficult.

In a previous post, “Staff Software Engineer Responsibilities“, I mention that the act of “Transforming Competing Opinions Into Decisions” as one of a Staff Engineer’s primary responsibilities. I note that prolonged technical debates are a risk for organizations and that Staff Engineers must “push through friction” in order to arrive at decisions. However, once you feel the friction, it’s never clear how you’re supposed to push through it.

This article presents a generalizable 6-step outline to help engineers drive consensus. This is a set of heuristics, not an algorithm.

Software Engineering Decision-Making

Preliminaries

• Decision-making is unique to every organization. It’s part of the culture.

• Decision-making and consensus-gathering take significant time and energy. Reserve this process for business-impacting problems.

• As long as the desired outcome is achieved, less is more. Ideally, you don’t need any of this.

• Any decision-making process should optimize for good-will amongst your peers.

• Decision-making relies on empathy and communication more so than technical skills and previous experiences.

• These ideas may be relevant for other situations where you hear the words “vision”, “alignment”, or “strategy.”

Step 1 — Assign A Directly Responsible Individual

Ensure a directly responsible individual is assigned to the decision outcome.

Step 2 — Research And Form Your Opinion

To successfully facilitate the journey towards consensus, you must achieve two prerequisites: understand the current state of affairs and form an opinion.

In certain cases, you’re an existing stakeholder and have already established your opinion. In other cases, you’re coordinating a decision outcome with minimal context. The latter often occurs when existing stakeholders are entrenched in their biases and objectivity is needed.

The outcome of this step is to have complete organizational and technical context. You must track stakeholders and listen to their perspectives. You must sift through prior art and read documentation related to the topic. This research will naturally lead you to an opinion. For the veterans, remember that your opinion should form from your newly gathered context more so than your past experiences.

Finally, be clear about who is a key stakeholder and who is an interested observer. If you were to survey the engineering organization, a hundred engineers may want to take part in the process. Unfortunately, it’s not feasible to gain consensus with one hundred people. Be intentional. Who is actually affected by the decision and who just wants some extra time on their soapbox?

Step 3 — Key Stakeholder Empathy

After the research legwork is done, you will have sufficient context to be able to engage with key stakeholders. You may have identified one or two acute areas of conflict. Software developers are opinionated; just one unresolved issue is enough to stall a company.

Meet with the individuals who hold the strongest opinions. Understand their philosophies—do they prefer generalized solutions or specific solutions? Talk to them about their previous experiences—are they avoiding the pitfalls of a failed project or a failed company?

Do not try to achieve consensus or reconcile conflict yet. During this step, your goal is to develop empathy; this is a prerequisite before you moderate broader meetings.

Step 4 — Gather Stakeholders And Moderate

With key stakeholders identified and empathy in hand, it is now time to gather everyone together.

The meetings should be casual. No productive conversations occur in formal councils.

The meeting should be documented. Throughout these meetings, you will arrive at mini-decisions and mini moments of consensus. Document these wins so the group does not repeat itself; talking in circles is tiring and wasteful.

Aggressively scope the meeting. Refrain from adding too many attendees or discussion topics. Answering one or two open questions per meeting is a great outcome.

Finally, the meetings should be moderated. Even casual meetings require structure. Ensure people are staying on topic, high-level goals are reiterated, and the majority of minutes are spent discussing. Constantly remind people about the outcome and its significance.

Step 5 — Timeboxed Arrival At A Decision

Due to the heavy amount of communication required, assume this process is at risk of taking too long. Once discussions begin, the arrival at a decision should be timeboxed. You do not want to spend months or even quarters discussing this topic.

Culture matters. Compromises must be made and stubbornness must be minimized. Explicitly tell everyone upfront that this decision-making process will be timeboxed. If consensus is not gathered in a reasonable timeframe, the team must agree to disagree and commit on moving forward. If a decision is still impossible, you may need to resort to heavier-handed alternatives (see later).

Step 6 — Document, Ratify, and Communicate

Congratulations! A decision has been made. There’s only a few logistical follow-ups left that will put a bow on the entire process.

The final decision must be documented. Other engineers will be interested to hear about the outcome and how the company chose to move forward. The problem must be clear, the solution must be laid out, and the trade-offs must be acknowledged.

Explicit stakeholder signatures go a long way. A ratified document gives the decision outcome weight and the authority to be referenced later.

Don’t be shy—broadcast the results to the company.

Finally, ensure that teams follow through. If an architecture decision was made, is the software actually being built to reflect the architecture? If an organizational process was changed, did it actually happen?

3 Heavier-Handed Alternatives

If consensus-gathering strategies are not effective, you may need to resort to heavier-handed alternatives. One method is to defer the decision to a higher authority. This may be the resident Staff Engineer, a select Principal Engineer, or perhaps even the CTO. This method is decisive, but it is not optimized for goodwill amongst your colleagues.

Another alternative is to align engineers by utilizing non-engineers. If engineers can’t align on a technical decision, it’s likely that they’re rooted in different product and business goals. In practice, I have seen that enlisting the help of a high-ranking product manager to reaffirm company goals may help with engineering alignment.

Another medium-handed approach is to utilize the wedding ceremony rule of, “Speak Now Or Forever Hold Your Peace.” This is a method to keep the process moving when people are lagging; this is not an excuse to skip communication. This is a risky strategy for significant decisions and is also not optimized for goodwill.

Conclusion

Software engineering decision-making can be exhausting and is never a simple sequence of steps. Consensus relies heavily on communication and empathy; it relies less on technical expertise and previous experiences.

Throughout the process, remain firm, but never offensive. Engage the right engineers and hold them accountable to their stake in the decision.

Finally, don’t stress too much. Significant decisions have significant expectations. If you need help, ask for help. If the process isn’t going anywhere and needs to be reframed, make the call and have a reset. The fact that you have taken on this massive responsibility deserves to be applauded!

Best of luck.

“Runtime” and “Runtime Environment” are some of the most overloaded terms in software development. It’s confusing for everyone; this word means many different things in many different contexts. This post’s goal is to provide you with an intuition behind the many use-cases of “runtime.”

Word Overload

Let’s enumerate the common use-cases of “runtime.”

Runtime — The Lifecycle (Noun)

The first meaning of runtime is with regards to program lifecycle. This refers to the period of time in which a program is executing. A intuitive comparison is often made between compile time and runtime.

The IDE is spitting out a lot of new warnings during compile time.

This process is outputting a lot of logs during runtime.

Runtime — During The Lifecycle (Adjective)

In the same context of program lifecycle, runtime is also commonly used as an adjective. This is sometimes hyphenated as “run-time.” The O’Reilly Style Guide recommends using “runtime” for both nouns and adjectives.

This program seems to throw a lot of runtime errors.

What kind of runtime metrics do you want to capture for this application?

One gotcha is that while O’Reilly recommends “runtime” for both adjectives and nouns, they recommend “compile-time” as an adjective and “compile time” as a noun.

We should probably fix some of these compile-time warnings.

If you prefer to catch errors at compile time rather than at runtime, Typescript might be a good choice.

Run time — How Long Did It Take?

“Run time” as two words refers to the raw execution time of a program. This usage doesn’t appear as often as the others.

Q: Did you measure the run time of the data cleanup script last week?

A: Yea—it took about 3 hours.

To avoid runtime overload confusion, simply asking “How long did the script take to execute?” is more clear.

Runtime Environment

Armed with an intuition of runtime as a lifecycle, let’s discuss runtime environments. These two concepts are related, but subtlety different. Confusion is created because people abridge “runtime environment” to just “runtime.”

You have to be careful that your Javascript program doesn’t have any runtime errors when it’s running in the Node.js runtime [environment].

Q: Did you see the release notes for the new JRE?

A: Yea, there’s a lot of fancy stuff going on in the new Java Runtime [Environment].

In these examples, the standalone “runtime” refers to a “runtime environment.” In these situations, it’s better to be precise with your language.

The new Java Runtime Environment seems to be doing a better job with garbage collection.

Is there a way to control how many threads handle the I/O within the Node.js runtime environment?

Culinary Intuition

After developing a craving for Chinese Yu Xiang Eggplant, you decide to hop on Google to look for a recipe. After sifting through a few search results and scrolling past a few walls of ads, you find a clean step-by-step recipe that seems like a winner.

Unfortunately, a recipe on its own is useless. The recipe you found on Google isn’t actually going to get you your eggplant—you need a kitchen! The eggplant recipe’s runtime environment is your kitchen.

Your code is just code. Whatever code you write, in whatever language you choose, needs to eventually execute on a computer. Runtime environments enable this execution.

Everyone’s Runtime Environment — The Operating System

The universal runtime environment for any kind of programmatic execution is the operating system. The operating system is the only way you can get the CPU to execute your code. The OS is the silent hero that ensures your program gets some memory, gets scheduled fairly, and doesn’t disturb its neighbors. It doesn’t matter if you’re using C, Python, or Node.js—at the end of the day, the operating system is everyone’s runtime environment.

An Executable’s Runtime Environment

An executable’s runtime environment is the operating system.

Every operating system defines a binary file format for executable code. In Unix-Like operating systems, this is the ELF file format. In addition, operating systems ship with programs that are able to take these files and give them to the hardware to be executed. In Linux, one example is the execve() program. For all other operating systems, there will be a similar set of file formats and loader programs.

A C Program’s Runtime Environment

What does it take to execute a C program on your computer? This is very close to an executable’s runtime environment; all we need to do is to make sure our C source can turn into a proper executable.

This step is familiar to everyone. Compilers like GCC or LLVM take your C code and create a properly formatted ELF file, targeted for your processor’s architecture, waiting to be passed into execve().

A subtler part of C compilers is that they also provide a C runtime library that is automatically compiled into your program. The purpose of these libraries is to provide you—the programmer—with basic facilities to interact with the runtime environment. For a compiled C program as an ELF executable, this environment is the operating system. For example, the C runtime library provides useful functions like malloc() and free() that allow you to manage memory, automatically injects startup routines into your program to help prepare it for execution, and many, many other things.

The concept of the “C Runtime Environment” isn’t a real concept. A C program’s runtime environment is the universal runtime environment, the operating system. The important point to remember is that all code—from assembly to Javascript—needs some kind of environment in order execute, just like all recipes need some kind of kitchen to turn into food.

Higher-Level Runtime Environments

As programming languages evolved, people wanted an environment that could handle additional tasks that felt cumbersome for developers. Do you really enjoy using malloc() and free() to manage all your dynamic memory? Wouldn’t some kind of automatic reference counting and garbage collection be extremely convenient? Some people say yes and this is how fancier runtime environments like the JRE developed.

Another popular higher-level runtime environment is Node.js. Many developers refer to this as the “Node.js Runtime” or simply just “Node.” Node is a runtime environment for the Javascript language, similar to how the JRE is a runtime environment for the Java language. Node comes with fancy features like a callback queue, an event loop, and a thread pool. Just like the JRE, these bells and whistles exist to make our lives easier as developers.

With higher-level runtime environments, features can become blurred across the language and its environment. Within the Java ecosystem, the automatic garbage collection feature is not technically a feature of the Java language itself, but is actually a feature of the JRE. 95% of developers won’t bother making this distinction. The statement “Java has automatic memory management” can more precisely be stated as, “The Java Runtime Environment has automatic memory management.” As a warning, you’ll get some unpleasant glares if you ever decide to correct someone over these kind of semantics.

Runtime Environment Layers

Runtime environments have their own runtime environments. If you download the Node binary for Linux, you’ll find that it’s just another ELF executable waiting to be run by the OS. So we could say—Javascript’s runtime environment is Node, and Node’s runtime environment is the operating system. Or you could also say—Javascript’s runtime environment is a combination of Node and the operating system. Again, dive deep enough and everyone’s runtime environment ends up being the operating system.

Interpreting Languages

Code does not always have to be compiled to OS/architecture-specific binaries. A common pattern is to execute your programs via interpreters. For example, a Python interpreter can read your Python source and produce corresponding machine instructions for your computer to execute. This conveniently makes Python source very portable. However, the Python interpreter itself is a compiled executable, built for a specific OS/architecture, who’s runtime environment is the operating system.

Conclusion

“Runtime” is one of the most overloaded terms in software development. The two most common contexts are either a program’s execution lifecycle or the environment in which a program executes.

A recipe is only a recipe; source code is only source code. A recipe turns into food via a kitchen; source code moves electricity through metal via a runtime environment.

Runtime environments have their own layers of abstraction. Your Java source gets turned into bytecode and gets interpreted and executed within the JRE. The JRE itself executes via the OS. Everyone’s runtime environment ends up being the operating system.

So the next time your colleague says, “Hey, I was measuring the run time of our program and I noticed we’re getting a lot of runtime errors after we upgraded to the new Node.js runtime“—you’ll know exactly what they mean.

Company culture. Extremely important, impossible to define.

Impossible to define doesn’t mean impossible to understand. A surface-level understanding of culture is available via Glassdoor and company onboarding documents. However, a deep understanding of culture is only available via experience.

This article focuses on a set of values often associated with software engineering culture. Instead of the discussing the values themselves, I discuss the activities which measure them.

Software Engineering Culture — A Basic Understanding

Culture Via Definitions

I will defer the definitions to the Internet:

Culture Via Questions

• Do you enjoy your job?

• Who do you look up to at work?

• How do your colleagues respond to difficult situations?

• What actions are rewarded? What actions are punished?

Culture Via Characteristics

• Heavily dependent on people

• Extremely difficult to change, yet constantly changing

• Influenced by parent cultures, but locally unique

• Experienced differently by different people

Culture Via Values (Software Engineering Specific-ish)

• Freedom — How To Work

• Freedom — Solution Choice

• Collaboration

• Personal Growth

• Operational Excellence

• Transparency

• Feedback

• Developer Happiness

How Is This Useful?

A long list of cliché bullet points is not useful; I’m sure “Operational Excellence” and “Personal Growth” are stamped inside every company’s culture handbook.

At any given software organization, infinite tasks are being concurrently executed by a perpetually understaffed employee thread pool. Code is being written, architecture is being reviewed, retrospectives are being run, interviews are being held, compensation is being planned. A unique culture is reflected and measured in each and every one of these processes.

In this article, I present three standard-issue software engineering activities that can be used as metrics for software engineering culture. Improving these activities is not for the sake of the activity, but for the sake of your company’s culture.

3 Interesting Software Engineering Culture Metrics

#1 — Adoption Of Frameworks & Tools

Values Measured: Feedback / Collaboration

Measurement Example: What ratio of teams have adopted the shared {rate limiting, caching, background job, migration, etc.} framework versus rolling their own solution?

Every software company has a set of teams that are designated as “Platform” teams. These teams are tasked with developing high-leverage software with the goal of accelerating experience-focused teams.

Despite their best intentions, platform developers still struggle to get other teams to adopt their technologies—even when that other team sits down the hallway. Why would they want to use your software over building their own solution or paying a little extra money on AWS features?

The level of adoption of internal platform features measures two (probably more) important cultural values.

The first value is feedback. In order to get a potential customer to adopt your software, you need to talk to the customer. In this case, the customer is your colleague. Do you understand their requirements? Is your software actually solving their problems? Have you met them yet?

A common risk is for platform teams to create a set of “standards” based on their personal perceptions of another team’s requirements. This happens more often than you might expect; it’s in a developer’s nature to build frameworks that bias towards their own problems. As a platform developer, never assume that you fully understand another developer’s problem without engaging him or her. Continuously gathering feedback is critical to develop this understanding and to build trust with your colleagues. This begets adoption.

The second value is collaboration. The adoption of a framework is driven by framework creators. There is a stark difference between these two strategies:

1. Hey, our team just developed a new set of security penetration testing tools. Here’s the documentation, please try it out and let us know how it goes.

2. Hey, our team just developed a new set of security penetration testing tools. Does your team have some time to walk us through your product so we can figure out how these tools might help you?

Both strategies have the same goal—adopt the penetration testing tools. Strategy #1 is more scalable, but it’s the wrong move for early adoption. I argue that Strategy #2 is the more effective route for platform developers to foster sustainable, long-term adoption of their software.

Strategy #2 reflects collaboration and empathy. Strategy #1 reflects throwing things over walls. If you’re willing to sacrifice your time and work alongside your colleagues, it won’t be long before every developer at the company has a new feature request for you. Once Strategy #2 is repeated successfully enough times, auxiliary actions—word of mouth, self-service assets, bug support—will be icing on the cake.

If a software organizations notices that a platform team keeps producing software that no one opts to use, it may reflect a lack of value in feedback and collaboration.

#2 — Effectiveness Of Retrospectives

Values Measured: Operational Excellence / Freedom — How To Work

Measurement Example: In a company survey, ask developers to rate the effectiveness of last quarter’s retrospectives from 1-5.

Look around any software organization and you will find retrospectives surrounding sprints, incidents, and projects. No matter the context, all retrospectives share a similar purpose—deeply understand a situation, cordially discuss areas for improvement, and grow as an organization. Running effective retrospectives is a clear metric for operational excellence.

Retrospectives also measure a more nuanced software engineering cultural value—freedom. Specifically, the freedom to choose how to work.

Retrospectives, alongside other software ceremonies, represent process. Every organization wants their process to be as effective as possible.

Ironically, effective process comes out of no process. Effective process is not built out of process, but is built out of values, or culture. Let’s take incident retrospectives as an example—how would you introduce this process to your organization?

1. We have decided to implement incident retrospectives because Amazon does incident retrospectives. From now on, every incident must be followed by a retrospective that follows this template.

2. We value operational excellence. We make mistakes and learn from them. After an incident is resolved, every team is responsible for implementing a retrospective process so we can collectively improve our operations.

Again, both strategies have the same goal, but may produce drastically different results. Strategy #1 runs the risk of mandating process for the sake of process. This is useless if the culture is not ready for it; it’s impossible to copy Amazon’s ceremonies if your company inherently has different values than them.

Furthermore, process before culture inadvertently stifles the ability for developers to choose how they work. Not every team will be enthusiastic about copying Amazon’s incident response. This doesn’t mean that they are neglecting operational excellence; they might just want to create their own solution.

Strategy #2 reflects freedom. Freedom is the preferred operating mode for software developers. Our goal is operational excellence and we want to be empowered to create a unique solution to reach this goal. For incident retrospectives, perhaps teams want to use a custom template, introduce a “minimum viable retrospective”, or execute their follow-ups asynchronously. If Strategy #2 is repeated enough times, it inevitably turns into a natural process for the organization and the end goal is reached.

This idea is not unique to engineering. Consider how consulting firms engage with their enterprise clients. No one appreciates a consultant barging into an organization and changing up how everything works. Seasoned consultants know that sustainable change does not come from mandating change, but from changing the client’s culture. If a consultant can shift a company’s values, they will inevitably shift that company’s processes, and hopefully that company’s success.

As a developer, examine the various processes that you participate in. Which ones are not effective? Perhaps process has been introduced before the culture is ready for it. If you had the freedom to change the process, while achieving the same goal, what would you do?

#3 — Effectiveness of RFCs (Or Any Other Approval Process)

Values Measured: Collaboration / Freedom — Solution Choice

Measurement Example: RFC creation count. RFC average approval time. Frequency of negative sentiment towards software approval processes.

Another pervasive process found in software organizations is the journey for consensus. Examples include gaining signatures on an RFC document, winning a majority vote from a technical committee, or arriving at an important architecture decision.

Frustration surrounding approval processes may signal that engineers value technology preference over collaboration. These two attitudes are very different:

1. No, that’s not going to work.

2. No, that’s not going to work, but let me help you solve your problem.

To belabor one more time—different strategies, different results, different culture.

Honorable Mentions

Every single software engineering activity measures culture. I’ve only touched upon three, but there are many, many more.

Engagement During All-Hands Meetings

Values Measured: Feedback / Transparency

If leadership is opaque and feedback falls on deaf ears, no developer will bother asking questions during meetings.

Diversity Of Team Ceremonies

Values Measured: Freedom — How To Work

Not every team has to follow the two-week agile sprint cadence to a tee.

Frequency of Performance Recalibrations

Values Measured: Transparency / Personal Growth

Recalibrating an engineer and delaying a promotion can be tough for both the developer and their manager. However, this is a healthy process that shows that management is transparent with each other and takes personal growth seriously.

Average Time To Merge Code

Values Measured: Operation Excellence / Collaboration

Average time to merge is an important health metric that is difficult to trace back to a specific engineering value; a thousand factors contribute to slow code reviews.

Regarding operation excellence—developers should be trained to write well-tested code and be able to present it clearly to their colleagues.

Regarding collaboration—developers have the responsibility to inform their stakeholders of upcoming changes. If you open up a code review in another team’s repository and their reaction is, “Where is this requirement even coming from?”—don’t expect your code to be merged any time soon.

Deployment Health

Values Measured: Developer Happiness

Developer happiness is directly correlated to healthy, predictable, and frequent deployments.

How Often The Career Ladder Changes

Values Measured: Feedback / Personal Growth

Leaderships doesn’t change engineering level descriptions for the sake of changing copy. When leadership fine-tunes the career ladder, it signals that they are listening and putting in an effort to support the growth of the engineers.

Conclusion

As an organization, don’t spend too much time attempting to describe your culture. It is already understood and reflected through process. Spend that energy paying attention to the processes that break down and be wary of introducing process for the sake of process.

As an individual, pay attention to your personal strategy of work. There’s a difference between “No, that’s not going to work” and “No, that’s not going to work—let me help you.” Your strategy affects your actions, which affects the company’s processes, which changes culture.

The path of the Staff Software Engineer represents the career progression of a technologist. Senior Engineer leads to Staff Engineer which leads to Principal Engineer which hopefully leads to a comfortable retirement. Unfortunately, Staff Software Engineer responsibilities are often poorly defined by management. The goal of this article is to give clarity to these responsibilities so aspiring Staff Engineers—and their managers—can perform their jobs more effectively.

Context

These responsibilities make the most sense within a specific context:

• An organization that makes money by selling software

  • This is important because it forces alignment between engineering and business goals.

• An organization that has separated the technical track (Staff/Principal) from the managerial track (Director/VP)

  • This is important because it implies that the company has created a space for technologists to make an impact. The risk is that this space can be ambiguous.

• An organization with at least few hundred software developers

  • This is important because a minimum people scale implies a basic organizational hierarchy and an ample need for collaboration.

These responsibilities will not make as much sense if you’re in a completely different context.

Preface: Product <> Engineering

Software companies strive to create inspirational product visions and effective product strategies. The vision is the what and the strategy is the how. If done well, these two items are extremely powerful. The vision of “Become A Space-Bearing Civilization” is enough for SpaceX to recruit thousands of engineers to work on rockets.

The company’s vision is set by product management—not engineering. The engineering organization is in service to the product organization which is in service to the business. Software must be designed to support the product strategy and ultimately realize the product vision.

Product management respects engineering; they know that tech debt is no laughing matter and that nothing ships without sustainable software. Similarly, engineering respects product management; they are fully aware of the technical implications baked into product goals.

• Does the company want to expand internationally?

• Does the company want to provide a Developer Platform?

• Is the target customer an enterprise or a person? Both?

• How quickly do we have to ship given the competition and market?

In this context, product and engineering collaboration is paramount. Staff Software Engineers are responsible for understanding product goals and aligning their work with what’s best for the business.

Staff Software Engineer Responsibilities

Problem-First Mindset

Every engineering organization has an overwhelming amount of technical problems. A Staff Engineer is responsible for discovering the problems that pose a risk to the business.

• The developer experience in a particular domain is terrible, which kills developer velocity and hurts product development.

• The responsibilities of a microservice are nebulous, which causes teams to stall and hurts product development.

• A business-critical codepath can’t handle a 10X magnitude bump and needs a re-architecture.

A large list of problems is not useful by itself. In additional to discovering problems, a Staff Engineer also has the responsibility to prioritize them.

A common risk is for Staff Engineers to hand off prioritization to managers. While an Engineering Manager can help with operational logistics, they may not be as informed on the technical implications of the problems at hand.

Transform Competing Opinions Into Decisions

Software developers inevitably grow attached to their work. Attachments often lead to conflicting technical opinions, and in the worst case scenario, block progress for the company.

Tensions are high surrounding the responsibilities of a particular microservice. There is no clarity for the service’s future. One team wants to expand its scope; another team wants to narrow it. Both groups have valid arguments and concerns, but development has stalled. Staff Engineers are responsible for breaking technical stalemates. They do this by talking to stakeholders, weighing trade-offs, and arriving at strong, opinionated recommendations.

Recommendations are only recommendations. Staff Engineers must push through friction and create decisions. Decision-making is a difficult process and will leave certain engineers upset; this is something that can’t simply be delegated to management. As a Staff Engineer, turning a prolonged technical debate into a recorded decision is a huge win for the company.

Concede With Humility

As a Staff Engineer, you may find yourself holding on to an opinion. Discussions surrounding an important architecture decision are going nowhere and now a Principal Engineer needs to conduct a tiebreaker. Even though arguments can be healthy, these situations are costly and must be minimized. Staff Engineers should practice egoless programming and know when to compromise; there is a difference between standing up for what you believe in and being stubborn.

Create More Staff Software Engineers

As with any other role, the end goal is to replicate yourself. For Staff Engineers, this means creating more Staff Engineers. A convenient byproduct of creating a prioritized list of technical problems is that you’ve automatically created a set of high-impact projects for growing senior engineers. This is a great way to elevate your colleagues while giving yourself extra bandwidth.

Choose The Right Problem To Solve

Staff Engineers must thoughtfully choose which problems they want to personally handle. A Staff Engineer can solve problems that most engineers cannot. By definition, there are fewer of these problems and they must be properly identified out of the sea of technical issues.

There are two common risks if business-impacting problems are not properly assigned. First, if a Staff Engineers chooses to work on a “comfortable” problem, then a senior engineer misses a growth opportunity and the Staff Engineer’s time is underutilized. This is counterproductive because the engineering organization doesn’t grow. Second, if a complex problem is delegated to an engineer who isn’t ready for it yet, an inadequate solution implemented for a critical business problem could put the company in worse shape than how it started.

Broadcast

A common sentiment heard around software organizations:

The communication required of Staff Engineers is very different than that of individual contributors, team leads, or engineering managers. Engineers mistakenly believe that the Staff Engineer role requires less communication—it’s not less, just different.

Staff Engineer projects are often isolated. This might be a deep untangling of debt or a broad, self-navigated exploration. It is impossible for the organization to benefit from this work if they’re unaware of it. Common mediums for this type of communication are sending newsletters, holding Q&A Meetings, and providing progress updates at relevant All-Hands Meetings.

A focus on consistent broadcasting is a quality that sets a Staff Engineer apart from his or her peers. This benefits both the company and the individual. For the company, you never know when a developer might pivot strategies based on a Staff Engineer’s explorations and recommendations. For the individual, actively communicating your work gives you a voice in the organization. This is essential for Staff Engineers because they must be able to influence without authority.

Observe

As a Staff Engineer, you have deeper technical context and experience than other engineers. You will see nuances where other engineers do not. Since it’s impossible to be involved in every project, the next best thing to do is to observe how the software is being built.

1. Actively participate in architecture review meetings

2. Read through architecture proposals and product briefs (new and old)

3. Sift through chatrooms, paying extra attention to cross-team inquiries

You notice that an engineer pops into another team’s Slack channel and asks about a certain piece of critical code. While you can easily tell them how the code works, it’s more important to determine why they are asking about it in the first place.

The goal of active observation is to get ahead of technical problems. It’s not practical to expect every engineer to ask for guidance when necessary; they don’t know what they don’t know. Staff Engineers have the responsibility to keep a pulse on the software before a minor gotcha turns into a technical mess. There is never an expectation to catch everything, but there is an expectation to speak up when necessary.

Expedite Instead Of Block

As a Staff Engineer, one of your main responsibilities is to provide guidance. This includes approving architecture proposals, reading pull requests, and giving up your time for other engineers. This guidance is essential if it’s in the critical path of a project. If a developer needs your sign-off for the sake of moving product development forward, then that item should immediately be re-ordered to the top of your TODO list.

When too many projects depends on your guidance, there is a risk of creating an organizational single point of failure. A common cause of this is when select engineers accumulate too much historical context and have neglected knowledge dissemination. This is irresponsible; SPOFs inevitably slow down projects and must be avoided at all costs. As a Staff Engineer, you are responsible for using your position to expedite work, not block it.

Conclusion

Many people mistake the path towards Staff Engineer as a way to stay out of the limelight, a way to avoid meetings and dealing with people. While these positions exist, their impact and growth are limited, especially in the context of a growing software company.

Successful Staff Engineers are much more than technical experts. They are role models that understand how their actions affect the engineering culture. They respect product management and optimize their efforts in service of the business. Finally, they don’t work in ivory towers, but are actively engaged with the organization.

I recently read through a Hacker News thread discussing the article “Kafka Is Not A Database”, by Arjun Narayan and George Fraser. The opinions behind this topic are fascinating and I enjoyed sifting through comments from both sides of the table. For the purposes of this post, I’ve labeled these two broad groups of thoughts as Team Blue and Team Red.

Team Blue believes that Kafka, a popular streaming platform, has the potential to be the source-of-truth for your data—replacing one of the key responsibilities of conventional databases.

Team Red strongly disagrees.

The following is high-level summary of these opinions.

Kafka As A Database

TL;DR Streams,Events,Kafka

First, a conceptual model of streams:

Imagine that you’ve hired an invisible assistant. This assistant’s only responsibility is to record everything you do:

1. Wakes up @ 8 am

2. Brushes teeth @ 8:35 am

3. Begins showering @ 8:40 am

4. …

As you go about your day, your assistant meticulously captures your activity and adds new entries into your daily log. This growing log represents a stream of your day-to-day activity. The data elements that comprise the stream are known as events. Once an event is recorded, it is immutable; the fact that you brushed your teeth at 8:35 am yesterday will never change.

While your invisible assistant is following you around and producing new events to your stream, who is consuming them? Anyone who cares! I’m sure your mom would be thrilled to process through your activities and call you if there are any anomalies.

What about Kafka?

Kafka is a platform (large set of software tools) that helps programmers work with streams.

Team Blue – Streams Everywhere

Team Blue believes that streams are a natural model for both life and computing. Your users produce a stream of clicks, sign-ups, and orders. Your code produces a stream of logs, metrics, and programmatic events. Team Blue believes that the intuitive nature of streams can lead to intuitive software architectures.

An essential component of any system is the database—a convenient place to read and write whatever you want. How can a stream replace this concept that we are so used to? If streams are a never-ending series of immutable events, then the property of state is the culmination of events up to a certain point in time. For example, the latest state of a database is the culmination of all operations made to that database. In computing, this is referred to as Event Sourcing. This paradigm is very different to how many developers are used to thinking about their data.

Beyond data storage, Team Blue is bullish on the idea of stream-focused architectures in general. Streams work well with microservices, promote loose coupling of systems, and have the potential to create leverage in your software. If you’re thinking about streams, you will inevitably think about Kafka—one of the industry’s leading streaming platforms.

Can you use Kafka as a database? Team Blue says yes.

Should you use Kafka as a database? That’s up to you.

Team Red – You Need Databases

On the other side of the table is Team Red who believes that programmers are making a mistake by replacing conventional databases with Kafka.

Team Red believes that 99% of applications need the features of a conventional database—especially the features handling complex concurrency issues. Many of us use these features and they are critical for the success of our applications. Modern databases are battle-tested and have been iterated on for decades. Team Red believes that the message of “Kafka As A Database” is a dangerous message that leads developers into architectures they’re not equipped to handle.

The People On Each Team

There is no shortage of influential people who align with Team Blue. One notable engineer is Jay Kreps, CEO of Confluent. Confluent is an organization that sells Kafka as a solution to enterprises. There is a direct correlation between using streams and using Kafka; I’m sure Jay has a few extra reasons to be bullish on streams beyond his personal software architecture preferences.

Another influencer aligned with Team Blue is Martin Kleppmann, author of the extremely popular book Designing Data-Intensive Applications. Martin Kleppmann is a researcher who loves the idea of streams. His goal is to educate as many programmers as possible. Martin’s only qualm with Team Red’s opinion is the tone of their blog post. In his own words:

Nice blog post [Kafka Is Not A Database] explaining why the “database inside out” approach is not for everyone. @narayanarjun and @frasergeorgew are right — if you need to maintain constraints before events are written (e.g. not selling more items than you have in stock), it’s easier to use a DB and CDC.

On the other hand, the message is also a bit patronising. It basically says “you’re not clever enough to use an event log correctly — leave database stuff to the experts”. I respectfully disagree: it is good to explore new approaches. Learn about the trade-offs, educate yourself.

Martin Kleppmann

Engineers on Team Red’s side include Arjun Narayan and George Fraser, co-authors of the article which refutes the idea of “Kafka As A Database.” They have decided to put a large, yellow caution sign in front of Team Blue’s message. Team Red has no issue with Kafka or streams; they have an issue with eager developers throwing out their RDMSs and jumping on the stream bandwagon. Their only extra agenda might be to get extra eyeballs on their engineering blogs.

Arguments From Team Blue

The following points are summaries of topics discussed in Martin Kleppmann’s video.

High Integrity Data – Never Deleted

With a database, your information is easily susceptible to being lost. Once your Postgres row is updated, it’s difficult to recover the previous state of that row without introducing custom accountability software. Databases and the records inside them are designed to be mutable. It’s both a great convenience and a great risk. If a table accidentally gets deleted, you might find yourself scrambling to restore your database from a backup. This is particularly painful in production.

On the other hand, if your data exists as a log of immutable events—and that log happens to be retained forever—the history of your changes never runs the risk of being erased. Accounting is a great example of this—accountants balance their books by only appending new entries to their ledgers. An error in a financial transaction is always fixed by another transaction.

Team Blue believes that this characteristic can be an asset to software architectures. As a permanent log of events, data will always have its historical context and can easily be audited. Furthermore, the lack of an “eraser” forces programmers and processes to be held highly accountable for their actions.

Separation Of Concerns – Reading / Writing

Team Blue points out that one of the great conveniences of databases is also one of its greatest challenges. The convenience of databases is undeniable—an easy one-stop shop to read and write anything. The drawback to being a one-stop shop is that you can’t be the best at any one thing, which becomes problematic as you scale.

Conventional databases easy conflate reads and writes, two very different styles of operations. As you scale, the access patterns in your software will inevitably force you to optimize for a certain set of operations. If an operation-specific optimization is done in a centralized location, it will likely degrade the performance of other operations in that location. If you optimize reads, you complicate writes—and vice versa.

One of the first steps in optimizing SQL reads is to begin denormalizing (duplicating) data across various tables. Once your data is duplicated, queries require less joins and respond faster. However, the drawback of denormalization is that it creates additional complications on the write pathway. Programmers need to remember to update and synchronize the duplicated data across multiple tables.

This pattern grows in complexity as you scale. If denormalization isn’t getting you the required read performance, the usual next step is to introduce a cache in front of your database. A cache is extremely convenient and can be designed to precisely answer the questions your clients are asking. However, as many programmers are intimately aware of, maintaining a cache is never easy. You now to need synchronize writes across network boundaries, worry about distributed transactions, and debug subtle invalidation issues.

Going back to our make-believe example, my assistant only has one responsibility—log my daily activity. One day, I decide that I want to look back in time to see how my activities are categorized. Unfortunately, I can’t ask my assistant this question because it’s too specific; all he knows how to do is produce events. To get insights on my activity, I need to hire a category manager to consume my activity stream and categorize everything. Assuming my manager is caught up, she’ll be able to immediately and accurately answer any question I have regarding my activity categorization.

With streams, there is a clear separation of concerns between readers and writers. Producers only worry about producing. Their access pattern is straightforward and lightning fast. Consumers only worry about consuming. They can be programmed to handle any use-case and never have to worry about affecting the performance of their producer counterparts. How fast you’re able to push updates to Elastic Search is completely independent from how quickly Kafka can write to a file.

For Team Blue, this separation of concerns is an advantage of streams. Databases are convenient, but they conflate concerns and quickly create performance trade-offs as you scale.

Easier Caching

One of Martin Kleppmann’s most interesting ideas is that you can get “Fully Precomputed Caches” with stream architectures. A common setup for Kafka consumers is to process logs in order to create various materialized views. A materialized view has a similar end goal as a cache—provide a precomputed view of the data that is useful for a client. However, the way in which they are maintained is drastically different.

With a materialized view, the write pattern is incredibly simple. The materialized view is created by simply processing through the log of events and updating the view accordingly. The process is straightforward, but potentially time-consuming. On the other hand, caches often have complex and error-prone write patterns. Logic to update and invalidate caches is often scattered around business logic, executed randomly depending on how customers use an application.

The concept of “warm vs. cold” caching also goes away with materialized views. If a materialized view is fully created (completely “caught up”), then it represents the complete state of the data. There is never a cache miss because a complete materialized view is completely hot by definition. Instead of worrying about cold starts and how to properly warm up a cache, programmers can create materialized views offline and start using them whenever they’re ready.

Arguments From Team Red

“Streams Everywhere” Is A Dangerous Message

Team Red believes that the ideas behind “Streams Everywhere” are hurting the software industry by causing eager engineering organizations to adopt streams without understanding their consequences. George comments:

We’re trying to address a real problem that is happening in our industry: VPs of eng and principal engineers at startups are adopting the “Kappa Architecture” / “Turning the Database Inside Out”, without realizing how much functionality from traditional database systems they are leaving behind. This has led to a barrage of consistency bugs in everything from food-delivery apps to the “unread message count” in LinkedIn. We’re at the peak of the hype cycle for Kafka, and it’s being used in all kinds of places it doesn’t belong. For 99% of companies, a traditional DBMS is the right foundation.

George Fraser

George’s points are valid. Software engineers have a tendency to adopt new and shiny objects without fully understanding them. Just because Kafka is trending, it doesn’t mean you need to solve every problem with it. Engineers need to be responsible, understand trade-offs, and choose technologies that fit their requirements.

Losing Critical Database Features

Team Red warns that many critical features will be lost if you forgo a database in favor of logs. In particular, features that help enforce data integrity are extremely useful and are extremely difficult to replicate. These features include foreign key constraints, the atomic (all-or-nothing) nature of transactions, and complicated isolation techniques that help mitigate concurrency issues. Arjun of Team Red writes:

In principle, it is possible to implement this architecture in a way that supports both reads and writes. However, during that process you will eventually confront every hard problem that database management systems have faced for decades. You will more or less have to write a full-fledged DBMS in your application code. And you will probably not do a great job, because databases take years to get right. You will have to deal with dirty reads, phantom reads, write skew, and all the other symptoms of a hastily implemented database.

Arjun Narayan

Arjun also has valid concerns. For a majority of applications—especially the smaller ones—these data integrity features are convenient and may be essential for your application’s success.

The Complexity Of Asynchronous Data Integrity

In one of his presentations, Martin Kleppmann reviews the ACID properties of a database transaction and how they can be implemented with a stream-based architecture:

If your system has strict requirements for validating data, you need to be very thoughtful when designing a stream-based architecture. As shown in the video, the complexity of ensuring all your customers have unique email addresses with streams is significantly more involved than adding a uniqueness constraint to a database column.

With streams, basic features that we take for granted become new responsibilities for your code and operations. You’ll need to separate the intent to write versus committing the write. You’ll need to serialize consumption of specific streams while parallelizing others. You’ll consume events only to immediately produce other events. All these complexities, combined with the fact that the software industry is extremely inexperienced with stream architectures, should give developers pause before choosing to adopt any of these new paradigms.

Kafka Alongside A Database

Team Red proposes that Kafka shines when used alongside a database.

In the article, Team Red argues that the data integrity features of databases perform the crucial functionality of “access control.” A common architecture setup is to use Kafka alongside a Change Data Capture (CDC) system. In the diagram above, there are three basic steps:

1. User requests hit the application and kick off transactional write operations to the database.

2. If all the checks look good, the writes land and update the database.

3. CDC reacts accordingly, translates the valid changes into valid events, and ships them off to Kafka to be consumed.

So, Kafka As A Database?

There is never a one-size-fits-all solution in software development. The discussions surrounding “Kafka As A Database” are deep and I encourage everyone to dig further into the opinions of both Team Red and Team Blue.

I personally have never used a Kafka log as the source-of-truth for my data. Stream-focused paradigms are fascinating to read about and I conceptually understand their benefits, but I’m personally a little wary of their practicality in large software organizations—especially ones with a wide range of opinions. Software development is hard enough as it is, even when trying to go “by the book.”

The difference between the titles Software Engineer vs Programmer vs Hacker vs {other_coding_title} is rooted in a person’s ideals. This difference is succinctly summarized in the wonderful article 3 Tribes of Programming, where the author buckets programmers into three distinct “tribes”:

You are a poet and a mathematician. Programming is your poetry

You are a hacker. You make hardware dance to your tune

You are a maker. You build things for people to use

josephg.com

The author asserts that all coders lean towards one of three tribes—The Poet, The Hacker, or The Maker. Even though a stark contrast is made to separate these groups, these tribes are not mutually exclusive—many programmers are Makers by day, Hackers by night, and aspiring Poets by weekend.

Software Engineer vs Programmer is a false dichotomy I’ve created to separate Makers from Non-Makers. I believe this separation to be relevant because the majority of us are Makers by profession—myself included. Without a doubt, the most accessible path to make a career out of code is the path of the Maker, or the Software Engineer.

If the Maker represents the Software Engineer, who is the Programmer? These are the Hackers and the Poets. These are the coders that code for the raw love of computing, the people who rightfully put the science in computer science. You’ll find these “purist” programmings developing new languages, tweaking assembly, and occasionally scoffing at the work of Makers. Pursuing a career as a Programmer will not be as accessible as pursuing a career as a Software Engineer.

Practically speaking however, this kind of separation is not helpful; everyone writes code for their own reasons. Since the majority of us are Makers, let’s dive into what’s actually important for a Maker to focus on.

Software Engineer Vs Programmer

For the Poets and Hackers, the work of Makers (Software Engineers) may appear dull. This is true to a degree, the average work of a Maker will not push the envelope of computer science. As a Maker, your work is removed from theory and physics; your work revolves around value and impact. From a technical perspective, the work may seem mundane, but this does not mean the the job of a Maker is easy.

You Need To Be A Hacker (A Little)

As a Maker, your objective is to build scalable and reliable software that solves a problem. To have a chance of doing this successfully, you need the foundations of a Hacker. It’s difficult to build software when you don’t understand how a computer works.

Choosing Technologies

As a Maker, your success depends on your ability to leverage the endless shelf of available technologies. Makers optimize for quick iterations in the face of constant change. To do this well, we must avoid re-inventing the wheel and choose the opportune moments to take advantage of existing solutions.

Integrating Technologies

After the relevant technologies have been selected, they must be integrated. A large portion of your work will be gluing these modules together in a coherent way. Makers need to be comfortable using and adapting to other people’s software. Do not underestimate glue-work; it is not a trivial.

Maintenance

As you’re iterating and applying glue, you must also pay attention to maintainability. If your software succeeds and creates a valuable product, then that product—and the corresponding code—will live on for many years. The longer software exists, the more important maintainability becomes. This is a good problem to have! You and your colleagues need to be able to iterate on the code without losing all of your hair.

Domain Modeling

As a Maker, you are a problem-solver. Problems belonging to real-life domains don’t always map nicely to 1s and 0s. It’s critical to understand the domain you’re working in so you can map it onto software. This means talking to your business colleagues, interviewing customers, and constantly measuring your product’s usability. Hackers and Poets benefit from thinking like a computer; Makers benefit from thinking like a human.

Creating Leverage

In addition to using technology as leverage, software engineers must always be looking for ways to create leverage. In organizations, this is commonly referred to as “Platforms.” Many companies have created successful business models out of monetizing leverage. Examples include Amazon Web Services, Twilio, and Stripe’s Platform of Platforms.

As a Maker, keep an eye out for opportunities that create leverage. This could be a small Git script to normalize your team’s branching patterns. This could be a lightweight library to handle JSON marshaling for a common use-case. This could be a unified way of handling Authentication across all your organization’s services. Whatever it may be, leverage is wonderful—create as much of it as you can.

Working With People

Finally, the importance of collaboration can not be neglected. All problems are people problems. Examine any public engineering career ladder, and you will notice how small of a role coding plays as you reach higher levels.

For those of us pursuing careers as Makers, collaboration and teamwork are crucial. In additional to your technical chops, devote time to develop soft skills—read about the psychology of programming, put yourself on projects with different personalities, and debug collaboration issues as you would a bug. The people aspect of software development will remain constant in perpetuity.

Software Engineer Vs Programmer Summary

• There are 3 tribes of programmers—The Poets, The Hackers, and the Makers.

• These tribes are not mutually exclusive, but everyone leans towards one tribe based on their ideals and goals.

• Most software engineers are Makers. This is where a majority of the jobs and money are at.

• As a Maker, you must have some Hacker in you. To build a reliable and scalable product, you mustn’t neglect the basics of computer science.

• As a Maker, you must familiarize yourself with existing technologies.

• As a Maker, a significant amount of your work will be gluing technologies together. Don’t underestimate integration work.

• As a Maker, make time for maintenance—at least 20% of your time. As software becomes more successful, maintenance becomes more important.

• As a Maker, optimize for leverage. Use it. Create it.

• As a Maker, collaboration and teamwork will get you farther than code.

Debugging is a critical skill. More important than the skill is the mindset. The debugger’s mindset is the attitude that you must always understand the why behind a problem; any ambiguities or unknowns are unacceptable. This mindset has the potential to carry you from debugging small functions to solving difficult organizational issues.

The Virtuous Cycle Of Debugging and Understanding

There is a virtuous cycle between debugging a program and understanding a program. 95% of time, you’ll be contributing to a pre-existing codebase. In order to make coherent changes to a repository with a long history and numerous authors, you need put in work to understand the system.

A tried and true way of expanding your understanding is through debugging. Every time you squash a bug, you understand the system a little better. Once you understand the system a little better, you are empowered to debug larger issues. Once you solve larger issues, you begin to understand deeper nuances of the code. This creates a virtuous cycle where both your debugging skills and your comprehension build on each other.

This virtuous cycle is your only option. If you fail to debug a small issue, you have failed to understand a scoped area of the code. Without a basic understanding of the system, you’ll lack the support structures for any further understanding. As the system evolves, you will find it harder and harder to keep up.

Be Scientific About It

Debugging is a science. It’s not guessing. It’s not throwing log statements in random locations. To debug effectively, you need a method. Your actions must be deliberate, your hypotheses must be proven (or disproven), and you must conclusively arrive at solutions.

If you work alongside experienced developers, you may get the impression that they can naturally and effortlessly debug anything. You may feel that they work by intuition rather than science. It’s difficult to pinpoint why their methods are so effective. How can this be?

This “intuition” is just experience. After putting years of coding under your belt, you will begin to leverage past experiences to find heuristics in debugging. You’ll see the same problem manifest itself across many projects. This intuition is the culmination of years spent methodically squashing bugs.

Debugging Has Its Own Scope

Debugging a bug in application code is very different than debugging a complicated distributed system. Similar to your projects, debugging has varying degrees of scope.

When you first start debugging, you will begin by fixing issues in your local domain. This might be checking for an extra edge case or preventing a particular runtime exception. If you’re doing this on a regular basis, it won’t be long before you start debugging larger issues.

As you gain more experience, you will begin to compartmentalize information. You’ll move specific modules into black boxes, each with their specific inputs and outputs. You will begin to focus on interfaces and integrations. You will skim through implementations, only taking a deeper look out of curiosity.

With a growing level of mental organization, the scope of your debugging abilities will increase. You’ll be able to consistently perform process of elimination across a variety of possibilities. You’ll be able to quickly isolate issues to specific modules. Most importantly, you’ll be able to offer robust, practical solutions for the system.

Debugging Skills Extend into Organization Problems

Your debugging skills will extend to different domains. A debugger’s mindset can give you a more sophisticated product sense, assist you with managerial duties, and even enable you to improve business outcomes. No matter which direction you decide to steer your career, debugging is critical.

For example, if you go down the engineering manager’s path, there will be times where you need to debug a dysfunctional team. Similar to a software defect, you must be scientific and diagnose why the team isn’t operating well. The project might be lacking adequate product specification, or perhaps two team members are butting heads. As a manager, you will need to debug organizational issue with the same attitude that you handle software bugs. This means attend extra meetings, reading through contentious code reviews, or taking the extra time to speak with engineers. It’s a different kind of debugging, but it’s still debugging.

Conclusion

The debugger’s mindset is crucial. You must always understand the “why” behind any problem; anything less is unacceptable. This mindset will enable you to solve a variety of problems, constantly reinforce your understanding, and allow you to implement robust solutions. As for your career, these skills will assist you from handling edge cases in functions, to analyzing nuanced system failures, to fixing organizational inefficiencies.