Skip to main content

Beyond CRUD: What I Learned About Building Reliable Backend Systems

Updated 23 min readMarkdown

A practical learning journey through databases, event-driven architecture, distributed systems, concurrency, and scalability.

Introduction: Beyond Making APIs Work

When I first started working on backend systems, my primary focus was building functionality: designing APIs, implementing business logic, interacting with databases, and making sure everything worked as expected.

As I gained more experience, I started realizing that building a backend system involves much more than making individual requests succeed.

What happens when multiple requests try to update the same resource simultaneously? What if a service crashes after committing a database transaction but before publishing an event? What happens when a message is delivered more than once, or when two services have different views of the same business operation?

These questions introduced me to a deeper side of backend engineering: reliability, consistency, concurrency, and distributed systems.

I started exploring these topics more deeply, trying to understand not only how different technologies work, but also why certain architectural patterns exist, what problems they solve, and what trade-offs they introduce.

In this article, I want to share some of the concepts and lessons that shaped my understanding of building reliable backend systems. We'll explore databases, event-driven architecture, messaging, idempotency, distributed transactions, concurrency, and scalability.

Rather than discussing these concepts in isolation, I'll use an e-commerce inventory reservation workflow as a recurring example to connect them and demonstrate how they address real engineering challenges.

The goal isn't to present a universal architecture or claim that there's one correct solution to every problem. It's to explore how we can move beyond simply building APIs and start thinking about the reliability and correctness of the systems behind them.

1. Understanding What's Happening Inside the Database

A backend application can have well-designed APIs and clean business logic, but its behavior still depends heavily on how it interacts with the database.

As I explored database performance and reliability, I found it useful to think beyond writing queries that return the expected result. We also need to understand how those queries execute, how concurrent operations interact, and what consistency guarantees the database provides.

Query performance and indexes

Consider an inventory table with millions of products. A query that looks simple in code can become expensive as the table grows:

SELECT *
FROM inventory
WHERE product_id = 'P-101';

Without a suitable index, the database may need to scan a large part of the table to find the matching row. An index can make lookups much more efficient, but indexes are not free. They consume storage and add work to inserts, updates, and deletes because the index itself must also be maintained.

That's why I started paying more attention to query plans rather than assuming that adding an index is always the answer.

In PostgreSQL, EXPLAIN shows the plan the database expects to use, while EXPLAIN ANALYZE executes the query and reports actual runtime information. Comparing estimated and actual row counts, scan types, and execution time helps identify where a query may be doing more work than expected.

The important lesson is that database optimization should be driven by evidence. An index should support a real access pattern, not simply exist because a column appears frequently in a WHERE clause.

Transactions and protecting business invariants

Performance is only one part of the problem. A database also needs to protect the correctness of business operations.

Imagine a customer placing an order containing three products. The system needs to reserve the requested quantity of each product. If two products can be reserved but the third is unavailable, we generally don't want to leave the first two partially reserved.

This is where a database transaction becomes important. The reservation operations and order creation can be performed within one transaction. If any required reservation fails, the application rolls back the transaction so that none of the changes are committed.

For an individual product, a conditional update is safer than reading the available quantity into the application and then updating it:

UPDATE inventory
SET available_quantity = available_quantity - 2
WHERE product_id = 'P-101'
  AND available_quantity >= 2;

The application checks the number of affected rows. One affected row means the reservation succeeded; zero means the product was unavailable or the requested quantity could not be reserved.

For an order containing multiple products, the application performs the required updates in the same transaction and rolls everything back if any update fails.

This is a useful example of letting the database enforce an important invariant: the available quantity must never become negative. The application still coordinates the business operation, but it doesn't rely on a separate read followed by an unprotected write.

Primary databases, read replicas, and replication lag

As an application grows, it may use read replicas to handle some read traffic without sending every query to the primary database.

A common arrangement is to send writes to the primary and distribute eligible reads across one or more replicas. This can improve read capacity, but it introduces an important consideration: replication is not necessarily instantaneous.

With asynchronous replication, a write can be committed on the primary before the change has reached a replica. A subsequent read from that replica may therefore return an older value.

For example, a customer might successfully reserve the last available item, but an immediate read from a lagging replica could still show the previous quantity.

This is why the choice of database endpoint matters. Reads that require the latest committed state---especially decisions that affect inventory correctness---should use an authoritative source rather than assuming every replica is immediately up to date.

Read replicas are useful, but they introduce consistency trade-offs that the application needs to understand.

What I took away from this section is that database engineering is about more than storing and retrieving data. It's about understanding query execution, protecting business invariants, and choosing the right consistency guarantees for each operation.

2. Moving Toward Event-Driven Architecture

As systems grow, a single business operation often needs to trigger work in several parts of the application.

Consider an order being placed. The system may need to reserve inventory, initiate payment, send a notification, and update analytics. One option is for the Order Service to call every other service synchronously.

That approach can work, but it creates dependencies between services. If the notification service is slow or unavailable, should that prevent a customer from placing an order? Does the Order Service really need to wait for every downstream operation to finish?

These questions led me to explore asynchronous communication and event-driven architecture.

Synchronous communication versus asynchronous messaging

With synchronous communication, a service sends a request to another service and waits for a response. HTTP APIs are a common example.

This is useful when the caller needs an immediate answer. For example, an API may need to check a service's response before it can continue.

With asynchronous communication, a service publishes a message or event and continues without waiting for every consumer to finish processing it.

For example, after an order is accepted, the Order Service can publish an OrderPlaced event. The Payment Service can consume it and start the payment process, while a Notification Service independently sends a confirmation message.

This reduces direct runtime coupling between the services. However, asynchronous communication introduces other concerns: messages may be delayed, consumers may fail, and processing may happen more than once.

It is not automatically better than synchronous communication. The right choice depends on whether the caller needs an immediate result, how much coupling is acceptable, and what failure behavior the business can tolerate.

Understanding Kafka topics, partitions, and consumer groups

A message broker such as Apache Kafka provides a way to publish and consume events without requiring producers to know the internal implementation of every consumer.

In Kafka, events are written to topics. A topic is divided into partitions, which allow data to be distributed and consumed in parallel.

A producer can use a message key to influence which partition receives an event. For example, using order_id as the key can keep events for the same order on the same partition, assuming the topic's partitioning configuration remains stable.

Kafka preserves the order of records within a partition. It does not provide a global ordering guarantee across all partitions. Also, if a consumer processes records concurrently, the order in which processing completes may differ from the order in which records were read.

Consumer groups provide another important part of the model. Consumers within the same group share the work of reading a topic's partitions. A partition is assigned to one consumer in a group at a time.

For example, if a topic has six partitions and a consumer group has three consumers, each consumer can be assigned two partitions. If the group has ten consumers, some consumers will remain idle because there are fewer partitions than consumers.

Different services can use separate consumer groups to process the same topic independently. The Payment Service might have one group, while the Notification Service has another. Each group tracks its own progress.

Kafka also retains records according to its configured retention policy, which means consumers can sometimes replay earlier events by resetting their offsets. This is useful, but it also means consumers must be designed with replay and duplicate processing in mind.

The key lesson for me is that a message broker helps services communicate asynchronously, but it doesn't automatically make the business workflow reliable. We still need to think about delivery guarantees, retries, duplicate processing, and what happens when a consumer fails.

3. Designing for Failure: Idempotency, Retries, and Reliable Event Publishing

Once a system uses asynchronous messaging, failure handling becomes part of the design rather than an edge case.

A consumer can process a message successfully and then crash before recording its progress. When it restarts, the broker may deliver that message again.

This is a normal possibility in an at-least-once processing approach. The goal is not to assume that every message will be delivered exactly once, but to make repeated processing safe.

Idempotency: making repeated operations safe

An operation is idempotent when repeating it has the same intended effect as performing it once.

Imagine the Inventory Service receives a request to reserve four units for an order. It updates the database successfully, but the consumer crashes before committing the Kafka offset. The message is delivered again after the service restarts.

If the consumer blindly subtracts four units again, inventory becomes incorrect.

One approach is to maintain a processed_events table with a unique event identifier. The consumer checks whether the event has already been processed and records it as processed as part of the same database transaction as the business update.

Conceptually, the transaction includes:

  1. Check or insert the event identifier under a uniqueness constraint.
  2. Apply the inventory reservation.
  3. Store the result of the operation.
  4. Commit the transaction.

If the event is delivered again, the uniqueness constraint prevents the same event from applying the business update a second time. The consumer can then acknowledge the message by committing its offset after the database transaction has committed.

There is an important distinction here: event-level idempotency and business-level idempotency are not identical.

Two different event IDs could still represent the same logical business operation. For example, two separate messages might both request a reservation for the same order. A unique constraint on the reservation's business identifier, such as order_id or a dedicated reservation ID, can protect against that case.

The correct idempotency key depends on what the system considers to be the same operation.

Retries: not every failure is temporary

Retries are useful when an operation fails because of a temporary condition, such as a network timeout or a temporarily unavailable dependency.

But retrying every failure indefinitely can make a system less reliable. An invalid request, for example, is unlikely to become valid simply because we send it again.

A more deliberate retry strategy distinguishes between transient and permanent failures.

For transient failures, exponential backoff increases the delay between attempts. Jitter adds randomness to those delays so that many clients don't retry at exactly the same time. Retries should also be bounded by a maximum attempt count or time limit.

Messages that continue to fail can be moved to a dead-letter queue for investigation or controlled reprocessing.

There is also an ordering consideration. If a consumer moves a failed event aside and continues processing later events, the business operation may observe events out of order. Whether that is acceptable depends on the workflow and the event's meaning.

The dual-write problem and the Transactional Outbox

Another reliability problem appears when a service needs to update its database and publish an event.

A naive implementation might do this:

  1. Commit the order to the database.
  2. Publish an OrderPlaced event to Kafka.

But what happens if the database commit succeeds and the application crashes before publishing the event?

The order exists, but downstream services never learn about it.

Reversing the order doesn't solve the problem. If the event is published first and the database transaction later fails, consumers may act on an order that was never committed.

This is known as the dual-write problem: the application is trying to coordinate two independent systems without a shared atomic transaction.

The Transactional Outbox pattern addresses this by writing the business data and an event record into the same database transaction.

For example, the Order Service can:

  • Create the order.
  • Reserve inventory.
  • Insert an OrderPlaced record into an outbox table.

If the transaction commits, both the business changes and the outbox record exist. If it rolls back, neither is committed.

A separate background publisher reads unpublished outbox records and sends them to Kafka. After successful publication, it marks the records as published.

This closes the gap between committing business data and recording the intention to publish an event.

However, it does not guarantee exactly-once delivery. The publisher might successfully send an event to Kafka and crash before marking the outbox record as published. It may send that event again after restarting.

That's why the outbox pattern and idempotent consumers complement each other: the outbox makes event publication recoverable, while consumers make repeated delivery safe.

The main lesson is that reliable event publishing requires more than calling a broker after a database update. We need to design for the possibility that the process stops between any two steps.

4. Concurrency and Scalability: Protecting Shared Resources

Concurrency becomes especially important when multiple requests attempt to modify the same resource at the same time.

Imagine an inventory item has five units available. Two customers place orders simultaneously: one requests four units, and the other requests three.

If both requests read the available quantity before either updates it, both may conclude that enough stock exists. Without appropriate protection, the system can oversell inventory.

Prefer atomic database operations where possible

For a resource already managed by a relational database, an atomic conditional update is often the simplest solution:

UPDATE inventory
SET available_quantity = available_quantity - 4
WHERE product_id = 'P-101'
  AND available_quantity >= 4;

The database coordinates concurrent updates to the same row. If one transaction updates the row first, a competing update must wait as needed, and the condition is evaluated against the current row state. The affected-row count tells the application whether the reservation succeeded.

For multi-product operations, updates should happen in a transaction. Acquiring or updating rows in a consistent order can also help reduce deadlock risk when concurrent transactions touch multiple products.

The important principle is to enforce the invariant as close as possible to the data being protected. A separate application-level read followed by a write leaves a race window.

Database locks and distributed locks

Sometimes an operation involves several steps that need coordinated access to a resource. A database transaction can use row-level locking---for example, SELECT ... FOR UPDATE---to lock selected rows while the transaction performs its work.

Distributed locks are a different tool. They may be useful when multiple application instances need to coordinate access to a resource that is not protected by one database transaction.

Redis is sometimes used for this purpose. A simplified lock acquisition command might look like:

SET lock:payment:order_123 unique-token NX PX 10000

The NX option makes the operation conditional on the key not already existing, while PX sets an expiration time. The unique token identifies the owner that acquired the lock.

When releasing the lock, the application should verify that the stored token still matches its own token before deleting it. Otherwise, an expired lock could have been acquired by another process, and the original process might accidentally delete the newer owner's lock.

Even with an expiration time, distributed locks have subtle failure modes.

Imagine Worker A acquires a lock and then pauses for longer than the lock's time-to-live. The lock expires, and Worker B acquires it. If Worker A resumes, both workers may now believe they can perform the operation.

A fencing-token approach can help protect the resource from stale lock holders. Each lock acquisition receives a monotonically increasing token. The resource being modified must atomically reject operations carrying a token older than the latest accepted token.

The important detail is that the resource itself must enforce the token. Checking it in application code and then performing a separate write can introduce another race.

For operations that can be protected by a normal database transaction or atomic update, that is usually preferable to introducing a distributed lock. Distributed locks add operational and correctness considerations of their own.

Caching and the consistency trade-off

Caching can reduce database load and improve response times, especially for data that is read frequently and changes relatively infrequently.

A common strategy is Cache-Aside:

  1. The application checks the cache.
  2. If the value is present, it returns the cached result.
  3. If the value is missing, it reads from the database.
  4. It stores the result in the cache with a time-to-live (TTL).

This works well for data such as product descriptions or category information.

But cache invalidation introduces a consistency problem. If the database is updated and the cache is not invalidated correctly, users may continue seeing stale data. A TTL limits how long stale data can remain, but it doesn't eliminate the inconsistency window.

Another issue is a cache stampede. If a popular cache entry expires, many requests may attempt to rebuild it simultaneously, creating a sudden burst of database traffic. Techniques such as request coalescing, TTL jitter, or carefully designed locking can reduce this effect.

For inventory reservation, the database should remain the authoritative source of truth. A cached quantity may be useful for display, but it should not be trusted to make the final decision about whether a reservation can succeed.

What I took away from this section is that adding more application instances does not automatically make a system safe under concurrency. We need to protect shared resources explicitly, choose caching strategies with their consistency costs in mind, and use coordination mechanisms only when the problem actually requires them.

5. Putting Everything Together: Designing an Inventory Reservation Workflow

So far, we've looked at several concepts individually: database transactions, event-driven architecture, idempotency, retries, the transactional outbox, and concurrency control.

But these concepts become much more interesting when we combine them to solve an actual engineering problem.

Let's bring back our e-commerce example and design an inventory reservation workflow.

The problem

Imagine a customer places an order containing several products.

Before confirming the order, the system needs to:

  1. Verify that all requested products have sufficient inventory.
  2. Reserve the requested quantities.
  3. Create the order.
  4. Notify other services that the order has been placed.
  5. Continue the payment process.

At first, this might look like a straightforward sequence of API calls. But several failure scenarios make the design more challenging.

What if two customers try to purchase the last available units simultaneously? What if the database transaction succeeds but the application crashes before publishing the event? What if a consumer processes the same event twice? And what happens if the payment service is temporarily unavailable?

Let's walk through one possible design.

Step 1: Receive the order request

The customer submits an order request to the Order Service.

The service validates the request and starts a database transaction. Within that transaction, it attempts to reserve the requested quantities.

For each product, we use a conditional update rather than reading the quantity into the application and checking it separately:

UPDATE inventory
SET available_quantity = available_quantity - 2
WHERE product_id = 'P-101'
  AND available_quantity >= 2;

The application checks the number of affected rows. If any requested product cannot be reserved, the transaction is rolled back.

This is important when an order contains multiple products: we don't want to reserve some items while silently failing to reserve the others.

If every reservation succeeds, the service creates the order and its reservation records.

Step 2: Persist the event using the Transactional Outbox

Once the order and inventory reservation have been created, the system needs to notify other services.

A naive implementation might commit the database transaction and then publish an OrderPlaced event to Kafka.

However, this creates a dual-write problem:

  • The database transaction may commit successfully.
  • The application may crash before publishing the event.
  • The order exists, but downstream services never receive the notification.

To avoid this, the Order Service writes an event to an outbox table as part of the same database transaction.

The transaction now contains three important operations:

  • Update inventory quantities.
  • Create the order and reservation records.
  • Insert an OrderPlaced event into the outbox.

All three either commit together or roll back together.

A separate background publisher reads unpublished outbox records and sends them to Kafka.

If publishing succeeds but the publisher crashes before marking the record as published, the event may be published again. That's why downstream consumers must still be idempotent.

Step 3: Process the event asynchronously

Once the event reaches Kafka, the Payment Service can consume it and begin the payment process.

The Inventory Service or other interested services can also consume relevant events independently, without requiring the Order Service to call each one directly.

Each service has its own consumer group, allowing it to process events independently.

This keeps the order-creation path from depending on every downstream service being available at that exact moment.

For example, a temporary notification-service outage shouldn't prevent a customer from placing an order.

Step 4: Make consumers idempotent

Kafka consumers can receive the same event more than once, so the Payment Service must account for duplicate delivery.

Suppose the Payment Service receives an OrderPlaced event and successfully creates a payment attempt. It then crashes before committing the Kafka offset.

When the service restarts, Kafka may deliver the same event again.

Without protection, the service might create another payment attempt.

A common approach is to use an idempotency key derived from the order ID or payment operation ID.

The service records the operation and its result in the database, enforcing uniqueness with a database constraint. Repeated requests for the same logical operation return or reuse the existing result rather than creating another operation.

The important point is that idempotency should be enforced at the business-operation level, not only by checking whether a particular Kafka event ID has been processed.

Step 5: Handle failures and retries

Not every failure should be handled in the same way.

For example:

  • A temporary database connection issue may justify a retry.
  • A temporary payment-provider timeout may be retried with exponential backoff and jitter.
  • An invalid payment request should not be retried indefinitely.

Retries should be bounded, and messages that repeatedly fail may need to be moved to a dead-letter queue for investigation or later processing.

We also need to think about the order's business state.

An order shouldn't be considered paid just because a payment request was sent. It should move to a paid state only after the payment outcome has been confirmed.

If payment ultimately fails, the system needs a compensating business operation to release the inventory reservation. This is not a rollback of the original database transaction: the reservation may have been committed already, and other services may have acted on the event.

Instead, the system performs a new operation, such as ReleaseInventoryReservation, which must itself be safe to retry and execute only once for that reservation.

The complete flow

Putting everything together, the workflow looks like this:

  1. The Order Service receives the customer's request.
  2. It atomically reserves inventory and creates the order within a database transaction.
  3. It inserts an OrderPlaced event into the outbox in that same transaction.
  4. An outbox publisher sends the event to Kafka.
  5. The Payment Service consumes the event and starts the payment process.
  6. Consumers use idempotency to protect against duplicate delivery.
  7. Transient failures are retried, while permanent failures are handled explicitly.
  8. If payment fails, the system releases the reservation through a separate compensating operation.

This design doesn't eliminate failures. Instead, it makes failures explicit and gives the system a controlled way to recover from them.

What I took away from this exercise

The biggest lesson for me is that reliable backend systems aren't built by choosing a single technology or architectural pattern.

Kafka doesn't solve database consistency. Transactions don't solve communication failures between services. Retries don't automatically make operations safe. And adding more application instances doesn't protect shared resources from concurrent updates.

Each mechanism addresses a particular problem, and the engineering challenge is understanding how they fit together.

More importantly, the design should follow the business requirements. A small application may not need Kafka, distributed locks, or a complex event-driven architecture. A simpler transactional design might be easier to operate and more than sufficient.

The goal isn't to introduce complexity. It's to understand the failure modes that matter and choose the simplest design that handles them correctly.

6. Conclusion: Thinking in Systems, Not Just Services

Looking back at these concepts, one thing has become increasingly clear to me: building reliable backend systems requires a different way of thinking.

It's not enough to know how to build an API, write database queries, or connect services through a message broker. We also need to understand what happens when requests overlap, dependencies become unavailable, messages are delivered more than once, or a process fails halfway through an operation.

Concepts like database transactions, event-driven architecture, idempotency, the transactional outbox, and concurrency control are not just technical patterns to learn. They represent different ways of reasoning about failure, consistency, and correctness.

I've also learned that there is rarely one universally correct architectural solution. Some problems require asynchronous communication and more sophisticated reliability mechanisms, while others can be handled perfectly well with a simple database transaction and a well-designed API.

The important part is understanding the trade-offs behind each decision and knowing when the additional complexity is actually justified.

For me, this journey has been about moving beyond making individual features work and developing a deeper understanding of the systems behind them.

And I think that's one of the most important steps in growing as a backend engineer: not just learning how to build services, but learning how to build systems that continue to behave correctly when things don't go as planned.

No comments yet. Be the first to comment.