Dex at 19:30, 12 July 2026

2026-07-12T19:30:03Z

Dex at 19:29, 12 July 2026

2026-07-12T19:29:31Z

Dex: Created page with "= Not Everything Belongs in a Database = == Understanding Storage Repositories in Enterprise and Cloud Systems == ''By Dex White'' == Introduction == One of the most common architectural mistakes made by developers is assuming that every piece of information should be stored in a database. Need users?
Create a Users table.
Need files?
Create a Files table.
Need logs?
Create a Logs table.
2026-07-12T19:28:47Z

Created page with "= Not Everything Belongs in a Database = == Understanding Storage Repositories in Enterprise and Cloud Systems == ''By Dex White'' == Introduction == One of the most common architectural mistakes made by developers is assuming that every piece of information should be stored in a database. Need users? <blockquote> Create a Users table. </blockquote> Need files? <blockquote> Create a Files table. </blockquote> Need logs? <blockquote> Create a Logs table. </blockq..."

New page
= Not Everything Belongs in a Database =
== Understanding Storage Repositories in Enterprise and Cloud Systems ==

''By Dex White''

== Introduction ==

One of the most common architectural mistakes made by developers is assuming that every piece of information should be stored in a database.

Need users?

<blockquote>
Create a Users table.
</blockquote>

Need files?

<blockquote>
Create a Files table.
</blockquote>

Need logs?

<blockquote>
Create a Logs table.
</blockquote>

Need permissions?

<blockquote>
Add some more tables.
</blockquote>

While this approach may work for smaller applications, enterprise and cloud systems quickly reveal its limitations.

The reality is that modern systems rarely rely on a single storage technology. Instead, they employ multiple repositories, each selected because it excels at a particular task.

A large cloud platform might simultaneously use:

* LDAP for identity management
* SQL databases for business transactions
* Object storage for files and documents
* Search indexes for discovery
* Graph databases for relationships
* Time-series databases for monitoring and telemetry

The question is not:

<blockquote>
Which repository is best?
</blockquote>

The real question is:

<blockquote>
Which repository is best suited to the data being stored?
</blockquote>

== Understanding the Purpose of a Repository ==

Every repository technology was created to solve a particular problem.

Before selecting a storage technology, architects should ask four fundamental questions:

=== 1. What is being stored? ===

Are you storing:

* User identities?
* Customer orders?
* Documents?
* Media files?
* Telemetry?
* Relationships?

Different data types often require different storage technologies.

=== 2. How will it be accessed? ===

Will the system perform:

* Mostly reads?
* Mostly writes?
* Full-text searches?
* Analytics?
* Transactions?

Storage solutions are often optimized for specific access patterns.

=== 3. How large will it become? ===

A solution designed for thousands of records may struggle with billions.

Scale matters.

=== 4. How important is consistency? ===

Does every transaction need to be immediately correct?

Or is eventual consistency acceptable?

The answer significantly influences repository selection.

== The Enterprise View of Data ==

Enterprise architects typically separate data into several categories:

<pre>
Identity Data
Business Data
Content Data
Operational Data
Analytical Data
</pre>

Each category tends to have a natural home.

Attempting to force all five categories into a single database often leads to unnecessary complexity, reduced performance, and maintenance difficulties.

== LDAP Directories ==

=== What LDAP Was Designed To Do ===

LDAP (Lightweight Directory Access Protocol) is frequently misunderstood.

LDAP is not a database replacement.

It is a directory service designed to answer questions about identities.

For example:

<pre>
Who is this user?

Which groups do they belong to?

What permissions do they have?

Can they authenticate?
</pre>

These questions are read-heavy and rarely involve complex transactions.

LDAP was specifically engineered for this type of workload.

=== Strengths of LDAP ===

==== Fast Reads ====

Directory services are heavily optimized for lookup operations.

Finding a user, group, or permission is extremely efficient.

==== Hierarchical Structure ====

Directories naturally represent organisational structures.

<pre>
Company
└── Departments
├── Finance
├── HR
└── IT
</pre>

==== Standards-Based ====

LDAP is widely supported by:

* Active Directory
* OpenLDAP
* FreeIPA
* Enterprise Identity Platforms

==== Centralized Identity ====

Multiple applications can authenticate against a single repository.

This creates a true Single Sign-On foundation.

=== Weaknesses of LDAP ===

LDAP is not ideal for:

* Financial transactions
* Order processing
* Complex reporting
* Relational business data

While these tasks are technically possible, they are not what LDAP was designed for.

=== Best Use Cases ===

LDAP excels at:

* Authentication
* Authorization
* Identity Management
* Group Membership
* Enterprise SSO

== SQL Databases ==

=== What SQL Was Designed To Do ===

Relational databases were designed to model business information.

Examples include:

<pre>
Customers
Orders
Invoices
Projects
Assets
Contracts
Tickets
</pre>

Relationships between data are a core strength of SQL systems.

=== Strengths of SQL ===

==== Data Integrity ====

SQL databases provide:

* Constraints
* Relationships
* Referential integrity

This ensures that business rules remain enforced.

==== Transactions ====

ACID-compliant databases guarantee consistency.

For example:

<pre>
Transfer £100

Debit Account A
Credit Account B

Commit
</pre>

Either both actions succeed or neither does.

==== Query Power ====

SQL remains one of the most powerful data query languages ever created.

Complex reporting often requires only a single query.

=== Weaknesses of SQL ===

SQL databases can become challenging when:

* Scaling globally
* Handling billions of records
* Managing highly variable schemas

Modern cloud platforms frequently supplement relational systems with specialised repositories.

=== Best Use Cases ===

SQL excels at:

* ERP systems
* CRM systems
* E-Commerce
* Workflow systems
* Financial systems
* Business applications

== Why Most Web Applications Store Users in SQL ==

Many developers encounter authentication systems that look something like this:

<pre>
Users
├── Id
├── Username
├── PasswordHash
├── Email
└── Role
</pre>

For many applications this is entirely acceptable.

A system with:

* 100 users
* 1,000 users
* 10,000 users

can operate perfectly well using SQL-based authentication.

The approach is simple, familiar and easy to maintain.

== The Problem Appears Later ==

Imagine an organisation running:

<pre>
CRM
HR
Finance
Projects
Ticketing
Knowledgebase
</pre>

If each application maintains its own user table, the organisation inherits several problems:

* Multiple passwords
* Multiple user accounts
* Multiple role systems
* Multiple account lockout policies
* Multiple password reset processes

Sooner or later someone asks:

<blockquote>
Why can't users just sign in once?
</blockquote>

This question is one of the reasons directory services became so important.

== Why Enterprises Prefer LDAP ==

Enterprise environments typically treat Identity as a platform rather than an application feature.

Instead of every application storing users independently:

<pre>
Application A
Application B
Application C
Application D
</pre>

all applications authenticate against a common directory.

<pre>
Applications
│
▼
Identity Provider
│
▼
LDAP Directory
</pre>

This creates:

* Centralized authentication
* Centralized authorization
* Consistent security policies
* Simplified auditing
* Simplified user management

== Modern Cloud Identity ==

Many people consider LDAP old technology.

In reality, most modern identity systems still follow directory principles.

Examples include:

* Microsoft Entra ID
* Okta
* Auth0
* Ping Identity
* Keycloak

Although the underlying implementation may differ, the conceptual model remains familiar:

<pre>
User
Group
Role
Claim
Permission
Organisation
</pre>

The industry has evolved, but the concepts have endured.

== Beyond LDAP and SQL ==

Modern cloud architectures frequently use several repository technologies simultaneously.

=== Object Storage ===

Best for:

* Documents
* Images
* Video
* Backups

Examples:

* Azure Blob Storage
* Amazon S3
* Google Cloud Storage

=== NoSQL Databases ===

Best for:

* Massive scale
* Flexible schemas
* Distributed applications

Examples:

* MongoDB
* DynamoDB
* Cosmos DB

=== Search Engines ===

Best for:

* Full-text search
* Content discovery
* Indexing

Examples:

* Elasticsearch
* Solr
* OpenSearch

=== Graph Databases ===

Best for:

* Relationship analysis
* Social networks
* Dependency mapping

Examples:

* Neo4j
* Amazon Neptune

=== Time-Series Databases ===

Best for:

* Monitoring
* Metrics
* Telemetry
* IoT

Examples:

* InfluxDB
* Prometheus
* TimescaleDB

== Repository Selection Matrix ==

{| class="wikitable"
! Repository Type
! Best Suited For
|-
| LDAP
| Identity and Authentication
|-
| Active Directory
| Enterprise Identity Management
|-
| SQL Database
| Transactional Business Data
|-
| NoSQL Database
| Massive Scale Distributed Data
|-
| Object Storage
| Files and Documents
|-
| Search Engine
| Full-Text Search
|-
| Graph Database
| Relationships and Dependencies
|-
| Time-Series Database
| Metrics and Telemetry
|}

== Designing for the Future ==

When designing a new platform, it is tempting to choose a single repository and store everything there.

Initially this appears simpler.

However, long-lived systems benefit greatly from separation of concerns.

A good rule of thumb is:

<pre>
Identity Data
↓
Directory Service

Business Data
↓
SQL Database

Files
↓
Object Storage

Logs
↓
Telemetry Platform

Search
↓
Search Engine
</pre>

Each repository performs the task it was designed to do.

== Conclusion ==

Storage repositories are not competitors; they are specialists.

LDAP is not better than SQL.

SQL is not better than object storage.

Object storage is not better than a graph database.

Each exists because different categories of information require different treatment.

The most successful enterprise and cloud architectures recognise this reality and place data where it naturally belongs.

The goal is not to find a universal repository.

The goal is to provide every type of data with the most appropriate home.

In architecture, as in engineering, the right tool for the right job remains one of the most important principles of all.

Not Everything Belongs in a Database - Revision history

Dex at 19:30, 12 July 2026

Dex at 19:29, 12 July 2026