Not Everything Belongs in a Database

Understanding Storage Repositories in Enterprise and Cloud Systems

One of the most common architectural mistakes made by developers is assuming that every piece of information should be stored in a database.

Need users?

Create a Users table.

Need files?

Create a Files table.

Need logs?

Create a Logs table.

Need permissions?

Add some more tables.

While this approach may work for smaller applications, enterprise and cloud systems quickly reveal its limitations.

The reality is that modern systems rarely rely on a single storage technology. Instead, they employ multiple repositories, each selected because it excels at a particular task.

A large cloud platform might simultaneously use:

• LDAP for identity management
• SQL databases for business transactions
• Object storage for files and documents
• Search indexes for discovery
• Graph databases for relationships
• Time-series databases for monitoring and telemetry

The question is not:

Which repository is best?

The real question is:

Which repository is best suited to the data being stored?

Understanding the Purpose of a Repository

Every repository technology was created to solve a particular problem.

Before selecting a storage technology, architects should ask four fundamental questions:

1. What is being stored?

Are you storing:

• User identities?
• Customer orders?
• Documents?
• Media files?
• Telemetry?
• Relationships?

Different data types often require different storage technologies.

2. How will it be accessed?

Will the system perform:

• Mostly reads?
• Mostly writes?
• Full-text searches?
• Analytics?
• Transactions?

Storage solutions are often optimized for specific access patterns.

3. How large will it become?

A solution designed for thousands of records may struggle with billions.

Scale matters.

4. How important is consistency?

Does every transaction need to be immediately correct?

Or is eventual consistency acceptable?

The answer significantly influences repository selection.

The Enterprise View of Data

Enterprise architects typically separate data into several categories:

Identity Data
Business Data
Content Data
Operational Data
Analytical Data

Each category tends to have a natural home.

Attempting to force all five categories into a single database often leads to unnecessary complexity, reduced performance, and maintenance difficulties.

LDAP Directories

What LDAP Was Designed To Do

LDAP (Lightweight Directory Access Protocol) is frequently misunderstood.

LDAP is not a database replacement.

It is a directory service designed to answer questions about identities.

For example:

Who is this user?

Which groups do they belong to?

What permissions do they have?

Can they authenticate?

These questions are read-heavy and rarely involve complex transactions.

LDAP was specifically engineered for this type of workload.

Strengths of LDAP

Fast Reads

Directory services are heavily optimized for lookup operations.

Finding a user, group, or permission is extremely efficient.

Hierarchical Structure

Directories naturally represent organisational structures.

Company
└── Departments
    ├── Finance
    ├── HR
    └── IT

Standards-Based

LDAP is widely supported by:

• Active Directory
• OpenLDAP
• FreeIPA
• Enterprise Identity Platforms

Centralized Identity

Multiple applications can authenticate against a single repository.

This creates a true Single Sign-On foundation.

Weaknesses of LDAP

LDAP is not ideal for:

• Financial transactions
• Order processing
• Complex reporting
• Relational business data

While these tasks are technically possible, they are not what LDAP was designed for.

Best Use Cases

LDAP excels at:

• Authentication
• Authorization
• Identity Management
• Group Membership
• Enterprise SSO

SQL Databases

What SQL Was Designed To Do

Relational databases were designed to model business information.

Examples include:

Customers
Orders
Invoices
Projects
Assets
Contracts
Tickets

Relationships between data are a core strength of SQL systems.

Strengths of SQL

Data Integrity

SQL databases provide:

• Constraints
• Relationships
• Referential integrity

This ensures that business rules remain enforced.

Transactions

ACID-compliant databases guarantee consistency.

For example:

Transfer £100

Debit Account A
Credit Account B

Commit

Either both actions succeed or neither does.

Query Power

SQL remains one of the most powerful data query languages ever created.

Complex reporting often requires only a single query.

Weaknesses of SQL

SQL databases can become challenging when:

• Scaling globally
• Handling billions of records
• Managing highly variable schemas

Modern cloud platforms frequently supplement relational systems with specialised repositories.

Best Use Cases

SQL excels at:

• ERP systems
• CRM systems
• E-Commerce
• Workflow systems
• Financial systems
• Business applications

Why Most Web Applications Store Users in SQL

Many developers encounter authentication systems that look something like this:

Users
├── Id
├── Username
├── PasswordHash
├── Email
└── Role

For many applications this is entirely acceptable.

A system with:

• 100 users
• 1,000 users
• 10,000 users

can operate perfectly well using SQL-based authentication.

The approach is simple, familiar and easy to maintain.

The Problem Appears Later

Imagine an organisation running:

CRM
HR
Finance
Projects
Ticketing
Knowledgebase

If each application maintains its own user table, the organisation inherits several problems:

• Multiple passwords
• Multiple user accounts
• Multiple role systems
• Multiple account lockout policies
• Multiple password reset processes

Sooner or later someone asks:

Why can't users just sign in once?

This question is one of the reasons directory services became so important.

Why Enterprises Prefer LDAP

Enterprise environments typically treat Identity as a platform rather than an application feature.

Instead of every application storing users independently:

Application A
Application B
Application C
Application D

all applications authenticate against a common directory.

Applications
       │
       ▼
Identity Provider
       │
       ▼
LDAP Directory

This creates:

• Centralized authentication
• Centralized authorization
• Consistent security policies
• Simplified auditing
• Simplified user management

Modern Cloud Identity

Many people consider LDAP old technology.

In reality, most modern identity systems still follow directory principles.

Examples include:

• Microsoft Entra ID
• Okta
• Auth0
• Ping Identity
• Keycloak

Although the underlying implementation may differ, the conceptual model remains familiar:

User
Group
Role
Claim
Permission
Organisation

The industry has evolved, but the concepts have endured.

Beyond LDAP and SQL

Modern cloud architectures frequently use several repository technologies simultaneously.

Object Storage

Best for:

• Documents
• Images
• Video
• Backups

Examples:

• Azure Blob Storage
• Amazon S3
• Google Cloud Storage

NoSQL Databases

Best for:

• Massive scale
• Flexible schemas
• Distributed applications

Examples:

• MongoDB
• DynamoDB
• Cosmos DB

Search Engines

Best for:

• Full-text search
• Content discovery
• Indexing

Examples:

• Elasticsearch
• Solr
• OpenSearch

Graph Databases

Best for:

• Relationship analysis
• Social networks
• Dependency mapping

Examples:

• Neo4j
• Amazon Neptune

Time-Series Databases

Best for:

• Monitoring
• Metrics
• Telemetry
• IoT

Examples:

• InfluxDB
• Prometheus
• TimescaleDB

Repository Selection Matrix

Repository Type	Best Suited For
LDAP	Identity and Authentication
Active Directory	Enterprise Identity Management
SQL Database	Transactional Business Data
NoSQL Database	Massive Scale Distributed Data
Object Storage	Files and Documents
Search Engine	Full-Text Search
Graph Database	Relationships and Dependencies
Time-Series Database	Metrics and Telemetry

Designing for the Future

When designing a new platform, it is tempting to choose a single repository and store everything there.

Initially this appears simpler.

However, long-lived systems benefit greatly from separation of concerns.

A good rule of thumb is:

Identity Data
    ↓
Directory Service

Business Data
    ↓
SQL Database

Files
    ↓
Object Storage

Logs
    ↓
Telemetry Platform

Search
    ↓
Search Engine

Each repository performs the task it was designed to do.

Conclusion

Storage repositories are not competitors; they are specialists.

LDAP is not better than SQL.

SQL is not better than object storage.

Object storage is not better than a graph database.

Each exists because different categories of information require different treatment.

The most successful enterprise and cloud architectures recognise this reality and place data where it naturally belongs.

The goal is not to find a universal repository.

The goal is to provide every type of data with the most appropriate home.

In architecture, as in engineering, the right tool for the right job remains one of the most important principles of all.