Virtual Hard Disk Management in VMware ESXi

From PiRho Knowledgebase
Jump to navigationJump to search

Summary:

Virtual Hard Disks (VMDKs) are the primary storage mechanism used by VMware virtual machines. Understanding how to create, migrate, convert, expand, troubleshoot, and recover virtual disks is an essential skill for any VMware administrator.

This article explains how VMware virtual hard disks work, common disk formats, migration scenarios, troubleshooting techniques, and practical methods for resolving compatibility issues between VMware products such as VMware Workstation and VMware ESXi.

Context

A virtual machine is only as useful as its storage. While CPU and memory resources are relatively straightforward, virtual storage introduces additional layers of abstraction and complexity.

Administrators commonly encounter virtual disk management tasks when:

  • Migrating virtual machines between VMware platforms.
  • Expanding storage capacity.
  • Recovering damaged virtual machines.
  • Consolidating snapshots.
  • Migrating workloads between datastores.
  • Converting physical servers into virtual machines.

A common misconception is that a VMDK is simply a large disk image. In reality, a VMDK typically consists of a descriptor file and one or more data files which together describe both the virtual disk and its contents.

Core Concepts

What is a VMDK?

A VMware Virtual Machine Disk (VMDK) is a virtual storage device presented to a guest operating system.

A VMDK commonly consists of: 

SERVER01.vmdk
SERVER01-flat.vmdk

The descriptor file: 

SERVER01.vmdk

Contains metadata describing:

  • Disk geometry.
  • Provisioning type.
  • Adapter type.
  • References to the disk data extents.

The flat file: 

SERVER01-flat.vmdk

Contains the actual virtual disk data.

Thin and Thick Provisioning

Thin Provisioning

A thin-provisioned disk consumes physical storage only as data is written.

Example: 

Configured Size : 100 GB
Actual Usage    : 24 GB

Advantages:

  • Efficient utilisation of datastore capacity.
  • Faster deployment.

Disadvantages:

  • Potential datastore overcommitment.
  • Greater requirement for storage monitoring.

Thick Provisioning

A thick-provisioned disk reserves its entire allocation immediately.

Example: 

Configured Size : 100 GB
Actual Usage    : 100 GB

Advantages:

  • Predictable storage consumption.
  • Consistent performance characteristics.

Disadvantages:

  • Higher initial storage requirements.

Virtual Storage Controllers

Virtual disks are presented through a virtual storage controller.

Common controller types include:

  • LSI Logic Parallel
  • LSI Logic SAS
  • VMware Paravirtual (PVSCSI)
  • BusLogic

Changing controller types incorrectly may result in boot failures or inaccessible storage devices within the guest operating system.

Reference Architectures

Small Environment

ESXi Host
│
├── Local Datastore
│   ├── VM01
│   ├── VM02
│   └── VM03
│
└── Backup Repository

Suitable for:

  • Home labs.
  • Training environments.
  • Small businesses.

Enterprise Environment

VMware Cluster
│
├── Shared SAN/NAS Storage
├── Multiple Datastores
├── High Availability
├── Storage vMotion
├── VM Replication
└── Backup Infrastructure

Suitable for:

  • Production workloads.
  • Critical business systems.
  • High-availability environments.

Practical Application

Migrating a Virtual Disk from VMware Workstation to ESXi

A common migration scenario involves moving a virtual machine from VMware Workstation to VMware ESXi.

A recommended approach is:

  1. Shut down the virtual machine cleanly.
  2. Remove or consolidate any snapshots.
  3. Upload the virtual disk files to the target datastore.
  4. Convert the virtual disk into an ESXi-native format.
  5. Attach the converted disk to a newly created virtual machine.

Example: 

vmkfstools -i source.vmdk destination.vmdk -d thin

This operation imports and converts the disk into a native ESXi format.

Expanding a Virtual Disk

Virtual disks can be expanded without recreating the virtual machine.

Example: 

vmkfstools -X 150G SERVER01.vmdk

This increases the virtual disk size to 150 GB.

After expanding the virtual disk, the guest operating system must be instructed to extend its partition or filesystem.

Cloning a Virtual Disk

Virtual disks can be cloned for migration, backup, or testing purposes.

Example: 

vmkfstools -i SERVER01.vmdk SERVER01-CLONE.vmdk -d thin

Common use cases include:

  • Migration between datastores.
  • Converting provisioning formats.
  • Creating test environments.

Common Pitfalls

Copying Only the Descriptor File

A frequent mistake is uploading: 

SERVER01.vmdk

Without: 

SERVER01-flat.vmdk

The descriptor contains metadata only. Without the associated data file, the virtual machine has no usable storage.

Snapshot Chains

Snapshot-enabled virtual machines may contain: 

SERVER01.vmdk
SERVER01-000001.vmdk
SERVER01-000002.vmdk

Copying only part of the snapshot chain can produce:

  • Inconsistent data.
  • Failed virtual machine boots.
  • Disk consolidation errors.

Always consolidate snapshots before migration where possible.

Unsupported Disk Types

A common migration error is: 

Unsupported or invalid disk type 7

This often occurs when a VMware Workstation virtual disk has been copied directly into an ESXi datastore without being imported or converted.

Resolution: 

vmkfstools -i source.vmdk imported.vmdk -d thin

This rewrites the disk into an ESXi-compatible format.

Unsupported Guest Operating Systems

A guest operating system may be newer than the target hypervisor.

Examples include:

  • Windows Server 2016 on ESXi 5.5.
  • Windows Server 2019 on older ESXi releases.

In these situations, virtual disk conversion may succeed while guest operating system compatibility issues remain.

Security Implications

Virtual hard disks frequently contain:

  • User data.
  • Active Directory databases.
  • Certificates.
  • Application configuration.
  • Sensitive business information.

Best practices include:

  • Restricting datastore access.
  • Encrypting backups.
  • Removing temporary migration copies.
  • Following least-privilege administration principles.
  • Performing secure disposal of retired virtual disks.

Troubleshooting & Diagnostics

Verify the Disk Descriptor

Check whether ESXi can successfully interpret the virtual disk descriptor: 

vmkfstools -e SERVER01.vmdk

Successful output confirms that the descriptor is readable.

Examine the VMDK Descriptor

Display the descriptor contents: 

cat SERVER01.vmdk

Look for values such as: 

createType="monolithicSparse"

or 

createType="streamOptimized"

These formats are commonly associated with VMware Workstation and OVF exports.

Verify Datastore Capacity

Check available storage space: 

df -h

Lack of free datastore space is a common cause of failed imports and conversions.

Review VMware Logs

Examine: 

vmware.log

Within the virtual machine directory.

This file frequently contains detailed information regarding:

  • Storage controller issues.
  • Disk compatibility problems.
  • Missing files.
  • Snapshot errors.

Operational Lifecycle

Build

Create VM
   ↓
Create Virtual Disk
   ↓
Install Guest OS

Run

Monitor Storage
   ↓
Backup Data
   ↓
Maintain Capacity

Maintain

Expand Disks
   ↓
Consolidate Snapshots
   ↓
Migrate Storage

Retire

Backup
   ↓
Archive
   ↓
Securely Delete

Design & Architecture Considerations

When designing VMware environments:

  • Prefer thin provisioning for general-purpose workloads unless specific performance requirements dictate otherwise.
  • Monitor datastore utilisation carefully.
  • Avoid leaving snapshots in place for extended periods.
  • Use shared storage for production workloads where possible.
  • Keep VMware product versions reasonably aligned during migration projects.
  • Validate guest operating system compatibility before undertaking large-scale migrations.
  • Test recovery procedures regularly.

Many virtual disk issues originate not from corruption but from:

  • Format incompatibilities.
  • Incomplete snapshot chains.
  • Storage controller mismatches.
  • Unsupported guest operating systems.

Lessons from the Field

One of the most useful troubleshooting techniques is reducing the scope of the problem.

Consider the following scenario: 

VM Fails to Start

Initially, the fault could exist in:

  • The VM configuration.
  • Virtual hardware.
  • Storage controller.
  • Guest operating system.
  • Virtual hard disk.
  • Snapshot chain.

By temporarily removing the virtual hard disk and successfully starting the virtual machine shell, the problem space becomes significantly smaller: 

VM Without Disk = Works
VM With Disk    = Fails

The investigation can then focus entirely on the virtual hard disk rather than the whole virtual machine.

A useful engineering principle is:

Never solve two problems at the same time.

Isolating components often turns a complex migration failure into a straightforward disk conversion exercise.

Related Topics

References

  • VMware ESXi Documentation
  • VMware Workstation Documentation
  • VMware vCenter Converter Documentation
  • VMware Virtual Disk Development Kit (VDDK)
  • VMware VMDK Specification
Retrieved from "https://knowledgebase.pirho.net/index.php?title=Virtual_Hard_Disk_Management_in_VMware_ESXi&oldid=413"