Virtual DOS machine (VDM) is Microsoft's technology that allows running legacy DOS and 16-bit Windows programs on Intel 80386 or higher computers when there is already another operating system running and controlling the hardware.
Virtual DOS machines rely on the virtual 8086 mode of the Intel 80386 processor, which allows real mode 8086 software to run in a controlled environment by catching and forwarding to the normal operating system (as exceptions) all operations which involve accessing hardware. The operating system can then perform an emulation and resume the execution of the DOS software.
VDMs generally also implement support for running 16- and 32-bit protected mode software (DOS extenders), which has to conform to the DOS Protected Mode Interface.
When a DOS program running inside a VDM needs to access a peripheral, Windows will either allow this directly (rarely), or will present the DOS program with a Virtual Device Driver (VxD in short) which emulates the hardware using operating system functions. A VDM will systematically have emulations for the Intel 8259A interrupt controllers, the 8254 timer chips, the 8237 DMA, etc.
In general, the VDMs and similar technologies do not satisfactorily run many older DOS programs on today's computers. Emulation is only provided for the most basic peripherals, although Windows 2000 and Windows XP added emulation of the Sound Blaster and other multimedia devices. Emulation of supported peripherals is incomplete and quirky. NT-family versions of Windows only update the real screen a few times per second when a DOS program writes to it, and do not emulate higher resolution graphics modes. Because software runs mostly native, all timing loops will expire prematurely. This either makes a game run much too fast or causes the software not even to notice the emulated hardware peripherals, because it does not wait long enough for an answer.
VDMs appeared with Windows/386 2.1 and are present in all subsequent 32-bit versions of Windows. In the Windows NT family, they are, however, relegated to running DOS and Windows 3.x programs and no longer participate in the implementation of the Windows API. The Windows NT executable which is used to handle a single DOS (and Windows 3.x) environment is called ntvdm.exe.
VDMs were also used in OS/2 2.0 and later.
Versions of Windows NT for 64-bit architectures (x64 and IA-64) do not include the NTVDM and are unable to run DOS or 16-bit Windows applications. This is because, in an x86-64 CPU, virtual 8086 mode is available as a sub-mode only in its legacy mode (for running 16- and 32-bit operating systems), not in the native, 64-bit long mode; a hard reset of the CPU is required to switch to legacy mode.
Wine also includes a VDM, which it utilizes for running Win16 and DOS applications.[1] Wine's VDM is compatible with both 32-bit and 64-bit operating systems; however, compatibility is more limited on 64-bit systems.
In January 2010, Google security researcher Tavis Ormandy revealed a serious security flaw in Windows NT's VDM implementation that allows unprivileged users to escalate their privileges to SYSTEM level, affecting all versions of the Windows NT kernel since 1993. This includes all 32-bit versions of Windows NT, 2000, XP, Server 2003, Vista, Server 2008, and Windows 7. Ormandy did publish a proof-of-concept exploit for the vulnerability. The workaround for this issue is to turn off 16-bit application support, which prevents older programs (those written for DOS and Windows 3.1) from running. 64-bit versions of Windows are not affected since they do not include the vulnerable feature.[2][3]