Install SCO OpenServer under VMware Workstation

By Lytle David Smith
I have been a SCO reseller since 1988, and I still have several clients running SCO OpenServer. I don't have a computer dedicated to running OpenServer, but it can frequently be very convenient to have access to an OpenServer system for testing. And it is not always acceptable to remotely access a client's system, because the desired testing may require rebooting or other actions that would prove disruptive to users of the system.

I have been using VMware Workstation version 4 for running Windows 98 and various distributions of Linux, so it was natural to want to do the same with OpenServer. However, I soon realized that VMware does not support OpenServer. The reason apparently has to do with how OpenServer interfaces with the hardware.
I did manage to find a brief outline of a procedure for installing OpenServer under VMware. As I am writing this, it is still available at http://www.chinaunix.net/bbsjh/4/11249.html (if you can read Chinese).
After several failed attempts, I managed to get OpenServer installed and running quite successfully under VMware. I have been using it for some time. It is very stable, and has proven very useful to me.
I wanted to fully document the procedure for installing OpenServer under VMware, for myself and for the public at large, so I went through the entire procedure and documented every step along the way. Here is the product of that process.

• Start VMware.
• Create a new virtual machine by selecting "File / New / New Virtual Machine...".
• Skip through intro to "New Virtual Machine Wizard" by clicking "Next".
• Specify a "Virtual machine configuration" of "Custom", and click "Next".
• Specify a "Guest operation system" of "Other", and click "Next".
• Specify a "Virtual machine name" that suits your taste, accept the default "Location", and click "Next".

• Specify a "Memory: Guest size (MB)" of at least "48", and click "Next".
  (I tried 32MB and the installation failed with the error "/ramFs/extractProd/extractAll terminated by signal 8".)
• Specify a "Network connection" that suits your requirements, and click "Next".
  (I prefer "Bridged networking", which makes your virtual machine appear as if it is just another system on your existing network.)
• Specify to "Create a new virtual disk", and click "Next".
• Specify a "Disk size (GB)" of at least "1", specify "Allocate all disk space now" and "Split disk into 2 GB files" as appropriate for your requirements, and click "Next".
• Specify a unique "Disk file" name, click "Advanced >>", specify a "Virtual device node" of "SCSI 0:0", and click "Finish".
• Click "Edit virtual machine settings".
• Click the "Audio" device, and click "Remove".
• Click the "CD-ROM 1 (IDE 1:0)" device, specify a "Connection" that points to the CD-ROM drive from which you will be installing, and specify a "Virtual device node" of "IDE 0:1".

• Start adding a device by clicking "Add...".
• Skip through intro to "Add Hardware Wizard" by clicking "Next".
• Specify a "Hardware type" of "Hard Disk", and click "Next".
• Specify to "Create a new virtual disk", and click "Next".
• Specify a "Disk size (GB)" of only "0.1", leave "Allocate all disk space now" and "Split disk into 2 GB files" unchecked, and click "Next".
• Specify a unique "Disk file" name, click "Advanced >>", specify a "Virtual device node" of "IDE 0:0", and click "Finish".Note regarding driver for BusLogic MultiMaster BT-958 SCSI host adapter:
  VMware virtual machines emulate the BusLogic MultiMaster BT-958 SCSI host adapter. OpenServer includes drivers for this host adapter, but for some reason installing OpenServer under VMware without using the BusLogic MultiMaster BT-958 BTLD (which is becoming increasingly difficult to find on the Internet) will eventually result in a "no root disk controller found" error.

• Download the BusLogic MultiMaster BT-958 BTLD diskette creator.
• Start a command-line session by clicking "Start / Run...", typing "cmd", and pressing [Enter].
• Make the drive where "mmunix.exe" resides the current drive by typing "{drive}:" (where {drive} is the drive letter for the drive where mmunix.exe resides) and pressing [Enter].
• Make the directory where "mmunix.exe" resides the current directory by typing "CD {path}" (where {path} is the path where mmunix.exe resides) and pressing [Enter].
• Extract the "BusLogic Multimaster BT-958 BTLD" diskette creator by typing mmunix and pressing [Enter].
• Create the "BusLogic Multimaster BT-958 BTLD" diskette by typing "unix a:", pressing [Enter], and confirming that you do wish to continue.
• Close the command-line session by typing "exit" and pressing [Enter].
• Insert the "SCO OpenServer CD-ROM" into the CD-ROM drive, and insert the "BusLogic Multimaster BT-958 BTLD" diskette into the diskette drive.
• Return to VMware and start the new virtual machine by selecting "Power / Power On" (or by pressing [Ctrl-B]).
• Enter the virtual machine's BIOS setup utility by quickly clicking inside the virtual machine window and pressing [F2].
• Select the "Boot" settings page by pressing [Right] three times.
• Use [Up], [Down], [+] and [-] to rearrange the boot order so that "CD-ROM Drive" is first and "Hard Drive" is second.
• Use [Up] or [Down] to select the "Hard Drive" and press [Enter] to expand the selection.
• Use [Up] or [Down] to select the "VMware Virtual IDE Hard-(PM)", then use [-] to rearrange the boot order so that it is last in the boot order.
• Save the BIOS settings and exit the BIOS setup utility by pressing [F10] and confirming that you wish to "Save configuration changes and exit now?".

Note regarding floppy drives and VMware:
For some unknown reason, VMware seems to have frequent problems connecting to the diskette drive when booting a virtual machine. If you receive a warning stating that VMware could not connect to the floppy drive, click "OK" to dispense with the warning(s). Then, click the "Reset" toolbar icon (or press [Ctrl-R]) to recycle power to the virtual machine. After a few tries, you should eventually be able to start the virtual machine with the floppy drive connected.

• At the "Boot:" prompt, type "restart link=blc Sdsk=blc(0,0,0,0) Srom=wd(0,0,0,0)" and press [Enter].
• At the "Please insert the fd(65)blc volume and press , or 'q' to quit:" prompt, press [Enter] (and then be patient).
• At the "Please enter ..." prompt, type "a" and press [Enter].
• At the "Press to begin installation" prompt, press [Enter].
• At the "Please insert (blc) package BTLD disk ..." prompt, press [Enter].
• At the "Installing SCO OpenServer(TM) software ..." prompt, press [Enter] to "Continue" and press [Enter] again to "Accept".
• At the "Media device to be used:" prompt, specify "IDE CD ROM".
• At the "IDE Controller:" prompt, specify "primary".
• At the "Master or Slave:" prompt, specify "slave".
• Press [Enter] until you confirm the "Accept above choices" prompt.
• At the "Insert your installation CD-ROM ..." prompt, press [Enter].
• At the "Keyboard:" prompt, specify "US English" (or whatever is appropriate for your circumstances), and press [Enter] until you confirm the "Accept above choices" prompt.
• At the "License number:" prompt, enter a valid license number and press [Enter].
• At the "License code:" prompt, enter a valid license code and press [Enter].
• Press [Enter] until you confirm the "Accept above choices" prompt.
• At the "Suitability for automatic Upgrade" prompt, choose "Fresh" and press [Enter].
• At the "Select OK to continue." prompt, choose "OK" and press [Enter].
• On the "Configuring the basic system" screen, complete the choices as appropriate and press [Enter] until you confirm the "Accept above choices" prompt.
• On the "Initial system profile" screen, complete the choices as appropriate and press [Enter] until you confirm the "Accept above choices" prompt.
• On the "Preparing your disk and choosing software" screen, specify a "Hard disk setup:" of "UNIX only : Bad Blocking OFF".
• At the "Optional software:" prompt, specify whatever optional software is appropriate for your circumstances and press [Enter] until you confirm the "Accept above choices" prompt.
• On the "Configuring optional software" screen, specify a "Network card:" of "Autodetect", which should result in "AMD PCNet- PCI" being detected.
• Still on the "Configuring optional software" screen, "Video and graphics:" should already specify "VESA SVGA".
• Still on the "Configuring optional software" screen, specify whatever remaining settings are appropriate for your circumstances and press [Enter] until you confirm the "Accept above choices" prompt.
• On the "Setting your root password" screen, specify and confirm your root password and press [Enter] until you confirm the "Accept above choices" prompt.
• At the "The installation can now proceed unattended." prompt, choose "OK" and press [Enter].
• At the "Please select the floppy device you are using:" prompt, choose "(2) /dev/fd1".
  (I know this is the wrong floppy device, but there is a method to the madness, so stick with me here. We already loaded the blc driver when we booted, so the installation can complete without reloading it. After the installation, at boot-time, we will load the driver once again, and then manually install the blc driver to the root filesystem using btldinstall.)
• At the "Please insert the floppy for BTLD package: blc and press :" prompt, press [Enter].
• At the "mount: Bad address, Could not mount the BTLD floppy ..." prompt, choose "(b) Abort BTLD load, but continue installation".
• At this point, the installation will continue unattended. (Be patient.)
• At the "Press to continue:" prompt, press [Enter].
• At the "Press to continue" prompt, press [Enter].
• At the "Safe to Power Off ..." prompt, press any key.
• Leave the "SCO OpenServer CD-ROM" and the "BusLogic MultiMaster BT-958 BTLD" diskette in place.
• See note above about floppy drives and VMware, and ensure that the diskette drive is connected.
• At the "Boot:" prompt, type "fd(64)unix link=blc root=hd(42) swap=hd(41)" and press [Enter].
• At the "Please insert the fd(65)blc volume and press , or 'q' to quit:" prompt, press [Enter] (and then be patient).
• At the "Please enter ..." prompt, type "a" and press [Enter].
• At the "Type CONTROL-d to proceed ..." prompt, enter your chosen root password and press [Enter].
• At the "#" prompt, type "mount /dev/fd0135ds18 /mnt" and press [Enter].
• At the "#" prompt, type "btldinstall /mnt" and press [Enter].
• At the "Please enter the names of the packages you wish to install, or q to quit" prompt, type "blc" and press [Enter].
• At the "There is already a blc installed. ..." prompt, choose "y" and press [Enter].
• At the "A previous installation of BusLogic driver, 'blc', has been detected. ..." prompt, choose "n" and press [Enter].
• At the "Are you installing on a PCI-compliant host adapter? (y/n):" prompt, choose "y" and press [Enter].
• At the "Do you want to enable Tag Queuing now? (y/n):" prompt, choose "y" and press [Enter].
• At the "Do you want to rebuild the kernel now? (y/n):" prompt, choose "y" and press [Enter].
• At the "Do you want this kernel to boot by default? (y/n):" prompt, choose "y" and press [Enter].
• At the "Do you want the kernel environment rebuilt? (y/n):" prompt, choose "y" and press [Enter].
• At the "#" prompt, type "scologin disable" and press [Enter].

Note regarding running the SCO GUI under VMware:
The SCO GUI does not appear to work well (if at all) with VMware. This is probably largely because, since VMware does not support OpenServer, (unlike Linux and Windows) there is no accelerated video driver provide for it. If you require the GUI, then consider using VNC to access the GUI remotely from your host operating system. I have complete instructions and installation scripts for use installing TightVNC onto OpenServer at http://David.ComputerLandInc.com