Ran into a situation where an “x” was not available for removing a USB Controller from a Guest.

To resolve, first shut down the guest (if running) and temporarily Unregister it from the host. Then from the datastore, download the .vmx file associated with the guest. Create a backup of the .vmx file, then edit the original with a text editor such as Notepad. Find and delete all of the lines that begin with “usb.”  For example:

usb.present = “TRUE”
usb.pciSlotNumber = “32”
usb:0.present = “TRUE”
usb:0.deviceType = “hid”
usb:0.port = “0”
usb:0.parent = “-1”
usb:1.speed = “2”
usb:1.present = “TRUE”
usb:1.deviceType = “hub”
usb:1.port = “1”
usb:1.parent = “-1”

Once these lines are removed from the .vmx file, save it and then upload the file back to the datastore, overwriting the original.

The final step is to re-register the guest with the host by pointing to the vmx file we just updated on the datastore. You should then find that the USB Controller is gone from the guest configuration.

If you have a fresh install of a mysql server running on Debian (e.g., after running apt-get install mysql-server) you may find that any mysql client connections from other computers to your Debian server fail (one popular example is Kodi / XBMC, etc.). You may also find that mysql connections made from Debian itself (such as a local WordPress database install) are working fine.

One quick test is to use a mysql client on another linux machine to try to connect to the Debian mysql server. From the console of the other linux machine, use the following command (replace the x.x.x.x with the IP address of the Debian server). In this example we are trying to connect with the root user.

mysql -h x.x.x.x -u root

You may then get something like:

ERROR 2003: Can’t connect to MySQL server on ‘x.x.x.x’ (111 “Connection refused”)

By default, mysql server on Debian will bind itself to the loopback address (127.0.0.1). This setting can cause all remote connections to the server to be blocked with Error 111, even though all local connections will work fine (such as a WordPress install noted earlier). As a comparison, you can test local connections by issuing the above command directly from the Debian console, which should yield an error such as:

ERROR 1045: Access denied for user ‘root’@’localhost’ (using password: NO)

This basically confirms local connections are hitting the Debian mysql server. To enable remote connections for Debian’s mysql server, modify the mysql configuration file using these steps:

sudo nano /etc/mysql/my.cnf

1. Find the line that says
   bind-address     127.0.0.1
2. and comment it out by inserting a hash in front of it
   #bind-address     127.0.0.1
3. Save and close the my.cnf file.
4. Then restart the mysql server:

sudo /etc/init.d/mysql restart

Then try connecting to the Debian mysql server from another computer.

After installing open-vm-tools, ESXi reported that VMware Tools were Installed and running on the Debian 8 guest. However any attempt to gracefully power off the guest within ESXi would fail with these errors:

[msg.tools.haltFailed] The request to Power off this virtual machine failed because the corresponding VMware Tools script did not run successfully.

Also present in the logs was this error:

msg.tools.poweronFailed] The VMware Tools power-on script did not run successfully in this virtual machine. If you have configured a custom power-on script in this virtual machine, make sure that it contains no errors.

For this scenario, the solution was to first uninstall open-vm-tools. For Debian / Ubuntu:

sudo apt-get remove open-vm-tools

then install open-vm-tools-desktop

sudo apt-get update
sudo apt-get install open-vm-tools-desktop

You should then be able to power off and power on the guest within ESXi.

Ran into a second error after swapping out a pass-through device in ESXi. This device was the same make and model as the previous device.

Failed - Module 'DevicePowerOn' power on failed.

* Module 'DevicePowerOn' power on failed.
* Missing pciPassthru0.id entry in the configuration file.
* Failed to start the virtual machine.

To resolve it, try these steps:

1. Completely remove the old and/or new pass-through device from the Guest configuration
2. Save the Guest configuration
3. Confirm you are able to successfully boot the Guest without any pass-through device
4. After the Guest starts, shut it down
5. Add the new pass-through device to the Guest configuration.
6. Save the Guest configuration
7. You should now be able to boot it successfully with the new pass-through device

Failed - Module 'DevicePowerOn' power on failed.

* Module 'DevicePowerOn' power on failed.
* The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at 1:0.0.
* Failed to start the virtual machine.

To resolve, first shut down the guest (if running without the pass through device) and temporarily Unregister it from the host. Then from the datastore, download the .vmx file associated with the guest. Create a backup of the .vmx file, then edit the original with a text editor such as Notepad. Find and delete the following lines:

Note: the old pass through device ID of 01:00.0" was specific to my configuration as noted in the original error message, so yours may differ. I’ve also cleared out all unique device information below with xxxxx since it will be based on your specific configuration and is not relevant.

pciPassthru0.id = "01:00.0"
pciPassthru0.deviceId = "xxx"
pciPassthru0.vendorId = "xxx"
pciPassthru0.systemId = "xxxxxxxxxxxxx"

Once these lines are removed from the .vmx file, save it and then upload the file back to the datastore, overwriting the original.

The final step is to re-register the guest with the host by pointing to the vmx file we just updated on the datastore. You should then be able to add the new pass through device and power on the guest.

During the install of FreeNAS, the process may hang with the following errors:

UNMAP failed, disabling BIO_DELETE
UNMAP. CDB ...
CAM status: SCSI Status Error
SCSI status: Check Condition
SCSI sense: ILLEGAL REQUEST ... (Invalid field in parameter list)
Command byte 0 is invalid
Error 22, Unretryable error

This was with both FreeNAS-9.10.1-U2 and FreeNAS-9.10.2-U1 (though it is unlikely that this specific incompatibility is isolated to only these two builds). One thing to try is to first shut down the guest and then reconfigure the VM. For the Hard Disk configuration options of the VM, the default setting for Virtual Device Node is SCSI Controller. Changing it to SATA Controller resolved the error and FreeNAS was able to then install successfully.

I ran into a situation where the host had crashed because of a driver problem (vmw_ahci). After changing the driver and rebooting the host, one of the guest VMs then failed to power on with the following errors:

• File system specific implementation of LookupAndOpen[file] failed
• Object type requires hosted I/O
• Cannot open the disk ‘…vmdk’ or one of the snapshot disks it depends on.
• Module ‘Disk’ power on failed.
• Failed to start the virtual machine.

The solution was to repair the disk using vmkfstools. This particular VM was split into two files, vmname.vmdk and vmname-s001.vmdk. You will want to perform the commands on the primary (pointer) vmdk (not vmname-s###.vmdk).
From the ESXi console, issue the following command substituting your actual path:

vmkfstools -x check /path/to/vmname.vmdk

In this case vmkfstools reported that the disk should be repaired. It would be prudent at this point to back up your vmdk files before trying the repair. Then initiate the repair on the primary vmdk with:

vmkfstools -x repair /path/to/vmname.vmdk

Once you get a notification that the repair has completed, try to start your VM again.

Upgrading from ESXi 5.5 to 6.5, performance of the host was relatively poor on a JetWay JNF9G-QM77. Guests were extremely sluggish.

There were also several errors  tracked in the various logs related to the boot drive and the datastore associated with the boot drive.

In the vobd log:

[vmfsCorrelator]: [esx.problem.vmfs.heartbeat.timedout] datastore1
[vmfsCorrelator]: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume (datastore1): [Timeout] 
[vmfsCorrelator]: [esx.problem.vmfs.heartbeat.recovered] datastore1

In the vmkwarning log:

WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "" state in doubt; requested fast path state update...

In the vmauthd log:

ScsiDeviceIO ... CmdSN 0x4305 from world 69114 to dev "" failed ... Invalid sense data ...

The solution was to disable the new AHCI driver of ESXi 6.5 (vmw_ahci). From the ESXi console issue the following command:

esxcli system module set --enabled=false --module=vmw_ahci

Then after a reboot ESXi should be using the sata-ahci driver. You can confirm this by checking the list of storage adapters for the host, looking for one titled Panther Point AHCI Controller.
After this driver swap, performance of the host should be restored.

If you are having trouble getting ESXi to discover a Synology iSCSI target with CHAP authentication, check your vobd log for errors that may look similar to the below:

[iscsiCorrelator]: [vob.iscsi.discovery.login.error] discovery failure on vmhba64 to x.x.x.x because the target returned a login status of 0201.

[esx.problem.storage.iscsi.discovery.login.error] iSCSI discovery to x.x.x.x on vmhba64 failed. The Discovery target returned a login error of: 0201.

This basically confirms some kind of problem negotiating the CHAP authentication. After double checking that you are using the correct CHAP credentials and you are still not able to get the ESXi host to discover the Synology target, try the steps below:

1. In the Synology Storage Manager for iSCSI Target, first disable Mutual CHAP (if enabled) and check that CHAP is enabled.
   Then in ESXi Configure iSCSI as follows:
2. Set only a Dynamic Target using the IP address and port of the Synology target. Remove any Static Target that you may have set.
3. Temporarily set CHAP Authentication to “Do not use CHAP”
4. Set Mutual CHAP Authentication to “Do not use CHAP”
5. Click Save Configuration
6. Click Refresh on the Devices tab and confirm the Synology is not being discovered
7. Now go back the iSCSI configuration in ESXi
8. Set CHAP Authentication back to “Use CHAP”
9. Click the down arrow and enter your CHAP credentials
10. Confirm the Static Target is now populated with the Synology target address information. If not, try populating it now.
11. Click Save Configuration
12. Click Refresh on the Devices tab

The Synology target should then show up in ESXi.

During the upgrade process from 5.5 to 6.5 (using the ISO method), I ran into the following error:

CONFLICTING_VIBS ERROR: Vibs on the host are conflicting with vibs in metadata. 
Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO 
providing newer versions of the conflicting vibs.

The vib listed as the problem was HUAWEI_bootbank_hio_2.0.0.42-1OEM.550.0.1331820

The next step was to cancel the upgrade process and reboot back into 5.5. Once you are back into ESXi, bring up the console and list all installed vibs by using the following command (pipe in the more command to break the list into pages):

esxcli software vib list | more

In this particular case the problematic vib was called hio

To remove it, initiate the following command:

esxcli software vib remove --vibname hio

Once it completes, reboot for the removal to cleanly take effect. Then reboot again and boot off your ISO installer to restart the ESXi upgrade process.