During the install of FreeNAS, the process may hang with the following errors:

UNMAP failed, disabling BIO_DELETE
UNMAP. CDB ...
CAM status: SCSI Status Error
SCSI status: Check Condition
SCSI sense: ILLEGAL REQUEST ... (Invalid field in parameter list)
Command byte 0 is invalid
Error 22, Unretryable error

This was with both FreeNAS-9.10.1-U2 and FreeNAS-9.10.2-U1 (though it is unlikely that this specific incompatibility is isolated to only these two builds). One thing to try is to first shut down the guest and then reconfigure the VM. For the Hard Disk configuration options of the VM, the default setting for Virtual Device Node is SCSI Controller. Changing it to SATA Controller resolved the error and FreeNAS was able to then install successfully.

I ran into a situation where the host had crashed because of a driver problem (vmw_ahci). After changing the driver and rebooting the host, one of the guest VMs then failed to power on with the following errors:

• File system specific implementation of LookupAndOpen[file] failed
• Object type requires hosted I/O
• Cannot open the disk ‘…vmdk’ or one of the snapshot disks it depends on.
• Module ‘Disk’ power on failed.
• Failed to start the virtual machine.

The solution was to repair the disk using vmkfstools. This particular VM was split into two files, vmname.vmdk and vmname-s001.vmdk. You will want to perform the commands on the primary (pointer) vmdk (not vmname-s###.vmdk).
From the ESXi console, issue the following command substituting your actual path:

vmkfstools -x check /path/to/vmname.vmdk

In this case vmkfstools reported that the disk should be repaired. It would be prudent at this point to back up your vmdk files before trying the repair. Then initiate the repair on the primary vmdk with:

vmkfstools -x repair /path/to/vmname.vmdk

Once you get a notification that the repair has completed, try to start your VM again.

Upgrading from ESXi 5.5 to 6.5, performance of the host was relatively poor on a JetWay JNF9G-QM77. Guests were extremely sluggish.

There were also several errors  tracked in the various logs related to the boot drive and the datastore associated with the boot drive.

In the vobd log:

[vmfsCorrelator]: [esx.problem.vmfs.heartbeat.timedout] datastore1
[vmfsCorrelator]: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume (datastore1): [Timeout] 
[vmfsCorrelator]: [esx.problem.vmfs.heartbeat.recovered] datastore1

In the vmkwarning log:

WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "" state in doubt; requested fast path state update...

In the vmauthd log:

ScsiDeviceIO ... CmdSN 0x4305 from world 69114 to dev "" failed ... Invalid sense data ...

The solution was to disable the new AHCI driver of ESXi 6.5 (vmw_ahci). From the ESXi console issue the following command:

esxcli system module set --enabled=false --module=vmw_ahci

Then after a reboot ESXi should be using the sata-ahci driver. You can confirm this by checking the list of storage adapters for the host, looking for one titled Panther Point AHCI Controller.
After this driver swap, performance of the host should be restored.

If you are having trouble getting ESXi to discover a Synology iSCSI target with CHAP authentication, check your vobd log for errors that may look similar to the below:

[iscsiCorrelator]: [vob.iscsi.discovery.login.error] discovery failure on vmhba64 to x.x.x.x because the target returned a login status of 0201.

[esx.problem.storage.iscsi.discovery.login.error] iSCSI discovery to x.x.x.x on vmhba64 failed. The Discovery target returned a login error of: 0201.

This basically confirms some kind of problem negotiating the CHAP authentication. After double checking that you are using the correct CHAP credentials and you are still not able to get the ESXi host to discover the Synology target, try the steps below:

1. In the Synology Storage Manager for iSCSI Target, first disable Mutual CHAP (if enabled) and check that CHAP is enabled.
   Then in ESXi Configure iSCSI as follows:
2. Set only a Dynamic Target using the IP address and port of the Synology target. Remove any Static Target that you may have set.
3. Temporarily set CHAP Authentication to “Do not use CHAP”
4. Set Mutual CHAP Authentication to “Do not use CHAP”
5. Click Save Configuration
6. Click Refresh on the Devices tab and confirm the Synology is not being discovered
7. Now go back the iSCSI configuration in ESXi
8. Set CHAP Authentication back to “Use CHAP”
9. Click the down arrow and enter your CHAP credentials
10. Confirm the Static Target is now populated with the Synology target address information. If not, try populating it now.
11. Click Save Configuration
12. Click Refresh on the Devices tab

The Synology target should then show up in ESXi.

During the upgrade process from 5.5 to 6.5 (using the ISO method), I ran into the following error:

CONFLICTING_VIBS ERROR: Vibs on the host are conflicting with vibs in metadata. 
Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO 
providing newer versions of the conflicting vibs.

The vib listed as the problem was HUAWEI_bootbank_hio_2.0.0.42-1OEM.550.0.1331820

The next step was to cancel the upgrade process and reboot back into 5.5. Once you are back into ESXi, bring up the console and list all installed vibs by using the following command (pipe in the more command to break the list into pages):

esxcli software vib list | more

In this particular case the problematic vib was called hio

To remove it, initiate the following command:

esxcli software vib remove --vibname hio

Once it completes, reboot for the removal to cleanly take effect. Then reboot again and boot off your ISO installer to restart the ESXi upgrade process.

In the process of flushing yum and trying to get CentOS to update, the system froze and I had to do a hard reset. Upon rebooting, the OS failed to load with several error messages:

XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1757 of file fs/xfs/xfs_alloc.c.
...
Corruption of in-memory data detected. Shutting down filesystem
Please unmount the filesystem and rectify the problem(s)
Failed to recover EFIs
task mount:365 blocked for more than 120 seconds.

Looks like some file system corruption and the last error about the system trying to perform a mount just kept repeating. Running xfs_repair in the console did nothing in this environment.

Next I tried booting from a CentOS install ISO and I chose the option to Rescue an existing installation. Choosing any of the 3 options in rescue mode would just result it the system hanging indefinitely without ever reaching a console. I chose Option 3 to skip to console and alt tabbing revealed:

Running ... mount -t xfs -o defaults,ro /dev/mapper/centos-root /mnt/sysimage

Now it is clear that the system is hanging on trying to mount the root partition which we know has problems. However I could not find a way to enter the console without rescue mode automatically trying to mount the root partition.

Next I tried booting a CentOS live ISO. Turns out it has no xfs tools so that was a dead end.

I decided to try booting a SystemRescueCd which has some xfs tools on it.

I chose the default boot option and reached a console. Ideally one should create a dump of the damaged partition by running xfs_metadump, restore the dump to an image using xfs_mdrestore, and then perform the repair on that image. That way you can evaluate the repair done to the image to see if it will work or irreversibly damage your actual data.

I went ahead and just did the repair on the actual data:

xfs_repair -L /dev/mapper/centos-root

The repair completed and after a reboot CentOS started successfully.

When sending a test email, it was hanging on the authentication method:

Auth method requested: XOAUTH2
Auth methods available on the server: 
LOGIN,PLAIN,XOAUTH2,PLAIN-CLIENTTOKEN,OAUTHBEARER,XOAUTH

Installing curl resolved it. For Debian and Ubuntu:

apt-get install curl libcurl3 php5-curl