ESXi local datastores missing after ESXi upgrade 6.7 to 7.0

After upgrading to 7.0 the system datastore was missing (attached via SATA).  Related to this article https://jc-lan.org/2017/02/01/poor-performance-in-esxi-6-5-with-jetway-jnf9g-qm77/ I had previously disabled the built in AHCI driver of ESXi due to poor performance.  Enabling it again followed by a reboot of the host resolves the issue:

You can verify if any modules are disabled with the following command (in this case we are looking for vmw_ahci)

esxcli system module list | grep false

Re-enable it:

esxcli system module set --enabled=true --module=vmw_ahci

Then reboot

Posted in Uncategorized | Leave a comment

Upgrade from ESXi 6.7 to 7.0 via command line (ssh)

This guide is for upgrading ESXi via the command line.

For this guide I upgraded from 6.7.0 Update 3 (Build 16075168) to 7.0.0 (Build 15843807)

Steps:

Set firewall

esxcli network firewall ruleset set -e true -r httpClient

Search for the GA release of ESXi 7.0 (to jump to later release see the note at the end)

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep ESXi-7.0.0

Verify the build number and try profile update

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

At this point I received the following error:

[HardwareError]
Hardware precheck of profile ESXi-7.0.0-15843807-standard failed with warnings: <CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly.>

<UNSUPPORTED_DEVICES WARNING: This host has unsupported devices [<PciInfo ‘ [1000:0072 1028:1f1c]’>, <PciInfo ‘ [1000:0072 1028:1f1c]’>]>

The ID 1000:0072 1028:1f1c  is related to a Dell HBA which will be set up to passthrough, so it is not important for them to be supported in ESXi. You can ignore this warning:

esxcli software profile update --no-hardware-warning -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

I then received a large list of dependency errors related to having previous versions of ESXi (you may not receive any on a fresh 6.7 install):

[DependencyError]
VIB Virident_bootbank_sas-vgc_5.1.75460.D1B-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_scsi-bfa_3.2.4.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB QLogic_bootbank_net-qlge_3.0.0.58-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Chelsio_bootbank_net-cxgb4_1.3.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Hitachi_bootbank_scsi-hfcldd_4.40.18.2406-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-esas4hba_1.01-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Fusion-io_bootbank_scsi-iomemory-vsl_3.2.8.1350-1OEM.550.0.0.1198610 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Fusion-io_bootbank_scsi-iomemory-vsl_3.2.8.1350-1OEM.550.0.0.1198610 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Memblaze_bootbank_block-pblaze3_0.9.0.7-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB QLogic_bootbank_net-qlge_3.0.0.58-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Hitachi_bootbank_scsi-hfcldd_4.40.18.2406-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Virident_bootbank_sas-vgc_5.1.75460.D1B-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity8fc_1.36-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Adaptec_bootbank_scsi-adp80xx_1.2.0.10624-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-esas4hba_1.01-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Stec_bootbank_scsi-stec-s1120_2.2.1.0202-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_net-bna_3.2.4.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_scsi-bfa_3.2.4.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Memblaze_bootbank_block-pblaze3_0.9.0.7-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_06.00.00.00.1vmw-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_scsi-arcmsr_1.20.00.19-1vmw.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity8fc_1.36-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Chelsio_bootbank_net-cxgb4_1.3.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Stec_bootbank_scsi-stec-s1120_2.2.1.0202-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Adaptec_bootbank_scsi-adp80xx_1.2.0.10624-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity16fc_1.06-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_scsi-arcmsr_1.20.00.19-1vmw.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_net-bna_3.2.4.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_06.00.00.00.1vmw-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity16fc_1.06-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
Please refer to the log file for more details.

Basically these are depreciated drivers from previous versions of ESXi. One option is to manually uninstall these vibs and then try the profile update again.  However I chose to switch to the profile install command, automatically removing the vibs (if you want to test before committing, add the —dry-run flag):

esxcli software profile install --no-hardware-warning --ok-to-remove -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

After it completes successfully you may receive a message that vibs were skipped during the upgrade. This is typical and the version displayed with depend on your system:

VIBs Skipped: VMware_locker_tools-light_11.1.1.16303738-16850804
VIBs Skipped: VMware_locker_tools-light_11.0.5.15389592-15843807

You can verify the correct version is installed with the below command.

esxcli software vib list | grep tools

Generally the build number of tools should match esx-base

esxcli software vib list | grep esx-base

Set firewall

esxcli network firewall ruleset set -e false -r httpClient

Then finally reboot (gracefully shutting down any VMs)

/sbin/reboot

Notes:

  1. You can jump all the way to Update 1 (or later) if you wish. Search with grep for 7.0U1d (instead of 7.0.0) which should present you with a profile of
    ESXi-7.0U1d-17551050-standard

    esxcli software profile install -p ESXi-7.0U1d-17551050-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --ok-to-remove --no-hardware-warning
  2. In one case I received the following error when using the profile update command. Switching to profile install resolved the issue:
    [InstallationError]
    Failed to update bootloader: [Errno 28] No space left on device
    cause = [Errno 28] No space left on device
    vibs = [‘VMware_bootbank_esx-base_7.0.1-0.30.17551050’]
  3. On one system the Intel 10 Gbe network card connection was failing (ixgben driver). In vSphere under Networking -> Physical NICs I noticed Auto-negotiate was showing as Enabled however that mode is not supported with this adapter/driver.  Editing the settings of the NIC, the Speed dropdown was coming up blank. Selecting 10000 Mpbs, full duplex and saving fixed the issue immediately.

 

Posted in Uncategorized | Leave a comment

Change Windows Server 2012 R2 Network Connection from Public to Private

One liner via PowerShell (run as administrator)

Change Ethernet 1 as appropriate based on what is shown in Network and Sharing Center

Set-NetConnectionProfile -InterfaceAlias “Ethernet 1” -NetworkCategory Private

Posted in Uncategorized | Leave a comment

Steps to Create an OS X Bootable ISO (El Capitan Example)

Apple has steps to create a bootable USB (linked below), but not an ISO file.
https://support.apple.com/en-us/HT201372

There are also several methods to create an ISO, such as colt2’s method linked below.
https://forums.macrumors.com/threads/el-capitan-bootable-dvd.1923931/post-22048516

The steps in this guide are what I found to be the most straight forward compared to other guides. These are performed on an OS X system to create a bootable ISO which can then be imaged to USB or used directly for installing on virtual machines.

Note – the ISO created from this method does not work when burned to DVD.  The resulting disk is bootable, but when trying to install the operating system you will receive the error:
The Install OS X El Capitan application can’t be used from this disk. Copy this application to a writable Mac OS Extended formatted disk and reopen it to continue installation.

The steps presented below can be used for other versions of OS X by increasing the size of the image as necessary and adjusting the name of the files appropriately (from El Capitan).

  1. Download El Capitan installer from Apple
    http://updates-http.cdn-apple.com/2019/cert/061-41424-20191024-218af9ec-cf50-4516-9011-228c78eda3d2/InstallMacOSX.dmg
  2. Install it
  3. Create a blank image file in the temp folder, overwriting any existing file that may exist
    hdiutil create -ov /tmp/ElCapitan -size 7316m -layout SPUD -fs HFS+J
  4. Mount the blank image to a temporary mount point
    hdiutil attach /tmp/ElCapitan.dmg -noverify -nobrowse -mountpoint /Volumes/my_bootdisk
  5. Write the installation files to the image (this step requires administrator privileges)
    sudo /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/Resources/createinstallmedia --volume /Volumes/my_bootdisk --applicationpath /Applications/Install\ OS\ X\ El\ Capitan.app
  6. Unmount the new image
    hdiutil detach /Volumes/Install\ OS\ X\ El\ Capitan
  7. Convert the new image to an ISO file
    hdiutil convert /tmp/ElCapitan.dmg -format UDTO -o /tmp/ElCapitan
  8. Move the file from the tmp folder to the Desktop of the logged in user and also change extension to ISO
    mv /tmp/ElCapitan.cdr ~/Desktop/ElCapitan.iso
  9. Delete the original image file
    rm /tmp/ElCapitan.dmg

The resulting ISO file is now ready to use.

 

Posted in Uncategorized | Leave a comment

Steps to Upgrade OMV 4.x (Arrakis) to 5.x (Usul)

I had an OMV system running 4.1.35-1 (Arrakis). Initially I tried using the steps from OMV’s ryecoaaron: https://forum.openmediavault.org/index.php?thread/27909-eta-for-omv-5-0-2020-2021-2022/&postID=219830#post219830
(This included having to use a Python fix for the weakref.py error: https://jc-lan.org/2020/04/26/openmediavault-weakref-python-error-when-updating)

However my progress came to a halt at this step:
omv-salt deploy run nginx

When I received this error
-bash: omv-salt: command not found

Below are the alternative steps I used to complete the upgrade. At a high level we first upgrade the Debian OS, then go back and upgrade/repair OMV package. It is based on this guide: https://blog.sakuragawa.moe/upgrade-omv-4-to-5-a-painful-path/

  1. sed -i 's/arrakis/usul/g' /etc/apt/sources.list.d/openmediavault.list
  2. sed -i 's/stretch/buster/g' /etc/apt/sources.list
  3. apt update
  4. On the next two steps, if prompted then choose: Don’t touch keymap, install all package maintainer’s configuration files (several of these) and restart services during package upgrades without asking
  5. apt -y --auto-remove --allow-downgrades --no-install-recommends upgrade
    apt -y --auto-remove --allow-downgrades --no-install-recommends dist-upgrade
  6. Now reboot
  7. apt -y purge openmediavault
  8. apt install -y --auto-remove --allow-downgrades openmediavault
  9. apt-get install usrmerge
  10. omv-confdbadm migrate conf 5.0.0
  11. omv-confdbadm populate
  12. sudo apt-get -y purge php7.0*
  13. omv-firstaid
  14. Choose option 9 – Clear web control panel cache
  15. Try to access OMV web interface
  16. Perform a final reboot to fix any broken services
  17. NOTE: if you had notifications enabled previously, log in to the web interface and disable them under the General Settings. Apply changes, then immediately revert back to Enabled and apply changes again.
  18. OPTIONAL: if you had a static IP and DNS server configured you will need to reconfigure your name server. In the web interface remove it, save then add it back. OR you can do the following:
    nano /etc/resolv.conf
    Add a line with the IP of your DNS: nameserver x.x.x.x
  19. OPTIONAL: reinstall OMV-extras (https://omv-extras.org)
Posted in Uncategorized | Leave a comment

Windows Installer Coordiantor Hangs on Windows Remote Desktop Servers

Attempting to install an application on Windows Server acting as a RDS resulted in Windows Installer Coordinator hanging indefinitely with the message:
Please wait while the application is preparing for the first use
followed by the Windows Installer error message:
Install server not responding
This occurred on both Server 2012 R2 and Server 2019.

The workaround is the same for both versions and involves temporarily changing a local group policy setting:

  1. Open Local Group Policy Editor (gpedit.msc)
  2. Navigate to:
    Computer Configuration -> Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Application Compatibility
  3. Locate the setting: Turn off Windows Installer RDS Compatibility
  4. Change it to Enabled and click Apply

At this point you should be able to retry or restart the installation of the application. Once complete, revert the group policy setting back to the previous value (default is Not Configured).

 

Posted in Uncategorized | Leave a comment

Unable to remove Floppy Drive from ESXi Virtual Machine

Problem: Removing the floppy drive and saving the VM configuration, the floppy drive would automatically appear again.

Solution: Shutdown the VM and unregister it. Manually edit the vmx file, find one or more lines similar to the below and delete them:

floppy0.startConnected = “FALSE”
floppy0.clientDevice = “TRUE”
floppy0.fileName = “vmware-null-remote-floppy”

Then add a new line :
floppy0.present = “FALSE”

Save and overwrite the old vmx file in the datastore, then re-register the VM. Your floppy drive should now be gone.

Posted in Uncategorized | Leave a comment

Microsoft Project License Error Loop

We had Project Professional (specifically 2016 version 1908) running on Windows Server 2012 R2.  Initially Project was installed with SharedComputerLicensing enabled. Later we reinstalled it via Office Deployment Tool and shared computer activation disabled in order to utilize volume licensing (rather than subscription).

The problem was a user was getting a license error loop that would repeat every time Project was launched. Project was still usable in this state, but the error was causing confusion. Uninstalling Project and installing fresh did not resolve the issue.

We noticed you just updated your license. Please reboot Microsoft Project for the changes to take effect.

The root cause was this particular user had run Project in the past when it had been installed with SharedComputerLicensing enabled.   Deleting the user profile from the system resolved the error loop.

Posted in Uncategorized | Leave a comment

Ubuntu 20.04 on ESXi Generating multipathd Errors

Brand new install of Ubuntu 20.04 LTS (Focal Fossa) running as a virtual machine on ESXi was generating a lot of errors in syslog like the below

multipathd[711]: sda: failed to get udev uid: Invalid argument
multipathd[711]: sda: failed to get sysfs uid: Invalid argument
multipathd[711]: sda: failed to get sgio uid: No such file or directory
multipathd[711]: sda: add missing path

The solution is to edit the configuration of the virtual machine using the steps shownb elow for ESXi 6.7. Source https://ubuntuforums.org/showthread.php?t=2441797

  1. Shut down the VM
  2. Edit the VM -> Edit settings -> VM Options
  3. Under Advanced ->Configuration Parameters -> click Edit Configuration button
  4. Add a new parameter with following properties:
    Key: disk.EnableUUID
    Value: TRUE
  5. Click OK -> Save
  6. Power on the VM and confirm the errors have stopped in /var/log/syslog
Posted in Uncategorized | Leave a comment

Mopidy fails to connect to Icecast server with Ubuntu 20.04

Mopidy 3.0.2 was installed on Ubuntu 20.04 (Focal Fossa) using the apt.mopidy.com method.
All attempts to connect to a known working Icecast2 server would not work and the Icecast server would not log any errors. However the Mopidy log would record the following error:
[MainThread] mopidy.audio.gst GStreamer error: Could not connect to server
The root cause is a bug in the version of the libshout3 library that is included in Ubuntu 20.04 (source: https://bugs.launchpad.net/mixxx/+bug/1833225). A workaround is to downgrade libshout3 library to a working version. In this example we will use the library in Ubuntu 19.10 (Eoan): 2.4.3 (2.4.3-1) –> 2.4.1 (2.4.1-2build1)
https://packages.ubuntu.com/eoan/libshout3

Steps:

  1. cd /tmp
  2. wget http://mirrors.kernel.org/ubuntu/pool/main/libs/libshout/libshout3_2.4.1-2build1_amd64.deb
  3. sudo dpkg -i libshout3_2.4.1-2build1_amd64.deb
  4. Restart Mopidy

To prevent the library from being replaced with every apt-get upgrade  you can hold it:

sudo apt-mark hold libshout3

Once the libshout3 package is fixed in 20.04 we will be able to undo it:

sudo apt-mark unhold libshout3

Then upgrade libshout3 using apt-get.

Posted in Uncategorized | Leave a comment