ESXi iSCSI datastore no longer mounts after HBA upgrade

For a storage upgrade, I swapped out two HBAs, an SAS9341-8i (SAS3) and a Dell PERC H310 (SAS2).  The 9341 was changed to a 9300-8i and the PERC was replaced with another 9300-8i.

The storage was acting as iSCSI targets for two ESXi datastores. For the LUN originally attached to the SAS2 card, the device could be seen in ESXi (under storage->devices) however the datastore would not mount. The datastore that was originally on the 9341 card was fine.

ESXi logs (monitor -> logs -> /var/log/vmkernel.log) showed that the device was “detected to be a snapshot.” VMware has as KB describing this problem where ESXi cannot confirm the identity of the LUN attached to the new SAS3 adapter.
https://kb.vmware.com/s/article/1011387

The steps used to fix the datastore that would not mount:

  1. Find the device
    esxcli storage vmfs snapshot list
  2. Resignature the device and mount
    esxcli storage vmfs snapshot resignature -l “LABEL OF THE VMFS”
  3. Check ESXi datastores (Storage-> datastores)
    You should see it appear with a new name “snap-xxxx-LABEL OF THE VMFS”
  4. Rename the datastore as appropriate

Notes: Registered VMs on that datastore had to be reregistered. Also VMs that used hard drives on that datastore had to updated in the configuration for the new signature.

Posted in Uncategorized | Leave a comment

PHP 8.0 Upgrade on Ubuntu 18.04 with phpmyadmin

After upgrading to PHP 8.0 and installing phpmyadmin using the official PPA, phpmyadmin would result in a blank page instead of a login.  The apache2 error.log had several lines of :

Got error ‘PHP message: PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /usr/share/php/tcpdf/tcpdf.php on line 16893’

The fix was to manually upgrade the php-tcpdf package:

sudo apt-get install php-tcpdf php-tcpdf

 

Posted in Uncategorized | Leave a comment

ESXi local datastores missing after ESXi upgrade 6.7 to 7.0

After upgrading to 7.0 the system datastore was missing (attached via SATA).  Related to this article https://jc-lan.org/2017/02/01/poor-performance-in-esxi-6-5-with-jetway-jnf9g-qm77/ I had previously disabled the built in (native) AHCI driver of ESXi due to poor performance.  Enabling it again followed by a reboot of the host resolves the issue:

You can verify if any modules are disabled with the following command (in this case we are looking for vmw_ahci)

esxcli system module list | grep false

Re-enable it:

esxcli system module set --enabled=true --module=vmw_ahci

Then reboot

Posted in Uncategorized | Leave a comment

Upgrade from ESXi 6.7 to 7.0 via command line (ssh)

This guide is for upgrading ESXi via the command line.

For this guide I upgraded from 6.7.0 Update 3 (Build 16075168) to 7.0.0 (Build 15843807)

Steps:

Set firewall

esxcli network firewall ruleset set -e true -r httpClient

Search for the GA release of ESXi 7.0 (to jump to later release see the note at the end)

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep ESXi-7.0.0

Verify the build number and try profile update

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

At this point I received the following error:

[HardwareError]
Hardware precheck of profile ESXi-7.0.0-15843807-standard failed with warnings: <CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly.>

<UNSUPPORTED_DEVICES WARNING: This host has unsupported devices [<PciInfo ‘ [1000:0072 1028:1f1c]’>, <PciInfo ‘ [1000:0072 1028:1f1c]’>]>

The ID 1000:0072 1028:1f1c  is related to a Dell HBA which will be set up to passthrough, so it is not important for them to be supported in ESXi. You can ignore this warning:

esxcli software profile update --no-hardware-warning -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

I then received a large list of dependency errors related to having previous versions of ESXi (you may not receive any on a fresh 6.7 install):

[DependencyError]
VIB Virident_bootbank_sas-vgc_5.1.75460.D1B-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_scsi-bfa_3.2.4.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB QLogic_bootbank_net-qlge_3.0.0.58-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Chelsio_bootbank_net-cxgb4_1.3.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Hitachi_bootbank_scsi-hfcldd_4.40.18.2406-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-esas4hba_1.01-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Fusion-io_bootbank_scsi-iomemory-vsl_3.2.8.1350-1OEM.550.0.0.1198610 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Fusion-io_bootbank_scsi-iomemory-vsl_3.2.8.1350-1OEM.550.0.0.1198610 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Memblaze_bootbank_block-pblaze3_0.9.0.7-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB QLogic_bootbank_net-qlge_3.0.0.58-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Hitachi_bootbank_scsi-hfcldd_4.40.18.2406-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Virident_bootbank_sas-vgc_5.1.75460.D1B-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity8fc_1.36-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Adaptec_bootbank_scsi-adp80xx_1.2.0.10624-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-esas4hba_1.01-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Stec_bootbank_scsi-stec-s1120_2.2.1.0202-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_net-bna_3.2.4.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_scsi-bfa_3.2.4.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Memblaze_bootbank_block-pblaze3_0.9.0.7-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_06.00.00.00.1vmw-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_scsi-arcmsr_1.20.00.19-1vmw.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity8fc_1.36-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Chelsio_bootbank_net-cxgb4_1.3.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Stec_bootbank_scsi-stec-s1120_2.2.1.0202-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Adaptec_bootbank_scsi-adp80xx_1.2.0.10624-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity16fc_1.06-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_scsi-arcmsr_1.20.00.19-1vmw.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_net-bna_3.2.4.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_06.00.00.00.1vmw-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity16fc_1.06-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
Please refer to the log file for more details.

Basically these are depreciated drivers from previous versions of ESXi. One option is to manually uninstall these vibs and then try the profile update again.  However I chose to switch to the profile install command, automatically removing the vibs (if you want to test before committing, add the —dry-run flag):

esxcli software profile install --no-hardware-warning --ok-to-remove -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

After it completes successfully you may receive a message that vibs were skipped during the upgrade. This is typical and the version displayed with depend on your system:

VIBs Skipped: VMware_locker_tools-light_11.1.1.16303738-16850804
VIBs Skipped: VMware_locker_tools-light_11.0.5.15389592-15843807

You can verify the correct version is installed with the below command.

esxcli software vib list | grep tools

Generally the build number of tools should match esx-base

esxcli software vib list | grep esx-base

Set firewall

esxcli network firewall ruleset set -e false -r httpClient

Then finally reboot (gracefully shutting down any VMs)

/sbin/reboot

Notes:

  1. You can jump all the way to Update 1 (or later) if you wish. Search with grep for ESXi-7.0U (instead of 7.0.0) which should present you with a list of all available updates. At the time of this guide, the latest is:
    ESXi-7.0U1d-17551050-standard

    esxcli software profile install -p ESXi-7.0U1d-17551050-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --ok-to-remove --no-hardware-warning
  2. In one case I received the following error when using the profile update command. Switching to profile install resolved the issue:
    [InstallationError]
    Failed to update bootloader: [Errno 28] No space left on device
    cause = [Errno 28] No space left on device
    vibs = [‘VMware_bootbank_esx-base_7.0.1-0.30.17551050’]
  3. On one system the Intel 10 Gbe network card connection was failing (ixgben driver). In vSphere under Networking -> Physical NICs I noticed Auto-negotiate was showing as Enabled however that mode is not supported with this adapter/driver.  Editing the settings of the NIC, the Speed dropdown was coming up blank. Selecting 10000 Mpbs, full duplex and saving fixed the issue immediately.
  4. On another system I received an error when running the esxcli software profile update command:
    Got no data from process: LANG=en_US.UTF-8 /usr/lib/vmware/esxcli-software profile.update
    This server did not have a USB boot drive or any modifications to the scratch drive. Rebooting the server and retrying the command resolved the issue.
Posted in Uncategorized | Leave a comment

Change Windows Server 2012 R2 Network Connection from Public to Private

One liner via PowerShell (run as administrator)

Change Ethernet 1 as appropriate based on what is shown in Network and Sharing Center

Set-NetConnectionProfile -InterfaceAlias “Ethernet 1” -NetworkCategory Private

Note: you can also use this command to pull all information
Get-NetConnectionProfile

 

Posted in Uncategorized | Leave a comment

Steps to Create an OS X Bootable ISO (El Capitan Example)

Apple has steps to create a bootable USB (linked below), but not an ISO file.
https://support.apple.com/en-us/HT201372

There are also several methods to create an ISO, such as colt2’s method linked below.
https://forums.macrumors.com/threads/el-capitan-bootable-dvd.1923931/post-22048516

The steps in this guide are what I found to be the most straight forward compared to other guides. These are performed on an OS X system to create a bootable ISO which can then be imaged to USB or used directly for installing on virtual machines.

Note – the ISO created from this method does not work when burned to DVD.  The resulting disk is bootable, but when trying to install the operating system you will receive the error:
The Install OS X El Capitan application can’t be used from this disk. Copy this application to a writable Mac OS Extended formatted disk and reopen it to continue installation.

The steps presented below can be used for other versions of OS X by increasing the size of the image as necessary and adjusting the name of the files appropriately (from El Capitan).

  1. Download El Capitan installer from Apple
    http://updates-http.cdn-apple.com/2019/cert/061-41424-20191024-218af9ec-cf50-4516-9011-228c78eda3d2/InstallMacOSX.dmg
  2. Install it
  3. Create a blank image file in the temp folder, overwriting any existing file that may exist
    hdiutil create -ov /tmp/ElCapitan -size 7316m -layout SPUD -fs HFS+J
  4. Mount the blank image to a temporary mount point
    hdiutil attach /tmp/ElCapitan.dmg -noverify -nobrowse -mountpoint /Volumes/my_bootdisk
  5. Write the installation files to the image (this step requires administrator privileges)
    sudo /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/Resources/createinstallmedia --volume /Volumes/my_bootdisk --applicationpath /Applications/Install\ OS\ X\ El\ Capitan.app
  6. Unmount the new image
    hdiutil detach /Volumes/Install\ OS\ X\ El\ Capitan
  7. Convert the new image to an ISO file
    hdiutil convert /tmp/ElCapitan.dmg -format UDTO -o /tmp/ElCapitan
  8. Move the file from the tmp folder to the Desktop of the logged in user and also change extension to ISO
    mv /tmp/ElCapitan.cdr ~/Desktop/ElCapitan.iso
  9. Delete the original image file
    rm /tmp/ElCapitan.dmg

The resulting ISO file is now ready to use.

 

Posted in Uncategorized | Leave a comment

Steps to Upgrade OMV 4.x (Arrakis) to 5.x (Usul)

I had an OMV system running 4.1.35-1 (Arrakis). Initially I tried using the steps from OMV’s ryecoaaron: https://forum.openmediavault.org/index.php?thread/27909-eta-for-omv-5-0-2020-2021-2022/&postID=219830#post219830
(This included having to use a Python fix for the weakref.py error: https://jc-lan.org/2020/04/26/openmediavault-weakref-python-error-when-updating)

However my progress came to a halt at this step:
omv-salt deploy run nginx

When I received this error
-bash: omv-salt: command not found

Below are the alternative steps I used to complete the upgrade. At a high level we first upgrade the Debian OS, then go back and upgrade/repair OMV package. It is based on this guide: https://blog.sakuragawa.moe/upgrade-omv-4-to-5-a-painful-path/

  1. sed -i 's/arrakis/usul/g' /etc/apt/sources.list.d/openmediavault.list
  2. sed -i 's/stretch/buster/g' /etc/apt/sources.list
  3. apt update
  4. On the next two steps, if prompted then choose: Don’t touch keymap, install all package maintainer’s configuration files (several of these) and restart services during package upgrades without asking
  5. apt -y --auto-remove --allow-downgrades --no-install-recommends upgrade
    apt -y --auto-remove --allow-downgrades --no-install-recommends dist-upgrade
  6. Now reboot
  7. apt -y purge openmediavault
  8. apt install -y --auto-remove --allow-downgrades openmediavault
  9. apt-get install usrmerge
  10. omv-confdbadm migrate conf 5.0.0
  11. omv-confdbadm populate
  12. sudo apt-get -y purge php7.0*
  13. omv-firstaid
  14. Choose option 9 – Clear web control panel cache
  15. Try to access OMV web interface
  16. Perform a final reboot to fix any broken services
  17. NOTE: if you had notifications enabled previously, log in to the web interface and disable them under the General Settings. Apply changes, then immediately revert back to Enabled and apply changes again.
  18. OPTIONAL: if you had a static IP and DNS server configured you will need to reconfigure your name server. In the web interface remove it, save then add it back. OR you can do the following:
    nano /etc/resolv.conf
    Add a line with the IP of your DNS: nameserver x.x.x.x
  19. OPTIONAL: reinstall OMV-extras (https://omv-extras.org)
Posted in Uncategorized | Leave a comment

Windows Installer Coordiantor Hangs on Windows Remote Desktop Servers

Attempting to install an application on Windows Server acting as a RDS resulted in Windows Installer Coordinator hanging indefinitely with the message:
Please wait while the application is preparing for the first use
followed by the Windows Installer error message:
Install server not responding
This occurred on both Server 2012 R2 and Server 2019.

The workaround is the same for both versions and involves temporarily changing a local group policy setting:

  1. Open Local Group Policy Editor (gpedit.msc)
  2. Navigate to:
    Computer Configuration -> Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Application Compatibility
  3. Locate the setting: Turn off Windows Installer RDS Compatibility
  4. Change it to Enabled and click Apply

At this point you should be able to retry or restart the installation of the application. Once complete, revert the group policy setting back to the previous value (default is Not Configured).

 

Posted in Uncategorized | Leave a comment

Unable to remove Floppy Drive from ESXi Virtual Machine

Problem: Removing the floppy drive and saving the VM configuration, the floppy drive would automatically appear again.

Solution: Shutdown the VM and unregister it. Manually edit the vmx file, find one or more lines similar to the below and delete them:

floppy0.startConnected = “FALSE”
floppy0.clientDevice = “TRUE”
floppy0.fileName = “vmware-null-remote-floppy”

Then add a new line :
floppy0.present = “FALSE”

Save and overwrite the old vmx file in the datastore, then re-register the VM. Your floppy drive should now be gone.

Posted in Uncategorized | Leave a comment

Microsoft Project License Error Loop

We had Project Professional (specifically 2016 version 1908) running on Windows Server 2012 R2.  Initially Project was installed with SharedComputerLicensing enabled. Later we reinstalled it via Office Deployment Tool and shared computer activation disabled in order to utilize volume licensing (rather than subscription).

The problem was a user was getting a license error loop that would repeat every time Project was launched. Project was still usable in this state, but the error was causing confusion. Uninstalling Project and installing fresh did not resolve the issue.

We noticed you just updated your license. Please reboot Microsoft Project for the changes to take effect.

The root cause was this particular user had run Project in the past when it had been installed with SharedComputerLicensing enabled.   Deleting the user profile from the system resolved the error loop.

Posted in Uncategorized | Leave a comment