pFsense Goes Unresponsive (no GUI or internet)

Running version 2.6.0.  After about 3 weeks of boot time, pFsense would lose connectivity. Requests to the internet would fail and the pfSense GUI web interface was unreachable. The console displayed a repeated error swap_pager_getswapspace(x): failed.  The dmesg log contained a list of errors for several services that were terminated (dhcpd, dnsmaasq, mpd5, nginx, php) with error message: pid xxxxx (service), jid x, uid y, was killed: out of swap space. In some cases you can use the top -o res -a command to sort by memory usage and find that bzip2 is consuming all resources.

The solution was to disable log compression under Status->System Logs-> Settings -> Log Compression -> then change to None.

Posted in Uncategorized | Leave a comment

Secure Wipe on OMV (and Debian Linux)

I wanted to use the command line to Secure Wipe a hard drive in OpenMediaVault, based on Debian Linux. I used the built in shred command, details are available at the link below:
https://manpages.debian.org/stretch/coreutils/shred.1.en.html

For example, I like to use the following combination (based on the device number found under Storage->Disks in the GUI):

 shred -vfz /dev/sdX

Verbose, force permissions and a final write of zeros (total 4 passes). This is in contrast to the OMW default, which is a single pass:
https://openmediavault.readthedocs.io/en/6.x/administration/storage/disks.html

shred -v -n 1 /dev/sdX
Posted in Uncategorized | Leave a comment

OMW Unable to Delete Shared Folder

When attempting to delete a shared folder with OpenMediaVault, I received the following error (after clicking show details):

Error #0: OMV\ExecException: Failed to execute command ‘export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; rm -f -r ‘/srv/dev-disk-by-label-EXMPLE/’ 2>&1′ with exit code ‘1’: rm: cannot remove ‘/srv/dev-disk-by-label-EXAMPLE/’: Device or resource busy in /usr/share/php/openmediavault/system/process.inc:196 Stack trace: #0 /usr/share/openmediavault/engined/rpc/sharemgmt.inc(519): OMV\System\Process->execute() #1 [internal function]: Engined\Rpc\ShareMgmt->delete(Array, Array) #2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array) #3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod(‘delete’, Array, Array) #4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call(‘ShareMgmt’, ‘delete’, Array, Array, 1) #5 {main}

In most cases this is because a service is still using the Shared Folder (such as a share). However in this case it was due to configuration of the Shared Folder.  The Relative Path of the Shared Folder was set to “/” in OMV. The fix was to adjust the Relative Path to be a folder, such as “/FolderName” then apply the configuration.

Posted in Uncategorized | Leave a comment

OMV ESXi Mount NFS Datastore Problems

I had created an NFS share on OpenMediaVault, named EXAMPLE, with the following ACL permissions:

Owner: root, read/write/execute
Group: users, None
Others: None

When trying to mount the share (EXAMPLE) in ESXi,  the following error would occur:

Failed to mount NFS datastore EXAMPLE – Operation failed, diagnostics report: Cannot open volume: /vmfs/volumes/xxxx

The issue was permissions on the OMV side. By default NFS will squash root requests to anonymous.  There are 2 options, either will work, depending on your preference:

  1. Enable anonymous access on the OMV Shared Folder:  set the ACLs for Others to read/write/execute. Alternatively, you can change the Owner to nobody.  Be sure to set the Recursive option.
  2. Enable no_root_squash on the NFS Share

Then try to mount again.  If you receive this error

Failed to mount NFS datastore EXAMPLE – The name ‘EXAMPLE’ already exists.  Then run this in ESXi CLI

esxcli storage nfs remove -v EXAMPLE

Posted in Uncategorized | Leave a comment

Add sysadmin role to a user in Microsoft SQL Express

  1. Stop SQLExpress service
  2. Run Command Prompt with administrator rights
  3. Find your install folder, for example
    C:\Program Files\Microsoft SQL Server\MSSQL12.SQLEXPRESS\MSSQL\Binn
  4. Start SQL Express in single-user mode:
    sqlservr.exe -sSQLEXPRESS -m”SQLCMD” -c
  5. Open another Command Prompt window (with administrator rights) and connect to the running instance of SQL Express:
    SQLCMD -S .\SQLEXPRESS
  6. Run three commands, modifying the first as appropriate for your user to grant them the desired role:
    sp_addsrvrolemember 'DOMAIN\user', 'sysadmin'
    
    go
    
    quit
  7. Go back to the running instance and kill it with Control-C
  8. Start SQLExpress service
  9. The user specified above should now have sysadmin Server Role
Posted in Uncategorized | Leave a comment

ESXi iSCSI datastore no longer mounts after HBA upgrade

For a storage upgrade, I swapped out two HBAs, an SAS9341-8i (SAS3) and a Dell PERC H310 (SAS2).  The 9341 was changed to a 9300-8i and the PERC was replaced with another 9300-8i.

The storage was acting as iSCSI targets for two ESXi datastores. For the LUN originally attached to the SAS2 card, the device could be seen in ESXi (under storage->devices) however the datastore would not mount. The datastore that was originally on the 9341 card was fine.

ESXi logs (monitor -> logs -> /var/log/vmkernel.log) showed that the device was “detected to be a snapshot.” VMware has as KB describing this problem where ESXi cannot confirm the identity of the LUN attached to the new SAS3 adapter.
https://kb.vmware.com/s/article/1011387

The steps used to fix the datastore that would not mount:

  1. Find the device
    esxcli storage vmfs snapshot list
  2. Resignature the device and mount
    esxcli storage vmfs snapshot resignature -l “LABEL OF THE VMFS”
  3. Check ESXi datastores (Storage-> datastores)
    You should see it appear with a new name “snap-xxxx-LABEL OF THE VMFS”
  4. Rename the datastore as appropriate

Notes: Registered VMs on that datastore had to be reregistered. Also VMs that used hard drives on that datastore had to updated in the configuration for the new signature.

Posted in Uncategorized | Leave a comment

PHP 8.0 Upgrade on Ubuntu 18.04 with phpmyadmin

After upgrading to PHP 8.0 and installing phpmyadmin using the official PPA, phpmyadmin would result in a blank page instead of a login.  The apache2 error.log had several lines of :

Got error ‘PHP message: PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /usr/share/php/tcpdf/tcpdf.php on line 16893’

The fix was to manually upgrade the php-tcpdf package:

sudo apt-get install php-tcpdf php-tcpdf

 

Posted in Uncategorized | Leave a comment

ESXi local datastores missing after ESXi upgrade 6.7 to 7.0

After upgrading to 7.0 the system datastore was missing (attached via SATA).  Related to this article https://jc-lan.org/2017/02/01/poor-performance-in-esxi-6-5-with-jetway-jnf9g-qm77/ I had previously disabled the built in (native) AHCI driver of ESXi due to poor performance.  Enabling it again followed by a reboot of the host resolves the issue:

You can verify if any modules are disabled with the following command (in this case we are looking for vmw_ahci)

esxcli system module list | grep false

Re-enable it:

esxcli system module set --enabled=true --module=vmw_ahci

Then reboot

Posted in Uncategorized | Leave a comment

Upgrade from ESXi 6.7 to 7.0 via command line (ssh)

This guide is for upgrading ESXi via the command line.

For this guide I upgraded from 6.7.0 Update 3 (Build 16075168) to 7.0.0 (Build 15843807)

Steps:

Set firewall

esxcli network firewall ruleset set -e true -r httpClient

Search for the GA release of ESXi 7.0 (to jump to later release see the note at the end)

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep ESXi-7.0.0

Verify the build number and try profile update

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

At this point I received the following error:

[HardwareError]
Hardware precheck of profile ESXi-7.0.0-15843807-standard failed with warnings: <CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly.>

<UNSUPPORTED_DEVICES WARNING: This host has unsupported devices [<PciInfo ‘ [1000:0072 1028:1f1c]’>, <PciInfo ‘ [1000:0072 1028:1f1c]’>]>

The ID 1000:0072 1028:1f1c  is related to a Dell HBA which will be set up to passthrough, so it is not important for them to be supported in ESXi. You can ignore this warning:

esxcli software profile update --no-hardware-warning -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

I then received a large list of dependency errors related to having previous versions of ESXi (you may not receive any on a fresh 6.7 install):

[DependencyError]
VIB Virident_bootbank_sas-vgc_5.1.75460.D1B-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_scsi-bfa_3.2.4.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB QLogic_bootbank_net-qlge_3.0.0.58-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Chelsio_bootbank_net-cxgb4_1.3.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Hitachi_bootbank_scsi-hfcldd_4.40.18.2406-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-esas4hba_1.01-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Fusion-io_bootbank_scsi-iomemory-vsl_3.2.8.1350-1OEM.550.0.0.1198610 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Fusion-io_bootbank_scsi-iomemory-vsl_3.2.8.1350-1OEM.550.0.0.1198610 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Memblaze_bootbank_block-pblaze3_0.9.0.7-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB QLogic_bootbank_net-qlge_3.0.0.58-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Hitachi_bootbank_scsi-hfcldd_4.40.18.2406-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Virident_bootbank_sas-vgc_5.1.75460.D1B-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity8fc_1.36-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Adaptec_bootbank_scsi-adp80xx_1.2.0.10624-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-esas4hba_1.01-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Stec_bootbank_scsi-stec-s1120_2.2.1.0202-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_net-bna_3.2.4.0-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_scsi-bfa_3.2.4.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Memblaze_bootbank_block-pblaze3_0.9.0.7-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_06.00.00.00.1vmw-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_scsi-arcmsr_1.20.00.19-1vmw.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity8fc_1.36-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Chelsio_bootbank_net-cxgb4_1.3.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Stec_bootbank_scsi-stec-s1120_2.2.1.0202-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB Adaptec_bootbank_scsi-adp80xx_1.2.0.10624-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity16fc_1.06-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_scsi-arcmsr_1.20.00.19-1vmw.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB Brocade_bootbank_net-bna_3.2.4.0-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_06.00.00.00.1vmw-1OEM.550.0.0.1331820 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.
VIB ATTO_Technology_Inc_bootbank_scsi-celerity16fc_1.06-1OEM.550.0.0.1331820 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.
Please refer to the log file for more details.

Basically these are depreciated drivers from previous versions of ESXi. One option is to manually uninstall these vibs and then try the profile update again.  However I chose to switch to the profile install command, automatically removing the vibs (if you want to test before committing, add the —dry-run flag):

esxcli software profile install --no-hardware-warning --ok-to-remove -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0.0-15843807-standard

After it completes successfully you may receive a message that vibs were skipped during the upgrade. This is typical and the version displayed with depend on your system:

VIBs Skipped: VMware_locker_tools-light_11.1.1.16303738-16850804
VIBs Skipped: VMware_locker_tools-light_11.0.5.15389592-15843807

You can verify the correct version is installed with the below command.

esxcli software vib list | grep tools

Generally the build number of tools should match esx-base

esxcli software vib list | grep esx-base

Set firewall

esxcli network firewall ruleset set -e false -r httpClient

Then finally reboot (gracefully shutting down any VMs)

/sbin/reboot

Notes:

  1. You can jump all the way to Update 1 (or later) if you wish. Search with grep for ESXi-7.0U (instead of 7.0.0) which should present you with a list of all available updates. At the time of this guide, the latest is:
    ESXi-7.0U1d-17551050-standard

    esxcli software profile install -p ESXi-7.0U1d-17551050-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --ok-to-remove --no-hardware-warning
  2. In one case I received the following error when using the profile update command. Switching to profile install resolved the issue:
    [InstallationError]
    Failed to update bootloader: [Errno 28] No space left on device
    cause = [Errno 28] No space left on device
    vibs = [‘VMware_bootbank_esx-base_7.0.1-0.30.17551050’]
  3. On one system the Intel 10 Gbe network card connection was failing (ixgben driver). In vSphere under Networking -> Physical NICs I noticed Auto-negotiate was showing as Enabled however that mode is not supported with this adapter/driver.  Editing the settings of the NIC, the Speed dropdown was coming up blank. Selecting 10000 Mpbs, full duplex and saving fixed the issue immediately.
  4. On another system I received an error when running the esxcli software profile update command:
    Got no data from process: LANG=en_US.UTF-8 /usr/lib/vmware/esxcli-software profile.update
    This server did not have a USB boot drive or any modifications to the scratch drive. Rebooting the server and retrying the command resolved the issue.
Posted in Uncategorized | Leave a comment

Change Windows Server 2012 R2 Network Connection from Public to Private

One liner via PowerShell (run as administrator)

Change Ethernet 1 as appropriate based on what is shown in Network and Sharing Center

Set-NetConnectionProfile -InterfaceAlias “Ethernet 1” -NetworkCategory Private

Note: you can also use this command to pull all information
Get-NetConnectionProfile

 

Posted in Uncategorized | Leave a comment