Errno 28 When Trying to Update VMware ESXi 6.7.0 Update 2 to Update 3

When trying to update/upgrade ESXi 6.7.0 Update 2 to Update 3 (via software profile update command) you may run into the following error below. This was a host running specifically Update 2 build 13644319.

[Errno 28] No space left on device
vibs = VMware_locker_tools-light_10.3.10.12406962-14141615
Please refer to the log file for more details.

The solution is to manually install VMware_locker_tools package by following these steps via SSH on the host:

  1. cd /tmp
  2. wget
  3. esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.10.12406962-14141615.vib
  4. Once you receive verification that the Operation finished successfully, try the profile update command again to patch ESXi.
Posted in Uncategorized | Leave a comment

Create a custom alias for a RemoteApp in Windows Server 2012 R2

For Windows Server 2012 R2 it’s not possible to change the alias of an existing RemoteApp. However it is possible to set a custom alias when creating a new RemoteApp using PowerShell.  The steps:

  1. Unpublish the existing RemoteApp
  2. Start PowerShell
  3. Issue the following command to publish the new app with your desired alias. Note that folder name is optional. The collection name you can find in Server Manager under Remote Desktop Services -> Collections
    New-RemoteApp -CollectionName “YourCollectionName” -Alias “YourAlias” -DisplayName “YourDisplayName” -FolderName “YourFolderName” -FilePath “C:\path\to\executable.exe”
Posted in Uncategorized | Leave a comment

Adjust Date/Time in Windows Server 2019

When attempting to change the timezone in Windows Server 2019 you may run into the following problems:

  • The change button in Windows Settings for Date & Time is greyed out
  • Receive an error message when clicking Change Time Zone button in Control Panel Date and Time, even though you are an administrator:
    Unable to continue
    You do not have permission to perform this task. Please contact your computer administrator for help.

The solution is to force the Control Panel Date and Time applet to run separately with administrator privileges:

  1. Click Start
  2. Type timedate.cpl
  3. Right click on the resulting Control panel item
  4. Choose Run as administrator
  5. Now you can adjust the time freely
Posted in Uncategorized | Leave a comment

Let’s Encrypt Suddenly Failing to Renew a Certificate

On a Ubuntu 18.04 bionic system I suddenly started getting errors with certbot for one domain while certificates for other domains on the same system were renewing without errors.  Performing a –dry-run would result in various error messages, such as:

DNS problem: SERVFAIL looking up CAA for …

Remote PerformValidation RPC failed

Unfortunately, an error on the ACME server prevented you from completing authorization. Please try again later.

Running certbot renew with the additional flag –debug-challenges and inspecting the letsencrypt.log revealed the following:

Invalid Content-Type header on POST. Content-Type must be “application/jose+json”

The solution was to simply update certbot on the system

sudo apt-get update
sudo apt-get upgrade
Posted in Uncategorized | Leave a comment

Unable to Delete User Profiles Windows 7

I had a situation where the Delete button was grayed out  / disabled when attempting to delete a user account on a Windows 7 machine. This occurred when logged in with an account with full administrator privileges.  The solution was to first navigate to the c:\Users\folder, find the desired user’s folder and try to enter it.  Windows should then prompt you:
You don’t currently have permission to access this folder. Click Continue to permanently get access to this folder.
Once that is complete and you can access the folder, you should then be able to go back to Advanced System Settings and delete the user profile.

Edit: 29-Jan-2020
If you still have the issue, make sure your account has full administrative privileges (e.g., if you had to elevate your domain account) and then reboot the system.

Posted in Uncategorized | Leave a comment

Solid Activity Light on Backplane for SAS Drive

I recently acquired some SAS hard drives but when installed in a Norco SATA/SAS backplane the activity light was always on when idle (for this backplane, the light was a constant green). This behavior was opposite of the SATA drives installed in the same backplane where the activity light was off when idle. When the SAS drives were active then the activity LED was consistent with the SATA drives and blinked as one would expect.

This behavior seems not uncommon with enterprise grade SAS drives but turns out it can be modified by using the program sdparm using most Linux distributions (this example was using Ubuntu). If you want to modify the behavior of the activity light, follow these steps.

If sdparm is not already installed :

sudo apt install sdparm

To turn off the activity light while idle, we need to modify the Ready Light Meaning (RLM) field in the HDD firmware. To read the current setting in the firmware:

sdparm --get=RLM /dev/ABC

where ABC is the SAS device name (e.g., /dev/sda).
Note: if in doubt about the device name, you can get a list of devices with the command: smartctl –scan

You should then get an output from sdparm similar to the below:

RLM 0 [cha: y, def: 0, sav: 0]

Note: If you specify the wrong device (e.g., a SATA drive instead of a SAS drive) you may get an output similar to the below:
RLM not found in Protocol specific port (SAS) mode page

To flip the behavior of the activity light, issue this command:

sdparm –set=RLM /dev/ABC

This immediately turned off the light for an Hitachi Ultrastar 7K3000 and you can verify it in the device’s firmware by issuing the same get=RLM command as before.

The output should then look similar to the below:

RLM 1 [cha: y, def: 0, sav: 0]

This change is not permanent so it would revert back on power cycle. To make the page field persistent, add the save flag the command we issued previously:

sdparm --set=RLM --save /dev/ABC

The activity light should now be permanently off while idle (or until you revert the change with sdparm). The output of the get=RLM command should now look similar to the below:

RLM 1 [cha: y, def: 0, sav: 1]

If you need to flip the activity light for other devices you can do it all in one step with the save flag.  If you want to revert the change for a device, issue this command:

sdparm --clear=RLM /dev/ABC

The change was again instant for this particular drive. To verify it in the firmware, the output of get=RLM should then be:

RLM 0 [cha: y, def: 0, sav: 1]

To commit the change, add the save flag again:

sdparm --clear=RLM --save /dev/ABC

The output of get=RLM should now be:

RLM 0 [cha: y, def: 0, sav: 0]

Now you can repeat the set/save or clear/save command as desired for each SAS device.

Posted in Uncategorized | Leave a comment

Acquiring Real IP of Client with NGINX, Apache and WordPress (also with Cloudflare)

For this particular example I was trying to block brute force authentication to WordPress using the Limit Login Attempts Reloaded plugin.  This setup had an NGINX proxy in front of Apache that was serving a WordPress instance as a Virtual Host all running on Ubuntu (specifically 18.04 bionic). The problem was Apache (and the WordPress plugin) were seeing the IP of NGINX proxy ( rather than the true IP of the client. This resulted in the IP of the NGINX proxy being blocked by WordPress.

Client—> NGINX—> Apache —> WordPress

If you are using Cloudflare, it may look like one of the following:
Client—> Cloudflare—> NGINX—> Apache —> WordPress
Client—> Cloudflare—> Apache —> WordPress

For the last example the Apache/WordPress plugin would see the IP of Cloudflare rather than the client. The steps below will work for that situation, but Cloudflare also has their own support article for an Apache behind Cloudflare configuration (using a slightly different approach):

To solve the issue (for any of these configurations) it requires several steps:

  1. First you need mod_remoteip for Apache
    sudo a2enmod remoteip
  2. Then create a configuration file for remoteip with any name you prefer. In this example the config file is named remoteip.
    sudo nano /etc/apache2/conf-available/remoteip
  3. This configuration file needs several lines depending on your setup (see Notes below)
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy x.x.x.x
    RemoteIPTrustedProxy y.y.y.y
    LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
    LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined


    1. RemoteIPInternalProxy directive should be added for a local NGINX proxy and x.x.x.x represents the internal address of the NGINX server (e.g.,
    2. For an external server such as Cloudflare, then you will need RemoteIPTrustedProxy and y.y.y.y represents the external address. For Cloudflare you would need several of these lines in CIDR notation, one for each of their IP ranges (
    3. If you have an internal proxy and an external proxy then you would need both of the above directives in your configuration file
    4. Adding LogFormat is not required to make this work however it will configure Apache to properly log the client IP.  The only change from default is replacing %h with %a.
  4. The last step is to configure the appropriate sites-enabled file of NGINX. In the same location block containing the proxy_pass IP of your Apache server, add the following lines:
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  5. Then restart the NGINX and Apache services
    sudo service apache2 restart
    sudo service nginx restart
Posted in Uncategorized | Leave a comment

Windows Server 2019 (and 2016) Evaluation Product Key Activation Error

When activating Windows Server 2019 (Standard or Datacenter) installed with an Evaluation ISO, you receive an error

The product key you entered didn't work. Check the product key and try again, or enter a different one. (0x80070032)

The solution is to activate via console (cmd.exe). This method should also work for Windows Server 2016.

  1. First get the target edition
    Dism /online /Get-CurrentEdition
  2. Next provide the target edition in the below command using the output from the previous step. In this example the target edition was: ServerDatacenter
    DISM /online /Set-Edition:ServerDatacenter /ProductKey:XXXXX-XXXXX-XXXXX-XXXXX-XXXXX /AcceptEula
  3. You should then see Removing package Microsoft-Windows-… which may take a while. If after a few minutes it is not progressing / freezes / hangs – right click in the window or press Control + D and restart the same command.
  4. Once the package is removed, the system should ask for a reboot and then configure updates to complete the process.
Posted in Uncategorized | Leave a comment

Openmediavault 4.x and Windows 10 Shared Folder Errors

When trying to access an SMB share running on openmediavault you may get some of the following error messages, even though permissions have been set up correctly on the server and the correct password is being provided by the client:

Attempting to access the root of the server \\x.x.x.x\ you receive:
You do not have permission to access \\x.x.x.x\. Contact your network administrator to request access.

Attempting to access a specific share on the same server \\x.x.x.x\share you receive the following after providing the username and password:
The specified network password is not correct.

The problem was due to a bad Group Policy setting. Using gpedit.msc, navigate to the following:


Network security: LAN Manager authentication level

In this case it was set to:
Send LM & NTLM – use NTLMv2 session security if negotiated

To fix the problem it was changed:
Send NTLMv2 response only

This is the default setting for Windows Server 2008 R2 and later. The change is immediate and no reboot is required for it to take effect.

Posted in Uncategorized | Leave a comment

VMware ESXi Hang on dma_mapper_iommu

ESXi (in this case version 6.7.0) can appear to hang / freeze during the boot up sequence at the following step:
dma_mapper_iommu loaded successfully
However the system is otherwise fully responsive (e.g., via vSphere). The root cause was that the video card used for displaying the local console had been enabled for passthrough.

Posted in Uncategorized | Leave a comment