Solid Activity Light on Backplane for SAS Drive

I recently acquired some SAS hard drives but when installed in a Norco SATA/SAS backplane the activity light was always on when idle (for this backplane, the light was a constant green). This behavior was opposite of the SATA drives installed in the same backplane where the activity light was off when idle. When the SAS drives were active then the activity LED was consistent with the SATA drives and blinked as one would expect.

This behavior seems not uncommon with enterprise grade SAS drives but turns out it can be modified by using the program sdparm using most Linux distributions (this example was using Ubuntu). If you want to modify the behavior of the activity light, follow these steps.

If sdparm is not already installed :

sudo apt install sdparm

To turn off the activity light while idle, we need to modify the Ready Light Meaning (RLM) field in the HDD firmware. To read the current setting in the firmware:

sdparm --get=RLM /dev/ABC

where ABC is the SAS device name (e.g., /dev/sda).
Note: if in doubt about the device name, you can get a list of devices with the command: smartctl –scan

You should then get an output from sdparm similar to the below:

RLM 0 [cha: y, def: 0, sav: 0]

Note: If you specify the wrong device (e.g., a SATA drive instead of a SAS drive) you may get an output similar to the below:
RLM not found in Protocol specific port (SAS) mode page

To flip the behavior of the activity light, issue this command:

sdparm –set=RLM /dev/ABC

This immediately turned off the light for an Hitachi Ultrastar 7K3000 and you can verify it in the device’s firmware by issuing the same get=RLM command as before.

The output should then look similar to the below:

RLM 1 [cha: y, def: 0, sav: 0]

This change is not permanent so it would revert back on power cycle. To make the page field persistent, add the save flag the command we issued previously:

sdparm --set=RLM --save /dev/ABC

The activity light should now be permanently off while idle (or until you revert the change with sdparm). The output of the get=RLM command should now look similar to the below:

RLM 1 [cha: y, def: 0, sav: 1]

If you need to flip the activity light for other devices you can do it all in one step with the save flag.  If you want to revert the change for a device, issue this command:

sdparm --clear=RLM /dev/ABC

The change was again instant for this particular drive. To verify it in the firmware, the output of get=RLM should then be:

RLM 0 [cha: y, def: 0, sav: 1]

To commit the change, add the save flag again:

sdparm --clear=RLM --save /dev/ABC

The output of get=RLM should now be:

RLM 0 [cha: y, def: 0, sav: 0]

Now you can repeat the set/save or clear/save command as desired for each SAS device.

Posted in Uncategorized | Leave a comment

Acquiring Real IP of Client with NGINX, Apache and WordPress (also with Cloudflare)

For this particular example I was trying to block brute force authentication to WordPress using the Limit Login Attempts Reloaded plugin.  This setup had an NGINX proxy in front of Apache that was serving a WordPress instance as a Virtual Host all running on Ubuntu (specifically 18.04 bionic). The problem was Apache (and the WordPress plugin) were seeing the IP of NGINX proxy ( rather than the true IP of the client. This resulted in the IP of the NGINX proxy being blocked by WordPress.

Client—> NGINX—> Apache —> WordPress

If you are using Cloudflare, it may look like one of the following:
Client—> Cloudflare—> NGINX—> Apache —> WordPress
Client—> Cloudflare—> Apache —> WordPress

For the last example the Apache/WordPress plugin would see the IP of Cloudflare rather than the client. The steps below will work for that situation, but Cloudflare also has their own support article for an Apache behind Cloudflare configuration (using a slightly different approach):

To solve the issue (for any of these configurations) it requires several steps:

  1. First you need mod_remoteip for Apache
    sudo a2enmod remoteip
  2. Then create a configuration file for remoteip with any name you prefer. In this example the config file is named remoteip.
    sudo nano /etc/apache2/conf-available/remoteip
  3. This configuration file needs several lines depending on your setup (see Notes below)
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy x.x.x.x
    RemoteIPTrustedProxy y.y.y.y
    LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
    LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined


    1. RemoteIPInternalProxy directive should be added for a local NGINX proxy and x.x.x.x represents the internal address of the NGINX server (e.g.,
    2. For an external server such as Cloudflare, then you will need RemoteIPTrustedProxy and y.y.y.y represents the external address. For Cloudflare you would need several of these lines in CIDR notation, one for each of their IP ranges (
    3. If you have an internal proxy and an external proxy then you would need both of the above directives in your configuration file
    4. Adding LogFormat is not required to make this work however it will configure Apache to properly log the client IP.  The only change from default is replacing %h with %a.
  4. The last step is to configure the appropriate sites-enabled file of NGINX. In the same location block containing the proxy_pass IP of your Apache server, add the following lines:
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  5. Then restart the NGINX and Apache services
    sudo service apache2 restart
    sudo service nginx restart
Posted in Uncategorized | Leave a comment

Windows Server 2019 (and 2016) Evaluation Product Key Activation Error

When activating Windows Server 2019 (Standard or Datacenter) installed with an Evaluation ISO, you receive an error

The product key you entered didn't work. Check the product key and try again, or enter a different one. (0x80070032)

The solution is to activate via console (cmd.exe). This method should also work for Windows Server 2016.

  1. First get the target edition
    Dism /online /Get-CurrentEdition
  2. Next provide the target edition in the below command using the output from the previous step. In this example the target edition was: ServerDatacenter
    DISM /online /Set-Edition:ServerDatacenter /ProductKey:XXXXX-XXXXX-XXXXX-XXXXX-XXXXX /AcceptEula
  3. You should then see Removing package Microsoft-Windows-… which may take a while. If after a few minutes it is not progressing / freezes / hangs – right click in the window or press Control + D and restart the same command.
  4. Once the package is removed, the system should ask for a reboot and then configure updates to complete the process.
Posted in Uncategorized | Leave a comment

Openmediavault 4.x and Windows 10 Shared Folder Errors

When trying to access an SMB share running on openmediavault you may get some of the following error messages, even though permissions have been set up correctly on the server and the correct password is being provided by the client:

Attempting to access the root of the server \\x.x.x.x\ you receive:
You do not have permission to access \\x.x.x.x\. Contact your network administrator to request access.

Attempting to access a specific share on the same server \\x.x.x.x\share you receive the following after providing the username and password:
The specified network password is not correct.

The problem was due to a bad Group Policy setting. Using gpedit.msc, navigate to the following:


Network security: LAN Manager authentication level

In this case it was set to:
Send LM & NTLM – use NTLMv2 session security if negotiated

To fix the problem it was changed:
Send NTLMv2 response only

This is the default setting for Windows Server 2008 R2 and later. The change is immediate and no reboot is required for it to take effect.

Posted in Uncategorized | Leave a comment

VMware ESXi Hang on dma_mapper_iommu

ESXi (in this case version 6.7.0) can appear to hang / freeze during the boot up sequence at the following step:
dma_mapper_iommu loaded successfully
However the system is otherwise fully responsive (e.g., via vSphere). The root cause was that the video card used for displaying the local console had been enabled for passthrough.

Posted in Uncategorized | Leave a comment

Errno 28 When Trying to Update VMware ESXi 6.7.0 Update 1 to Update 2

When trying to update/upgrade ESXi 6.7.0 Update 1 to Update 2 (via software profile update command) you may run into the following error:

[Errno 28] No space left on device
vibs = VMware_locker_tools-light_10.3.5.10430147-12986307
Please refer to the log file for more details.

The solution is to manually install VMware_locker_tools package by following these steps via SSH on the host:

  1. cd /tmp
  2. wget
  3. esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.5.10430147-12986307.vib
  4. Once you receive verification that the locker tools package has been installed successfully, try the profile update command again to patch ESXi.
    Note – in some cases the installation of this VIB may be skipped by the server and you receive the message: Host is not changed.  If so then simply try the profile update command again and you should not receive Errno 28 related to the locker tools VIB.

Posted in Uncategorized | Leave a comment

Errno 28 When Trying to Update VMware ESXi

When trying to update/upgrade ESXi 6.7 (via software profile update command) you may run into the following error message:

[Errno 28] No space left on device
Please refer to the log file for more details.

The solution is to set the host swap to use a datastore. In vSphere Web Access:

  1. Navigate to Host->Manage->System->Swap. On a fresh install, the Datastore setting  is usually set to No.
  2. Choose Edit Settings, select a valid Datastore in the drop down menu, then Save.
  3. Now you are ready to issue the profile update command again to patch ESXi


Posted in Uncategorized | Leave a comment

Windows Server 2016 Stuck at 0% Windows Updates

New install of Windows Server 2016 (build 14393.447 from ISO SW_DVD9_Win_Svr_STD_Core_and_DataCtr_Core_2016_64Bit_English_-3_MLF_X21-30350 with no updates) had an issue where Windows Updates would hang / freeze indefinitely at 0% progress. 

When Windows Update ran it would briefly flash a list of updates that were available before disappearing. Then it would simply state “Updates are available” with a progress of 0%.  Also the folder C:\Windows\SoftwareDistribution was roughly 500MB in size. 

The solution was to re-enable the Windows Firewall which had been disabled immediately after the initial install and prior to running Windows Update for the first time. After enabling Windows Firewall, restart the Windows Update service (in services.msc) and then have Windows check for updates again.

Posted in Uncategorized | Leave a comment

Windows Server 2016 Activation Error

Attempting to activate / convert an evaluation copy of Windows Server 2016 (specific build 14393.rs1_release.161220-1747) to the full version using a valid retail key, resulted in the following error:

This edition cannot be upgraded.

A solution is to perform the activation via console with the following command:

Dism /online /Set-Edition:<Edition> /AcceptEula /ProductKey:<ProductKey>

where <Edition> is either ServerDatacenter or ServerStandard and <ProductKey> is your product key with dashes.

The Deployment Image Servicing and Management tool may indicate that it has finished installing product key, but then hang at 10.0% on the next step:

Removing package Microsoft-Windows-Server.....

If after 30 minutes it has not progressed – cancel the entire process, then repeat the same steps and it should go through the second time.

Posted in Uncategorized | Leave a comment

Windows Server 2016 Remote Desktop Connection Fails

Brand new install of Windows 2016 Server Evaluation without any updates installed, specific build 14393.rs1_release.161220-1747 (also 14393.447)  Attempting to remote desktop to the server from a Windows 10 client resulted in an error:

An authentication error has occurred.
The function requested is not supported
Remote computer:
This could be due to CredSSP encryption oracle remediation. For more information, see

Note: For Windows 7 clients, the last line above in red color will be omitted from the error message.

This happened for both Datacenter and Standard installs of the same build. The fix is to simply install all available updates from Windows Update, rebooting in between as necessary. After all updates have been installed, try the remote connection again.

Posted in Uncategorized | Leave a comment