iOS 10 Device (iPhone iPad) Forgets Hidden Wireless Network

A particularly annoying issue where an iOS device will constantly forget about a hidden wireless network. The problem seem to start after upgrading it to iOS 10. In this case it was an iPhone 5.

Resetting the network settings would seem to work for a while but it would eventually occur again. Most often it would occur after power cycling the wireless access point or if the access point lost connection to the internet.  Unhiding the SSID would resolve it however that does not fix the root problem. Here are the steps to flush out the bad configuration in the device so that it no longer forgets:

  1. Connect to the hidden wireless network as usual
  2. Go to Settings -> Wi-Fi.  Click the info icon next to the name of the hidden network to which you are connected
  3. Choose Forget This Network
  4. Reboot the phone by holding the Home button and on/off button
  5. Navigate to Settings -> General -> Reset -> and choose Reset Network Settings
  6. The phone will automatically reboot
  7. Go to Settings -> Wi-Fi. Disable Ask to Join Networks
  8. Finally, reconnect to the hidden wireless network

Now you should no longer have the issue of the device randomly (or not so randomly) forgetting the hidden network.

Update: Eventually the device forgot the network again. The only thing that has been a permanent solution is to broadcast the SSID, such that the network is no longer hidden.

How to Grow RAID Array in OpenMediaVault After Upgrading All Existing Drives with Larger Drives

Scenario: You have an existing RAID array in OMV and you have replaced each drive one at a time with a larger drive. Now you wish to grow the array into the larger size but it turns out the GUI of OMV does not have that option. When you navigate to “RAID Management” and then click “Grow,” it comes up with an empty list of devices. This is expected since all of your drives are used in the array already.

To grow the array into the new larger size after all drives have been upgraded, access the console of OMV (e.g., via SSH) and issue the command:

where the letter X represents the number of your md device. The first md device is typically zero (i.e., md0).

After issuing the command, you should receive a confirmation immediately that the array has been resized. Something like this for an md0 device:

mdadm: component size of /dev/md0 has been set to [the new size in K]

You can confirm the change has taken place by navigating back to RAID Management where you will then see the RAID is in a resync process.

ESXi 6.5 Host Initiator Unable to Connect to OpenMediaVault 2.x Guest iSCSI Target

I ran into a problem when trying to have an OpenMediaVault 2.x Guest machine act as an iSCSI target for an ESXi 6.5 Host iSCSI initiator. The goal was to have OMV share a datastore for the ESXi host using the iSCSI protocol.

To create iSCSI targets on OMV, the easiest way is to install the available plugin (openmediavault-iscsitarget) from the web GUI. However a major limitation with this plugin is that you are not able to create fileio targets using the web GUI. In another post I detail how to get around that limitation but at this point we’ll assume you have successfully created a target (either fileio or blockio) on OMV and are now trying to get an ESXi host to discover and connect to it.

You may find that ESXi successfully discovers the OMV target but then fails to connect to it. I happened to be logged in to the console of the OMV guest at the time and saw several console error messages appear when ESXi was attempting to connect to the OMV target. They were similar to the below. You may need to reboot OMV and then try to rediscover the target in ESXi in order to reproduce them.

To flush the errors out I ended up having to reboot the OMV guest machine. The root cause appears to be a bugged version of iscsitarget that is installed from the OMV repositories when running the Linux 3.2 kernel. The solution is to install a backports kernel for OMV and then install updated iscsitarget packages that the OMV maintainers have made available.

The steps:

  1. Back up / make note of any iSCSI initiator settings in ESXi
  2. Remove the iSCSI configuration from the ESXi host
  3. Reboot OMV
  4. Back up / make note of any iSCSI target configuration settings you have configured for OMV
  5. Uninstall the iSCSI plugin (openmediavault-iscsitarget) from OMV using the web GUI
  6. Reboot OMV
  7. Log in to the OMV console
  8. Run several commands from the OMV console to fully remove and flush out the iSCSI configuration. One or more of these may not be necessary depending on your configuration:
    apt-get purge openmediavault-iscsitarget
    apt-get purge iscsitarget
    apt-get purge iscsitarget-dkms
    apt-get autoremove
    apt-get autoclean
  9. Install the OMV-Extras.org Plugin. This has to be downloaded and installed manually since it is not available by default within the web GUI. The official guide:
    http://forum.openmediavault.org/index.php/Thread/5549-OMV-Extras-org-Plugin/
  10. Navigate to the plugin configuration page in the web GUI and confirm that the OMV-Extras.org main repository is enabled.
  11. Install the 3.16 backports kernel.  At this point I configured it to be the default for OMV. The official guide:
    http://forum.openmediavault.org/index.php/Thread/8584-Install-Backports-Kernel/
  12. Reboot so that you are running 3.16 kernel
  13. Now reinstall the openmediavault-iscsitarget plugin. With the 3.16 kernel running, the plugin will automatically use the newer packages of iscsitarget available from the OMV-Extras.org repository (noted below for reference only).
    http://omv-extras.org/debian/pool/main/i/iscsitarget/
  14. At this point you should then be able to configure the iSCSI target on OMV and have ESXi successfully connect to it.

How to Utilize iSCSI fileio with OpenMediaVault 2.x

OMV (version 2.x) has an iscsitarget plugin available (openmediavault-iscsitarget), however the web interface only supports BlockIO type of targets and I was looking to implement FileIO.

After installing the plugin, it is possible to manually configure the OMV plugin for a fileio target using the  console by following the steps below. I ran into trouble when trying to access this target with ESXi which I cover in a separate article. If you are trying to access an iSCSI target with Microsoft Windows as an initiator, this method may work for you.

  1. Install openmediavault-iscsitarget plugin
  2. Enable the plugin in the web interface and confirm that it is running
  3. Access the console (locally, via ssh, etc.)
  4. Create an image file to hold your data (LUN). For OMV, Shared Folders are located at
    /media/your-volume-uuid/shared-folder-name
    . For example, to create a 1TB “thin” image using dd and the count parameter:
    dd if=/dev/zero of=/path/to/your/thinvolume.img bs=1M seek=1048576 count=0
  5. Edit the IET config file using the command:
    nano /etc/iet/ietd.conf
  6. In this configuration file you will need to add your target information. Add the following lines to the bottom of this file, modifying it as appropriate for your configuration.
    Target iqn.yyyy-mm.com.example:hostname.Target-1
            IncomingUser username password
            OutgoingUser
            Lun 0 Path=/path/to/your/thinvolume.img,Type=fileio
            Alias LUN1
  7. Restart iscsi-target with this console command
  8. Then use this command to confirm it is running
  9. Then try to connect to the ISCSI target running OMV either by IP address or by IQN name.

Bind iSCSI to a Single Network Card on OpenMediaVault

If you have multiple network cards for OMV (such as using one for a dedicated storage network), you may be looking to force OMV to assign all iSCSI traffic to one network card. A word of caution – if you are using the openmediavault-iscsitarget plugin, then following these steps may break the web interface for this plugin when upgrading or removing the plugin, since OMV will complain about this file having been modified.  It is possible to run iscsitarget without the plugin (which is recommended if you are looking to set up a filetypeio configuration), and I cover how to do that in another post.

To bind all iSCSI traffic (IET) to a single network card:

  1. Access the console
  2. Edit the configuration file for iscsitarget:
  3. Find the options line:
    ISCSITARGET_OPTIONS=””
    and change it to contain your IP address as follows (NOTE: you must use a double dash):
    ISCSITARGET_OPTIONS=”–address x.x.x.x″
    where x.x.x.x is the IP address of the network card in OMV corresponding to your dedicated storage network.
  4. Restart iscsitarget

Unable to Remove USB Controller from ESXi 6.5 Virtual Machine

Ran into a situation where an “x” was not available for removing a USB Controller from a Guest.

To resolve, first shut down the guest (if running) and temporarily Unregister it from the host. Then from the datastore, download the .vmx file associated with the guest. Create a backup of the .vmx file, then edit the original with a text editor such as Notepad. Find and delete all of the lines that begin with “usb.”  For example:

usb.present = “TRUE”
usb.pciSlotNumber = “32”
usb:0.present = “TRUE”
usb:0.deviceType = “hid”
usb:0.port = “0”
usb:0.parent = “-1”
usb:1.speed = “2”
usb:1.present = “TRUE”
usb:1.deviceType = “hub”
usb:1.port = “1”
usb:1.parent = “-1”

Once these lines are removed from the .vmx file, save it and then upload the file back to the datastore, overwriting the original.

The final step is to re-register the guest with the host by pointing to the vmx file we just updated on the datastore. You should then find that the USB Controller is gone from the guest configuration.

Unable to Connect to a MySQL Server Running on Debian Linux

If you have a fresh install of a mysql server running on Debian (e.g., after running apt-get install mysql-server) you may find that any mysql client connections from other computers to your Debian server fail (one popular example is Kodi / XBMC, etc.). You may also find that mysql connections made from Debian itself (such as a local WordPress database install) are working fine.

One quick test is to use a mysql client on another linux machine to try to connect to the Debian mysql server. From the console of the other linux machine, use the following command (replace the x.x.x.x with the IP address of the Debian server). In this example we are trying to connect with the root user.

You may then get something like:

ERROR 2003: Can’t connect to MySQL server on ‘x.x.x.x’ (111 “Connection refused”)

By default, mysql server on Debian will bind itself to the loopback address (127.0.0.1). This setting can cause all remote connections to the server to be blocked with Error 111, even though all local connections will work fine (such as a WordPress install noted earlier). As a comparison, you can test local connections by issuing the above command directly from the Debian console, which should yield an error such as:

ERROR 1045: Access denied for user ‘root’@’localhost’ (using password: NO)

This basically confirms local connections are hitting the Debian mysql server. To enable remote connections for Debian’s mysql server, modify the mysql configuration file using these steps:

  1. Find the line that says
    bind-address     127.0.0.1
  2. and comment it out by inserting a hash in front of it
    #bind-address     127.0.0.1
  3. Save and close the my.cnf file.
  4. Then restart the mysql server:

Then try connecting to the Debian mysql server from another computer.

Debian 8 Linux Guest Fails to Shut Down After Installing open-vm-tools

After installing open-vm-tools, ESXi reported that VMware Tools were Installed and running on the Debian 8 guest. However any attempt to gracefully power off the guest within ESXi would fail with these errors:

Also present in the logs was this error:

For this scenario, the solution was to first uninstall open-vm-tools. For Debian / Ubuntu:

sudo apt-get remove open-vm-tools

then install open-vm-tools-desktop

sudo apt-get update
sudo apt-get install open-vm-tools-desktop

You should then be able to power off and power on the guest within ESXi.

ESXi 6.5 Guest Fails to Power On After Changing Pass Through Device (Missing pciPassthru0.id entry)

Ran into a second error after swapping out a pass-through device in ESXi. This device was the same make and model as the previous device.

To resolve it, try these steps:

  1. Completely remove the old and/or new pass-through device from the Guest configuration
  2. Save the Guest configuration
  3. Confirm you are able to successfully boot the Guest without any pass-through device
  4. After the Guest starts, shut it down
  5. Add the new pass-through device to the Guest configuration.
  6. Save the Guest configuration
  7. You should now be able to boot it successfully with the new pass-through device

ESXi 6.5 Guest Fails to Power On with New Pass Through Device

Initially the guest had a pass through HBA controller that was working well. That controller was later upgraded in the host. To update the guest with the new controller, the old pass through device was removed and the new pass through device was added, however the guest then failed to power on with the following error:

Upon testing, if the new pass through device was removed from the guest then the guest would boot successfully. Other guests could successfully boot with this new pass through device, so there was something wrong with this particular guest’s configuration and not the pass through device.

The problem I found is that ESXi did not cleanly remove the old pass through device from this guest configuration. In this case the old pass through device had ID 01:00.0.

To resolve, first shut down the guest (if running without the pass through device) and temporarily Unregister it from the host. Then from the datastore, download the .vmx file associated with the guest. Create a backup of the .vmx file, then edit the original with a text editor such as Notepad. Find and delete the following lines:

Note: the old pass through device ID of 01:00.0" was specific to my configuration as noted in the original error message, so yours may differ. I’ve also cleared out all unique device information below with xxxxx since it will be based on your specific configuration and is not relevant.

pciPassthru0.id = "01:00.0"
pciPassthru0.deviceId = "xxx"
pciPassthru0.vendorId = "xxx"
pciPassthru0.systemId = "xxxxxxxxxxxxx"

Once these lines are removed from the .vmx file, save it and then upload the file back to the datastore, overwriting the original.

The final step is to re-register the guest with the host by pointing to the vmx file we just updated on the datastore. You should then be able to add the new pass through device and power on the guest.