Network issues when migrating Hyper-V servers

There are two very important rules for Fridays.  Never do a config change or code release unless absolutely necessary and make sure to reward yourself and your team by having a good lunch or a beer after work.

Sadly this Friday we forgot one of these rules (the good fish & chip lunch was never in doubt!) and ended up leaving the office around midnight after hitting a snag with a relatively simple Hyper-V guest server migration between physical Hyper-V host servers.

Here is a diagram of what we had hoped to achieve.

As with most server work of this nature it needs to occur out of hours to minimize disruption.  All we had to do was shutdown one of the Hyper-V guest virtual machines and export it to its new home on the other physical server.  I was confident we could do it in a couple of hours after work on Friday night.

Both physical servers have the same connectivity, one physical management interface and two iSCSI connections to the shared SAN.  These connections are mapped through to the virtual networking inside Hyper-V.  Shown below is the setup of one of the servers, note the static MAC address box has been changed back to dynamic.

The server had originally been running a specific MAC address.  When you first setup the virtual connections they are all named the same with #1, #2 and #3 between the multiple connections.  This makes discovering which connection is for the management interface and which is for the iSCSI interface extremely difficult without a fixed MAC address as a point of reference.

The server exported to the new host no problem and was back up and running in no time, however when we tried to ping the domain controller (also virtual) we got all kinds of mixed errors, destination host unreachable and Request timed out faults.  Other machines in the virtual network could still ping the domain controller and the domain controller could ping both physical host machines.

After much head scratching it was spotted in the configurations that the exported server had the same fixed MAC address as the domain controller!  Somehow it must have got misconfigured or reset when the server was exported.

Personally I blame myself for being so confident that it was such an easy move that we would be finished and in the pub for 8pm that night.  It was such a small mistake that lead to a lot of time being spent chasing ghosts in the networking.  It goes to show there is no such thing as a small job that doesn’t need a good plan to be implemented successfully and on time.

Note: if you like the Viso diagram check out Jonathan Cusson’s Hyper-V template