VMware vSphere 4.x was released for general availability nearly two years ago and now vSphere 5.x is rumored for release later this year. In June 2009, virtualization master Scott Lowe wrote a blog post illustrating the roughly 16 manual steps to upgrade virtual machines to VMxNet3 adapters and Paravirtual SCSI (PVSCSI) controllers. Interestingly, two years following I still encounter larger environments that are not fully taking advantage of the performance characteristics of these two virtual machine hardware types.
The reason is simple, this 16 step process can take up to 30 minutes of downtime per virtual machine if done manually, not to mention the inevitable human error. Unfortunately, what I have most frequently heard is that the time required to manually perform these virtual machine modifications enterprise-wide does not outweigh the advantages of utilizing VMxNet3 Adapters and PVSCSI Controllers. Instead, most larger environments update and implement these enhancements in their VM templates, but leave all other vSphere migrated/legacy virtual machines configured with either Flexible/E1000 NICs and/or LSI Logic Parallel SCSI/SAS Controllers.
What if we could change all that? What if this entire process could be automated, requiring less than 5 minutes of downtime for each virtual machine? What if we could cycle through all virtual machines in a given vCenter instance and automatically perform these modifications to all virtual machines? Two years after its release VMware certainly has not demonstrated an “Update Manager” migration approach for VMxNet3 and PVSCSI. Other scripts have attempted this in the past, but typically miss the bar when it comes to the Guest Operating System configuration. Most data center virtual machines have Static IP Addresses configured on the Virtual Machines and quite frankly, this can be a chore to remove and add network adapters, ensuring the correct settings are reconfigured for the new adapters.
The script provided in this post solves that problem by dynamically recording the static address and NIC configuration prior to removal, to ensure the original configuration is set on the new adapter. From a high-level, here are the processes performed by the script:
– Obtain a list of all virtual machines to process (Accepts * to process all machines within vCenter)
-Prompts for Guest Operating System credentials to use for all virtual machines
-Bypasses machines that are not online, VMware tools not up-to-date, VMware tools not running, or already have VMxNet3 & PVSCSI
-Obtain the current network configuration by invoking ipconfig /all
-Shutdown the virtual machine
-Upgrade the VM Hardware from Version 4 to Version 7
-Obtain the network name and remove the legacy (flexible/E100) adapter
-Add a VMxNet3 adapter on the same network as the legacy adapter (VM Network, etc)
-Add a temporary hard disk on a Paravirtual SCSI controller to install the PVSCSI drivers on boot
-Start the virtual machine
-Remove Hidden Devices utilizing DevCon.exe and automated RemoveHidden.bat file (Automated process for devmgr_show_nonpresent_devices)
-Use DevCon.exe to check for hardware changes
-Rename vmxnet3 network connection to Local Area Connection
-Apply original static ip address and network configuration to the new adapter
-Shutdown the virtual machine
-Remove the temporary hard disk and secondary PVSCSI controller
-Change the primary SCSI controller to Paravirtual
-Start the virtual machine
-Ensure the virtual machine boots with PVSCSI controller before moving forward
-Cycle through this process on the remaining virtual machines
Here is a screenshot of the output after running through this process from the beginning on a sample virtual machine:
Limitations:
– Only execute against virtual machines with static IP addresses configured. The current revision does not programatically determine if the NIC is set to DHCP and assumes all VMs have Static IP addresses. I intend to change this in future releases.
– Only execute against virtual machines with single NIC configurations and single SCSI controllers. The current revision does not address multi-homed systems or multiple SCSI controller configurations.
– Only execute against virtual machines with up-to-date VMware tools. Other scriptsare available to update VMware Tools using PowerCLI without a reboot. Outdated machines will be bypassed.
– Only execute locally from the virtual center server. For simplicity, in Revision 1 I intend for the script to be executed from the vCenter server locally. Therefore, the above components must be installed on the server locally.
– Test the script against test/dev VMs using the “comma-seperated virtual machines” option prior to running against “*” for all virtual machines within vCenter.
Requirements:
– PowerShell is installed on the Virtual Center server. Execution policy has been set to Unrestricted for the Local Machine scope (Run PowerShell command “Set-ExecutionPolicy Unrestricted -scope LocalMachine”)
– PowerCLI version 4.1.1 is installed on the Virtual Center server obtained here
– The included script (AutomatedMigration.VMxNet3.Paravirtual.ps1)
– DevCon.exe included in the zip file or available from http://support.microsoft.com/kb/311272. Devcon.exe for i386 must be placed in the same directory as the script.
– NVSPBind.exe included in this post or available from http://archive.msdn.microsoft.com/nvspbind. NVSPBind.exe for XP (Most stable for all operating systems and forward compatible) must be placed in the same directory as the script.
– Authentication credentials for the Guest Operating System of all virtual machines. Credentials will be requested once and used throughout the process for subsequent virtual machines.
– IMPORTANT: Outage windows have been scheduled and approved for virtual machines. Two reboots will be required in this process, so do not perform during production hours! 🙂
To execute, simply unzip all files and right click the Ps1 script and select “Run with PowerShell”. Do not execute from an existing PowerShell or PowerCLI instance. All necessary components will load automatically when you select Run with PowerShell.
Please feel free to leave a comment if you have any special requests or would like to leave feedback on results in your environment! Now that I have the framework fully laid out, I fully intend to utilize these processes to enable automated upgrades to components available in the new release of vSphere 5. Check back often for new revisions and updates on this script.
Click here to download the current version (Rev.3) along with Devcon, Devcon64 and nvspbind: AutomatedMigration.VMxNet3.Paravirtual
Updates for Revision 2:
Per Johnny Stanko’s request I have added functionality to read in the current WINS server list and assign these addresses to the new VMxNet3 adapter. Additionally, I have added functionality to remove the IPv6 component, QoS Packet Scheduler component, Link-Layer Topology Discovery Mapper component, Link-Layer Topology Discovery Responder component and enable NetBIOS over TCP/IP. To configure any of these settings, modify the Rev2 PS1 script in Notepad and change any of the variables at the top to $true before executing the script. These features required an additional command-line configuration utility NVSPBind.exe (Included in the updated Zip file). Here’s an updated screenshot of the script post-execution:
Updates for Revision 3:
Per Ryan’s request I have added functionality to check the Guest Operating System version prior to continuing. Processing will occur for the following “supported” operating systems:
– Windows Server 2003
– Windows Server 2008
– Windows Server 2008R2
– Windows XP
– Windows Vista
– Windows 7
All other operating system versions will report “Bypassing legacy or non-Windows virtual machine”.
Additionally, I have added functionality to support Devcon hidden device removal for 64-bit Operating Systems. Devcon64.exe is now included in the attached Zip file (Obtained from the Windows Driver Kit using the procedures documented here). The attached Zip file also includes an updated version of Devcon.exe for 32-bit systems (6.1.7600).
Thanks!
Does PowerShell need to be installed on all VMs or just on the local machine the script is being executed from?
Apologies if it is unclear, PowerShell and PowerCLI need to be installed on the vCenter server only. Script should be run locally on vCenter server since in the current version I’m pointing at localhost for the SDK connection. I’ll be releasing rev 2 shortly with some enhancements integrated from customer feedback. Thanks for the comment. Let me know if you have any recommendations once you migrate a couple test machines. Thanks!
I have sucessfully connected through PowerCLI locally on the vCenter server and was able execute Get-VM and other commands with no problem…however when I execute the script(with devcon in the same dir) I receive the following…any suggestions?
[vSphere PowerCLI] C:Program Files (x86)VMwareInfrastructurevSphere PowerCLI
Scripts> c:scriptsAutomatedMigration.VMxNet3.Paravirtual
The term ‘-f’ is not recognized as a cmdlet, function, operable program, or scr
ipt file. Verify the term and try again.
At line:1 char:3
+ -fi <<<
Try either launching the script directly or right clicking the Ps1 and selecting “Run with PowerShell”.
Thanks
Thank you very much. I was able to execute the script and it worked perfectly!!! Well, almost perfect…it only added only 1 DNS entry…probably something on my end. I have about 45 QA instances and 90 Prod instances to tackle and this will ease my worries of human error and not to mention all the time you have saved me.
I have one more question…is there code I can insert to add WINS addresses and also disable/uncheck QoS Packet Scheduler, TCP/IPv6, the two link layer topologies, register the connection in DNS and force NetBIOS over TCP/IP? I know…I am asking too much of you. 🙂
If not, I will look around and try to tweak on my own but I am horrible with scripts.
Again, thanks so much this is an AWESOME script!!!
Johnny, I have some ideas on how to tackle WINS, QoS, IPv6, LLTDMIO and LLTDR using NVSPBind (Will also need to be in the same directory along with Devcon). Also, I will be re-writing the DNS query function for cleaner and more consistent results. Stay tuned, I should have Rev2 with these modifications within the next several days. Thanks for the ideas!
Johnny, please download Rev. 2 and give it a go in your environment! Thanks
This is perfect!!! Works great and I really appreciate you tweaking this out. Now if I can just get someone to click the button for me… 🙂
Thanks again Tech Guy you made my job a lot easier.
I meant YoungTech…DOH!
Hi “youngtech”,
This is Jean with Fidelity. The script worked great for a while. From sometime last week, I got timeout error for waiting VMtools running on each VM at the step “Running Devcon and remove hidden.bat…”. I saw the vmtools are on OK status. Don’t know why it behaved like this. I’ve changed the script from sleep 3 seconds to 10 seconds, rebooted vCenter server. Nothing seems to help.
Any idea?
Jean,
This may be related to running against 2008 R2. See Ryan’s feedback below. Up to this point I think all testing had been performed on 32-bit Operating Systems (mostly Server 2003 R2 and Server 2008 non-R2).
Please try the Revision that I’ll post later this week to see if that resolves the issue with the script hanging at “Running Devcon and RemoveHidden.bat”
Thanks!
Hi,
Haven’t tried this yet but looks really good. 2 questions before I run it:
1. How does this go against Windows 2000 servers? Windows 2000 doesn’t support Paravirtual or VMXNET3. What is its behaviour against this?
2. Devcon on Windows 2008 R2 64 bit and Windows 7 64 bit. I was attempting to do something similar however this got me stuck in my automation previously as Devcon 32bit will not work in the above OS’s. I was attempting to do some scripting to determine the architecture and this consequently selected which Devcon to use however I couldn’t find a devcon for the above OS’s.
Regards
Ryan,
I’ll repost later this week with an updated version that supports 2008R2 and Win7 64-bit. I’m not sure how I missed this, but the version that I packaged most certainly only works with 32-bit Operating Systems. According to a Technet blog post “If you don’t match the platform DevCon.exe might be partially functional. For example if you try to use x86 version of DevCon.exe with x64 OS it would correctly read and display information but return errors if you try to make some changes (e.g. update a driver).”
I’ve obtained the 64-bit version of devcon from the 600Mb Windows Deployment Kit (v 7.1.0) and validated that it removes hidden devices using the script as expected. At this point I just need to repackage it all up in a Rev. 3, which I can post later this week.
Thanks for the heads up Ryan. Good input!
youngtech
Ryan,
Sorry for the delay. Rev. 3 has been posted above. Please download using the link above and send me any feedback from use in your environment! Thanks
Hi youngtech
this looks great, any updates on the last post?
Is the rev3 version ready as most of the machines we need to address are Win 2008 R2
tx a lot
Harry,
Sorry for the delay. Rev. 3 has been posted above. Please download using the link above and send me any feedback from use in your environment! Thanks
Here’s a question,
I’m running into issues with the script attempting to write files to the root of c: (like ipconfig.log for example) as well as c:windowstemp. Windows 2008 R2 with UAC has issues creating/writing files to these directories. Were you able to run this on Windows 2008 R2 guests? Any workarounds necessary?
Jason,
Unfortunately, UAC must be temporarily disabled in order for the migration script to work properly against 2008 and 2008 R2 VMs. As you would expect, this is by design as UAC is intended to prevent unwanted scripts (potentially malicious) from running without manual user intervention. You’re not the first to bring this up (See: http://communities.vmware.com/thread/289529), and currently there is no workaround for PowerCLI.
UAC can be disabled rather easily through group policy or even the registry (HKLMSoftawreMicrosoftWindowsCurrentVersionPoliciesSystemEnableLUA = 0), but requires a reboot to apply. This can be done in conjunction with running the migration script since you will likely be planning an outage window to migrate the virtual machines. If this is resolved in future releases of PowerCLI, I will be sure to adjust the scripts appropriately. Sorry to be the bearer of bad news…Thanks–
youngtech
This works !
Is ther possible to change the VM IP configuration and recreate credentials to migrate IP addresses of VMs ?
Thanks.
Not sure I quite understand. What is it you’re looking to accomplish exactly?
Thanks,
youngtech
Invalid credentials detected.”`n”Please re-run the script and provide valid credentials
i know for sure the credentials are valid, where did it go wrong?
Hey man,
Can you make the script to where you can modify it to only do VMXNET3 or PVSCSI? We don’t want to use pvscsi for whatever reason but we are doing the conversion of VMXNET3 soon. like tomorrow lol.
cwjking,
The code is all open so you’re free to make any minor modifications like what you’re requesting. The easiest thing to do would be to remark out the following line:
Write-Host `t(get-date -uformat %I:%M:%S) “Changing Primary SCSI controller to paravirtual” -ForegroundColor Green
$invokeOut = Get-HardDisk -VM $vm | Select -First 1 | Get-ScsiController | Set-ScsiController -Type paravirtual
Everything else as it relates to PVSCSI can remain in there, this is just the final “cut-over” if you will. If you remark (#) or remove this line, the script will only process VMXNET3 changes as you requested. Hope this helps!
Thanks,
youngtech
So it is working fine but with 2 exceptions which i cannot seem to get around, it will not inject the default gateway IP and also gives an error on the DNS and WINS although WINS isnt an issue.
Method invocation failed because [System.Boolean] doesn’t contain a method name
d ‘Trim’.
At C:Program Files (x86)VMwareInfrastructurevSphere PowerCLIScriptsAutoma
tedMigration.VMxNet3.Paravirtual.ps1:35 char:73
+ $dns = $dnslog -match “d{1,3}.d{1,3}.d{1,3}.d{1,3}” | % {$_.Trim <
<<< ()}
+ CategoryInfo : InvalidOperation: (Trim:String) [], RuntimeExcep
tion
+ FullyQualifiedErrorId : MethodNotFound
Cannot index into a null array.
At C:Program Files (x86)VMwareInfrastructurevSphere PowerCLIScriptsAutoma
tedMigration.VMxNet3.Paravirtual.ps1:36 char:17
+ $dns[0] = $dns[ <<<< 0].Split(":")[1].Trim()
+ CategoryInfo : InvalidOperation: (0:Int32) [], RuntimeException
+ FullyQualifiedErrorId : NullArray
Method invocation failed because [System.Boolean] doesn't contain a method name
d 'Trim'.
At C:Program Files (x86)VMwareInfrastructurevSphere PowerCLIScriptsAutoma
tedMigration.VMxNet3.Paravirtual.ps1:40 char:75
+ $wins = $winslog -match "d{1,3}.d{1,3}.d{1,3}.d{1,3}" | % {$_.Trim
<<<< ()}
+ CategoryInfo : InvalidOperation: (Trim:String) [], RuntimeExcep
tion
+ FullyQualifiedErrorId : MethodNotFound
Outstanding! The power of Powershell never ceases to amaze me. One question, anyway to set the NIC to 1GB Full duplex?
Thanks,
JB
Hi, thanks for the script.
I’m facing to strange issue…. with vcenter 4.1 U1 no problem …
We updated to vCenter 5 and I’m receiving each time an invalid credentials ..
I updated powercli to 5 version .. but problem persists ..
Anyone can help me ? 🙂 thank you
First off, I am running PowerCLI v 5.01 on an ESXi 5.0 environment with a full Windows vCenter5 server.
I ran into the same issue as Marshcroft in comment #24, getting the exact same errors. The DNS and WINS info was not displayed during the initial step, and therefore, did not populate after the adapter was changed. This occurred on a Windows 2003 VM.
I tried it on a Windows 2008R2 VM and it failed to apply the static IP, giving it all DHCP instead. It did not get the aforementioned errors, however, and did display all of the correct IP information during the initial steps of the script, it just never applied any of them. It also left the IPv6 component active and did not remove the hidden network device.
The hard disk controller change worked great for everything, which was fantastic.
Another point that nobody has mentioned is that this process changes the MAC address for the adapter. If you have special applications that care about the MAC address (we have a credit card billing system that is fussy over this), then make sure you record the MAC prior to the update, and then manually apply it by running Edit Settings for the VM in the vCenter GUI.
So, there are a few issues, probably related to ESXi 5 and PowerCLI 5. I’ll check back here for a new update if you happen to get around to it. Thanks for all of your efforts.
Amitj,
These issues all sound related to UAC being enabled within the guest operating system. This is a terrible pain and I’m going to re-evaluate if I can pre/post disable re-enable UAC within the guest. Have you tried against a VM with UAC disabled?
Thanks!
–youngtech
Hey. Thanks for providing such a great script to the world of VM users. I want to know if we can define list of VM’s on which we want to run this script instead of doing it on all VM’s. So that we can test it on 1 or few VM’s and then go ahead with prod environment.
Swapnil,
This is not a problem, you just enter the list seperated by commas after selecting not to process all VMs in vCenter. Let me know if this works for you. Thanks!
–youngtech
Hi,
Awesome script!
But i too have the error: Method invocation failed because [System.Boolean] doesn’t contain a method name
d ‘Trim’.
And no mask, dns and wins are filled. Tested on windows 2003 R2.
HI, regarding the error above. I think it’s related to the netsh command you use to poll the DNS and WINS information.
Command:
netsh interface ip show dnsservers / is not known on 2003 (R2)
Should be:
netsh interface ip show dns
And:
netsh interface ip show wins
Hi, also again regarding above.
You copy the DNS.log from VM to PowerCLI host. If this is a UAC enabled destination host, the script is not allowd to copy to C: … I recommend copieing to %TEMP%
Also, there’s perhaps an easyer way to get the aditional IP information. Please see: http://communities.vmware.com/message/1618413
The removehidde.bat file is not in the most recent zip. Where can I find that?
Nevermind.. I found the removehidden.bat was built in the script.
Hi
I tried running the script following your instructions but the powershell window open and closes. I have Power CLI installed and I choose right click run with Power Shell. I have powershell 1.0 though.
Running CLI 5.01, any idea on how to fix this:
Please provide authentication credentials for VM Guest Operating System(s) Get-VM : 02-07-12 16:04:36 Get-VM VM with name ‘TEST-DLFFO01’ was not found, using the specified filter(s).
At C:\Scripts\AutomatedMigration.VMxNet3.Paravirtual\AutomatedMigration.VMxNet3.Paravirtual.ps1:18char:14
+ $vm = Get-VM <<< .\AutomatedMigration.VMxNet3.Paravirtual.ps1
Hello, just found this blog post and am really impressed! I’m looking forward to testing your script! However, it seems the link to dl the zip file is not working? In fact, the virtualyoungtech blog seems to be offline entirely…
Could you please help with below error
Invalid credentials detected.”`n”Please re-run the script and provide valid credentials
i know for sure the credentials are valid, what might be the issue?
Invoke-vmscript remote server retured an error 407 proxy autheindication require
First of all, THANK YOU for this script. Anything significant I need to know for changing the target hardware version from version 7 to version 9?
Mahesh, I received the invalid credentials error as well. I found it was because I am running a 64-bit OS and you have to run the command below in you (x86) PowerShell runtime as well before it will work. The execution policy setting is separate between x86 and x64.
“Set-ExecutionPolicy Unrestricted -scope LocalMachine”
Hi Young,
My environment consists of VM’s with two NIC’s. Any idea or suggestions on how to make this script work with dual NIC’s VM.
Thanks,
Siva
Looks like this may work on a script I was trying to build. I have a VDI cluster that all I’d like to do is change the boot disk to PVSCSI. NIC’s use DHCP and VMXNET3 NIC’s are installed. Would it be possible or easily changed to just migrate to PVSCSI?