vrijdag, maart 09, 2007

VMware: (Auto)patching your ESX host

As you all know, patching an ESX host is not exactly fun. VMware has promised to deliver a better patching system in their new version (ESX 3.1 and VC 2.1). In the meantime, we have to make sure that our ESX hosts are patched. In this post, I will use an IIS repository to deploy ESX patches from a central server (well, its not a push but rather a pull technology).

A special thanks goes out to Arne who wrote an article (in Dutch) on his excellent ictfreak blog on configuring IIS for ESX patches. I decided to use it and add a Perl script and a patchList to make the process more manageable.

First part: installing & configuring IIS
  1. Create a folder called VIPatches
  2. Download all patches from the VMware site
  3. Extract them to the VIPatches folder. It should look something like this:

  4. In the VIPatches, create a file called patchesList.txt with the relevant patch numbers IN THE CORRECT ORDER. Make sure no additional carriage returns are available after the final patch (in our case 3199476).
  5. Install IIS
  6. Go to IIS manager and create a new website. Call it VIPatches (or something similar)
  7. Change the port number to a free port (example: port 8082)
  8. Make sure to browse to the correct folder (in our case E:\VIPatches) and to activate Directory browsing

  9. On the directory security tab: make sure Anonymous logon is selected.
  10. On the HTTP Headers tab, MIME Types button: add .* and ‘application/octet-stream’
  11. Browse with your preferred internet browser to http://<servername>:8082/VI3Patches. You should be able to see all patches.
Second part: configuring & patching your ESX host
  1. Use the Service Console
  2. Open the appropriate firewall port by issuing the following command (depending on your configured port in IIS):
    esxcfg-firewall -o 8082,tcp,out,httpClient
  3. Create (touch) a script under /tmp called patchESX.pl (or create it on a central location so you can copy it to all your ESX hosts with WinSCP or FastSCP).
  4. Open with nano (or vim) and add the following content:
    #!/usr/bin/perl
    # patchESX.pl -- auto update esx perl script
    # by Vincent Vlieghe
    # Version 6/03/2007

    use LWP::Simple;

    $patchlist = get 'http://<yourservername>:8082/patchesList.txt';
    @array = split(/\n/, $patchlist);
    foreach $item (@array)
    {
    print $item;
    $item = trim($item);
    $cmdQuery = "esxupdate query | grep ESX-$item";
    if(system($cmdQuery) == 0)
    {
    print "\n$item is already installed - skipping\n";
    }
    else
    {
    print "\n$item is not yet installed - installing\n";
    $cmdUpdate = "esxupdate -n -r http://<yourservername>:8082/ESX-$item update";
    system($cmdUpdate);
    }
    }
    sub trim($)
    {
    my $string = shift;
    $string =~ s/^\s+//;
    $string =~ s/\s+$//;
    return $string;
    }
  5. Replace <yourservername> with the IIS servername. Make sure the ESX host can contact it (check your DNS!).
  6. Make sure your ESX host is running in maintenance mode.
  7. Run the script by issuing perl patchESX.pl. Watch and enjoy!
  8. Reboot your ESX host when all updates are installed.

25 opmerkingen:

Kent zei

Thanks for the script.

I was having some difficulty with line for detecting if a patch is installed or not. (i.e. $cmdQuery = "esxupdate query grep ESX-$item";
if(system($cmdQuery) == 0))
It was pulling everyone as already being installed as the grep was not effective. I believe you missed a pipe symbol "|" between query and grep.

Vincent Vlieghe zei

Bowulf,

Yes you are correct! I didn't notice that my pipe was deleted when posting it with the blogger tool :(. I updated the script now! Tnx for the remark

Unknown zei

Nice article and script. :)

I do have one question, though. You note in the article to "... Make sure no additional carriage returns are available after the final patch." Unfortunately, every text editor I've tried (nano and gedit, so far) to use on my Linux system adds a carriage return on the last line there wasn't one there.

Any recommendations? My guess would be to enhance the segment of code where the array is populated to only include numeric values, but my skills in Perl are lacking. ;)

Duncan Sluijter zei

Thanks for the script,


Added the firwall rule to the script enabling it and disabling it after install.

Near start

$cmdFirewallOn = "esxcfg-firewall -o 80,tcp,out,httpClient";
system($cmdFirewallOn);


Near end

$cmdFirewallOff = "esxcfg-firewall -c 80,tcp,out,httpClient";
system($cmdFirewallOff);



@ricky

You really should start nano -w
It won't wrap the lines. Maybe this helps.


Greetings,


Duncan
The Netherlands

Also added some logging.

Lemme know if your interested.

Duncan Sluijter zei

P.S. Maybe not important but I got it running under Apache.


Grtz,


Duncan

Unknown zei

Hi guys, it's almost working for me: 1) for each patch i am getting message "ESX-6856573 is not yet installed - installing", but the patch was already installed time ago. 2) And then when installing the patch: "ERROR: Invalid repository at http://czchowsint058.prg-dc.dhl.com:8082/ESX-ESX-6856573/: HTTP Error 404: Not Found" - it seems that for some reason it search "ESX-ESX-6856573" folder via IIS. any ideas?
thanks

Unknown zei

Hi, just to let you know. problem was in "ESX-$item". I've shorted that to "$item" and it's working with no problem. thanks guys.

Vincent Vlieghe zei

Hi Guys,

I'm glad it's working OK. Duncan: indeed, opening & closing the firewall port is a good addition to the script!

Monty and Mari Beth zei

Got it working in a couple hours today.

A couple notes to save someone troubleshooting time. These may have been mentioned in other posts in different contexts, but I will mention them with my experience:

1. When entering Duncan's suggested firewall open/close entries, make sure to change the port to match what you assigned. (that one is obvious, but it took me a few minutes to spot my mistake).

2. When editing the script, make sure to use an editor that does not insert carriage returns. nano without opening with the -w option inserted a cr in my script between "esxupdate -n -r" and "http://[server]..." that caused No such file or directory errors.

Great script! This will save me hours.

ericsc00 zei
Deze reactie is verwijderd door de auteur.
ericsc00 zei
Deze reactie is verwijderd door de auteur.
afokkema zei

With the following command, you can change the maintenance mode of an esx server from the command line:

vimsh -n -e /hostsvc/maintenance_mode_enter to enable maintenance mode

and

vimsh -n -e /hostsvc/maintenance_mode_exit to disable it.

maybe it's a nice addition to the script ;-)

johanteekens zei

Nice script, it worked the first time right out of the box(page).

Unknown zei

Excellent script. Saved us a ton of time

Unknown zei

I am VERY new to scripting. I keep getting an HTTP 404 error.
Can I copy and paste the script from the web page? Where are the carriage returns in the script? I am missing something simple. I will keep trying.
Looks like and excellent time saver once I get it to work.

Unknown zei

I've used this procedure and it works great, but you have to be careful to handle batches with patches (like ESX-6431040, which contains 8 patches). When you extract this patch you get subfolders that you have to move to the root of your repository. Don't forget to add the extra patch numbers to your patcheslist.txt!! I've added them instead of the original patch number that contained the 8 patches.

staplep zei
Deze reactie is verwijderd door de auteur.
staplep zei
Deze reactie is verwijderd door de auteur.
staplep zei
Deze reactie is verwijderd door de auteur.
Monty and Mari Beth zei

I found that this script caused some problems when an error occurred during patching. So, to avoid applying patches out of order (known problems caused by this) we developed an error check. The main issue with our method is that we command out and you cannot see the patching in progress. You do get report of success. Also, you can check logs to verify installation.

I have posted the entire script with the error check addition here so you can see what was commented out and the script that replaced it. I also include the firewall open/close.
-----------------------


#!/usr/bin/perl
# patchESX.pl -- auto update esx perl script
# by Vincent Vlieghe
# 20070326 - added firewall open and close
# 20070327 - Added error check

$DEBUG=1;
use LWP::Simple;

# open esx firewall
$cmdFirewallOn = "esxcfg-firewall -o 0000,tcp,out,httpClient"; #(replace 0000 with the port you want to use)
system($cmdFirewallOn);

# List patches from patch list on web server - detect if they are installed - install if not

$patchlist = get 'http://server:port/patchesList.txt';

@array = split(/\n/, $patchlist);
foreach $item (@array)
{
print $item;

$item = trim($item);
$cmdQuery = "esxupdate query | grep ESX-$item";

if(system($cmdQuery) == 0)
{
print "\n$item is already installed - skipping\n";
}
else
{
print "\n$item is not yet installed - installing\n";

$cmdUpdate = "esxupdate -n -r http://servername:port#/ESX-$item update 2>&1";

# ORIGINAL that I replaced with ERROR CHECK below: system($cmdUpdate);

# Error check (added this section)
print "COMMAND: $cmdUpdate \n" if ($DEBUG > 0);

open (CMDOUT,"$cmdUpdate|");
for (< CMDOUT >) { # NOTE - there should not be spaces around CMDOUT - this html editor did not like the correct perl syntax, so I had to add spaces to get it to post.
print "$_ \n" if ($DEBUG);
if ($_ =~ /ERROR/i ) {
print "Error has occurred: '$_' \n";
exit 1;
}
}
close CMDOUT;
}
}
sub trim($)
{
my $string = shift;
$string =~ s/^\s+//;
$string =~ s/\s+$//;
return $string;
}

# close esx firewall
$cmdFirewallOff = "esxcfg-firewall -c 0000,tcp,out,httpClient"; #(replace 0000 with your port again)
system($cmdFirewallOff);

Unknown zei

I ran across this script and incorporated the firewall piece into it. It worked beautifully!!! I don't know who you guys are but I did want to thank you all for saving me a lot of time patching VMware servers. Thanks again!

Unknown zei

Most Exelent Uber fast way to simply update your esx servers.

Kenny Moy zei

I just ran the script, how can you tell if the updates ran and installed? Because it runs straight through with no wait between install of patches.

Unknown zei

Thanks very much, this has helped me a lot.

Apart from a few mistakes my end, this worked very easily. I would like a tutorial about patching from the local ESX host if you are bored!!

Many thanks again

Unknown zei

That is cool. Thanks!
More would be great!
Web Hosting Video Tutorial