Oh No! Vmware ESXi bug.

Interesting details on the bug affecting vmware. For me, specifically ESXi - I just killed ntpd for now.

--

As of tomorrow morning, VM's running on all hosts with ESX 3.5U2 in enterprise configurations will not power on. VMotion/HA/DRS will probably also not work.

Boom.

Apparently, there is some bug in the vmware license management code. VMware is scrambling to figure out what happened and put out a patch.

Running VM's will not be immediately impacted.

There is a major discussion going on in the vmware communities about the issue: http://communities.vmware.com/thread/162377?tstart=0

OK, while we're all remaining calm....just imagine the implications that bugs like this can occur and get past QA testing....5 years down the road, nearly all server apps worldwide running in VM's if you believe a lot of forecasts ......some country decides to initiate cyberwarfare and manages to get a backdoor into whatever is the prevailing hypervisor of the day.....boom. All your VM's are belong to us.

I honestly think a lot of the hype from those who want to build a vm security industry is crap, but god protect us if the baseline code for critical hypervisors like ESX isn't kept secure and regularly audited.

I'd love to find out what happened here.

What regression testing on new releases does vmware do to check for date based bugs? I'd think they'd at least check for simple things like changing the date to 1 year or 1 month in the future.

UPDATE: Frank Wegner has posted the following suggestions:

You can see the latest status here: http://kb.vmware.com/kb/1006716 Please check back often, because it will notify you when this issue has been fixed. Until then the best workaround I can think of is:

* Do nothing
* Turn DRS off
* Avoid VMotion
* Avoid to power off VM's

I'd council against turning DRS off as that actually deletes resource pool settings....instead, set sensitivity to 5 which should effectively disable it w/ minimal impact.

UPDATE 2: VMware Website appears to be having trouble keeping up with people requesting updates.

UPDATE 3: VMware has stated they will have fixes available in 36hrs at the earliest.

UPDATE 4: Anand Mewalal comments:

We used the following workaround to power on the VM's.
Find the host where a VM is located
run ' vmware-cmd -l ' to list the vms.
issue the commands:
service ntpd stop
date -s 08/01/2008
vmware-cmd /vmfs/volumes/vm path/vmname.vmx start
service ntpd start

UPDATE 5: Apparently, there are no easily seen warnings in logs/etc or VC prior to hitting the bug. VC will continue to show the hosts as licensed and no errors will appear in vmkernel log file until you try to start up a new vm, reboot a vm, or reboot the host.

UPDATE 6: Welcome Slashdot readers! I've temporarily disabled comments to allow the server vm to handle the load. Apparently Movable Type 4.1 executes a seperate perl cgi script to handle comments on each page load. Load times might have been slow for the last 45 minutes, but should be OK now.

UPDATE 7: I made some minor corrections to this entry that others have requested.

SSH enable in VMWARE ESXi

ESXi 3.5 does ship with the ability to run SSH, but this is disabled by default (and is not supported).

1) At the console of the ESXi host, press ALT-F1 to access the console window.
2) Enter unsupported in the console and then press Enter. You will not see the text you type in.
3) If you typed in unsupported correctly, you will see the Tech Support Mode warning and a password prompt. Enter the password for the root login.
4) You should then see the prompt of ~ #. Edit the file inetd.conf (enter the command vi /etc/inetd.conf).
5) Find the line that begins with #ssh and remove the #. Then save the file. If you're new to using vi, then move the cursor down to #ssh line and then press the Insert key. Move the cursor over one space and then hit backspace to delete the #. Then press ESC and type in :wq to save the file and exit vi. If you make a mistake, you can press the ESC key and then type it :q! to quit vi without saving the file.
6) Once you've closed the vi editor, run the command /sbin/services.sh restart to restart the management services. You'll now be able to connect to the ESXi host with a SSH client.

ESXi Adventures -

From: http://traviskensil.wordpress.com/2008/08/11/esxi-adventures/

ESXi Adventures

August 11, 2008 · No Comments

This post will focus on some of the positives and negatives and some of my experiences thus far that I’ve had with ESXi. Part of this post will also outline some of my plans for backup/restore with the hopes of generating other’s opinions of the viability of these options, since very few exist from VMware currently.

COMPATIBILITY: Personally, I’ve always gone against the “norm” and run stuff that shouldn’t be run on things…I guess I like to tempt the odds. Same thing with ESXi. So far I’ve run ESXi on a Dell Poweredge 1900 and my Intel-motherboard based workstation with zero issues. My theory goes that most modern day servers/workstations should be compatible with it; or at least should have some way of making it work. Will soon be testing it on a Dell 2600 as well. Also tested on a Gateway workstation (E series) and confirmed IT DOESN’T work. The PERC controllers seem to be recognized and play well with ESXi. I know I’ve seen that HP and some Sun stuff also works well…we are 100% Dell for servers so thats the limit of my testing ability. I am sure as time rolls on that ESXi will mature in its compatibility ability.

INSTALL: Install is a piece of cake. Basically goto VMware’s site, give them your life history, then download the 200MB .ISO file and receive your license key and your good to go. Install should find your RAID/SATA controller right off the bat and present it as an option….if it doesn’t…..better switch hardware. I have also been investigating USB-boot…some guy on the Vmware communities’ seems to have got it figured out.

http://communities.vmware.com/blogs/Knorrhane/2008/01/21/installing-esx-3i-on-usb-stick

For my platform I just installed to the local disks; will be migrating to the USB drive soon!

BACKUP: I hate to complain about FREE but come on VMware….at least give us something to backup with. Maybe we don’t need enterprise features but a basic tool would be nice. Now, apparently according to the documentation there is the CLI-based way but there still should be some kind of GUI to it all…especially when you consider how nice the Infrastructure client is. This is one area where ESXi DOES NOT shine at all….in fact its a large disappointment! Later this year some 3rd party products are suppost to begin supporting ESXi backup but most can’t wait 3-6 months while thats figured out. If Microsoft Hypervisor has this it definitely could turn the tables for our support of VMware. I have however, devised a simple way to provide a basic level of backup. It is as follows….

• ESX CONFIG/HOST

1) Create image of USB drive contents (using Winimage http://www.winimage.com/winimage.htm or other similiar tools). This should in theory preserve the entire ESXi config, directory/drive structure required to get the system going. The first part is to take the image and save it somewhere.

2) The second part is to take the image of USB1 and copy it to USB2 which should allow for a backup USB stick should the primary fail or become corrupt.

• Virtual Machines

1) After successfully creating/importing the virtual machine, power it done properly. Go into the Datastore Browser and manually download the virtual machine files. Store in a safe place.

2) Obviously take the usual backups from within the virtual machine, use a backup proxy or however you perform “normal” backups.

That is my backup plan for ESXi currently. I have tested parts of it and found success…others need work. Ideally this would allow you to restore the ESXi config, restore your virtual machine then merge the most current data and be back up and running in a short period of time. I have yet to test this process. Please….let me know your thoughts as well….anyone got a better solution?

VMware Infrastructure Client

This is the one place ESXi shines and shines bright! The interface is clean and very easy. Tabs give access to everything one would need to know. I LOVE the networking visualization it shows; groups physical cards to virtual switches/groups…very nice. The performance and reporting per virtual machine or per server is sweet! My only gripe is that I experienced some bad delay when switching virtual machine consoles. I can’t completely rule out the workstation I was on but it still seemed pretty noticable. Also…be aware that ESXi itself doesn’t have any kind of network/IP checking in it. (Realized too late in the process that two machines had conflicting IPs…the ESXi console made no mention of that) To install the client just simply goto the IP address of the box (listed on the physical server’s console)…a splash page will load with download links to the software.

Thats pretty much it for now. Also, the VMware converter does a nice job of bringing in existing VMs into ESXi. My only complaint is the lack of efficient backup in the product. I realize VMware needs to make money, but I think a basic feature like backup, even in a simple form, should be included. Just having a simple export config and VM option would be fine. Most companies use 3rd party software for the majority of their backup needs anyway. I do feel this is one area that will limit ESXi’s deployment. Microsoft Hypervisor includes stuff like LIVE backup in its product without additional licensing (http://www.microsoft.com/windowsserver2008/en/us/virtualization-consolidation.aspx or so its site claims) which has VMware at an extreme disadvantage. The Infrastructure client is nice, but without backup included it is pretty useless I think.

So theres my results starting out. I failed to mention…my test environment is a Dell Poweredge 1900….running Domain Controller, Exchange 2003, Ubuntu Web/DNS services, File/Print servers and a test WinXP. So far so good.

With that said….anyone else using ESXi in testing or production use? What are your backup thoughts? Please share!

How does VMware ESXi Server compare to ESX Server?

From: http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-articles/general/vmware-esxi-server-compare-esx-server.html?printversion

How does VMware ESXi Server compare to ESX Server?

• Published: Aug 06, 2008
• Updated: Aug 06, 2008
• Section: General
• Author: David Davis
• Printable Version

In this article we explain how VMware ESXi (the “thin” version) and ESX Server (the “full version”) compare to each other and why it is important to know the differences, as a VMware Virtualization Admin.

Introduction

Most of you are familiar with VMware ESX Server as it has been around for so many years. ESX Server offers the “service console” built in and it is a rather large installation (in comparison to ESXi). The latest version of ESXi is “thinner” and lacks the service console. You should note that ESXi is NOT a replacement for the traditional ESX Server but, instead, an alternate version available. In my opinion, neither of these versions is “better” than another. Instead, these two versions are just “different” from one another. Let us learn how these two differ and help you determine which one is best for you.

What are the 10 major differences between VMware ESX Server and ESXi Server?

1. VMware ESXi Server has no service console

The traditional (full) ESX Server has a special built-in virtual machine called the “service console”. This service console is really a modified version of Red Hat Enterprise Linux that is installed and running in every ESX Server by default. The service console has special access to the VMware-proprietary VMFS file system. 3rd party applications can be installed in the service console and Linux-based utilities can be run in the service console. Additionally, VMware includes a number of ESX-related tools in the service console, most of which start with “esxcfg-“ and they are run by accessing the service console with SSH.

As VMware ESXi Server has no service console, there is no SSH access to the server and there are no 3rd party applications that can be installed on the server. However, there are also benefits to NOT having these features (discussed more below).

2. VMware ESXi Server uses RCLI instead of service console utilities

As ESXi doesn’t have any CLI with VMware-related or Linux utilities, VMware needed to provide a CLI interface to ESXi. What VMware came up with is the Remote Command line Interface (RCLI). This is an application that you typically install as a VM and it is used to perform scheduled or ad hock scripting on the VMware Infrastructure. The ESXi RCLI is its own command line where ESX server service console scripting would be made up of mostly Linux utilities.

For more information on how to manage ESXi, take a look at Managing VMware ESXi.

3. VMware ESXi Server is extremely thin = fast installation + faster boot

Because the service console has been removed from ESXi, the footprint in memory has been reduced to just 32MB. In my opinion, it is truly amazing that you can run a hypervisor, allowing you to run virtual machines on your server, with just 32MB of RAM overhead. In comparison, the full ESX Server on disk footprint is about 2GB.

Because the hypervisor is so small, the installation happens in about 10 minutes (or so) and the server boots up in 1-2 minutes. This is quite different from the full ESX server installation and boot, both of which are longer.

4. VMware ESXi Server can be purchased as an embedded hypervisor on hardware

While ESXi is so small that it can be easily installed and can even be booted from a USB Flash disk, what is truly unique about ESXi is that it is being sold by hardware vendors as a built-in hypervisor. That means that, say, you buy a Dell server, ESXi can be built inside the server (embedded) on a flash chip, on the motherboard. There is no installation of ESXi on disk.

5. VMware ESXi Server’s service console (firewall) is configured differently

As there is no service console to protect with the ESX Server security profile (software firewall), the security profile configuration in ESXi is very simplistic. The ESXi security profile configuration consists of a couple of services that you can either enable or not enable with inbound access. Here is a comparison between the two:

Figure 1: ESXi Security Profile – only 2 services

Figure 2: VMware ESX Server (full) Security Profile

For more information on how to configure VMware ESX Server Security Profiles – see my VirtualizationAdmin.com article How to schedule tasks with the VMware Infrastructure Client and ESX Server.

6. VMware ESXi Server has a “yellow firmware console”

Instead of the full ESX Server “service console” boot (which looks like a Linux server booting), ESXi has a tiny “Direct Console User Interface (DCUI)”. Unofficially, I like to call this the “yellow firmware console”. In this ESXi console, all that you can configure are some very basic ESXi server options such as the root user password, network settings, and a couple other items. In the graphic below, you can see why I call it “yellow”:

Figure 3: ESXi yellow firmware console / DCUI

Because this tiny firmware console (did I mention that it’s yellow?) has so few features, the server is virtually “stateless”. A new server can be configured in seconds because there is almost nothing to configure.

7. VMware ESXi Server has server health status built in

With ESXi some hardware monitoring features are built into the hypervisor. With ESX Server, this is not yet built in. Instead, you must install hardware monitoring software in the service console. For more information on ESXi server health status and how to install vendor-specific utilities to provide similar information on ESX Servers, please see my article: Obtaining server health status in VMware ESX and VMware ESXi.

Figure 4: ESXi Health Status

8. Some networking features are configured through the service console are not available or are experimental

As ESXi is relatively new and as ESX server has the option to install code for advanced ESX Server features, not all features available in the full ESX Server are also available in ESXi. In fact, I have had issues getting VMware High Availability (VMHA) to work in ESXi. VMHA was not officially supported on ESXi until some recent patches came out for ESXi. Still, even after the patches, I had difficulties with ESXi and VMHA.

There are other ESX Server features that are “experimental” on ESXi. For the full list visit: Differences in Supported Networking Features Between ESX Server 3.5 and ESX Server 3i

9. VMware ESXi Server requires fewer patches and less rebooting

Because the full ESX server essentially has a modified Linux system as the service console, there are many patches that have to be deployed to keep it secure. With ESXi, on the contrary, the server has very few patches that need to be applied. Because ESXi has no service console and it is considered more secure and more reliable. Security, Reliability, and Maintainability, are all major factor when considering a hypervisor.

10. You can buy VMware ESXi Server for as little as $495

With the full version of ESX Server, the least expensive purchase option is the Foundation (Starter) kit for about $1,500, while you can purchase ESXi only (with no support) for $495. On the other hand, if you do get the Foundation kit, you not only get the full ESX Server but also ESXi and a number of VMware Infrastructure Suite options. Still, obtaining ESXi for under $500 allows a server to do so much more than it ever could before.

Which version of VMware ESX Server is best for you?

I am not here to sell you on VMware, on ESX Server, or ESXi Server, what I am here to do is to inform you of the drastic differences between these two versions of “ESX Server”. In my opinion, ESX Server (full) must be used if you have 3rd party apps or if you just want to have access to the “Linux-style” service console.

On the other hand, if you are willing to give up those two benefits, with ESXi, you will get an ESXi Server that boots faster, has fewer patches to deploy, and is more reliable. ESXi is also the least expensive option.

I recommend testing both VMware ESX Server and ESXi server. Both are available for a free evaluation download from VMware Inc.

Conclusion

In this article, you learned about VMware ESX Server the differences between ESXi and ESX Server. Additionally, you learned about how to make the right choice for you. Both of these hypervisors from VMware can be evaluated at no cost.

For more information on VMware ESXi and its architecture, take a look at: VMware.com Whitepaper – the Architecture of VMware ESXi.

For information on ESXi’s features, visit: VMware.com ESXi Features.

iSCSI Initiator Support on FreeBSD 7

Thanks to http://www.cyberciti.biz/faq/freebsd-iscsi-initiator-howto/

Q. How do I install and configure iSCSI initiator service under FreeBSD server?

A. FreeBSD 7.x has full support for iSCSI. Older version such as FreeBSD 6.3 requires backport for iSCSI. Following instruction are known to work under FreeBSD 7.0 only.
FreeBSD iscsi_initiator driver

The iscsi_initiator implements the kernel side of the Internet SCSI (iSCSI) network protocol standard, the user land companion is iscontrol and permits access to remote virtual SCSI devices via cam.
Compile driver

Please note that FreeBSD 7.x has this driver compiled by default. You can skip this step if driver exists at /boot/kernel/iscsi_initiator.ko. To compile this driver into the kernel,
# cd /usr/src/sys/i386/conf
# cp GENERIC ISCSIKERNEL
# vi ISCSIKERNEL
Place the following lines in your kernel configuration file:
device iscsi_initiator
Save and close the file. Building a Kernel, type the following commands:
# cd /usr/src
# make buildkernel KERNCONF=ISCSIKERNEL
Install the new kernel:
# make installkernel KERNCONF=ISCSIKERNEL
Now reboot the system:
# reboot
Install iSCSI Initiator driver under FreeBSD

You need FreeBSD kernel driver for the iSCSI protocol. You need to use driver called /boot/kernel/iscsi_initiator.ko. You can load this driver by typing following command as root user:
# kldload -v iscsi_initiator.ko
Output:

Loaded iscsi_initiator.ko, id=6

Alternatively, to load the driver as a module at boot time, place the following line in /boot/loader.conf:
# vi /boot/loader.conf
# Beginning of the iSCSI block added by Vivek
iscsi_initiator_load="YES"
# End of the block added by Vivek
Save and close the file.
FreeBSD iscontrol command to login / negotiator / control for an iSCSI initiator session

Now, you need to use iscontrol command. First, do a discovery session and exit:
# iscontrol -d targetaddress=iSCSI-SERVER-IP-ADDRESS initiatorname=nxl
# iscontrol -v -d targetaddress=192.168.1.100 initiatorname=nxl
Please note down the list of available targetnames/targetadresses. Once you know the target name, edit /etc/iscsi.conf file:
# vi /etc/iscsi.conf
Append config directives as follows:

officeiscsi {
authmethod = CHAP
chapIName = YOUR-ISCSI-USERNAME
chapSecret = YOUR-ISCSI-PASSWORD
initiatorname = nxl
TargetName = iqn.XYZZZZZZZZZZZZZ # whatever "iscontrol -v -d " gives you
TargetAddress = 192.168.1.100:3260,1 # your iscsi server IP
}

Save and close the file.
Where,

* officeiscsi { : Start config for iSCSI.
* authmethod : Set authentication method to chap
* chapIName : Your username
* chapSecret : Your password
* initiatorname : if not specified, defaults to iqn.2005-01.il.ac.huji.cs:
* TargetName : is the name by which the target is known, not to be confused with target address, either obtained via the target administrator, or from a discovery session.
* TargetAddress : is of the form domainname[:port][,portal-group-tag] to quote the RFC: The domainname can be specified as either a DNS host name, a dotted-decimal IPv4 address, or a bracketed IPv6 address as specified in [RFC2732].
* } : End of config

Start an iSCSI session

The following command will read options from /etc/iscsi.conf, use the targetaddress found in the block nicknamed myiscsi, login and negotiate whatever options are specified, and start an iscsi-session.
# iscontrol -c /etc/iscsi.conf -n officeiscsi
Once you run the iscontrol command it should create a new device in /dev directory. To see the device name run dmesg command:
# dmesg
Format iSCSI volume

Now run sysinstall command to format just discovered iSCSI device name at /dev location:
# sysinstall
Select Custom > 3 Partition > Select iSCSI device name such as da1. Once formatted just mount device, enter:
# mkdir /iscsi
# mount /dev/da1s1 /iscsi
You may also need to update /etc/fstab file:
/dev/ad1s1 /iscsi ufs rw 3 3

get in shape with little or no equipment:

http://lifehacker.com/400053/get-in-shape-with-little-or-no-equipment

ESXi - [Unsupported] Console Access

ESXi - [Unsupported] Console Access

To access the ESXi console in unsupported mode do the following.

1. Open the console of your ESXi host, you should see a screen that look something like the following ( altered to protect the innocent ):


2. Use the key-combo Alt+F1, and you'll bounce to a virutal terminal (vtty1) with some log messages in it. Though you won't get a response to any characters that you type in here type 'unsupported' and then hit enter.


3. Once you access the unsupported mode, you'll be greeted with a friendly reminder that you shouldn't be accessing the console without the help of your friendly neighborhood VMware support engineer. Enter your root password here and you'll have unfettered access to the console. Some of your old friends are still here like esxtop and esxcfg-mpath, and so are some new ones like esxcfg-locker.



Standard warnings apply here. You really shouldn't be messing around in here if you don't know what you're doing.