Tag Archives: pvscsi

Oracle versus RedHat and VMware

Uncategorized, , , , , , ,

As I wrote earlier, there are many things to consider about when deciding what distribution is best for your environment when running Oracle under VMware. Over the past couple of years there’s been a quiet battle with Oracle on one side and RedHat with VMware on the other. As a customer, your dollars fund the game. Your choices will help determine who wins this battle.

First up, Oracle

In Oracle Database 11g, Oracle introduced a new features called Database Smart Flash Cache. The feature leverages any flash storage (aka Solid State Drives (SSDs) or Enterprise Flash Drives (EFDs)) on the system to act a second level cache behind RAM, increasing the database buffer cache without adding RAM to the system. There was a recent post on this topic on Oracle’s Linux blog which links to an interesting white paper if you want more detail on the inner workings of this feature.

Sounds great right? Here’s the thing: It’s only available on Oracle Solaris or Oracle Linux. Now Oracle Solaris on SPARC processors I could see – it’s a Big-Endian Unix whereas x86/x86-64 Linux is Little-Endian OS and Solaris on SPARC is a much more mature OS that Linux.

Oracle Linux is binary compatible with RedHat Enterprise Linux – so there is no technical reason why Database Smart Flash Cache shouldn’t work. I’m not the first to point this out. Dave Welch of House of Brick Technologies wrote about this in October 2009. I suggest you read his eloquent blog post on the subject. Note that this was all before the Oracle UEK Kernel was released and nothing in Oracle’s documentation states that UEK is required for Database Smart Flash Cache.

In September of 2010 at Oracle OpenWorld 2010, Oracle announced Oracle Unbreakable Enterprise Kernel (UEK). As you can read in this datasheet on the UEK,the big selling points are that it’s a modern 2.6.32 linux kernel that shows “tremendous performance improvements” compared to a standard Enterprise Linux 5 kernel (which is a 2.6.18 kernel). Amazing right? Not really. RHEL 5 was released in March of 2007 and hits the end of Production 1 life cycle in Q4 of 2011. RHEL 6 was released in November of 2010 is also a modern 2.6.32 linux kernel. Key benefits of UEK according to Oracle’s FAQ are that it’s fast, modern, reliable and optimized for Oracle. RHEL 6 is fast modern and reliable too. Yes, the UEK does have some optimizations and bug fixes for Oracle but they or may not be relevant in your environment.

So why does Oracle beat up on RedHat? It’s about money. Oracle wants that OS support revenue.

How is Oracle beating up on VMware? It’s also about money. Virtualizing Oracle allows you to run more Oracle servers on the same hardware. That’s lost revenue to Oracle, unless you’re using their hardware (Exadata) or software (OracleVM). Oracle tries to discourage VMware use with Oracle’s infamous supported but not certified argument against VMware. There is also Oracle’s policy of having to license all the cores on a host regardless of how many cores your VM can see (unless you’re using Oracle’s virtualization solution of course). Here’s another one I came across recently: In the release notes for Oracle Linux 6 is this little nugget:

Unbreakable Enterprise Kernel doesn’t contain vmw_pvscsi driver (11697522)
As a workaround, when creating a new VM in VSphere, do not pick Red Hat Enterprise Linux 6 x86-64 as the OS type but use Oracle Linux 5 x86-64 (or Red Hat Enterprise Linux 5). ESX will then expose the LSI Logic SCSI controller in the VM and the 2.6.32-100.28.* kernel will see the devices properly.

I did a default install of RHEL 6 (which uses the RedHat Kernel instead of Oracle Kernel) and lo and behold, RHEL 6 automatically uses PVSCSI to get you better disk performance under VMware. Oracle’s Unbreakable Kernel is based off of RedHat’s Kernel. That means Oracle went out of it’s way to cripple performance by removing PVSCSI support when using their kernel under VMware. Oracle’s own virtualization product OracleVM doesn’t have any sort of PVSCSI enhancements, so what better way to erase the performance benefits you’d see virtualizing under VMware? As a customer, I find this behavior deplorable.

Next Up, RedHat

With the release of RedHat Enterprise Linux 6, RedHat stopped providing the source of a vanilla kernel and all their patches in different packages and now provides this all in one large tarball. It’s totally legal by the requirements of the GPL, but it makes it much more difficult for downstream vendors (such as Oracle) to figure out what optimizations RedHat has made to the vanilla kernel and to then incorporate those changes into their own customized kernel. Now why would RedHat do this? Let’s see what Brian Stevens, CTO, VP Worldwide Engineering for RedHat had to say in this press release

“When we released RHEL 6 approximately four months ago, we changed the release of the kernel package to have all our patches pre-applied. Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.

Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.”

Who is the number one seller of RHEL support behind RedHat? Oracle.

Finally, VMware

VMware has done a few things I can think of to fight back against Oracle:
1) Quite simply, Oracle’s UEK (Unbreakable Enterprise Kernel) is NOT supported under VMware. Yes, I’m sure it requires resources to test yet another kernel, but I don’t believe that’s the case here. Could it be because of the “optimizations” Oracle has made such as the deadline scheduler (not typically good when used under a hypervisor) or stripping PVSCSI support out of the UEK? Maybe.

2) Up until the last few weeks, Hot Memory Add wasn’t listed as supported with Oracle Enterprise Linux 5.X under VMware according to VMware’s Guest OS Compatibility List, even though RedHat Enterprise Linux (on which OEL is based) was supported. Maybe it was an innocuous typo … or maybe not.

3) In November 2010, Oracle changed their support policy to explicitly INCLUDE Oracle RAC 11.2.0.2 when virtualized. Before that time, Oracle RAC was explicitly NOT supported by Oracle under VMware. Oracle’s yearly conference, Oracle OpenWorld, was held in September – before the change in the Oracle RAC on VMware support policy – and VMware had a booth there. I attended a number of sessions at that booth and at least two of them (one by Todd Muirhead of VMware, one by David Welch of House of Brick Technologies) talked about running Oracle RAC virtualized under VMware. Both presenters made it very clear that this wasn’t supported by Oracle, but one has to wonder why VMware would take the time to talk about a solution that wasn’t supported by Oracle? Maybe they knew Oracle was going to change their support policy and were just letting customers know it could be done “if only Oracle would support it”… or maybe VMware was preparing a more offensive role regarding Oracle RAC on VMware. I don’t know, but it’s interesting to think about.

So what’s a customer to do? To me the answer is simple: If you’re going to run Oracle virtualized, I’d highly recommend running it on RHEL Linux on VMware and buying my support for RHEL Linux from RedHat. If you buy support for RHEL from Oracle, assuming the support quality level is the same as RedHat’s, you’re helping Oracle to stifle true innovation that benefits everyone. If you decide to run OL, not only are you helping Oracle to stifle innovation that benefits everyone, you’re also telling Oracle you’re fine with them limiting features to customers (such as Smart Flash Cache) entirely for anti-competitive marketing reasons.

Don’t reward bad behavior.

What Linux distribution should you use for Oracle virtualized on VMware

Uncategorized, , , , , ,

Recently Tim Hall of Oracle-Base fame wrote an article “Which Linux do you pick for Oracle Installations?” which addresses Oracle on non-virtualized Linux. Tim’s article is excellent but doesn’t take VMware virtualization into account, so without further ado, which Linux distribution should you use for Oracle virtualized on VMware?

When virtualizing Oracle with VMware, most Oracle DBAs are going to run it on some flavor of Linux. Oracle generally supports three distributions of Linux for their enterprise products: Novell’s SuSE Linux Enterprise Server (SLES), RedHat Enterprise Linux (RHEL), and Oracle Linux (OL). Each Operating System has costs, features and support implications that make it unique. You need to determine which is best for your environment. In the United States, RHEL is most popular, whereas in Europe SLES is most prevalent. Almost all of my experience is with RHEL or on downstream distributions (such as CentOS or OL) of it, but my biases shouldn’t have an impact on this evaluation. The file system for VMware’s vSphere ESX and vMA (vSphere Management Assistant) and many of the VMware appliances from EMC and PHD Virtual are RedHat/CentOS based. This shouldn’t be a deciding factor when deciding what OS for your database system, but this does come in useful in the event of the occasional esoteric troubleshooting situation.

With some minor exceptions, RHEL and OL are the same to operate — the files are almost entirely in the same location, the commands are the same, etc.

For the purpose of this evaluation, I am limiting my comparison to the latest two versions of the 64-bit x86 platform for each distribution and how they differ when run on VMware’s latest released version of vSphere (4.1U1). Partially this is being done to save me time and effort, but also these are the platforms you would decide between if you were looking to maximize database system performance.

Note: At the time of this writing, RHEL 6 and OL 6 were NOT certified for most Oracle products. This is due to the fact these versions are relatively recently released and Oracle is still certifying their products on the new versions. Also note that the VMware / SLES promotion is limited to SLES 11.

Cost:
Your main consideration here is whether you just want access to patches and updates or if you want actual support with your issues. In my nine years of running Oracle on Linux, I’ve had to open a total of two tickets on Linux support – once with RedHat, once with Oracle.

o SLES – If you’re running vSphere 4.0U2 or later and are active on qualifying VMware vSphere Software and Services SKUs, you can run an unlimited number of virtual machines and get free subscriptions to patches and updates of SLES 11 SP1. Phone and online support has varying levels and costs. You can read more about VMware’s SuSE agreement.
o OL – Oracle Linux is free to download in compiled form. If you want a subscription to patches and updates only, the cost is $119 per year per server for an unlimited number of physical CPUs. Phone and online support has varying levels and costs. You can read more about Oracle Linux in the Oracle Linux FAQ. You can also check out the Oracle Linux support pricing guide.
o RHEL – RedHat Linux can only be downloaded in compiled form with a subscription. The cheapest subscription is a Self-Support subscription which comes with a subscription to patches and updates and no other support for $349 per year per server, where each server is limited to a 2 socket configuration with 1 virtual guest. Phone and online support has varying levels and costs. You can check out the various support options and their cost on the Redhat website.

Features offered by VMware:
Do you want to use VMware features such as Paravirtualized SCSI (PVSCSI), Hot Add Memory or Hot Plug vCPUs? Do you have a specific requirement for Enhanced VMXNET Networking?

Not all the distributions and versions support all these features. For example, if your database workloads are very I/O intensive, SLES is probably not a good choice.

o SLES 10 – Networking: e1000, Enhanced VMXNET and VMXNET3 are supported. A standard install will default to e1000.
– Storage: PVSCSI is NOT supported
– Hot Add: Hot Add Memory supported, Hot Plug vCPUs NOT supported
o SLES 11 – Networking: e1000, Enhanced VMXNET and VMXNET3 are supported. A standard install will default to e1000.
– Storage: PVSCSI is NOT supported
– Hot Add: Hot Add Memory supported, Hot Plug vCPUs supported
o RHEL 5.6 – Networking: e1000, Enhanced VMXNET and VMXNET3 are supported. A standard install will default to e1000.
– Storage: PVCSCI is supported. PVSCSI is NOT supported on hard disk 1 of the virtual machines.
– Hot Add: Hot Add Memory supported, Hot Plug vCPUs NOT supported
o RHEL 6.0 – Networking: e1000, VMXNEXT3 supported. Enhanced VMXNET NOT supported. A standard install will default to VMXNET3.
– Storage: PVCSCSI is supported. A standard install will default to PVCSCI.
– Hot Add: Hot Add Memory supported, Hot Plug vCPUs supported
o OL 5.6 – Networking: e1000, Enhanced VMXNET and VMXNET3 are supported. A standard install will default to e1000.
– Storage: PVCSCI is supported. PVSCSI is NOT supported on hard disk 1 of the virtual machines.
– Hot Add: Hot Add Memory supported, Hot Plug vCPUs NOT supported
o OL 6.0 – Networking: e1000, VMXNEXT3 supported. Enhanced VMXNET NOT supported. A standard install will default to VMXNET3.
– Storage: PVCSCSI is supported. A standard install will default to PVCSCI.
– Hot Add: Hot Add Memory supported, Hot Plug vCPUs supported

Note: Previously OL 5.6 was listed in VMware’s certified list as NOT supporting Hot Add memory, but this has been changed recently.

Note: On OL, the new Unbreakable Enterprise Kernel (UEK) is NOT supported under VMware. You will have issues installing the VMware Tools if you are running this kernel.

Many companies standardize on one or two operating systems for their organization to minimize support costs. When bringing virtualization into the mix, your organization should re-evaluate your operating system choices to to get the performance and features you need.