Oracle versus RedHat and VMware

As I wrote earlier, there are many things to consider about when deciding what distribution is best for your environment when running Oracle under VMware. Over the past couple of years there’s been a quiet battle with Oracle on one side and RedHat with VMware on the other. As a customer, your dollars fund the game. Your choices will help determine who wins this battle.

First up, Oracle

In Oracle Database 11g, Oracle introduced a new features called Database Smart Flash Cache. The feature leverages any flash storage (aka Solid State Drives (SSDs) or Enterprise Flash Drives (EFDs)) on the system to act a second level cache behind RAM, increasing the database buffer cache without adding RAM to the system. There was a recent post on this topic on Oracle’s Linux blog which links to an interesting white paper if you want more detail on the inner workings of this feature.

Sounds great right? Here’s the thing: It’s only available on Oracle Solaris or Oracle Linux. Now Oracle Solaris on SPARC processors I could see – it’s a Big-Endian Unix whereas x86/x86-64 Linux is Little-Endian OS and Solaris on SPARC is a much more mature OS that Linux.

Oracle Linux is binary compatible with RedHat Enterprise Linux – so there is no technical reason why Database Smart Flash Cache shouldn’t work. I’m not the first to point this out. Dave Welch of House of Brick Technologies wrote about this in October 2009. I suggest you read his eloquent blog post on the subject. Note that this was all before the Oracle UEK Kernel was released and nothing in Oracle’s documentation states that UEK is required for Database Smart Flash Cache.

In September of 2010 at Oracle OpenWorld 2010, Oracle announced Oracle Unbreakable Enterprise Kernel (UEK). As you can read in this datasheet on the UEK,the big selling points are that it’s a modern 2.6.32 linux kernel that shows “tremendous performance improvements” compared to a standard Enterprise Linux 5 kernel (which is a 2.6.18 kernel). Amazing right? Not really. RHEL 5 was released in March of 2007 and hits the end of Production 1 life cycle in Q4 of 2011. RHEL 6 was released in November of 2010 is also a modern 2.6.32 linux kernel. Key benefits of UEK according to Oracle’s FAQ are that it’s fast, modern, reliable and optimized for Oracle. RHEL 6 is fast modern and reliable too. Yes, the UEK does have some optimizations and bug fixes for Oracle but they or may not be relevant in your environment.

So why does Oracle beat up on RedHat? It’s about money. Oracle wants that OS support revenue.

How is Oracle beating up on VMware? It’s also about money. Virtualizing Oracle allows you to run more Oracle servers on the same hardware. That’s lost revenue to Oracle, unless you’re using their hardware (Exadata) or software (OracleVM). Oracle tries to discourage VMware use with Oracle’s infamous supported but not certified argument against VMware. There is also Oracle’s policy of having to license all the cores on a host regardless of how many cores your VM can see (unless you’re using Oracle’s virtualization solution of course). Here’s another one I came across recently: In the release notes for Oracle Linux 6 is this little nugget:

Unbreakable Enterprise Kernel doesn’t contain vmw_pvscsi driver (11697522)
As a workaround, when creating a new VM in VSphere, do not pick Red Hat Enterprise Linux 6 x86-64 as the OS type but use Oracle Linux 5 x86-64 (or Red Hat Enterprise Linux 5). ESX will then expose the LSI Logic SCSI controller in the VM and the 2.6.32-100.28.* kernel will see the devices properly.

I did a default install of RHEL 6 (which uses the RedHat Kernel instead of Oracle Kernel) and lo and behold, RHEL 6 automatically uses PVSCSI to get you better disk performance under VMware. Oracle’s Unbreakable Kernel is based off of RedHat’s Kernel. That means Oracle went out of it’s way to cripple performance by removing PVSCSI support when using their kernel under VMware. Oracle’s own virtualization product OracleVM doesn’t have any sort of PVSCSI enhancements, so what better way to erase the performance benefits you’d see virtualizing under VMware? As a customer, I find this behavior deplorable.

Next Up, RedHat

With the release of RedHat Enterprise Linux 6, RedHat stopped providing the source of a vanilla kernel and all their patches in different packages and now provides this all in one large tarball. It’s totally legal by the requirements of the GPL, but it makes it much more difficult for downstream vendors (such as Oracle) to figure out what optimizations RedHat has made to the vanilla kernel and to then incorporate those changes into their own customized kernel. Now why would RedHat do this? Let’s see what Brian Stevens, CTO, VP Worldwide Engineering for RedHat had to say in this press release

“When we released RHEL 6 approximately four months ago, we changed the release of the kernel package to have all our patches pre-applied. Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.

Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.”

Who is the number one seller of RHEL support behind RedHat? Oracle.

Finally, VMware

VMware has done a few things I can think of to fight back against Oracle:
1) Quite simply, Oracle’s UEK (Unbreakable Enterprise Kernel) is NOT supported under VMware. Yes, I’m sure it requires resources to test yet another kernel, but I don’t believe that’s the case here. Could it be because of the “optimizations” Oracle has made such as the deadline scheduler (not typically good when used under a hypervisor) or stripping PVSCSI support out of the UEK? Maybe.

2) Up until the last few weeks, Hot Memory Add wasn’t listed as supported with Oracle Enterprise Linux 5.X under VMware according to VMware’s Guest OS Compatibility List, even though RedHat Enterprise Linux (on which OEL is based) was supported. Maybe it was an innocuous typo … or maybe not.

3) In November 2010, Oracle changed their support policy to explicitly INCLUDE Oracle RAC 11.2.0.2 when virtualized. Before that time, Oracle RAC was explicitly NOT supported by Oracle under VMware. Oracle’s yearly conference, Oracle OpenWorld, was held in September – before the change in the Oracle RAC on VMware support policy – and VMware had a booth there. I attended a number of sessions at that booth and at least two of them (one by Todd Muirhead of VMware, one by David Welch of House of Brick Technologies) talked about running Oracle RAC virtualized under VMware. Both presenters made it very clear that this wasn’t supported by Oracle, but one has to wonder why VMware would take the time to talk about a solution that wasn’t supported by Oracle? Maybe they knew Oracle was going to change their support policy and were just letting customers know it could be done “if only Oracle would support it”… or maybe VMware was preparing a more offensive role regarding Oracle RAC on VMware. I don’t know, but it’s interesting to think about.

So what’s a customer to do? To me the answer is simple: If you’re going to run Oracle virtualized, I’d highly recommend running it on RHEL Linux on VMware and buying my support for RHEL Linux from RedHat. If you buy support for RHEL from Oracle, assuming the support quality level is the same as RedHat’s, you’re helping Oracle to stifle true innovation that benefits everyone. If you decide to run OL, not only are you helping Oracle to stifle innovation that benefits everyone, you’re also telling Oracle you’re fine with them limiting features to customers (such as Smart Flash Cache) entirely for anti-competitive marketing reasons.

Don’t reward bad behavior.

12 thoughts on “Oracle versus RedHat and VMware

  1. Hi.

    Nice post.

    Not sure I agree 100% with your opinions on Oracle Linux, but I understand your reasoning and it’s certainly worth saying.

    Regarding the RAC on VMware issue, I don’t think those sessions at last years OOW were as a result of any pending support statement, although it adds some legitimacy to stance. VMware have been promoting VMware RAC for a few years. I think it was about 4 years ago they asked me to write the 10g and 11g RAC on ESX articles. At the time I was only doing VMware Server RAC installations, so they sent me a bunch of licenses and asked me to repeat the articles but using ESX instead. That year they had VMware RAC talks on their OOW stand also.

    Regarding the kernel version issue, I agree that if you were running RHEL6 you would have many of the benefits of OL5.6+UEK, but Oracle’s products are not certified against RHEL6 (or OL6) yet. You can claim something underhand in this, but if try a RAC install on RHEL6 it is a complete pain, mostly because the change in inittab behavior between RHEL5 and RHEL6. It wouldn’t surprise me if they didn’t support RHEL6/OL6 until Oracle 12.x. If they do support 11gR2 RAC on RHEL6/OL6, they will need some changes to the installation process compared to that for RHEL5. With this in mind, OL5+UEK is a very nice option, provided you are not using VMware. :)

    Keep up the good work. Really enjoying you posts.

    Cheers

    Tim…

  2. A couple of remarks:

    * The licensing policy for Oracle on OVM is the same as on VMWare.
    You need to license the number of cores in the physical servers, using the max(nbr cores in a server) * min(physical servers,virtual guests with Oracle) formula.

    * ADVM / ACFS is not supported when using the UEK kernel, only when using the RHEL kernel.
    Are they now benefiting Redhat, or is there a good technical reason for this?

    * Neither Redhat 6 nor OL 6 are supported for Oracle databases.

    I do agree that Oracle has a tendency to spread FUD about running Oracle databases on VMWare.

    Question: how relevant is the PVSCSI support when running Oracle databases under VMWare? Are you normally not using ISCSI / NFS volumes directly from the guest or RDM volumes?

  3. Hi Freek!
    I admit I may be misunderstanding your formula for licensing Oracle on OVM, but I can state licensing for Oracle on Oracle VM is definitely different from licensing Oracle on VMware. If you check out my earlier post on Licensing Oracle on VMware vSphere, I reference the links to Oracle’s partitioning document which says something to the effect (I’m not legally allowed to quote according to the legalese at the bottom of every page) that Oracle VM is considered a valid type of hard partitioning when you follow the instructions in http://www.oracle.com/technology/tech/virtualization/pdf/ovm-hardpart.pdf … and therefore if you’re running Oracle guest VMs using 2 cores on a 8 core physical host running Oracle VM, you only need to pay Oracle for 2 cores worth of licenses … versus the same setup using VMware requiring you to pay Oracle for 8 cores worth of licenses. You can do the same type of CPU pinning with vSphere that Oracle allows with OracleVM, but for some reason beyond my understanding, Oracle doesn’t count that towards as a valid form of hard partitioning.

    ADVM / ACFS – I haven’t heard anything about ASM dynamic volume manager or ASM cluster file system not being supported when using a UEK kernel… can you send me some links on this so I can do my own research?

    Yes, you are correct, RHEL 6 / OL 6 is (still) not supported for Oracle DBs – I’ve got to think they’ll do that one of these days. You might want to check out Kevin Closson’s comment on my “On Oracles commitment to Linux” post a few days ago for his (and to a lesser degree, the oaktable private list) thoughts on the matter.

    Regarding PVSCSI, you may want to check out http://www.vmware.com/pdf/vsp_4_pvscsi_perf.pdf which is a performance white paper from VMware on the performance benefits on PVSCSI vs. LSI Logic SCSI adapters. You should note that white paper was based on vSphere 4.0 and PVSCSI in vSphere 4.0 could actually decrease performance compared to LSI if the VM is pushing less than 2000 IOPS consistently. This was rectified in vSphere 4.1 .

    I personally don’t use RDM volumes unless some clustering solution (for example a quorum drive with microsoft clustering) requires it. VMware and EMC both say there isn’t a performance advantage and the disadvantages (no snapshots, no thin provisioning, etc) outweigh any possible benefits, IMHO. So my iSCSI/NFS/FC VMware datastores have the VMDKs of my Oracle VMs on them, using a PVSCSI SCSI adapter if I’m running vSphere 4.1 or higher.

  4. I was unaware of the issues regarding RHEL 6 / OL 6 with RAC and your idea that Oracle may not support RHEL6/OL6 under Oracle DB 12.X matches up with Kevin Closson’s thoughts as well. If that’s Oracle’s reason, why not just state RHEL 6 / OL 6 with 11g is supported but only when not running RAC ? I’m unfamiliar with the actual changes made to inittab between 5 and 6 but since Oracle already makes an oracle-validated rpm, it seems they could possibly include a customized iniitab in a new (possibly RAC-specific) oracle-validated rpm.

    I rarely touch RAC these days, but it is on my to-do list to get more familiar with it.

  5. Excellent article.

    You can also add ASMlib to the list of thing oracle will no longer support under RHEL6.

    Although UDEV do replaces ASMlib for most installations, we still need ASMlib to get rid of stale file handles problems left by the RDBMS.

  6. Hi,

    Thanks for the pointers to the articles.

    You can find the statement about ACFS / ADVM not supporting the UEK kernel in the release notes and on Oracle support notes:

    http://download.oracle.com/docs/cd/E11882_01/relnotes.112/e16778/toc.htm#sthref1
    mos note: Certification Information for Oracle Database on Linux x86-64 [ID 1304727.1]

    You are right about the hard partitioning, but when used you can no longer make use of live migration and after a failover you need to manually pin the cores again.
    In my opinion this severely limits the benifits of using virtualization.

  7. Jean-Francois,

    I have a support case open about oracle processes keeping file handles on dropped asm disks. This seems to be a bug that they don’t get fixed (even after several patches)

    But I dont understand how the asmlib helps you to solve this?
    The rdbms processes are still going to access the block devices directly.

  8. Hello, It is a good read, thanks for insight. We are looking for deploy Oracle HRMS R12 on intel box. I need the comparison of LINUX vs SOLARIS on intel box, which platform to go?

    Anyone please give some lead.

    regards

  9. Please check certify on My Oracle Support – I’m pretty sure that Solaris on Intel isn’t a supported platform for Oracle Applications R12.

    Even if it is, I don’t know of anyone running a production Oracle system on Solaris x86 – Solaris on Sun/Oracle Hardware yes, but not Solaris on Intel.

    My personal recommendation these days is to go with Nahelm class or better x86 CPUs running RedHat Enterprise Linux under VMware vSphere.

    Good luck!

  10. Hi J, I just recently stumbled over this blog post and wanted to clarify two things, if you don’t mind.

    You wrote: “Oracle’s Unbreakable Kernel is based off of RedHat’s Kernel.” – this is actually not the case. The Unbreakable Enterprise Kernel is derived from a recent version of mainline Linux, we try to minimize the number of patches we apply on top of it. The UEK will be updated to a new version from mainline every 12-18 months and is fully maintained and supported by Oracle.

    With regards to the missing vmv_pvscsi driver – this was indeed a bug (as noted in the release notes) that was addressed in later errata releases of the Unbreakable Enterprise Kernel ( ELSA-2011-2010 for OL5/6: http://oss.oracle.com/pipermail/el-errata/2011-March/002008.html and http://oss.oracle.com/pipermail/el-errata/2011-March/002014.html)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>