Tag Archives: vsphere

Which would your CEO prefer

Uncategorized, , , ,

Would your CEO choose to give up a negligible amount of system performance at peak system load in exchange for a reduction of risk during system upgrades and an alternative to hours of downtime?

I suspect the answer is yes.

What’s the cost of such a technology? Believe it or not, it’s free. It’s the snapshot functionality that’s built into VMware’s vSphere Hypervisor (based on ESXi) that you can leverage once you virtualize your Oracle database or any other x86 systems. Is the performance impact truly negligible? Don’t take it from me; you can read it in Oracle’s own words.

Recently RedHat and Oracle have come out with updates to their mainstream distributions (RedHat Enterprise Linux 5.6 and Oracle Linux 5.6) and each has come out with an entirely new version (RedHat Enterprise Linux 6 and Oracle Linux 6). System and database administrators all around the world are updating their critical systems. Applying those updates to critical systems requires testing and downtime.

The best practice for operating system upgrades is to test the upgrade on the same exact hardware, software and data as in production. But are your test and production systems identical? Same motherboard? Same processors? Same amount and brand of RAM and same exact operating system packages installed? Pretty unlikely. With virtualization, all those configurations are identical.

With VMware virtualization, I can take take a clone of the entire production server while it’s live and being used by users. Now I have a truly identical copy of production to test my upgrade. Note that to do a clone of the entire production server while it’s live requires VMware vCenter which isn’t free, but vSphere Essentials (which includes vCenter) starts at $1000 as of the time of this writing.

With snapshots (which are free and don’t require vCenter), you take a snapshot (3 mouse clicks and less than 10 seconds) of the virtual machine and then do the upgrade. If you run into issues, just revert / rollback the snapshot to the state it was in when you took the snapshot. The time to do that revert / rollback is only the time necessary to reboot your virtual machine – 5 minutes? If the upgrade and testing goes smoothly, you just merge the snapshot into the virtual machine while the system is up and available to users. Total time spent doing non-upgrade activities such as backups or restores? Essentially zero.

Compare this to how you would handle a critical system without virtualization: you’d take the system offline for the upgrade, take a full backup of the system, do the upgrade (hoping it works just like it did in the similar but most likely not identical test system), have the users test, and, if there’s an issue, possibly spend hours restoring from backup. You do trust your backups… right?

Enterprise DBAs and the companies that employ them tend to be risk adverse when it comes to losing data or experiencing downtime. Virtualizing the hardware allows you to ensure your test and development systems are the exact same systems as production, thereby reducing risk of unforeseen issues during the upgrade. Utilizing snapshots allows you to very quickly take a save point of your entire server and rollback to it very quickly in the event of unforeseen issue, almost entirely eliminating (minutes vs. hours) the downtime associated with recovering from an unforeseen issue.

Licensing Oracle on VMware vSphere

Uncategorized, , , ,

Honestly, I thought this issue was done and buried, but over the past few weeks I’ve seen this question come up multiple times, so let’s get this cleared up.

Let’s go right to the source – Oracle’s own documentation. If you read Oracle’s partitioning document you will see that this is Oracle’s stance as of January 24, 2011. In it, it discusses soft partitioning and hard partitioning. Soft partitioning is leveraging the Operating System features to limit the number of CPUs an Oracle instance (or Oracle virtual machine) can run on. Hard partitioning physically partitions a large server into smaller self contained servers. The document lists what Oracle considers valid examples of each type of partitioning. In the document, Oracle specifically defines VMware’s partitioning (and Oracle VM’s partitioning) as soft partitioning. In the document, Oracle states that soft partitioning isn’t a “valid” means of restricting the amount of software licenses and you must license all the processors on a given server. Note that later in the document Oracle states that Oracle VM CAN be used for hard partitioning if you set it up as described in this document which goes into detail on how to bind an OracleVM VM to physical processors / cores. There is no mention in the documents if binding a VMware VM to a physical processor/core would also count as hard partitioning. Oracle does state that their list of partitioning technologies isn’t comprehensive, so things are left open to interpretation.

Please note I highly recommend you go and read these documents yourself and draw your own conclusions, and of course you can and should talk with an Oracle-employed licensing expert. In these documents Oracle states I cannot reproduce the document in any manner without express written consent so I am only telling you my interpretation.

VMware has three different techniques for restricting a VM to a specific subset of processors / cores. They are VMware vCenter clusters, VMware DRS affinity rules, and vSphere CPU affinity (pinning). I advise my clients to use the VMware vCenter cluster technique, but your organization might have a different interpretation. To describe the different VMware techniques, I will use an example of a 10 host VMware vCenter datacenter, with each host having 2 physical sockets and 4 cores per socket. Therefore, this entire VMware vCenter datacenter has 80 physical x86 cores (4*2*10) of processing power.

VMware vCenter clusters are logical clusters inside of vCenter made up of one or more hosts. By assigning a VM to that cluster, you are forcing that VM to run ONLY on the host(s) that make up that cluster. For example, if you create a 2 host VMware vCenter cluster inside your 10 host VMware datacenter, your VM can run on any processors / cores inside that 2 host cluster. As a result, Oracle licensing requires you to license all 16 (4*2*2) cores in that cluster. Note that you are restricting other non-Oracle workloads from also running on these hosts, so your Oracle VMs will get the best possible performance available on those hosts, possibly at the detriment to your non-Oracle workloads running on other hosts.

In vSphere 4.1, a DRS rule called “Virtual Machines to hosts” became available. That rule allows you to limit the location of a VM to specific host(s) in the VMware vCenter cluster. For example, if you create a DRS affinity rule assigning a VM to a single host inside your VMware cluster, your VM can run on any processors / cores inside that host. As a result, Oracle licensing requires you to license all 8 (4*2*1) cores on that host. You can read more about the VM to hosts affinity rule in this post by Frank Denneman who is a co-author (along with Duncan Epping) of vSphere 4.1 HA and DRS technical deepdive. Note that you aren’t restricting other non-Oracle workloads from also running on this host and thus you could have less than optimal Oracle performance.

VMware vSphere itself allows you to pin a virtual machine to one or more physical cores on a server using vSphere’s CPU affinity settings. You can read about the details of this in the vSphere resource management guide version 4.1 starting on page 20. This is the technical equivalent of the Oracle VM technique of binding a VM to a specific subset of physical processors / cores. For example, if you pin your Oracle VM to two physical cores, your VM can only run on those two physical cores. As a result, Oracle licensing requires you to license those 2 cores. Note that you aren’t restricting other non-Oracle workloads from also running on this core and thus you could have less than optimal Oracle performance.

Does Oracle consider VMware’s CPU affinity settings an acceptable form of partitioning? What about VMware DRS VM to host affinity rules? I have seen no official documentation from Oracle either way. I advise my clients to always buy enough Oracle licenses to allow Oracle to run on at least two hosts. This allows the customer to not be concerned about Oracle’s licensing ambiguity (as you’re licensing the entire hosts Oracle can run on) and also allows the customer to get the benefits of VMware such as vMotion, HA, DRS and FT to reduce and possibly eliminate downtime or less than optimal performance for their Oracle systems. I have had a client who went from running Oracle physical (with the one physical server having 8 processors / cores) to virtual (with the physical server having 8 processors / cores) and the client wanting the benefits of vMotion, HA, DRS and FT but without having to buy Oracle licenses for an additional 8 CPU host. According to the Oracle partitioning document I referenced earlier, Oracle does allow customers to only licenses processors / cores that are turned on. For this customer I therefore recommended that they turn off half the processors / cores in each host. Please note this limited their VMs to a maximum of 4 cores each- the amount of cores available on each host.

Licensing Oracle on VMware vSphere is an area of much confusion and disagreement due to Oracle not presenting clear public guidelines on whether DRS Affinity rules or vSphere CPU affinity are valid methods of partitioning.  I hope that Oracle addresses this licensing confusion soon, but until then, separate VMware vCenter clusters are the least legally risky way to virtualize Oracle.  I would love for someone from Oracle to officially and on the record address the techniques I mentioned in this post.

Oracle listened, customers WIN! RAC supported on VMware

vmware, , , , , , , ,

As I was flying home last night and downloading tweets before takeoff, I found out some amazing news. Ugh, not the time to have intermittent internet access! But eventually I got home, did the reading and confirmed the news.

Oracle RAC 11gR2 (11.2.0.2) is now supported by Oracle under VMware.

You can read the updated My Oracle Support (MOS) announcement yourself in note 249212.1 which now states:

NOTE:  Oracle has not certified any of its products on VMware.
For Oracle RAC, Oracle will only accept Service Requests as described in this note on
Oracle RAC 11.2.0.2 and later releases.


(Remember:  Certified is different than Supported .  Oracle doesn't certify hardware that isn't Oracle's own )


This is simply fantastic news.  I talked to an petroleum company in Houston earlier this year who wanted to virtualize their Oracle EBS system and move platforms from Sun Solaris to x86 architecture.  Their big concern was that they were using 8 SPARC Processors and they knew that 8 x86 CPUs is the limit for a virtual machine under VMware vSphere 4.1.  We discussed various steps they could take to ensure their environment would thrive under this limitation, but now it's a non-issue. In the event they need more computing power, they can implement Oracle RAC under vSphere and start up another RAC instance as necessary. 

I do need to point out that as of this moment, 11.2.0.2 database is not certified or supported with Oracle Application (Oracle EBS) 11i or R12.  These certifications usually come out a few months after the initial database announcement (which was Sept 10th for 11.2.0.2).  If you check out the blog of Steven Chan (a Senior Director in Oracle's Applications Technology Group - the group responsible for the Oracle E-Business Suite technology stack) and specifically these comments , you'll see that Steven wrote:


We haven't certified 11.2.0.2 with Oracle E-Business Suite Release
11i yet.  This project is underway now.  11.2.0.1 is the latest
certified database release for the E-Business Suite.
Oracle's Revenue Recognition rules prohibit us from discussing
certification and release dates, but you're welcome to monitor or
subscribe to this blog for updates, which I'll post as soon as soon as
they're available.





So 11.2.0.2 database certification with EBS 11i and R12 is coming.My main client doesn't use RAC (our business can survive the downtime associated with a HA event and we aren't near the 8 CPU limitation of VMware vSphere 4.1), but knowing its an option can only give upper management even more confidence that virtualizing our entire Oracle environment under VMware vSphere was the right thing to do.






For those wanting more information on Oracle RAC under VMware vSphere, I'd suggest watching this Oracle virtualization webcast put on by Embarcadero and VMware a few weeks ago. I'd also highly recommend following VirtualTodd on Twitter.  Todd Muirhead was at Oracle OpenWorld in the VMware booth and presented some very interesting performance data from running RAC under VMware.  I can't find a link to the presentation, but you can follow Todd's postings and perhaps find his testing results at his blog on the VMware communities site .



Think of the possibilities of combined Oracle RAC and VMware vSphere:



o  Multiple RAC nodes on different vSphere hosts means no database downtime during a hardware failure.



o  Combining multiple RAC databases on same vSphere host to consolidate workloads but still segregate environments 



o  Much faster provisioning of new RAC nodes with vSphere virtual machine cloning and VMware VAAI (vStorage APIs for Array integration)

o ... and many more I still need to wrap my head around

Webinars of interest to Oracle Apps DBAs on RedHat Linux with VMware

vmware, , , , , , , ,

So recently I’ve been getting notifications about a number of various interesting Webinars. Since they’re coming from all sorts of random sources, I’m sharing the links as a service to my readers.

I’m receiving no consideration or such from the companies mentioned, these are things I thought would be of interest to me, and perhaps my readers

—

Upcoming (all dates / times are based on Central US time zone)

Get The Facts: Oracle’s Unbreakable Enterprise Kernel for Linux Oct 26th 11am, put on by Oracle

EBS Workflow Purging – Best Practices Nov 10th 11am, put on by Oracle

How to use My Oracle Support for ATG issues Oct 27th 11am, put on by Oracle

E-Business Suite using a DB-Tier with RAC Oct 28th 11am, put on by Oracle

Top 10 Virtual Storage Mistakes Oct 28th 1pm, put on by Quest Software

On-Demand (already happened)

Oracle Virtualization Webcast put on by Embarcadero and VMware

RHCE Virtual Loopback: Unlocking the value of the Cloud put on by RedHat (pdf)

RHCE Virtual Loopback: Performance and Scalability RHEL5 -> RHEL6 put on by RedHat (pdf)

Managing Red Hat Enterprise Linux in an Increasingly Virtual World put on by RedHat

State of “Btrfs” File System for Linux put on by Oracle

Lower Your Storage Costs with Oracle Database 11g and Compression put on by Oracle

Linux Configuration and Diagnostic Tips & Tricks put on by Oracle

—

Feel free to share any other Seminars you think may be of use in the comments!

Oracle internal cloud session updates from VMworld Day 1

Uncategorized, , , ,

This week I’m at VMware VMworld in San Francisco. Yesterday was day one of the event and the Oracle related highlight for me was session

EA7061 Creating an Internal Oracle Database Cloud Using vSphere by Jeff Browning of EMC.

I’ve been to Jeff’s sessions before and always found them entertaining and informative. Below are some of my thoughts from what was covered at the session.

The most striking informative graphic was an X-Y graph where the X axis was scalability and Y was availability. At the high end of both were Oracle RAC. At the low end of both was MS Access and MySQL. In the sweet spot was Oracle standard edition coupled with VMware vSphere HA clusters.

What does this say to the DBAs? What many of us already knew – not every workload is appropriate for being virtualized under VMware. If your system or the business it’s supporting cannot survive the downtime you’d have in the event of a host failure and subsequent HA restart, you should spend the $$ for Oracle RAC. However, Jeff pointed out that in his experience roughly 90% of systems can survive the downtime associated with a HA event – that’s 90% of the databases out there being good candidates for virtualizing Oracle under VMware vSphere.

One of Jeff’s great examples of why to virtualize was to reduce database sprawl. He cited a Fortune 100 company with 824 physically booted Oracle databases and they pay $228 Million a year to support those machines.

To reduce this sprawl, you’ve got two approaches – according to Jeff, Oracle’s preferred way is to use RAC and come up with one global instance where you can put all your various products. Unfortunately that just doesn’t strike me as realistic in any sort of large company. I run primarily Oracle’s own products and even they can’t run on the same database version in many cases. Oracle E-Business requires Oracle 10g or Oracle 11gR2. Yet Oracle Email Center requires an Oracle 9i database (which needs RedHat 4). A global RAC instance just doesn’t make sense.

The other approach is to virtualize the machines – now I’ve got a RedHat 4 32-bit OS machine running Oracle 9i database on the same hardware as a RedHat 5 64-bit OS running a 11gR2 database. There’s lots of cost savings on both Oracle licensing and reducing the amount of hardware that one can gain with this approach.

One thing I hadn’t really thought about that Jeff brought up with regards to VMware vSphere and Oracle is that the time to vMotion your Oracle database can be longer than with other types of virtual machines – sometimes taking as long as twenty minutes. The reason for this has to deal with how vMotion works – its basically copying the contents of RAM for that VM to another server and then copying over memory blocks that have changed since the first copy, over and over till the delta is very small. Oracle heavily uses memory for its SGA (System Global Area) and so for heavy transaction OLTP systems, vMotions can take a longer than expected time.

The final thing I want to share from Jeff’s presentation was the relevant performance of different protocols and file systems with regards to Oracle and VMware. On the NAS (NFS) storage side, Jeff assigned a value of 95% efficiency when accessing database datafiles via Oracle Direct NFS (DNFS) offering. Compare this to 65% efficiency running VMDKs over traditional NFS. That’s a huge performance difference. As a result, Jeff recommends just using this for a boot / OS disk and definitely not for your database files. On the SAN side, Jeff noted the best performance (100% relative efficiency) comes from using Oracle’s Automatic Storage Management (ASM) with VMWare Raw Disk Mapping (RDM) containers. Compare this with a 98% efficiency with ASM using VMware Virtual Machine Disk Format (VMDK) containers. This is another example of how the Oracle DBAs need to communicate with the VMware administrators when planning out their environment. Many times DBAs don’t even realize they’re running in a virtual environment, and you can’t expect a VMware admin to know about the performance benefits of Oracle DNFS or ASM.

Overall it was a great session and I’m definitely looking forward to applying what I learned to my environments when I get back home.

Too big to fail: Virtualizing Big Iron databases

vmware, , ,

Recently I was talking with a company in Houston,TX running Oracle E-Business (EBS) 11i on Solaris with an Oracle 9i database. They run VMware for other parts of their business and wanted to leverage the features of VMware vSphere and Site Recovery Manager (SRM) to virtualize their EBS environment and have the ability to quickly move their EBS environment in the event of a hurricane or other natural disaster bearing down on Houston to their geographically diverse DR site.

This call had lots of moving parts. They were on Oracle 9i database and wanted to move to Oracle 11g to ease their support costs. They wanted to move from Solaris hardware to commodity x86 hardware and RedHat linux to ease their support costs. Their existing Oracle 9i database was running on and using 8 SPARC processors at peak levels throughout the business day.

in VMware vSphere, the maximum processors you can make visible to a VM is 8 virtual x86 processors. Is a virtualized x86 processor as fast as a physical SPARC processor? Would their SQL run faster on Oracle 11g than it did on Oracle 9i? Is RedHat Linux going to allow the database to process requests as fast as Solaris? Will their SAN storage and LUN layout be fast enough? Will their file system be a limiter?

Besides building up the environment and just going for it in production, how can you know?

By leveraging some very cool tools from both Oracle and VMware.

For Oracle database, Oracle offers an add-on called Real Application Testing which has a feature called Database Replay. Database replay allows you to capture the workload on your production database server and replay it on another environment to see if things are faster or slower. Although this was a new feature on Oracle 11g database, Oracle made backports of this available for 9i and 10g databases – exactly for purposes like this (well, maybe not to aid in virtualizing to VMware, but you get my drift).

Using Database Reply and Real Application Testing (both licensable features from Oracle Enterprise Edition) allows companies to test SAN changes, hardware changes, database upgrades, OS changes, etc., all with a production load, but without risking actual production issues.

Where does VMware fit into this? The way Real Application Testing and Database Reply work is by capturing all the transactions generated in production, massaging them a little bit, and then playing them back against a clone of Production. That clone needs to be at the exact same point in time (or SCN – System Change Number in database speak) as PROD so that the replay is playing back against an exact replica of the database. Although setting up a clone to an exact isn’t hard for an Oracle DBA, it does require time – time to build the test system, time to restore a backup of the database and time to apply archive logs and roll the database forward to match PROD’s SCN.

Even in cases such as this where the Production database isn’t virtualized, by making the test system virtualized, we can not only test all these changes, but we can leverage VMware snapshot technology to allow us to very quickly take our database back to the SCN we want to run Database Replay against, without having to continually restore the database. Using snapshots you just go thru that setup effort once, take a snapshot and then just keep rolling back to your snapshot as many times as necessary to test performance.

Of course, you may find that the 8 processor limit in VMware or the OS or the SAN can’t handle your production load. Time to give up and stay physical? No. In Oracle 10g and further refined in Oracle 11g, Oracle has greatly improved the ability the database has to help a DBA manage the system load and even to have the database tune itself. By leveraging features such as Advanced Compression and SecureFiles (to reduce the physical I/O), Automatic Optimizer Statistics Collection and Automatic SQL Tuning Advisor (to tune queries to use less CPU and/or disk resources), you can give your database more room to grow yet still stay on the same (or less!) hardware.

Oracle VM compared to VMware vSphere: Part 1

Uncategorized, , ,

I’ve been meaning to take a serious look at Oracle VM for a few months. In fact, it was this post [Live Migration of EBS Services Using Oracle VM] (and my long-winded reply) that was a major push for me to start this blog.

The final bit of impetus to learn all about Oracle VM came a few months ago when I saw the “Oracle VM for x86 Essentials” beta exam. If passed, you earn the certification “Oracle VM for x86 Certified Implementation Specialist”. It’s a certification geared for Oracle Partners. I figured the knowledge could help me to better understand Oracle’s offering. First and foremost, I’m an Oracle Applications DBA. If Oracle’s product could allow me to better serve my clients and do my job – awesome!

So I’ve been hitting all the Oracle VM resources I could find to learn about the product. I’ll post links to a number of the excellent resources I found at the end of this post. All the links at the bottom refer to information on the currently available product (Oracle VM 2.2). While compiling all of this information, I came across [Oracle Virtualization:Making Software Easier to Deploy, Manage, and Support] – a slide deck from a recent Sydney Australia Oracle meetup. It talks about upcoming features of Oracle VM 3.0. If those features come to pass, Oracle VM will become more enticing to many organizations.

Honestly, I’ve got *tons* of things I want to write about with regards to Oracle VM — so much that I don’t know where to begin.

General Impressions
Remember that first time you went from something with a nice GUI, like Windows (Thanks Apple Microsoft!) to something a little more “nerdy” like Linux ? The GUI, if there was one, was stripped down and clunky. Many of the things you could do with a couple of mouse clicks before now require specialized commands at a command line. All of these different steps you need to do just to get things working. Well, it’s the same type of thing going from VMware vCenter to Oracle VM Manager. It’s not that the product is bad — it isn’t. The Oracle VM interface is clunky and the product doesn’t have the richness of features of VMware vSphere. Simple as that. Are those differences worth it to you? Everyone’s needs are different. Both underlying products (Oracle VM Server, VMware vSphere ESX 4.0) run Linux and Windows VMs well enough for most enterprise-level systems.

As you can read in this Gartner report on Server Virtualization Infrastructure, VMware is the clear market leader. Oracle VM, although categorized as a niche player, is the strongest of the niche players and right on the border of being listed as a challenger to VMware.

Here are a few areas where Oracle VM has an advantage over VMware:

o Certified vs. Supported

(I hate talking about this but it needs to be addressed.) Is your VMware virtualized Oracle database supported by Oracle? YES. Is it Certified? No. I went into this in detail in this [Oracle Support on VMware] blog post so I won’t do it again. Short of running Oracle RAC, which is expressly NOT supported when virtualized under VMware, the question of whether you should care about the “certified” distinction is something each company needs to answer for themselves. To me, the whole thing smacks of FUD (Fear, Uncertainty, Doubt).

o Pricing

There are two parts to pricing. First is the effect virtualizing Oracle Database will have on your Oracle database licensing. I go into this in more detail in a post on Oracle licensing under VMware. One of Oracle VM’s main selling points is that Oracle considers Oracle VM (through hardcoding the CPU binding in the vm.cfg file) a type of hard partitioning and VMware vSphere a type of soft partitioning. When using hard partitioning, Oracle only requires you to license the processors (cores) in that hard partition (aka, the processors visible to the VM). When using soft partitioning, Oracle requires you to license ALL the processors (cores) in the server, even though there may be many more processors present than allocated to the VM. It should be noted that you can do the same type of CPU binding (called CPU affinity) with VMware vSphere, but that Oracle somehow still considers this soft partitioning.

This just seems like a way for Oracle to give their Oracle VM product preferential treatment. How does the joke go… Where does the 800 lb gorilla sit? Anywhere he wants to.

The second part of pricing involves the actual Oracle VM product versus the VMware vSphere product. Oracle basically has two pricing points
o Premier Limited — Up to 2 CPU sockets, regardless of the number of cores per socket in the physical server
o Premier — unlimited CPU sockets in the server

VMware, unlike Oracle, has four product feature levels (Standard, Advanced, Enterprise and Enterprise Plus) and so a head to head comparison is a complete pain to do. The short answer is that Oracle can be significantly cheaper.. The downside of this inexpensiveness is a lack of features. Yes, VMware generally costs more than Oracle, but you’re paying for additional features. Are those features worth it to your organization? That’s for you to decide. In my organization, we are willing to pay for VMware’s features, but my organization’s needs may be different than the needs of your organization.

Does your organization have a need for VM snapshots? Mine absolutely does. Oracle VM doesn’t have it and VMware does, even when you’re using the free version of each product.

Does your VM require more than 8 CPUs? VMware has a limit of 8 CPUs for a single VM. Oracle VM’s limit is 32 CPUs for a single VM. Through tuning and software improvements, my main client has managed to reduce the number of CPUs for our Production Oracle E-Business Suite database from an unvirtualized 8 cores to 2 cores virtualized, so the difference is immaterial… but maybe your organization needs 20 cores.

Does your organization have a need to do vMotions / Live Migrations? They come included with Oracle VM, but it’s not recommended to do more than one at a time. There is an additional cost to get VMware vMotion, but VMware supports a default configuration with up to 4 simultaneous moves and allows up to 8 simultaneous moves.

Does your organization need automated SAN level replication of your VMs so they can be brought up automatically in case of disaster? VMware has that functionality with Site Recovery Manager. Oracle doesn’t have anything like it.

o Oracle VM Templates

Do you want a pre-built VM you can download with the Oracle software already installed and configured? Oracle offers downloads of pre-built environments from a basic OEL 5 Linux box all the way through to a downloadable 38GB Oracle EBS R12.1.1 system. I admit, that could be pretty cool. However, it may not be right for your company. My main client never allows consultants to have console type access to our Linux servers. I don’t think my auditors would approve of a pre-built VM for production use, even if it was pre-built by Oracle. As something for quickly throwing up a demo or dev environment, I think it’s fantastic. I hope Oracle continues to do this for more and more of their products. Oracle Enterprise Manager 10gR5 took me roughly 2 weeks to install. Discoverer 11g about a week. Secure Enterprise Search about 2 weeks. It would be great to have a pre-built test system I could reference when building my production systems.

I’ve got numerous ideas for more blog posts with regards to Oracle VM. Feedback directing me to what interests others would be great.

Part 2 coming after the I take the exam later this week. Wish me luck!

Links
Live Migration of EBS Services with Oracle VM
Installing & Configuring OEL 5 with Database 11gR1 as a Paravirtualized Machine (PVM) on an Oracle VM Server
The underground Oracle VM Manual
Official Oracle VM Wiki home page
Oracle VM for x86 Essentials Exam 1Z0-540 Exam Topics Study Guide
Oracle VM 2.2 Documentation Library
Performing Physical to Virtual (P2V) and Virtual to Virtual (V2V) (aka VMware to Oracle) conversions Note the excellent pdf linked from the article too.
Installing, Configuring and Using Oracle VM Server for x86

Why run Oracle E-Business on vSphere?

Uncategorized, , ,

If I’ve got the dedicated hardware for my Oracle E-Business (EBS) install, why would I want to run it virtualized?

Most of the places I’ve seen typically have dedicated hardware for the Oracle EBS instances – so why virtualize it at all?

One major reason is the ability to rollback the entire environment in the event of a patching issue. My main client has an upcoming upgrade involving a database upgrades (11gR1 to 11gR2) OS level patches (updates to RedHat 5.4), Java updates (going to 1.6.0_20b5) and roughly 300 Oracle EBS patches. Yes, 300. Although we’re still staying at 11.5.10.2 ATG RUP 7, Oracle recently updated their minimum patch list you must be on to be received Extended Support on Apps 11i as of Nov. You can find the full up to date list at Metalink document Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10 (Note 883202.1).

Without virtualization, I’d have to take a full backup of my Database and App Tiers before starting the upgrade and then be prepared to go back to these if we can’t finish the upgrade by a certain time. Depending on your environment, this could be hours, days or not even practical.

Instead, at the start of the upgrade I merely take a snapshot of each VM involved. If I do this with the VMs shut down (i.e. not capturing the memory state of the VM as well) this takes about 10 seconds. Done. In the event we decide to rollback all our upgrade changes, it’s just a quick Revert to the Snapshot (might take as long as a minute per VM – OMG, a whole minute!!), restart the Apps services and poof, you’re done.

If only the upgrades themselves were so easy and foolproof.

Another reason to virtualize is of course getting the best usage of your Oracle licenses. Oracle EBS requires Oracle Enterprise Edition. I’ve blogged previously here about how to do that so I won’t go into that here.

Avoid the RAC tax. According to Oracle’s most recent Global Price list, RAC is an additional $23k per processor on top of Enterprise Edition. RAC makes sense for companies in two situations – 1) When their database must not go down. 2) Systems too large to run on 1 physical box. Sure, there are some installations where EBS requires more horsepower than can be run under 8 processors (the limit of a VM under vSphere 4.0 Update 2), but they are the vast minority. Many companies running EBS that implement RAC because they want to be protected from hardware failures. VMware has two features that protect you here – High Availability (HA) which will restart your VM on another vSphere host if the VM goes down and the much cooler and more RAC like feature, Fault Tolerance (FT). FT will actually run your VM on two ESX hosts simultaneously. They work in an active / passive arrangement – you aren’t having users connecting to both machines and load balancing between the two machines, but instead all users are connecting to one machine and in the event it fails, user sessions and other processes and connections are now active on the other node. Your users don’t notice anything, no disconnections, no restarts, no autoconfig required. It just simply works. The current drawback to FT is that it’s currently limited to 1 CPU VMs – that’s reasonable in my experience with an Apps 11i Forms/Web tier, but can frequently be a show stopper with a database server. However, if you’re willing to leverage the performance tuning features of 11g, you may be able to get past that. It is also rumored that VMware is working on getting FT to work with multiple processor VMs. When they do that it should really put a dent in RAC. Before I started the leveraging the SQL tuning available in 11gR2 and SQL Profiles, my main client system ran with CPU as the constraint and had a pretty constant load of 4 CPUs. After tuning the SQL, CPU load occasionally spikes at over 1 CPU and typical CPU load is 0.5 processors. The system is now a prime candidate for FT. My limiter became disk I/O which we addressed with more spindles.

Creating a clone of an EBS system for development or testing can be a big pain. Almost always you’re talking about cloning 2 servers (DB and Apps tiers), running autconfig, canceling scheduled concurrent jobs, etc. With VMware Lab Manager, you can have your developers / analysts create linked clones of your Production environment in minutes. No need to run autoconfig, the systems show up with their same instance names, same machine names, etc. This is done behind the scenes by putting the cloned VMs in their own private network. Instead of your copy of PROD taking up 100GB or so for EBS executables and another XXX GB for the database itself, it’s merely just the size of the changes between your production environment and your cloned VM. I have yet to do this in my own environment, so I can’t speak from experience, but what I have seen has been impressive.

It’s been a few years and your server hardware needs to be replaced due to expensive support costs? In the physical world that means building up new database / App servers and having a downtime to copy everything over, reconfigure the networking (and possibly the machine names) and then hoping you aren’t running into some sort of unexpected problem (like the AMD time drift issue a few years back that caused Kernel Panics – see here )… or you could just install vSphere onto your new host, make it visible to your shared storage and vMotion your system live and in production to the new hardware. What’s not to love?

Who really does run Oracle EBS under VMware though? Small little companies? No. Two of my favorite examples are VMware and EMC. Both run Oracle EBS virtualized.

Oracle licensing under VMware and how to get the best bang for your buck

Uncategorized, , ,

I’ve been meaning to write an article on how Oracle’s licensing works with regards to VMware and how to minimize costs, but Oracle Storage Guy (Jeff Browning) said it better in this blog post ( http://oraclestorageguy.typepad.com/oraclestorageguy/2010/05/oracle-license-costs-on-the-vmware-vsphere-platform.html ) than I could have said it.  Go read his blog post in another tab and then continue on..

Back?  Good.

So a real world example.  My main vSphere cluster is a 16 blade Dell Cluster.  11 of those blades are Dell M600 blades each with I2 Intel Xeon E5430 processors (those are pre-Nahelm for those curious) and 5 of those blades are Dell M610 blades each with 2 Intel Xeon 5550 (Nahelm).  In both cases, each blade has 2 sockets, with each core having 4 cores.  With the Xeon 5550 processors, they also have 2 logical (hyperthreading) cores per physical core, giving me a total of 16 logical processors per M610 blade.  Oracle licensing does not count logical cores, only physical ones.

Rather than paying to license 8 cores on 16 blades (which at list rates would work out to 8 * 16 * 0.5 * $47,500 ) for $3,040,000 or just over $3 MILLION DOLLARS in Oracle licensing, we have two clusters in vCenter.  One of these contains 8 M600s and 3 M610s and basically contains everything in our cluster with the exception of our Oracle products licensed by processor.  The other cluster is 2 M610 nodes and it holds all our Oracle databases and other Oracle products that are licensed by processor.  That works out to (8 * 2 * 0.5 * $47,500) $380,000 or the same performance for about 1/10 of the price.

Now, you may be asking how those two blades are handling the load – I’m writing this blog entry during the middle of our production day.  I’m running 20 Oracle database servers (one instance per VM server) combined on the two blades.

Blade #1 – 1044 MHz CPU utilized, 25897MB RAM utilized

Blade #2 – 4448 MHz CPU utilized, 27093MB RAM utilized

I didn’t make a typo.  I’m running 20 database servers essentially off of 2 CPUs worth of processing power (each Xeon 5550 is 2.67 MHz) and I’ve got 32 CPUs of processing power in the two node cluster (4 sockets, each with 4 physical cores, each physical core with 2 logical cores), so that’s not going to be a bottleneck at any time soon.

On the RAM side, each M610 has 48G of RAM and right now I’m utilizing about 48G of RAM.  Each M610 can handle up to 192G of RAM, but we went with 12 4G sticks due to that being the current cost sweet spot.  Again, not a bottleneck.

So like Jeff said, utilization on Oracle database servers is typically low.  If I wasn’t virtualized at all, I’d have to have 20 physical servers and license all those processors for Oracle Enterprise Edition.  If I allowed my Oracle VMs into my regular VMware cluster, I’d have all the advantages of VMware (HA (high availability), vMotion, Storage vMotion, etc) but I’d have to pay 10x the cost in Oracle licenses.  Instead I’ve got a separate VMware cluster only licensing what I need and getting all the advantages of VMware.

For those curious about what we’re doing with all those instances, some are development / support environments, but we’re running our production Oracle E-Business Suite, Oracle Hyperion, Oracle Agile, and various other smaller load systems on two nodes supporting hundreds of end users.

What can I say? VMware rocks.

Test lab

vmware, , , ,

These days I’m waiting on my functional users to finish testing a large upgrade of our Oracle Apps environment.  So far *fingers crossed*, they have yet to find many issues that require my time, so I’m having plenty of time to work on side projects.

This past week I build up a VMWare test lab with some de-commissioned hardware we had lying around.  The test lab consists of

  • Dell 2950 with 4 cores and 16GB of RAM, 300GB RAID 5 local storage
  • Dell 2850 with 4 cores and 9GB of RAM, 300GB RAID 5 local storage
  • Iomega Storcenter ix4-200d  2.7TB RAID5 NAS storage (NFS / iSCSI)
  • ESX 4.0 (vSphere) plus all patches thru 3/17/10
  • vCenter 4.0

I admit, for de-commissioned hardware, it’s still a pretty powerful little cluster.

We’re about 3/4 of the way thru our upgrade from ESX 3.5 to vSphere on both our primary cluster and our DR (Disaster Recovery) cluster.  At this point we’ve got all the ESX hosts upgraded to vSphere.  The next steps are to get those hosts with all the latest ESX patches and then update the VMWare tools (which will cause a momentary downtime for the network of the VM) and the VM hardware layer (which requires the VM be shut down) of each VM running in the cluster.  Due to business requirements for when we can bounce certain VMs, this is a process that will take a week or two to get done.

In addition to our primary and DR clusters, we also have 6 standalone ESX 3.5 boxes with no shared storage.  Some of these run somewhat critical virtual machines and so minimizing downtime is important.  That’s where the test lab is coming in.

Besides providing a non-production environment for me to learn and test out various things with VMWare, I’m utilizing the Iomega StorCenter as shared storage.  The plan is as follows.

  1. Add the NFS Share on the StorCenter to the ESX 3.5 Server.
  2. Utilize Storage Vmotion to move the datastore for the VMs on the ESX 3.5 Server to shared storage
  3. Utilize Vmotion to move the running VM from the ESX 3.5 Server to the ESX 4.0 Servers.

So far, no downtime.

Once we’ve moved all the VMs off the standalone ESX 3.5 server, we’ll upgrade the various BIOSs / Firmwares on the ESX host and then install a fresh and fully patched version of vSphere.  Then we’ll re-attach it to the shared storage and move our VMs back.  Finally, we’ll schedule downtime with the respective business owners of those VMs for some downtime to upgrade the VMWare tools and the VM hardware version.

It’s a good little project that lets us leverage many of the cool features in VMWare and should provide quite the fun learning experience.

And for those of you scratching your head and wondering why we are choosing to run somewhat critical VMs on only localized storage – we’re finishing up an expansion of our server room and upgrading to a larger SAN so we can eventually store those VMs in the SAN.  The StorCenter might be used once we’ve upgraded all these standalone ESX boxes as a temporary home, but we’ll have to see.