Tag Archives: vmware

VMware Knowledge base entries of interest for Oracle DBAs

Uncategorized, , , , ,

One of the things I love about VMware’s support site is their knowledge base. It’s not horrific flash like My Oracle Support (aka Metalink), and it’s freely searchable without a support contract. Also very cool is there is an RSS feed of new or updated knowledge base articles. It’s good to scan in an idle moment or two each day to have an idea of what issues other people are seeing.

Because of that RSS feed, I came across three knowledge base articles I’d like to highlight here:

KB Article 1023696: Oracle 11G R2 32 bit client fails with a segmentation fault when run in a RHEL 5.4 64 bit virtual machine

In this case, sqlplus would seg fault when you try running it. The issue it turns out isn’t a VMware issue – it’s an Oracle bug when running 32-bit 11gR2 client on a 64-bit RH OS with an AMD processor. The fix is Oracle patch 8670579.

KB Article: 1023898
RedHat and CentOS virtual machine show warning messages when starting the udev daemon

This issue actually cropped up in my VMware environments awhile ago. Basically you see messages like this when your VM starts:

udevd[572]: add_to_rules: unknown key ‘SUBSYSTEMS’
udevd[572]: add_to_rules: unknown key ‘ATTRS{vendor}’
udevd[572]: add_to_rules: unknown key ‘ATTRS{model}’
udevd[572]: add_to_rules: unknown key ‘SUBSYSTEMS’
udevd[572]: add_to_rules: unknown key ‘ATTRS{vendor}’
udevd[572]: add_to_rules: unknown key ‘ATTRS{model}’

On RHEL, the fix is to do the following

vi /etc/udev/rules.d/99-vmware-scsi-udev.rule

change

ACTION==”add”, BUS==”scsi”, SYSFS{vendor}==”VMware, ” , SYSFS{model}==”VMware Virtual S”, RUN+=”/bin/sh -c ‘echo 180 >/sys$DEVPATH/device/timeout'”

To:

ACTION==”add”, BUS==”scsi”, SYSFS{vendor}==”VMware ” , SYSFS{model}==”Virtual disk “, RUN+=”/bin/sh -c ‘echo 180 >/sys$DEVPATH/device/timeout'”

and then reboot the VM.

The final article I want to mention is
KB Article: 1023185 VMware Tools installation fails to start the guest operating system daemon on Red Hat Enterprise Linux 4 64-bit guests with the 32-bit glibc-common package installed

This issue relates to Oracle because 32-bit glibc-common is frequently required for Oracle DB installs. The issue occurs because VMware tools configuration is looking for the 64-bit tools (64-bit OS, generally you’d want to install the 64-bit RPMs…). The solution is to install VMware tools as normal, but before running the configuration script, to issue
ln –s /usr/lib/vmware-tools/lib64/libdnet.so.1/libdnet.so.1 /lib64/libdnet.so.1
ln –s /usr/lib/vmware-tools/lib64/libproc-3.2.7.so/libproc-3.2.7.so /lib64/libproc-3.2.7.so
and then run the configuration program for vmware tools
/usr/bin/vmware-config-tools.pl

Hopefully this is helpful to other Oracle on RHEL under VMware people out there.

Oracle VM compared to VMware vSphere: Part 1

Uncategorized, , ,

I’ve been meaning to take a serious look at Oracle VM for a few months. In fact, it was this post [Live Migration of EBS Services Using Oracle VM] (and my long-winded reply) that was a major push for me to start this blog.

The final bit of impetus to learn all about Oracle VM came a few months ago when I saw the “Oracle VM for x86 Essentials” beta exam. If passed, you earn the certification “Oracle VM for x86 Certified Implementation Specialist”. It’s a certification geared for Oracle Partners. I figured the knowledge could help me to better understand Oracle’s offering. First and foremost, I’m an Oracle Applications DBA. If Oracle’s product could allow me to better serve my clients and do my job – awesome!

So I’ve been hitting all the Oracle VM resources I could find to learn about the product. I’ll post links to a number of the excellent resources I found at the end of this post. All the links at the bottom refer to information on the currently available product (Oracle VM 2.2). While compiling all of this information, I came across [Oracle Virtualization:Making Software Easier to Deploy, Manage, and Support] – a slide deck from a recent Sydney Australia Oracle meetup. It talks about upcoming features of Oracle VM 3.0. If those features come to pass, Oracle VM will become more enticing to many organizations.

Honestly, I’ve got *tons* of things I want to write about with regards to Oracle VM — so much that I don’t know where to begin.

General Impressions
Remember that first time you went from something with a nice GUI, like Windows (Thanks Apple Microsoft!) to something a little more “nerdy” like Linux ? The GUI, if there was one, was stripped down and clunky. Many of the things you could do with a couple of mouse clicks before now require specialized commands at a command line. All of these different steps you need to do just to get things working. Well, it’s the same type of thing going from VMware vCenter to Oracle VM Manager. It’s not that the product is bad — it isn’t. The Oracle VM interface is clunky and the product doesn’t have the richness of features of VMware vSphere. Simple as that. Are those differences worth it to you? Everyone’s needs are different. Both underlying products (Oracle VM Server, VMware vSphere ESX 4.0) run Linux and Windows VMs well enough for most enterprise-level systems.

As you can read in this Gartner report on Server Virtualization Infrastructure, VMware is the clear market leader. Oracle VM, although categorized as a niche player, is the strongest of the niche players and right on the border of being listed as a challenger to VMware.

Here are a few areas where Oracle VM has an advantage over VMware:

o Certified vs. Supported

(I hate talking about this but it needs to be addressed.) Is your VMware virtualized Oracle database supported by Oracle? YES. Is it Certified? No. I went into this in detail in this [Oracle Support on VMware] blog post so I won’t do it again. Short of running Oracle RAC, which is expressly NOT supported when virtualized under VMware, the question of whether you should care about the “certified” distinction is something each company needs to answer for themselves. To me, the whole thing smacks of FUD (Fear, Uncertainty, Doubt).

o Pricing

There are two parts to pricing. First is the effect virtualizing Oracle Database will have on your Oracle database licensing. I go into this in more detail in a post on Oracle licensing under VMware. One of Oracle VM’s main selling points is that Oracle considers Oracle VM (through hardcoding the CPU binding in the vm.cfg file) a type of hard partitioning and VMware vSphere a type of soft partitioning. When using hard partitioning, Oracle only requires you to license the processors (cores) in that hard partition (aka, the processors visible to the VM). When using soft partitioning, Oracle requires you to license ALL the processors (cores) in the server, even though there may be many more processors present than allocated to the VM. It should be noted that you can do the same type of CPU binding (called CPU affinity) with VMware vSphere, but that Oracle somehow still considers this soft partitioning.

This just seems like a way for Oracle to give their Oracle VM product preferential treatment. How does the joke go… Where does the 800 lb gorilla sit? Anywhere he wants to.

The second part of pricing involves the actual Oracle VM product versus the VMware vSphere product. Oracle basically has two pricing points
o Premier Limited — Up to 2 CPU sockets, regardless of the number of cores per socket in the physical server
o Premier — unlimited CPU sockets in the server

VMware, unlike Oracle, has four product feature levels (Standard, Advanced, Enterprise and Enterprise Plus) and so a head to head comparison is a complete pain to do. The short answer is that Oracle can be significantly cheaper.. The downside of this inexpensiveness is a lack of features. Yes, VMware generally costs more than Oracle, but you’re paying for additional features. Are those features worth it to your organization? That’s for you to decide. In my organization, we are willing to pay for VMware’s features, but my organization’s needs may be different than the needs of your organization.

Does your organization have a need for VM snapshots? Mine absolutely does. Oracle VM doesn’t have it and VMware does, even when you’re using the free version of each product.

Does your VM require more than 8 CPUs? VMware has a limit of 8 CPUs for a single VM. Oracle VM’s limit is 32 CPUs for a single VM. Through tuning and software improvements, my main client has managed to reduce the number of CPUs for our Production Oracle E-Business Suite database from an unvirtualized 8 cores to 2 cores virtualized, so the difference is immaterial… but maybe your organization needs 20 cores.

Does your organization have a need to do vMotions / Live Migrations? They come included with Oracle VM, but it’s not recommended to do more than one at a time. There is an additional cost to get VMware vMotion, but VMware supports a default configuration with up to 4 simultaneous moves and allows up to 8 simultaneous moves.

Does your organization need automated SAN level replication of your VMs so they can be brought up automatically in case of disaster? VMware has that functionality with Site Recovery Manager. Oracle doesn’t have anything like it.

o Oracle VM Templates

Do you want a pre-built VM you can download with the Oracle software already installed and configured? Oracle offers downloads of pre-built environments from a basic OEL 5 Linux box all the way through to a downloadable 38GB Oracle EBS R12.1.1 system. I admit, that could be pretty cool. However, it may not be right for your company. My main client never allows consultants to have console type access to our Linux servers. I don’t think my auditors would approve of a pre-built VM for production use, even if it was pre-built by Oracle. As something for quickly throwing up a demo or dev environment, I think it’s fantastic. I hope Oracle continues to do this for more and more of their products. Oracle Enterprise Manager 10gR5 took me roughly 2 weeks to install. Discoverer 11g about a week. Secure Enterprise Search about 2 weeks. It would be great to have a pre-built test system I could reference when building my production systems.

I’ve got numerous ideas for more blog posts with regards to Oracle VM. Feedback directing me to what interests others would be great.

Part 2 coming after the I take the exam later this week. Wish me luck!

Links
Live Migration of EBS Services with Oracle VM
Installing & Configuring OEL 5 with Database 11gR1 as a Paravirtualized Machine (PVM) on an Oracle VM Server
The underground Oracle VM Manual
Official Oracle VM Wiki home page
Oracle VM for x86 Essentials Exam 1Z0-540 Exam Topics Study Guide
Oracle VM 2.2 Documentation Library
Performing Physical to Virtual (P2V) and Virtual to Virtual (V2V) (aka VMware to Oracle) conversions Note the excellent pdf linked from the article too.
Installing, Configuring and Using Oracle VM Server for x86

Lessons learned from a virtualized Oracle upgrade

Uncategorized, , ,

So about a week ago, we did a rather massive upgrade at my main client to the Oracle E-Business infrastructure. The main things in this upgrade were:

Licensing modules necessary for us to have a full installation of Oracle HR
Upgrade Oracle database from 11.1.0.7 64-bit to 11.2.0.1 64-bit
Apply all CPU security patches thru Apr 2010
Upgrade memory on DB server from 8G to 12G
Upgrade server side java from 1.6.0_16 to 1.6.0_20
Upgrade client side java from 1.6.0_16 to 1.6.0_20b5 (see this link on why the special b5 version)
Apply approximately 350 (not a typo) individual E-Business patches, for the following things:
o Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10 (Note 883202.1)
o Upgrading from Financials Family Pack F to Family Pack G (FIN_PF.G)
o Recommended 11i Apps patches for all our products
o Java related patches
o Latest DST v11 related patches (see here)
o Implement WebADI

As you might gather from this list, it was a rather large upgrade. The apps patches alone totaled about 10GB of patches once merged into one patch and the backup directory for the merged patches ended up totaling 6GB. Test runs had the upgrade running about 24 hours with 8 CPUs on some scratch disk storage I had in the SAN . Like I mentioned in previous posts, we utilized VMware snapshots on our boxes at various points in the upgrade in case we needed to roll back or experienced an unforeseen issue.

One of the VMware best practices we follow with our VMs is to break the boot “disk” and the data “disk” for our VMs into their own virtual disks. Besides during booting up / shutting down of a VM, the boot disk generally experiences very low traffic. So it’s pretty typical, especially with a replicated SAN system such as ours, to put your boot “disks” (VMDKs) for a bunch of VMs on one VMware datastore, possibly with slower disks, and your data “disks” (VMDKs) on another dedicated datastore. In our case, the boot disk datastore is a 2 disk RAID 1 (mirrored) set with Fiber Channel drives and the data disk datastore is a 9 disk (8+1) RAID 5 datastore of SSDs (aka EFDs aka super super fast disks).

Although I had run multiple dry runs before the upgrade, one thing I failed to notice / realize is that by default VMware snapshots are stored where the VM lives, or more specifically, where the VM’s configuration file lives… in this case on my slowest disks.

This became extremely clear during our large merged patch of 330+ Apps patches – things got slower and slower. At that point, shutting down the VM and moving the snapshots wasn’t really an option. It was just a matter of suffering thru and learning for next time. Luckily the business had fully planned on the upgrade taking 24 hours for the patching even though I expected us to be at roughly 1/2 that time with SSDs.

By the time the upgrade was done and the business analysts had finished their testing and calling the upgrade good (and hence when we were ready to delete the 5 sets of snapshots), the snapshots for my two VMs that utilize about 450GB of space had grown to about 200GB. It took about 5 hours for the snapshots to be merged into the base VMDKs. Although the system was usable during that time, it was quite laggy. Luckily it was still the weekend for most of our users and they weren’t too inclined to utilize Oracle.

On the subject of VMware snapshot deletions, I recently came across two notes that should be of use to other VMware admins
1) With the latest version of vSphere (4.0 Update 2), VMware has greatly improved the speed and efficiency of deleting all the snapshots for a VM. You can read more about it here. Unfortunately at the time of my Oracle upgrade I was on vSphere 4.0 Update 1.
2) When you delete a large snapshot, it will frequently appear to “hang” at 95% – check out this knowledge base article on how to monitor snapshot deletions.

Overall the upgrade was a success and minus the occasional user issues Monday morning (first business day after the upgrade) was pretty much a non-event.

These are the sorts of situations that make sending your people to training, or giving them the time and inclination to read manuals and blogs, so essential. Not as a result of this, but somewhat related, I’ll be attending the VMware vSphere troubleshooting class in the next month or two and will be (assuming I pass the test) earning my VCP and possibly trying to earn a VCAP-DCA by end of year.

How virtualization can magnify your architecture problems

Uncategorized, , , ,

I recently started working with a new client who has a hosting provider hosting their Oracle database on Linux under VMware. An excellent choice, but this client is experiencing major performance issues – data for forms taking a minute or more to come up is just one example.

As I learned more about their environment I found that virtualization (VMware in this case, though the issue isn’t specific to any particular virtualization vendor) actually made their system performance worse. I know, I’m a VMware groupie (heck a VMware vExpert!) and we’re all amazed I’d write such a thing, but alas, it’s true.

The database is around 80GB in size. Each day this hosting provider would take a full (level 0 incremental) backup of the Oracle database via RMAN. The hosting provider wrote this RMAN backup to the same mount point in the VM that the database uses.

Please take a moment to catch your breath and stop clenching your hands into fists over this very very bad idea.

So why is this such a bad idea? For a couple of reasons.

One is performance – you’re now greatly degrading the performance of your database by writing a full backup to the same disks that are trying to handle database requests. You have, at the least, doubled the amount of I/O going to those disks.

Two is the ability to recover. If your ESX host or your VM experiences an issue (running out of disk space, disk corruption, fire, whatever), you can no longer access the mount point in the VM where you backed up the data.

Best practice for implementing RMAN in a situation like this is to backup your database to another set of disks on another machine in another physical location. A typical example is to have an NFS export on your backup destination server (in another datacenter) and have RMAN write direLet’s say ctly to that NFS mount. This way you aren’t writing your backup to the same disks (thereby not impacting production performance much) and you’re covered in the case of issues with the hardware or VM itself.

So where does VMware fit into this? I mentioned that the hosting provider was also performing VM-level backups. In particular, they were performing VM-level backups at the same time they were running RMAN backups. All to the same set of disks.

Now I’ve got the VMware Admins and the Oracle DBAs cringing.

When you initiate a VM level backup, VMware takes a snapshot of the VM. This means it makes a delta file on the same ESX datastore and stops writing to the VMDK(s) that make up the VM. All changes to the VM get written to the delta file instead. That delta file can grow (8 megs at a time) up till it’s the same size as the original VMDK.

When you are taking a VM level backup, you want to choose a time when you’re not doing many writes to the VM. This way the delta file won’t grow so big that you could run out of space on the datastore (LUN) and your performance impact is decreased.

So here they are writing their full Oracle backup of 80GB out to a mount point inside their VM. That’s 80GB of writes you’re doing. VMware see those writes and has to write them to it’s snapshot (delta file). So now not only are you serving up database queries on your disks, you’re also scanning every block of your database on those disks for changes (this database did not employ Oracle Changed Block Tracking), you’re writing a full RMAN backup to those disks and VMware is having to copy all those writes into a delta file on those same disks.

Virtualization can be wonderful and solve or simplify many of the issues an administrator faces, but it can also magnify fundamental architecture flaws.

Why run Oracle E-Business on vSphere?

Uncategorized, , ,

If I’ve got the dedicated hardware for my Oracle E-Business (EBS) install, why would I want to run it virtualized?

Most of the places I’ve seen typically have dedicated hardware for the Oracle EBS instances – so why virtualize it at all?

One major reason is the ability to rollback the entire environment in the event of a patching issue. My main client has an upcoming upgrade involving a database upgrades (11gR1 to 11gR2) OS level patches (updates to RedHat 5.4), Java updates (going to 1.6.0_20b5) and roughly 300 Oracle EBS patches. Yes, 300. Although we’re still staying at 11.5.10.2 ATG RUP 7, Oracle recently updated their minimum patch list you must be on to be received Extended Support on Apps 11i as of Nov. You can find the full up to date list at Metalink document Minimum Baseline Patch Requirements for Extended Support on Oracle E-Business Suite 11.5.10 (Note 883202.1).

Without virtualization, I’d have to take a full backup of my Database and App Tiers before starting the upgrade and then be prepared to go back to these if we can’t finish the upgrade by a certain time. Depending on your environment, this could be hours, days or not even practical.

Instead, at the start of the upgrade I merely take a snapshot of each VM involved. If I do this with the VMs shut down (i.e. not capturing the memory state of the VM as well) this takes about 10 seconds. Done. In the event we decide to rollback all our upgrade changes, it’s just a quick Revert to the Snapshot (might take as long as a minute per VM – OMG, a whole minute!!), restart the Apps services and poof, you’re done.

If only the upgrades themselves were so easy and foolproof.

Another reason to virtualize is of course getting the best usage of your Oracle licenses. Oracle EBS requires Oracle Enterprise Edition. I’ve blogged previously here about how to do that so I won’t go into that here.

Avoid the RAC tax. According to Oracle’s most recent Global Price list, RAC is an additional $23k per processor on top of Enterprise Edition. RAC makes sense for companies in two situations – 1) When their database must not go down. 2) Systems too large to run on 1 physical box. Sure, there are some installations where EBS requires more horsepower than can be run under 8 processors (the limit of a VM under vSphere 4.0 Update 2), but they are the vast minority. Many companies running EBS that implement RAC because they want to be protected from hardware failures. VMware has two features that protect you here – High Availability (HA) which will restart your VM on another vSphere host if the VM goes down and the much cooler and more RAC like feature, Fault Tolerance (FT). FT will actually run your VM on two ESX hosts simultaneously. They work in an active / passive arrangement – you aren’t having users connecting to both machines and load balancing between the two machines, but instead all users are connecting to one machine and in the event it fails, user sessions and other processes and connections are now active on the other node. Your users don’t notice anything, no disconnections, no restarts, no autoconfig required. It just simply works. The current drawback to FT is that it’s currently limited to 1 CPU VMs – that’s reasonable in my experience with an Apps 11i Forms/Web tier, but can frequently be a show stopper with a database server. However, if you’re willing to leverage the performance tuning features of 11g, you may be able to get past that. It is also rumored that VMware is working on getting FT to work with multiple processor VMs. When they do that it should really put a dent in RAC. Before I started the leveraging the SQL tuning available in 11gR2 and SQL Profiles, my main client system ran with CPU as the constraint and had a pretty constant load of 4 CPUs. After tuning the SQL, CPU load occasionally spikes at over 1 CPU and typical CPU load is 0.5 processors. The system is now a prime candidate for FT. My limiter became disk I/O which we addressed with more spindles.

Creating a clone of an EBS system for development or testing can be a big pain. Almost always you’re talking about cloning 2 servers (DB and Apps tiers), running autconfig, canceling scheduled concurrent jobs, etc. With VMware Lab Manager, you can have your developers / analysts create linked clones of your Production environment in minutes. No need to run autoconfig, the systems show up with their same instance names, same machine names, etc. This is done behind the scenes by putting the cloned VMs in their own private network. Instead of your copy of PROD taking up 100GB or so for EBS executables and another XXX GB for the database itself, it’s merely just the size of the changes between your production environment and your cloned VM. I have yet to do this in my own environment, so I can’t speak from experience, but what I have seen has been impressive.

It’s been a few years and your server hardware needs to be replaced due to expensive support costs? In the physical world that means building up new database / App servers and having a downtime to copy everything over, reconfigure the networking (and possibly the machine names) and then hoping you aren’t running into some sort of unexpected problem (like the AMD time drift issue a few years back that caused Kernel Panics – see here )… or you could just install vSphere onto your new host, make it visible to your shared storage and vMotion your system live and in production to the new hardware. What’s not to love?

Who really does run Oracle EBS under VMware though? Small little companies? No. Two of my favorite examples are VMware and EMC. Both run Oracle EBS virtualized.

vExpert 2010

Uncategorized,

VMware has an advocacy recognition program called VMware vExpert.  The program’s goal is to recognize individuals who significantly contribute to the community of VMware users.  Such individuals include authors, bloggers, VMUG leaders, tool builders, and other IT professionals.

I waited until the last minute to submit my application because I debated if I was even qualified to apply.  This blog is very new.  I’m not the knowledge expert on VMware products like many of the other vExperts.  Heck, I don’t even have VCP (VMware Certified Professional) credentials. I’m one of the IT professionals out there buying the books that many of the other vExperts publish!  The VMware vExpert judges deemed me worthy of the recognition and I’m thrilled and proud to have been awarded it on Friday.

Although I’m not the pure VMware subject matter expert many are, I have and will continue to contribute my subject matter expertise with Oracle products on Linux under VMware.  My main client made the switch to running our Tier 1 production Oracle systems under VMware about 2 years ago.  I believe it was one of the best decisions we ever made, from both a technical and financial perspective.

Since then in addition to my main duties as Sr. Oracle Applications DBA and Sr. Linux Administrator, I’ve been learning and leveraging many of the features VMware virtualization buys us.  I’ve done many reference calls both for Oracle (my main client was one of the first companies in the world to run Oracle 11g database in Production), EMC (we utilize many EMC products in our environment) and VMware and will continue to in the future

Oracle licensing under VMware and how to get the best bang for your buck

Uncategorized, , ,

I’ve been meaning to write an article on how Oracle’s licensing works with regards to VMware and how to minimize costs, but Oracle Storage Guy (Jeff Browning) said it better in this blog post ( http://oraclestorageguy.typepad.com/oraclestorageguy/2010/05/oracle-license-costs-on-the-vmware-vsphere-platform.html ) than I could have said it.  Go read his blog post in another tab and then continue on..

Back?  Good.

So a real world example.  My main vSphere cluster is a 16 blade Dell Cluster.  11 of those blades are Dell M600 blades each with I2 Intel Xeon E5430 processors (those are pre-Nahelm for those curious) and 5 of those blades are Dell M610 blades each with 2 Intel Xeon 5550 (Nahelm).  In both cases, each blade has 2 sockets, with each core having 4 cores.  With the Xeon 5550 processors, they also have 2 logical (hyperthreading) cores per physical core, giving me a total of 16 logical processors per M610 blade.  Oracle licensing does not count logical cores, only physical ones.

Rather than paying to license 8 cores on 16 blades (which at list rates would work out to 8 * 16 * 0.5 * $47,500 ) for $3,040,000 or just over $3 MILLION DOLLARS in Oracle licensing, we have two clusters in vCenter.  One of these contains 8 M600s and 3 M610s and basically contains everything in our cluster with the exception of our Oracle products licensed by processor.  The other cluster is 2 M610 nodes and it holds all our Oracle databases and other Oracle products that are licensed by processor.  That works out to (8 * 2 * 0.5 * $47,500) $380,000 or the same performance for about 1/10 of the price.

Now, you may be asking how those two blades are handling the load – I’m writing this blog entry during the middle of our production day.  I’m running 20 Oracle database servers (one instance per VM server) combined on the two blades.

Blade #1 – 1044 MHz CPU utilized, 25897MB RAM utilized

Blade #2 – 4448 MHz CPU utilized, 27093MB RAM utilized

I didn’t make a typo.  I’m running 20 database servers essentially off of 2 CPUs worth of processing power (each Xeon 5550 is 2.67 MHz) and I’ve got 32 CPUs of processing power in the two node cluster (4 sockets, each with 4 physical cores, each physical core with 2 logical cores), so that’s not going to be a bottleneck at any time soon.

On the RAM side, each M610 has 48G of RAM and right now I’m utilizing about 48G of RAM.  Each M610 can handle up to 192G of RAM, but we went with 12 4G sticks due to that being the current cost sweet spot.  Again, not a bottleneck.

So like Jeff said, utilization on Oracle database servers is typically low.  If I wasn’t virtualized at all, I’d have to have 20 physical servers and license all those processors for Oracle Enterprise Edition.  If I allowed my Oracle VMs into my regular VMware cluster, I’d have all the advantages of VMware (HA (high availability), vMotion, Storage vMotion, etc) but I’d have to pay 10x the cost in Oracle licenses.  Instead I’ve got a separate VMware cluster only licensing what I need and getting all the advantages of VMware.

For those curious about what we’re doing with all those instances, some are development / support environments, but we’re running our production Oracle E-Business Suite, Oracle Hyperion, Oracle Agile, and various other smaller load systems on two nodes supporting hundreds of end users.

What can I say? VMware rocks.

Oracle Support on VMware – your Cisco switches and DELL servers aren’t certified with Oracle

Uncategorized, , , ,

So I came across this article last week on Oracle VM and how companies are trying it and how it’s, and I’m quoting here “not half bad”

If I ever come out with my own Enterprise level software and have a couple of public releases under my belt and the best thing I can get in the press is “not half bad”, shoot me.  Seriously.  That’s embarrassing.  Even if you do offer it for free.

So let’s look past the headline and see what the article has to say, shall we?

First sentence “Oracle’s continued refusal to support its applications virtualized on something other than the Oracle VM hypervisor has forced the hands of some users, pushing them to try the Xen-based virtualization offering.”

It’s this sort of FUD (Fear, Uncertainty, Doubt) spreading that’s one of the things I dislike about our industry.  Let me be completely clear ORACLE FULLY SUPPORTS ITS APPLICATIONS VIRTUALIZED ON SOMETHING OTHER THAN ORACLE VM HYPERVISOR.  Don’t believe me?  Check out Metalink My Oracle Support note 249212.1 – Support Position for Oracle Products Running on VMWare Virtualized Environments. Here’s the first paragraph

Oracle has not certified any of its products on VMware virtualized
environments. Oracle Support will assist customers running Oracle products
on VMware in the following manner: Oracle will only provide
support for issues that either are known to occur on the native OS, or
can be demonstrated not to be as a result of running on VMware.

Did you know your Cisco switches aren’t certified by Oracle?  Your DELL servers aren’t certified by Oracle either.

Very very rarely does Oracle certify another vendor’s hardware products.  Certify != support.  Oracle support’s your installation of Oracle productions on VMware just fine.  They take the very reasonable point of view that they’ll help you until they come to believe the issue is VMware at which point they’ll refer you to VMware support.  Seems completely rational to me.

I’ve been an Oracle Apps DBA running Oracle products under VMware for about 4 years.  I have never had Oracle not help me because I run many of my environments under VMware. I’ve spoken with numerous other Oracle DBAs running Oracle products under VMware.  None of us have ever had an issue where Oracle wouldn’t support our environments because we were running under VMware.

I’ll address the rest of the article in another blog post shortly, but seeing this FUD still slung about makes me too annoyed to write coherently.

Test lab

vmware, , , ,

These days I’m waiting on my functional users to finish testing a large upgrade of our Oracle Apps environment.  So far *fingers crossed*, they have yet to find many issues that require my time, so I’m having plenty of time to work on side projects.

This past week I build up a VMWare test lab with some de-commissioned hardware we had lying around.  The test lab consists of

  • Dell 2950 with 4 cores and 16GB of RAM, 300GB RAID 5 local storage
  • Dell 2850 with 4 cores and 9GB of RAM, 300GB RAID 5 local storage
  • Iomega Storcenter ix4-200d  2.7TB RAID5 NAS storage (NFS / iSCSI)
  • ESX 4.0 (vSphere) plus all patches thru 3/17/10
  • vCenter 4.0

I admit, for de-commissioned hardware, it’s still a pretty powerful little cluster.

We’re about 3/4 of the way thru our upgrade from ESX 3.5 to vSphere on both our primary cluster and our DR (Disaster Recovery) cluster.  At this point we’ve got all the ESX hosts upgraded to vSphere.  The next steps are to get those hosts with all the latest ESX patches and then update the VMWare tools (which will cause a momentary downtime for the network of the VM) and the VM hardware layer (which requires the VM be shut down) of each VM running in the cluster.  Due to business requirements for when we can bounce certain VMs, this is a process that will take a week or two to get done.

In addition to our primary and DR clusters, we also have 6 standalone ESX 3.5 boxes with no shared storage.  Some of these run somewhat critical virtual machines and so minimizing downtime is important.  That’s where the test lab is coming in.

Besides providing a non-production environment for me to learn and test out various things with VMWare, I’m utilizing the Iomega StorCenter as shared storage.  The plan is as follows.

  1. Add the NFS Share on the StorCenter to the ESX 3.5 Server.
  2. Utilize Storage Vmotion to move the datastore for the VMs on the ESX 3.5 Server to shared storage
  3. Utilize Vmotion to move the running VM from the ESX 3.5 Server to the ESX 4.0 Servers.

So far, no downtime.

Once we’ve moved all the VMs off the standalone ESX 3.5 server, we’ll upgrade the various BIOSs / Firmwares on the ESX host and then install a fresh and fully patched version of vSphere.  Then we’ll re-attach it to the shared storage and move our VMs back.  Finally, we’ll schedule downtime with the respective business owners of those VMs for some downtime to upgrade the VMWare tools and the VM hardware version.

It’s a good little project that lets us leverage many of the cool features in VMWare and should provide quite the fun learning experience.

And for those of you scratching your head and wondering why we are choosing to run somewhat critical VMs on only localized storage – we’re finishing up an expansion of our server room and upgrading to a larger SAN so we can eventually store those VMs in the SAN.  The StorCenter might be used once we’ve upgraded all these standalone ESX boxes as a temporary home, but we’ll have to see.