Monthly Archives: August 2010

Oracle internal cloud session updates from VMworld Day 1

Uncategorized, , , ,

This week I’m at VMware VMworld in San Francisco. Yesterday was day one of the event and the Oracle related highlight for me was session

EA7061 Creating an Internal Oracle Database Cloud Using vSphere by Jeff Browning of EMC.

I’ve been to Jeff’s sessions before and always found them entertaining and informative. Below are some of my thoughts from what was covered at the session.

The most striking informative graphic was an X-Y graph where the X axis was scalability and Y was availability. At the high end of both were Oracle RAC. At the low end of both was MS Access and MySQL. In the sweet spot was Oracle standard edition coupled with VMware vSphere HA clusters.

What does this say to the DBAs? What many of us already knew – not every workload is appropriate for being virtualized under VMware. If your system or the business it’s supporting cannot survive the downtime you’d have in the event of a host failure and subsequent HA restart, you should spend the $$ for Oracle RAC. However, Jeff pointed out that in his experience roughly 90% of systems can survive the downtime associated with a HA event – that’s 90% of the databases out there being good candidates for virtualizing Oracle under VMware vSphere.

One of Jeff’s great examples of why to virtualize was to reduce database sprawl. He cited a Fortune 100 company with 824 physically booted Oracle databases and they pay $228 Million a year to support those machines.

To reduce this sprawl, you’ve got two approaches – according to Jeff, Oracle’s preferred way is to use RAC and come up with one global instance where you can put all your various products. Unfortunately that just doesn’t strike me as realistic in any sort of large company. I run primarily Oracle’s own products and even they can’t run on the same database version in many cases. Oracle E-Business requires Oracle 10g or Oracle 11gR2. Yet Oracle Email Center requires an Oracle 9i database (which needs RedHat 4). A global RAC instance just doesn’t make sense.

The other approach is to virtualize the machines – now I’ve got a RedHat 4 32-bit OS machine running Oracle 9i database on the same hardware as a RedHat 5 64-bit OS running a 11gR2 database. There’s lots of cost savings on both Oracle licensing and reducing the amount of hardware that one can gain with this approach.

One thing I hadn’t really thought about that Jeff brought up with regards to VMware vSphere and Oracle is that the time to vMotion your Oracle database can be longer than with other types of virtual machines – sometimes taking as long as twenty minutes. The reason for this has to deal with how vMotion works – its basically copying the contents of RAM for that VM to another server and then copying over memory blocks that have changed since the first copy, over and over till the delta is very small. Oracle heavily uses memory for its SGA (System Global Area) and so for heavy transaction OLTP systems, vMotions can take a longer than expected time.

The final thing I want to share from Jeff’s presentation was the relevant performance of different protocols and file systems with regards to Oracle and VMware. On the NAS (NFS) storage side, Jeff assigned a value of 95% efficiency when accessing database datafiles via Oracle Direct NFS (DNFS) offering. Compare this to 65% efficiency running VMDKs over traditional NFS. That’s a huge performance difference. As a result, Jeff recommends just using this for a boot / OS disk and definitely not for your database files. On the SAN side, Jeff noted the best performance (100% relative efficiency) comes from using Oracle’s Automatic Storage Management (ASM) with VMWare Raw Disk Mapping (RDM) containers. Compare this with a 98% efficiency with ASM using VMware Virtual Machine Disk Format (VMDK) containers. This is another example of how the Oracle DBAs need to communicate with the VMware administrators when planning out their environment. Many times DBAs don’t even realize they’re running in a virtual environment, and you can’t expect a VMware admin to know about the performance benefits of Oracle DNFS or ASM.

Overall it was a great session and I’m definitely looking forward to applying what I learned to my environments when I get back home.

Too big to fail: Virtualizing Big Iron databases

vmware, , ,

Recently I was talking with a company in Houston,TX running Oracle E-Business (EBS) 11i on Solaris with an Oracle 9i database. They run VMware for other parts of their business and wanted to leverage the features of VMware vSphere and Site Recovery Manager (SRM) to virtualize their EBS environment and have the ability to quickly move their EBS environment in the event of a hurricane or other natural disaster bearing down on Houston to their geographically diverse DR site.

This call had lots of moving parts. They were on Oracle 9i database and wanted to move to Oracle 11g to ease their support costs. They wanted to move from Solaris hardware to commodity x86 hardware and RedHat linux to ease their support costs. Their existing Oracle 9i database was running on and using 8 SPARC processors at peak levels throughout the business day.

in VMware vSphere, the maximum processors you can make visible to a VM is 8 virtual x86 processors. Is a virtualized x86 processor as fast as a physical SPARC processor? Would their SQL run faster on Oracle 11g than it did on Oracle 9i? Is RedHat Linux going to allow the database to process requests as fast as Solaris? Will their SAN storage and LUN layout be fast enough? Will their file system be a limiter?

Besides building up the environment and just going for it in production, how can you know?

By leveraging some very cool tools from both Oracle and VMware.

For Oracle database, Oracle offers an add-on called Real Application Testing which has a feature called Database Replay. Database replay allows you to capture the workload on your production database server and replay it on another environment to see if things are faster or slower. Although this was a new feature on Oracle 11g database, Oracle made backports of this available for 9i and 10g databases – exactly for purposes like this (well, maybe not to aid in virtualizing to VMware, but you get my drift).

Using Database Reply and Real Application Testing (both licensable features from Oracle Enterprise Edition) allows companies to test SAN changes, hardware changes, database upgrades, OS changes, etc., all with a production load, but without risking actual production issues.

Where does VMware fit into this? The way Real Application Testing and Database Reply work is by capturing all the transactions generated in production, massaging them a little bit, and then playing them back against a clone of Production. That clone needs to be at the exact same point in time (or SCN – System Change Number in database speak) as PROD so that the replay is playing back against an exact replica of the database. Although setting up a clone to an exact isn’t hard for an Oracle DBA, it does require time – time to build the test system, time to restore a backup of the database and time to apply archive logs and roll the database forward to match PROD’s SCN.

Even in cases such as this where the Production database isn’t virtualized, by making the test system virtualized, we can not only test all these changes, but we can leverage VMware snapshot technology to allow us to very quickly take our database back to the SCN we want to run Database Replay against, without having to continually restore the database. Using snapshots you just go thru that setup effort once, take a snapshot and then just keep rolling back to your snapshot as many times as necessary to test performance.

Of course, you may find that the 8 processor limit in VMware or the OS or the SAN can’t handle your production load. Time to give up and stay physical? No. In Oracle 10g and further refined in Oracle 11g, Oracle has greatly improved the ability the database has to help a DBA manage the system load and even to have the database tune itself. By leveraging features such as Advanced Compression and SecureFiles (to reduce the physical I/O), Automatic Optimizer Statistics Collection and Automatic SQL Tuning Advisor (to tune queries to use less CPU and/or disk resources), you can give your database more room to grow yet still stay on the same (or less!) hardware.

Kicking the tires on the CloudShare cloud

vmware, , , , ,

I’m extremely lucky as an Oracle Apps DBA – I’ve got literally tons of hardware for me to experiment and play on to learn.   

But not everyone has hardware and software lying around and the time and ability to set it up. I may know my way around Oracle and RedHat Linux, but setup Windows with SQL Server? Active Directory? Sharepoint? Not a chance. That’s what my (very nice and understanding) Windows Administrator coworkers are the experts in. But I came across a way to create a prebuilt environment with all this and more, for free.

Recently I stumbled across CloudShare and their free Pro offering. This isn’t a sales pitch, I promise.

Right now they are in public beta and while it is in public beta, it’s free to use. All you need to sign up is to provide your name and an email address.

With the free account you can create an environment of up to 6 VMs (Linux or Windows) at a time based on the available templates:

Xubuntu 8.04 Desktop

Windows XP With Office 2007

Windows XP Professional

Windows XP Pro with Office 2010

Windows Server 2008 Enterprise Edition x64

Windows Server 2008 Enterprise

Windows Server 2003 R2 Enterprise Edition

Windows 7 Pro

Windows 2008 with SQL Server 2008

Windows 2008 with Microsoft CRM Dynamics

Windows 2008 with Active Directory

Windows 2003 with Sharepoint 2007

Windows 2003 R2 with Oracle 11g

Ubuntu 8.04 Desktop

CentOS 5 With RubyOnRails

CentOS 5 with MySQL

CentOS 5 with KDE

CentOS 5

Sure the machines aren’t the most powerful things out there, but that’s not the point. You want to go through an Oracle install? Go for it. Want to kick the tires on Windows 7 without buying it? 5 minutes and you’re up and running. Want to mess with Sharepoint or Active Directory but avoid all those downloads, installs and configuration? Poof – done. Want to setup a 6 VM environment and share it with friends over the internet? Done.

It’s actually pretty cool.

There’s one feature they have that I especially like: FastUpload functionality. With this you download one of their existing templates (which are good for 90 days), fire it up with VMware Workstation, and make whatever changes you want and then you can upload your VM back to their servers. The magic of this is it doesn’t re-upload the whole VM (Gigabytes and gigabytes of data), but just the changed blocks.

How cool is that? Short of having friends in the know, this is the only legal way I know of to download a prebuilt Windows VM (please note you should have your own Microsoft licenses) and be able to check it in after configuring it just so.

So now you’ve made this super cool environment and want to share it with your friends – you can send them an invitation to use your environment. You can send 10 invitations each month. When your friend opens their invitation, they see the last snapshot of your environment and can make whatever changes they want, without it effecting your original. Your friend can decide to take ownership of the environment and then they become authors of the copy for as long as they want. Think linked clones in VMware speak.

So while your VM is in CloudShare’s cloud, how do you access it? It’s got a External address ( XXXXXXXX.env.cloudshare.com ) so you can access it from anywhere on the Internet and also an Internal IP so it can see the other VMs in your environment. Want to RDP to your private Windows XP Pro with Office 2010 box from your iPhone to open a document in your email? Go for it. Their platform supports http/https, ssh, RDP and any fat client that uses public IPs.

I spent a couple hours messing around in my environment this past weekend and setup an Oracle 11gR1 database on a windows box, downloaded some evaluation software from Oracle and setup a little client server environment involving 3 boxes. I was able to upgrade the Centos box (it was Centos 5.4 – latest is 5.5), and get it configured to talk to my Oracle server.

CloudShare has some big Partners , notably to most of my readers, VMware. There’s even a quote from Paul Maritz (President and CEO of VMware) on the Partners page: “We see the CloudShare platform as an attractive technology that strengthens, supports and extends our ecosystem.”

Why Oracle VM isn’t enterprise ready

Uncategorized, ,

Starting this week, Oracle has publicly started really pushing Oracle’s virtualization products. I attended a seminar on Tuesday covering the road map and yesterday was an all day online Virtualization forum.

Oracle’s server virtualization is focused mainly on two products – Oracle VM for Sparc and Oracle VM for x86. I’m going to focus on Oracle VM for x86, as commodity x86 hardware is the big industry focus and Oracle is really focusing on why you should go with Oracle VM versus VMware.

I’m hear to tell you Oracle VM just isn’t ready for the enterprise. Sure, there are large reference customers out there, but Oracle VM doesn’t have the features I consider necessary to be called enterprise ready. I run VMware vSphere in my enterprise environments and so I’ll compare Oracle VM to VMware vSphere, since I believe VMware vSphere is enterprise ready.

Load Balancing – with virtualization you can run many virtual servers on one physical server. Oracle VM’s load balancing works by performing automated load balancing at each virtual machine power on. Basically what that means is when you start a VM it gets placed on the least busy (in terms of memory and CPU usage) physical server in your server pool. That’s it. VMware’s load balancing called DRS (Distributed Resource Scheduling) not only does this but also checks the load on each host in the cluster every 5 minutes and (if you have it set to fully automated – the VMware best practice) automatically redistributes VMs for the best possible performance.

In my environments, and I suspect almost everyone’s, the workload on the servers changes throughout the day. During the business day, much of the system load is OLTP type loads – users entering data, querying data, placing orders, etc. After the primary business hours, the system load becomes much more batch intensive as things like reports are generated, statistics are gathered, and backups are performed. With Oracle VM, this isn’t taken into account. I could have some Oracle VM servers completely idle while others are overwhelmed. I believe overall system performance to be critical to a product being enterprise ready.

Snapshots – being able to take a snapshot of a VM is, in my belief, critical to an enterprise virtualized environment. Oracle VM doesn’t do snapshots. Simple as that. When I asked on Tuesday at road map seminar with Oracle if that would be available in the next version (officially due sometime in the next 12 months, though I suspect it might be released in the next month), I was told they couldn’t answer yes or no. The fact is, Oracle VM doesn’t have snapshots and VMware vSphere does. But what really is the big deal? Why do I want snapshots?

o Patching – enterprise systems frequently have patches and code changes and need to have a failback plan if something doesn’t go right. With Oracle VM I’m out of luck. Sure I can go back to the last full system backup I took, but we’re probably talking hours of downtime if I need to failback. With VMware vSphere, I take a snapshot of the VM before I start patching (something that takes only a couple of seconds – no exaggeration) and then start my patching. If I need to fail back, I just go in the vSphere menu and choose “Revert to current snapshot” and the VM will restart right back to where it was when you took the snapshot. You even have the option to “snapshot the virtual machine’s memory” meaning if you revert back, your system won’t be in a state as if it had just rebooted, but will have all the processes running as if the machine never stopped.

o Backups – with Oracle VM, if I want to take a backup of the entire VM, I have to use a software agent running inside the VM. As anyone who has ever dealt with Windows knows, you frequently have troubles backing up open files… you know, like an Oracle database or the OS itself. As that backup runs, something that frequently takes hours, files are changing and you’re not getting a completely consistent image of the system. In VMware vSphere, there are many software packages, both from VMware and from third-party vendors that utilize snapshots to take a consistent image of the system. To me, enterprise ready includes good backups. Maybe I’m too demanding.

o Cloning – with Oracle VM if you want to clone a VM, you need to power it off. Yes, if I want to clone my production ERP system VMs, Oracle VM requires I turn VMs off to perform a clone. It’s on page 68 of the Oracle VM Manager 2.1.5 Manual . In VMware vSphere, I can clone with the VM up and available to users. In addition, with the latest version of VMware vSphere, vSphere uses public vStorage APIs to push much of this work onto the SAN itself, thereby reducing and almost eliminating network traffic AND freeing up compute resources on your cluster.

Memory usage – One of the benefits of virtualizing is consolidation – putting many VMs onto one physical server and thereby getting getting better usage of my resources. Oracle VM offers no memory consolidation technologies to increase your consolidation ratios (how many VMs you can put on a physical server). VMware vSphere offers FOUR technologies to increase your consolidation – Transparent Page Sharing, Ballooning, Memory Compression and Swapping. If I’m going to virtualize to consolidate my infrastructure, why not use the product that allows the best consolidation?

There’s many more scenarios where VMware vSphere is a much better and mature product than Oracle VM, but that’s not my point here. My point is that Oracle VM doesn’t meet what I would consider to be an enterprise ready product.

Oracle needs to improve their software installations and accompanying documentation

Uncategorized, , ,

Of the many things Oracle Corporation needs to fix, one big one is their installers and the accompanying documentation. They are, simply, negligent.

The definition of negligence is “failure to act with the prudence that a reasonable person would exercise under the same circumstances”

I’ve been spending the last few days setting up infrastructure for Oracle UCM (Universal Content Manager). Due to the “new-ness” of UCM 11g, we’re going with UCM 10gR3 which has been out for a few years and has been thru many updates. Yes, Oracle acquired this software when it bought Stellent. That was in 2006 and Oracle has released their own versions since then. So no excuses there.

Nowhere in those updates did Oracle think to improve their installation.

Here’s just some of the issues I’ve had to deal with on this software. Please understand this isn’t some cheap piece of software – licensing for our environment was somewhere north of $1M. We’re also using the latest version of Oracle’s flagship database (11gR2 Enterprise Edition) and used the dbca (Database Creation Assistant) to create the database.

11gR2

1) When creating a database using DBCA, which is Oracle’s recommended method, it doesn’t even follow Oracle’s standards for controlfiles or redologs – things that are critical to having a database setup for maximum resistance to disk corruption issues.

a) DBCA by default will create 2 controlfiles – Oracle’s standard is 3 controlfiles.

b) DBCA by default will create 3 redo log file groups each with one member and names each member redo0X.log (X is 1, 2 or 3) and makes them each 50Meg in size. Oracle’s standard is actually 3 groups each with 2 members. Although not Oracle’s standards, I cannot fathom why they would make the file extension .log – you’re just begging for someone to accidentally forget those are critical to database operation and just delete what could easily be construed as unnecessary logging files.

UCM

1) Nowhere in the installation guide does it tell you the characterset HAS to be AL32UTF8 for the automatic installation to succeed.

2) The scripts to automatically create the user for UCM don’t appear to work, but in Oracle’s defense that may somehow be my fault. I can’t get them to work.

3) In the UCM guide Oracle gives sample code for creating a tablespace for the UCM data – yet those create scripts are built for a database that doesn’t use locally managed tablespaces. Locally managed tablespaces was released with Oracle 9i database and UCM requires 9i or higher, so there’s just no reason for this.

4) If you’re doing the install on Linux and using Apache (which I suspect would be the majority of installs), Oracle doesn’t automatically make the necessary changes to Apache needed to get the product working. The pre-installation tasks and considerations (Chapter 3) don’t mention this. The step by step installation instructions (Chapter 4) don’t mention anything about manual setup. The installer itself asks you what web server you wish to use and gives you the following options

Web Server

*1. Apache

2. Sun ONE

3. Configure manually

Doesn’t that imply that if you choose Apache it will be setup automatically?

In Chapter 5 (Post-Installation Tasks and Considerations), the section on Web Servers says

“If, during the installation of the Content Server software, you chose to configure the web server manually, you need to perform a number of tasks to set up and configure the web server for use with Content Server. For further details refer to Appendix A”

Again, doesn’t that imply that since I chose Apache it was setup automatically?

It isn’t until you end up at Appendix A do you find this “Since Apache cannot be configured automatically by the Content Server installer, you need to do it manually” Once you do try those steps, you’ll notice they use non-default filename paths and inconsistent server instance names in their examples.

Seriously Oracle, this is negligent. I could write out similar blog posts about most Oracle products I’ve had to install over the years.

Oracle, before you publish your documentation, take it to someone not on the development team, give them the documentation and have them try and follow the steps. I think you’ll be surprised by the results.

a rant about FUD about Oracle on VMware

Uncategorized, , , ,

You know those searchXXXXXXXXXXX.com websites (searchoracle.techtarget.com, searchvmware.techtarget.com, searchvirtualization.techtarget.com etc)? There are some good articles, but I keep seeing alot of plainly inaccurate articles about virtualized Oracle, especially Oracle under VMware.

I keep seeing these FUD (Fear, Uncertainty, Doubt) articles on searchXXXXXXXXXX.com that just bug the hell out of me.

Take this article about Oracle RAC on VMware . They start out with something reasonable and accurate

Oracle will not support customers running Oracle RAC on VMware, for reasons that many say are political and technically outdated.

and then say things that are just completely not true:

“In short, Oracle won’t support it unless the customer can prove that the problem wasn’t related to the virtual machine.

…

While getting support for single-instance Oracle on VMware is difficult…”

I run multiple Oracle databases and various Oracle products (Oracle E-Business Suite, Oracle Hyperion, Oracle Universal Content Manager, Oracle Hyperion, Oracle Agile, etc) and It is no different to get support for Oracle virtualized under VMware unless troubleshooting leads Oracle Support to suspect your issue is with VMware itself.

The rest of the article I pretty much agree with. I’ve met Dave Welch numerous times and find his outlook on Oracle on VMware similar to mine. Oracle’s stated “we do not support Oracle RAC on VMware” appears to be nothing more than Oracle’s whim with no current technical issues to back it up. As much as I don’t like it, that’s their choice. If / when VMware starts getting Fault Tolerance working with multiple CPUs in a VM, that’s going to mute the argument about needing to run Oracle RAC solely for uptime requirements. Sure, you’ll still have VMs that aren’t good candidates for virtualization (Oracle VM or VMware) but that’s not the bulk of installations out there.

Oracle uses VMware in its training classes – I attended an Oracle Hyperion installation and configuration class last year that utilized VMware Workstation running 3 or 4 VMs on each student’s machine. I’ve worked issues with Oracle Hyperion with Oracle Support and had the analyst not only notice my environment was under VMware, but state that roughly half their customers run Hyperion virtualized under VMware. With Oracle now having Oracle VM and Oracle Virtualbox, you’d think at least Oracle’s own training partners would be using Oracle products in their lab and you’d think if this support was such a big deal that I’d have Oracle’s support telling me about the benefits of Oracle VM when they noticed I was running VMware.

Here’s another article that bugged me, this time about how Oracle VM is not half bad . First line of the article:

“Oracle’s continued refusal to support its applications virtualized on something other than the Oracle VM hypervisor has forced the hands of some users, pushing them to try the Xen-based virtualization offering.”

Did you see what I did? ” Oracle’s continued refusal to support its applications virtualized on something other than the Oracle VM hypervisor..”. That’s simply and plainly wrong.

I’ll end this article quoting from the official stance of Oracle Support with regards to VMware, My Oracle Support (aka Metalink) note 249212.1

Support Status for VMware Virtualized Environments 
-------------------------------------------------- 
Oracle has not certified any of its products on VMware virtualized 
environments. Oracle Support will assist customers running Oracle products 
on VMware in the following manner: Oracle will only provide 
support for issues that either are known to occur on the native OS, or 
can be demonstrated not to be as a result of running on VMware. 

If a problem is a known Oracle issue, Oracle support will recommend the 
appropriate solution on the native OS.  If that solution does not work in 
the VMware virtualized environment, the customer will be referred to VMware 
for support.   When the customer can demonstrate that the Oracle solution 
does not work when running on the native OS, Oracle will resume support, 
including logging a bug with Oracle Development for investigation if required.


---

A battery improvement tip with VMware Fusion

Uncategorized, , , ,

So I recently switched from PCs running a RHEL base operating system to a MacBook Pro running MacOS. It’s been overall a pretty smooth transition, but with plenty of small bumps along the way.

In current MacBook Pros (MBPs) there are 2 graphics chipsets – an integrated Intel chipset and a NVIDIA discrete chipset. The NVIDIA gives much better graphics performance but at the expense of battery life.

Whenever I start up a Windows VM, I found the system would automatically switch to the NVIDIA chipset. Since I don’t use my Windows VMs for graphics intensive usage (they’re mainly to run those few Windows only business applications), I needed to find a way to force the system to stay using the Intel chipset.

I came across gfxCardStatus . With this program I can manually switch which graphics chipset is being used. I’ve found that I need to set my chipset to Intel only before starting the VM in order for things to work properly. If I try to change it while the Windows VM is already running, the VM will no longer respond to keyboard input.

This *may* also be the case with Linux and ESX VMs – I haven’t run any of them recently. It is definitely an issue with Windows XP VMs.

Hope this helps!