Category Archives: Uncategorized

Oracle VM Security: Sometimes you need hip waders

Have you ever read something and thought, “what a load of crap. I had better get my hip waders out.”

Well, as a cynical jaded DBA, I have that experience regularly.

Take this Oracle.com blog post on Oracle VM where Rene Kundersma who is a Technical Architect with Oracle explains Oracle’s reasons for NOT shipping Oracle VM with a “fancy Gnome X-Window” environment:

“Oracle has it reasons to NOT ship Oracle VM with all the bells and whistles of a fancy Gnome X-Window environment. This has to do with vulnerabilities, not tested situations of software combination’s and whatever reason that makes Oracle VM not to behave as tested and intended.”

Vulnerabilities as the reason for Oracle VM not having a “fancy X-Window environment”. Vulnerabilities… really? But isn’t Oracle VM running on a special version of Oracle Unbreakable Linux (hint: yes – they’re both based off of RedHat Enterprise Linux)?

Want to get to the console of a VM running under Oracle VM? It uses VNC. Sure, you need to know the password to connect to the VNC Desktop, but guess what, the VNC traffic isn’t encrypted. The password is sent in cleartext.

Unbreakable indeed.

I find this all the more contradictory when one of Oracle’s talking points for why to use Oracle VM is Secure Live Migration which SSL encrypts the live migration (aka vMotion) traffic. My favorite line: “No need to purchase special hardware or deploy special secure networks. “

No need to deploy special secure networks! VLANs? Who needs them? We’ve got encrypted live migration!

Oh wait, in Oracle’s own Oracle Real Application Clusters in Oracle VM Environments guide, there’s this tidbit

“While Secure Live Migration is enabled by default, it should be considered that a secure connection to the remote guest (using –ssl) adds overhead to the Live Migration operation. It is therefore recommended to avoid such secure connections, if permitted. In general, a secure connection can be avoided, if the Live Migration network is inherently secure. “

Seriously Oracle, which is it?

But let’s get back to the main point Rene was trying to get across – that Oracle VM doesn’t come with a GUI to reduce vulnerabilities. Oracle’s October 2010 CPU (Critical Patch Update) was released on October 12th, 2010 and for the current version of Oracle VM (2.2.1) it lists 4 vulnerabilities, 3 of which have a base score of 9.0 (the scale is from 0.0 to 10.0, with 10.0 representing the highest severity of vulnerability). All 3 of those 9/10 severity vulnerabilities have a low access complexity (they’re easy to do) and result in complete access.

Oracle, thank you for not including a “fancy Gnome X-Window” with Oracle VM so as to reduce vulnerabilities. Given how insecure your product appears without a GUI, I shudder to think what things would be like with a GUI.

An alternative to scrambling data: Restricting access with Virtual Private Database (VPD)

Back in June, I wrote a blog post on scrambling HR data in our EBS instances . Although effective, it was a bit of a kludge – it involved an excel spreadsheet, and giving everyone the same salary and banking info.

As we went with this solution in our development and test environments, we ran into some issues where the salary data would totally screw up the benefits data as it’s calculated as a percentage of salary. The solution was effective at keeping the data secure, but it wasn’t optimal. After some investigation, we turned to Oracle VPD – Virtual Private Database – functionality. With this we are able to restrict access to certain columns (such as salary or national identifiers) to all but necessary users. With an EBS database, where every connection is connect as APPS, this poses special considerations.

I’ll cover the technical details of implementing VPD in an EBS environment. Then I’ll talk about the changes you need to make to keep things functional for your business analysts and yet keep the data secure.

First it was necessary to create a policy function. In our case this is very generic, basically just returning the predicate.

CREATE OR REPLACE FUNCTION “APPS”.”LUM_HIDE_HR_COLS” (schema in varchar2, tab in varchar2)

return varchar2 as predicate varchar2(8) default ‘1=2’;

begin

return predicate;

end;

/

Next we add a policy on the column we want to restrict access to point it at the policy function we created

begin dbms_rls.add_policy(object_schema => ‘HR’, object_name => ‘PER_ALL_PEOPLE_F’, policy_name => ‘LUM_HIDE_HR_COLS’, function_schema => ‘APPS’, policy_function => ‘LUM_HIDE_HR_COLS’, statement_types => ‘select,’, update_check => FALSE , enable => TRUE , static_policy => FALSE , policy_type => dbms_rls.STATIC , long_predicate => FALSE , sec_relevant_cols => ‘NATIONAL_IDENTIFIER,’ , sec_relevant_cols_opt => DBMS_RLS.ALL_ROWS ); end;

/

In this case we’re ADDing a policy and ENABLing it, applying the LUM_HIDE_HR_COLS function on the NATIONAL_IDENTIFIER column of table PER_ALL_PEOPLE_F in the HR schema, preventing users from SELECTing data and stating that this is for ALL_ROWS.

Once we issued that, all users (besides SYS and SYSTEM) will get NULLs when they select NATIONAL_IDENTIFIER from that table. That took care of our social security number concern.

We also set up additional policies on other data:

begin dbms_rls.add_policy(object_schema => ‘HR’, object_name => ‘PER_PAY_PROPOSALS’, policy_name => ‘LUM_HIDE_HR_COLS’, function_schema => ‘APPS’, policy_function => ‘LUM_HIDE_HR_COLS’, statement_types => ‘select,’, update_check => FALSE , enable => TRUE , static_policy => FALSE , policy_type => dbms_rls.STATIC , long_predicate => FALSE , sec_relevant_cols => ‘PROPOSED_SALARY_N,’ , sec_relevant_cols_opt => DBMS_RLS.ALL_ROWS ); end;

/

The above policy restricts access to PROPOSED_SALARY_N column of HR.PER_PAY_PROPOSALS. That took care of our salary data concern.

begin dbms_rls.add_policy(object_schema => ‘HR’, object_name => ‘PAY_EXTERNAL_ACCOUNTS’, policy_name => ‘LUM_HIDE_HR_COLS’, function_schema => ‘APPS’, policy_function => ‘LUM_HIDE_HR_COLS’, statement_types => ‘select,’, update_check => FALSE , enable => TRUE , static_policy => FALSE , policy_type => dbms_rls.STATIC , long_predicate => FALSE , sec_relevant_cols => ‘SEGMENT3,SEGMENT4,’ , sec_relevant_cols_opt => DBMS_RLS.ALL_ROWS ); end;
/

The above policy restricts access to the SEGMENT3 and SEGMENT4 columns of HR.PAY_EXTERNAL_ACCOUNTS. That took care of our banking data concern.

begin dbms_rls.add_policy(object_schema => ‘HR’, object_name => ‘PER_ADDRESSES’, policy_name => ‘LUM_HIDE_HR_COLS’, function_schema => ‘APPS’, policy_function => ‘LUM_HIDE_HR_COLS’, statement_types => ‘select,’, update_check => FALSE , enable => TRUE , static_policy => FALSE , policy_type => dbms_rls.STATIC , long_predicate => FALSE , sec_relevant_cols => ‘ADDRESS_LINE1,ADDRESS_LINE2,’ , sec_relevant_cols_opt => DBMS_RLS.ALL_ROWS ); end;
/
The above policy restricts access to ADDRESS_LINE1 and ADDRESS_LINE2 columns of HR.PER_ADDRESSES. That took care of our concern of employees looking up addresses of other employees.

begin dbms_rls.add_policy(object_schema => ‘HR’, object_name => ‘PER_ALL_ASSIGNMENTS_F’, policy_name => ‘LUM_HIDE_HR_COLS’, function_schema => ‘APPS’, policy_function => ‘LUM_HIDE_HR_COLS’, statement_types => ‘select,’, update_check => FALSE , enable => TRUE , static_policy => FALSE , policy_type => dbms_rls.STATIC , long_predicate => FALSE , sec_relevant_cols => ‘GRADE_ID,’ , sec_relevant_cols_opt => DBMS_RLS.ALL_ROWS ); end;
/

The above policy restricts access to the GRADE_ID column of HR.PER_ALL_ASSIGNMENTS_F. That took care of our concern about employees looking up the pay grade of other employees.

begin dbms_rls.add_policy(object_schema => ‘HR’, object_name => ‘PAY_ELEMENT_ENTRY_VALUES_F’, policy_name => ‘LUM_HIDE_HR_COLS’, function_schema => ‘APPS’, policy_function => ‘LUM_HIDE_HR_COLS’, statement_types => ‘select,’, update_check => FALSE , enable => TRUE , static_policy => FALSE , policy_type => dbms_rls.STATIC , long_predicate => FALSE , sec_relevant_cols => ‘SCREEN_ENTRY_VALUE,’ , sec_relevant_cols_opt => DBMS_RLS.ALL_ROWS ); end;
/

The above policy restricts access to the SCREEN_ENTRY_VALUE column of HR.PAY_ELEMENT_ENTRY_VALUES_F. That took care of our figuring out salary based on insurance coverage concern.

That’s all there is to it. You can just issue the code above in an Apps 11i instance and at that point no user besides SYS and SYSTEM can see that data.
Now that you’ve handled the technical details, there’s the issue of cloning the instances and being able to test HR functionality in development environments while still restricting the data. Here’s what we do.
In PROD, in addition to the function and policies listed above, we create a read-only user calls APPSHR:

create user APPSHR identified by XXXXXXXXXXXXX;
GRANT ALTER SESSION TO “APPSHR”;
GRANT CREATE DATABASE LINK TO “APPSHR”;
GRANT CREATE PUBLIC SYNONYM TO “APPSHR”;
GRANT CREATE SESSION TO “APPSHR”;
GRANT CREATE SYNONYM TO “APPSHR”;
GRANT EXECUTE ANY PROCEDURE TO “APPSHR”;
GRANT SELECT ANY DICTIONARY TO “APPSHR”;
GRANT SELECT ANY SEQUENCE TO “APPSHR”;
GRANT SELECT ANY TABLE TO “APPSHR”;
GRANT UNLIMITED TABLESPACE TO “APPSHR”;
GRANT “RESOURCE” TO “APPSHR”;
ALTER USER “APPSHR” DEFAULT ROLE ALL;
That APPSHR user now has the ability to select any data in the system but it’s read only (no updating).
We then exempt the APPS and APPSHR user in PROD ONLY from the policies we created:

GRANT EXEMPT ACCESS POLICY to APPS;
GRANT EXEMPT ACCESS POLICY to APPSHR;

At this point only users who connect at the database level as APPS (that would be all forms based users) and APPSHR (our HR analysts) can see the restricted data. The APPS password in our PROD environment is known only to the DBAs. The APPSHR password is known only to the HR Business Analysts / HR Developers / DBAs. All other business analysts have access to another read-only account called APPSXXXX that is NOT exempt from the security policies. With that APPSXXXX account, the regular business analysts and developers can query the database directly for all but the restricted data and can access any data via the forms that their forms responsibilities allow.
When we clone an instance, we don’t have to do any scrambling. For all of our DEV, TEST and PSUP (Production Support) instances we merely have to change the APPS password to a commonly known password and issue

REVOKE EXEMPT ACCESS POLICY FROM APPS;

Now everyone can read and write data as APPS, but because APPS now is restricted by the policies we put in place, APPS can’t see the sensitive data. The only time this causes a problem is for the HR Business Analyst or HR Developers who need a non-PROD place to work issues or develop code. For them we created a special HR cloned instance with the same security setup as PROD but with the APPS password known to the HR Analysts and HR Developers.

This solution has worked out much better for us than the scrambling. Give it a shot and let me know how it goes for you in the feedback!

Oracle Advanced Compression Advisor

My main Oracle Applications Database has been growing steadily and is now around 270GB. In terms of databases this isn’t huge, but when you keep multiple development and test copies around on enterprise class storage, AND replicate it to your DR site, that can get expensive.

With Oracle 11g database, Oracle came out with two products to help manage space (and improve performance!) in your Oracle database – Oracle Advanced Compression and Oracle SecureFiles. Although both are for reducing disk usage, they are aimed at different areas of disk usage within an Oracle database.

SecureFiles is the next generation of storage for Oracle LOBs (Large OBjects). Oracle LOBs stored in the format before SecureFiles are said to be stored in BasicFiles. SecureFiles is aimed at attachments to the database – CLOBS (Character LOBs), BLOBs (Binary LOBs), and NCLOBs (multi-byte character LOBs). SecureFiles offers a number of benefits over BasicFiles. Two are relevant to reducing space usage – de-duplification and compression. SecureFiles is a free feature of Enterprise Edition and has no additional licensing costs. As a result, it’s the sort of low hanging fruit that should be of interest to any Oracle DBA out there – free improved performance and free reduced disk storage. What’s not to like? Because this feature is free, we’re actively testing with this in our environments and plan on rolling this out by end of year. I’ll post a much longer blog post with our space savings and details of converting data from BasicFiles to SecureFiles later.

Advanced Compression is aimed at table data – compressing the data stored in the tables. This not only saves space on the file system, but actually improves performance by reducing the amount of data that needs to be read from disk (reading data from disk is frequently the bottleneck with Oracle databases – which is why Oracle is so memory hungry and tries to cache much of the data it needs in the System Global Area (SGA)). Advanced Compression is a add-on feature to Enterprise Edition at a cost of $11,500 per x86 license (and remember it takes TWO x86 CORES to equal one x86 LICENSE) – and like everything Oracle, that is based on how many cores are in the box, not how many your database cpu_count is set to or VM (if you virtualize your Oracle database) utilizes.

With Oracle Enterprise Manager (OEM) 11g, one of the new features is a Compression Advisor. You can read about other reasons to upgrade to OEM 11g at this blog post on OEM 11g new features. When run against an Oracle 11gR2 database, this advisor will analyze your database objects, estimate the compression ratio you’ll achieve and even make recommendations on the best compression settings for your environment. Although my largest database is 11gR2, I have a variety of other database versions on those same physical hosts (gotta love virtualization!) that aren’t 11gR2 and hence don’t have the DBMS_COMPRESSION package.

Luckily, I stumbled across a standalone version on Oracle Technology Network. This standalone version will work with Oracle 9iR2 (9.2.0.X) through 11gR1 (11.1.0.X) and can give you the data you need to convince business areas to upgrade to 11g database.

One thing to be aware of with this script: it will create a temporary table of the compressed data so you may wish to reating a tablespace specifically for storing the temporary table and making that the default tablespace of the user executing the script. The temporary table gets dropped at the end.

Note: The example on the Oracle Technology Network link above is incorrect. It is using the DBMS_COMPRESSION package which is in 11gR2 Oracle database and NOT provided by this package. So if using an 11gR2 database, you use DBMS_COMPRESSION package, but if using a 9iR2 thru 11gR1 database, use the DBMS_COMP_ADVISOR package like in my example below

Here’s the output from running it against a 9.2.0.8 database with a table OM_DATA in a schema called OO_MAIL. The table has 4.5 million rows and is 9.5 GB in size. (The product that uses this database requires Oracle 9iR2, for those wondering)

SQL> exec DBMS_COMP_ADVISOR.getratio(‘OO_MAIL’,’OM_DATA’,’OLTP’,25);

Sampling table: OO_MAIL.OM_DATA

Sampling percentage: 25%

Compression Type: OLTP

Estimated Compression Ratio: 1.62

PL/SQL procedure successfully completed.


I also ran this against my largest table in my Oracle Applications (11gR2) instance (INV.MTL_TRANSACTION_ACCOUNTS) – a 2.5GB table with 14 million rows:


Sampling table: INV.MTL_TRANSACTION_ACCOUNTS

Sampling percentage: 25%
Compression Type: OLTP
Estimated Compression Ratio: 2.57

So that works out to 3.64GB space I would save on the 9i database and 1.57GB in my 11gR2 database. A total of about 5GB saved. Every database (and the data it contains) is different, so run the numbers against your database to decide if Advanced Compression is worth it in your environment… and check out SecureFiles. It’s free.

Oracle internal cloud session updates from VMworld Day 1

This week I’m at VMware VMworld in San Francisco. Yesterday was day one of the event and the Oracle related highlight for me was session

EA7061 Creating an Internal Oracle Database Cloud Using vSphere by Jeff Browning of EMC.

I’ve been to Jeff’s sessions before and always found them entertaining and informative. Below are some of my thoughts from what was covered at the session.

The most striking informative graphic was an X-Y graph where the X axis was scalability and Y was availability. At the high end of both were Oracle RAC. At the low end of both was MS Access and MySQL. In the sweet spot was Oracle standard edition coupled with VMware vSphere HA clusters.

What does this say to the DBAs? What many of us already knew – not every workload is appropriate for being virtualized under VMware. If your system or the business it’s supporting cannot survive the downtime you’d have in the event of a host failure and subsequent HA restart, you should spend the $$ for Oracle RAC. However, Jeff pointed out that in his experience roughly 90% of systems can survive the downtime associated with a HA event – that’s 90% of the databases out there being good candidates for virtualizing Oracle under VMware vSphere.

One of Jeff’s great examples of why to virtualize was to reduce database sprawl. He cited a Fortune 100 company with 824 physically booted Oracle databases and they pay $228 Million a year to support those machines.

To reduce this sprawl, you’ve got two approaches – according to Jeff, Oracle’s preferred way is to use RAC and come up with one global instance where you can put all your various products. Unfortunately that just doesn’t strike me as realistic in any sort of large company. I run primarily Oracle’s own products and even they can’t run on the same database version in many cases. Oracle E-Business requires Oracle 10g or Oracle 11gR2. Yet Oracle Email Center requires an Oracle 9i database (which needs RedHat 4). A global RAC instance just doesn’t make sense.

The other approach is to virtualize the machines – now I’ve got a RedHat 4 32-bit OS machine running Oracle 9i database on the same hardware as a RedHat 5 64-bit OS running a 11gR2 database. There’s lots of cost savings on both Oracle licensing and reducing the amount of hardware that one can gain with this approach.

One thing I hadn’t really thought about that Jeff brought up with regards to VMware vSphere and Oracle is that the time to vMotion your Oracle database can be longer than with other types of virtual machines – sometimes taking as long as twenty minutes. The reason for this has to deal with how vMotion works – its basically copying the contents of RAM for that VM to another server and then copying over memory blocks that have changed since the first copy, over and over till the delta is very small. Oracle heavily uses memory for its SGA (System Global Area) and so for heavy transaction OLTP systems, vMotions can take a longer than expected time.

The final thing I want to share from Jeff’s presentation was the relevant performance of different protocols and file systems with regards to Oracle and VMware. On the NAS (NFS) storage side, Jeff assigned a value of 95% efficiency when accessing database datafiles via Oracle Direct NFS (DNFS) offering. Compare this to 65% efficiency running VMDKs over traditional NFS. That’s a huge performance difference. As a result, Jeff recommends just using this for a boot / OS disk and definitely not for your database files. On the SAN side, Jeff noted the best performance (100% relative efficiency) comes from using Oracle’s Automatic Storage Management (ASM) with VMWare Raw Disk Mapping (RDM) containers. Compare this with a 98% efficiency with ASM using VMware Virtual Machine Disk Format (VMDK) containers. This is another example of how the Oracle DBAs need to communicate with the VMware administrators when planning out their environment. Many times DBAs don’t even realize they’re running in a virtual environment, and you can’t expect a VMware admin to know about the performance benefits of Oracle DNFS or ASM.

Overall it was a great session and I’m definitely looking forward to applying what I learned to my environments when I get back home.

Why Oracle VM isn’t enterprise ready

Starting this week, Oracle has publicly started really pushing Oracle’s virtualization products. I attended a seminar on Tuesday covering the road map and yesterday was an all day online Virtualization forum.

Oracle’s server virtualization is focused mainly on two products – Oracle VM for Sparc and Oracle VM for x86. I’m going to focus on Oracle VM for x86, as commodity x86 hardware is the big industry focus and Oracle is really focusing on why you should go with Oracle VM versus VMware.

I’m hear to tell you Oracle VM just isn’t ready for the enterprise. Sure, there are large reference customers out there, but Oracle VM doesn’t have the features I consider necessary to be called enterprise ready. I run VMware vSphere in my enterprise environments and so I’ll compare Oracle VM to VMware vSphere, since I believe VMware vSphere is enterprise ready.

Load Balancing – with virtualization you can run many virtual servers on one physical server. Oracle VM’s load balancing works by performing automated load balancing at each virtual machine power on. Basically what that means is when you start a VM it gets placed on the least busy (in terms of memory and CPU usage) physical server in your server pool. That’s it. VMware’s load balancing called DRS (Distributed Resource Scheduling) not only does this but also checks the load on each host in the cluster every 5 minutes and (if you have it set to fully automated – the VMware best practice) automatically redistributes VMs for the best possible performance.

In my environments, and I suspect almost everyone’s, the workload on the servers changes throughout the day. During the business day, much of the system load is OLTP type loads – users entering data, querying data, placing orders, etc. After the primary business hours, the system load becomes much more batch intensive as things like reports are generated, statistics are gathered, and backups are performed. With Oracle VM, this isn’t taken into account. I could have some Oracle VM servers completely idle while others are overwhelmed. I believe overall system performance to be critical to a product being enterprise ready.

Snapshots – being able to take a snapshot of a VM is, in my belief, critical to an enterprise virtualized environment. Oracle VM doesn’t do snapshots. Simple as that. When I asked on Tuesday at road map seminar with Oracle if that would be available in the next version (officially due sometime in the next 12 months, though I suspect it might be released in the next month), I was told they couldn’t answer yes or no. The fact is, Oracle VM doesn’t have snapshots and VMware vSphere does. But what really is the big deal? Why do I want snapshots?

o Patching – enterprise systems frequently have patches and code changes and need to have a failback plan if something doesn’t go right. With Oracle VM I’m out of luck. Sure I can go back to the last full system backup I took, but we’re probably talking hours of downtime if I need to failback. With VMware vSphere, I take a snapshot of the VM before I start patching (something that takes only a couple of seconds – no exaggeration) and then start my patching. If I need to fail back, I just go in the vSphere menu and choose “Revert to current snapshot” and the VM will restart right back to where it was when you took the snapshot. You even have the option to “snapshot the virtual machine’s memory” meaning if you revert back, your system won’t be in a state as if it had just rebooted, but will have all the processes running as if the machine never stopped.

o Backups – with Oracle VM, if I want to take a backup of the entire VM, I have to use a software agent running inside the VM. As anyone who has ever dealt with Windows knows, you frequently have troubles backing up open files… you know, like an Oracle database or the OS itself. As that backup runs, something that frequently takes hours, files are changing and you’re not getting a completely consistent image of the system. In VMware vSphere, there are many software packages, both from VMware and from third-party vendors that utilize snapshots to take a consistent image of the system. To me, enterprise ready includes good backups. Maybe I’m too demanding.

o Cloning – with Oracle VM if you want to clone a VM, you need to power it off. Yes, if I want to clone my production ERP system VMs, Oracle VM requires I turn VMs off to perform a clone. It’s on page 68 of the Oracle VM Manager 2.1.5 Manual . In VMware vSphere, I can clone with the VM up and available to users. In addition, with the latest version of VMware vSphere, vSphere uses public vStorage APIs to push much of this work onto the SAN itself, thereby reducing and almost eliminating network traffic AND freeing up compute resources on your cluster.

Memory usage – One of the benefits of virtualizing is consolidation – putting many VMs onto one physical server and thereby getting getting better usage of my resources. Oracle VM offers no memory consolidation technologies to increase your consolidation ratios (how many VMs you can put on a physical server). VMware vSphere offers FOUR technologies to increase your consolidation – Transparent Page Sharing, Ballooning, Memory Compression and Swapping. If I’m going to virtualize to consolidate my infrastructure, why not use the product that allows the best consolidation?

There’s many more scenarios where VMware vSphere is a much better and mature product than Oracle VM, but that’s not my point here. My point is that Oracle VM doesn’t meet what I would consider to be an enterprise ready product.

Oracle needs to improve their software installations and accompanying documentation

Of the many things Oracle Corporation needs to fix, one big one is their installers and the accompanying documentation. They are, simply, negligent.

The definition of negligence is “failure to act with the prudence that a reasonable person would exercise under the same circumstances”

I’ve been spending the last few days setting up infrastructure for Oracle UCM (Universal Content Manager). Due to the “new-ness” of UCM 11g, we’re going with UCM 10gR3 which has been out for a few years and has been thru many updates. Yes, Oracle acquired this software when it bought Stellent. That was in 2006 and Oracle has released their own versions since then. So no excuses there.

Nowhere in those updates did Oracle think to improve their installation.

Here’s just some of the issues I’ve had to deal with on this software. Please understand this isn’t some cheap piece of software – licensing for our environment was somewhere north of $1M. We’re also using the latest version of Oracle’s flagship database (11gR2 Enterprise Edition) and used the dbca (Database Creation Assistant) to create the database.

11gR2

1) When creating a database using DBCA, which is Oracle’s recommended method, it doesn’t even follow Oracle’s standards for controlfiles or redologs – things that are critical to having a database setup for maximum resistance to disk corruption issues.

a) DBCA by default will create 2 controlfiles – Oracle’s standard is 3 controlfiles.

b) DBCA by default will create 3 redo log file groups each with one member and names each member redo0X.log (X is 1, 2 or 3) and makes them each 50Meg in size. Oracle’s standard is actually 3 groups each with 2 members. Although not Oracle’s standards, I cannot fathom why they would make the file extension .log – you’re just begging for someone to accidentally forget those are critical to database operation and just delete what could easily be construed as unnecessary logging files.

UCM

1) Nowhere in the installation guide does it tell you the characterset HAS to be AL32UTF8 for the automatic installation to succeed.

2) The scripts to automatically create the user for UCM don’t appear to work, but in Oracle’s defense that may somehow be my fault. I can’t get them to work.

3) In the UCM guide Oracle gives sample code for creating a tablespace for the UCM data – yet those create scripts are built for a database that doesn’t use locally managed tablespaces. Locally managed tablespaces was released with Oracle 9i database and UCM requires 9i or higher, so there’s just no reason for this.

4) If you’re doing the install on Linux and using Apache (which I suspect would be the majority of installs), Oracle doesn’t automatically make the necessary changes to Apache needed to get the product working. The pre-installation tasks and considerations (Chapter 3) don’t mention this. The step by step installation instructions (Chapter 4) don’t mention anything about manual setup. The installer itself asks you what web server you wish to use and gives you the following options

Web Server

*1. Apache

2. Sun ONE

3. Configure manually

Doesn’t that imply that if you choose Apache it will be setup automatically?

In Chapter 5 (Post-Installation Tasks and Considerations), the section on Web Servers says

“If, during the installation of the Content Server software, you chose to configure the web server manually, you need to perform a number of tasks to set up and configure the web server for use with Content Server. For further details refer to Appendix A”

Again, doesn’t that imply that since I chose Apache it was setup automatically?

It isn’t until you end up at Appendix A do you find this “Since Apache cannot be configured automatically by the Content Server installer, you need to do it manually” Once you do try those steps, you’ll notice they use non-default filename paths and inconsistent server instance names in their examples.

Seriously Oracle, this is negligent. I could write out similar blog posts about most Oracle products I’ve had to install over the years.

Oracle, before you publish your documentation, take it to someone not on the development team, give them the documentation and have them try and follow the steps. I think you’ll be surprised by the results.

a rant about FUD about Oracle on VMware

You know those searchXXXXXXXXXXX.com websites (searchoracle.techtarget.com, searchvmware.techtarget.com, searchvirtualization.techtarget.com etc)? There are some good articles, but I keep seeing alot of plainly inaccurate articles about virtualized Oracle, especially Oracle under VMware.

I keep seeing these FUD (Fear, Uncertainty, Doubt) articles on searchXXXXXXXXXX.com that just bug the hell out of me.

Take this article about Oracle RAC on VMware . They start out with something reasonable and accurate

Oracle will not support customers running Oracle RAC on VMware, for reasons that many say are political and technically outdated.

and then say things that are just completely not true:

In short, Oracle won’t support it unless the customer can prove that the problem wasn’t related to the virtual machine.

While getting support for single-instance Oracle on VMware is difficult…”

I run multiple Oracle databases and various Oracle products (Oracle E-Business Suite, Oracle Hyperion, Oracle Universal Content Manager, Oracle Hyperion, Oracle Agile, etc) and It is no different to get support for Oracle virtualized under VMware unless troubleshooting leads Oracle Support to suspect your issue is with VMware itself.

The rest of the article I pretty much agree with. I’ve met Dave Welch numerous times and find his outlook on Oracle on VMware similar to mine. Oracle’s stated “we do not support Oracle RAC on VMware” appears to be nothing more than Oracle’s whim with no current technical issues to back it up. As much as I don’t like it, that’s their choice. If / when VMware starts getting Fault Tolerance working with multiple CPUs in a VM, that’s going to mute the argument about needing to run Oracle RAC solely for uptime requirements. Sure, you’ll still have VMs that aren’t good candidates for virtualization (Oracle VM or VMware) but that’s not the bulk of installations out there.

Oracle uses VMware in its training classes – I attended an Oracle Hyperion installation and configuration class last year that utilized VMware Workstation running 3 or 4 VMs on each student’s machine. I’ve worked issues with Oracle Hyperion with Oracle Support and had the analyst not only notice my environment was under VMware, but state that roughly half their customers run Hyperion virtualized under VMware. With Oracle now having Oracle VM and Oracle Virtualbox, you’d think at least Oracle’s own training partners would be using Oracle products in their lab and you’d think if this support was such a big deal that I’d have Oracle’s support telling me about the benefits of Oracle VM when they noticed I was running VMware.

Here’s another article that bugged me, this time about how Oracle VM is not half bad . First line of the article:

“Oracle’s continued refusal to support its applications virtualized on something other than the Oracle VM hypervisor has forced the hands of some users, pushing them to try the Xen-based virtualization offering.”

Did you see what I did? ” Oracle’s continued refusal to support its applications virtualized on something other than the Oracle VM hypervisor..”. That’s simply and plainly wrong.

I’ll end this article quoting from the official stance of Oracle Support with regards to VMware, My Oracle Support (aka Metalink) note 249212.1

Support Status for VMware Virtualized Environments 
-------------------------------------------------- 
Oracle has not certified any of its products on VMware virtualized 
environments. Oracle Support will assist customers running Oracle products 
on VMware in the following manner: Oracle will only provide 
support for issues that either are known to occur on the native OS, or 
can be demonstrated not to be as a result of running on VMware. 

If a problem is a known Oracle issue, Oracle support will recommend the 
appropriate solution on the native OS.  If that solution does not work in 
the VMware virtualized environment, the customer will be referred to VMware 
for support.   When the customer can demonstrate that the Oracle solution 
does not work when running on the native OS, Oracle will resume support, 
including logging a bug with Oracle Development for investigation if required.

---

A battery improvement tip with VMware Fusion

So I recently switched from PCs running a RHEL base operating system to a MacBook Pro running MacOS. It’s been overall a pretty smooth transition, but with plenty of small bumps along the way.

In current MacBook Pros (MBPs) there are 2 graphics chipsets – an integrated Intel chipset and a NVIDIA discrete chipset. The NVIDIA gives much better graphics performance but at the expense of battery life.

Whenever I start up a Windows VM, I found the system would automatically switch to the NVIDIA chipset. Since I don’t use my Windows VMs for graphics intensive usage (they’re mainly to run those few Windows only business applications), I needed to find a way to force the system to stay using the Intel chipset.

I came across gfxCardStatus . With this program I can manually switch which graphics chipset is being used. I’ve found that I need to set my chipset to Intel only before starting the VM in order for things to work properly. If I try to change it while the Windows VM is already running, the VM will no longer respond to keyboard input.

This *may* also be the case with Linux and ESX VMs – I haven’t run any of them recently. It is definitely an issue with Windows XP VMs.

Hope this helps!

VMware Knowledge base entries of interest for Oracle DBAs

One of the things I love about VMware’s support site is their knowledge base. It’s not horrific flash like My Oracle Support (aka Metalink), and it’s freely searchable without a support contract. Also very cool is there is an RSS feed of new or updated knowledge base articles. It’s good to scan in an idle moment or two each day to have an idea of what issues other people are seeing.

Because of that RSS feed, I came across three knowledge base articles I’d like to highlight here:

KB Article 1023696: Oracle 11G R2 32 bit client fails with a segmentation fault when run in a RHEL 5.4 64 bit virtual machine

In this case, sqlplus would seg fault when you try running it. The issue it turns out isn’t a VMware issue – it’s an Oracle bug when running 32-bit 11gR2 client on a 64-bit RH OS with an AMD processor. The fix is Oracle patch 8670579.

KB Article: 1023898
RedHat and CentOS virtual machine show warning messages when starting the udev daemon

This issue actually cropped up in my VMware environments awhile ago. Basically you see messages like this when your VM starts:

udevd[572]: add_to_rules: unknown key ‘SUBSYSTEMS’
udevd[572]: add_to_rules: unknown key ‘ATTRS{vendor}’
udevd[572]: add_to_rules: unknown key ‘ATTRS{model}’
udevd[572]: add_to_rules: unknown key ‘SUBSYSTEMS’
udevd[572]: add_to_rules: unknown key ‘ATTRS{vendor}’
udevd[572]: add_to_rules: unknown key ‘ATTRS{model}’

On RHEL, the fix is to do the following

vi /etc/udev/rules.d/99-vmware-scsi-udev.rule

change

ACTION==”add”, BUS==”scsi”, SYSFS{vendor}==”VMware, ” , SYSFS{model}==”VMware Virtual S”, RUN+=”/bin/sh -c ‘echo 180 >/sys$DEVPATH/device/timeout'”

To:

ACTION==”add”, BUS==”scsi”, SYSFS{vendor}==”VMware ” , SYSFS{model}==”Virtual disk “, RUN+=”/bin/sh -c ‘echo 180 >/sys$DEVPATH/device/timeout'”

and then reboot the VM.

The final article I want to mention is
KB Article: 1023185 VMware Tools installation fails to start the guest operating system daemon on Red Hat Enterprise Linux 4 64-bit guests with the 32-bit glibc-common package installed

This issue relates to Oracle because 32-bit glibc-common is frequently required for Oracle DB installs. The issue occurs because VMware tools configuration is looking for the 64-bit tools (64-bit OS, generally you’d want to install the 64-bit RPMs…). The solution is to install VMware tools as normal, but before running the configuration script, to issue
ln –s /usr/lib/vmware-tools/lib64/libdnet.so.1/libdnet.so.1 /lib64/libdnet.so.1
ln –s /usr/lib/vmware-tools/lib64/libproc-3.2.7.so/libproc-3.2.7.so /lib64/libproc-3.2.7.so
and then run the configuration program for vmware tools
/usr/bin/vmware-config-tools.pl

Hopefully this is helpful to other Oracle on RHEL under VMware people out there.

Oracle VM compared to VMware vSphere: Part 1

I’ve been meaning to take a serious look at Oracle VM for a few months. In fact, it was this post [Live Migration of EBS Services Using Oracle VM] (and my long-winded reply) that was a major push for me to start this blog.

The final bit of impetus to learn all about Oracle VM came a few months ago when I saw the “Oracle VM for x86 Essentials” beta exam. If passed, you earn the certification “Oracle VM for x86 Certified Implementation Specialist”. It’s a certification geared for Oracle Partners. I figured the knowledge could help me to better understand Oracle’s offering. First and foremost, I’m an Oracle Applications DBA. If Oracle’s product could allow me to better serve my clients and do my job – awesome!

So I’ve been hitting all the Oracle VM resources I could find to learn about the product. I’ll post links to a number of the excellent resources I found at the end of this post. All the links at the bottom refer to information on the currently available product (Oracle VM 2.2). While compiling all of this information, I came across [Oracle Virtualization:Making Software Easier to Deploy, Manage, and Support] – a slide deck from a recent Sydney Australia Oracle meetup. It talks about upcoming features of Oracle VM 3.0. If those features come to pass, Oracle VM will become more enticing to many organizations.

Honestly, I’ve got *tons* of things I want to write about with regards to Oracle VM — so much that I don’t know where to begin.

General Impressions
Remember that first time you went from something with a nice GUI, like Windows (Thanks Apple Microsoft!) to something a little more “nerdy” like Linux ? The GUI, if there was one, was stripped down and clunky. Many of the things you could do with a couple of mouse clicks before now require specialized commands at a command line. All of these different steps you need to do just to get things working. Well, it’s the same type of thing going from VMware vCenter to Oracle VM Manager. It’s not that the product is bad — it isn’t. The Oracle VM interface is clunky and the product doesn’t have the richness of features of VMware vSphere. Simple as that. Are those differences worth it to you? Everyone’s needs are different. Both underlying products (Oracle VM Server, VMware vSphere ESX 4.0) run Linux and Windows VMs well enough for most enterprise-level systems.

As you can read in this Gartner report on Server Virtualization Infrastructure, VMware is the clear market leader. Oracle VM, although categorized as a niche player, is the strongest of the niche players and right on the border of being listed as a challenger to VMware.

Here are a few areas where Oracle VM has an advantage over VMware:

o Certified vs. Supported

(I hate talking about this but it needs to be addressed.) Is your VMware virtualized Oracle database supported by Oracle? YES. Is it Certified? No. I went into this in detail in this [Oracle Support on VMware] blog post so I won’t do it again. Short of running Oracle RAC, which is expressly NOT supported when virtualized under VMware, the question of whether you should care about the “certified” distinction is something each company needs to answer for themselves. To me, the whole thing smacks of FUD (Fear, Uncertainty, Doubt).

o Pricing

There are two parts to pricing. First is the effect virtualizing Oracle Database will have on your Oracle database licensing. I go into this in more detail in a post on Oracle licensing under VMware. One of Oracle VM’s main selling points is that Oracle considers Oracle VM (through hardcoding the CPU binding in the vm.cfg file) a type of hard partitioning and VMware vSphere a type of soft partitioning. When using hard partitioning, Oracle only requires you to license the processors (cores) in that hard partition (aka, the processors visible to the VM). When using soft partitioning, Oracle requires you to license ALL the processors (cores) in the server, even though there may be many more processors present than allocated to the VM. It should be noted that you can do the same type of CPU binding (called CPU affinity) with VMware vSphere, but that Oracle somehow still considers this soft partitioning.

This just seems like a way for Oracle to give their Oracle VM product preferential treatment. How does the joke go… Where does the 800 lb gorilla sit? Anywhere he wants to.

The second part of pricing involves the actual Oracle VM product versus the VMware vSphere product. Oracle basically has two pricing points
o Premier Limited — Up to 2 CPU sockets, regardless of the number of cores per socket in the physical server
o Premier — unlimited CPU sockets in the server

VMware, unlike Oracle, has four product feature levels (Standard, Advanced, Enterprise and Enterprise Plus) and so a head to head comparison is a complete pain to do. The short answer is that Oracle can be significantly cheaper.. The downside of this inexpensiveness is a lack of features. Yes, VMware generally costs more than Oracle, but you’re paying for additional features. Are those features worth it to your organization? That’s for you to decide. In my organization, we are willing to pay for VMware’s features, but my organization’s needs may be different than the needs of your organization.

Does your organization have a need for VM snapshots? Mine absolutely does. Oracle VM doesn’t have it and VMware does, even when you’re using the free version of each product.

Does your VM require more than 8 CPUs? VMware has a limit of 8 CPUs for a single VM. Oracle VM’s limit is 32 CPUs for a single VM. Through tuning and software improvements, my main client has managed to reduce the number of CPUs for our Production Oracle E-Business Suite database from an unvirtualized 8 cores to 2 cores virtualized, so the difference is immaterial… but maybe your organization needs 20 cores.

Does your organization have a need to do vMotions / Live Migrations? They come included with Oracle VM, but it’s not recommended to do more than one at a time. There is an additional cost to get VMware vMotion, but VMware supports a default configuration with up to 4 simultaneous moves and allows up to 8 simultaneous moves.

Does your organization need automated SAN level replication of your VMs so they can be brought up automatically in case of disaster? VMware has that functionality with Site Recovery Manager. Oracle doesn’t have anything like it.

o Oracle VM Templates

Do you want a pre-built VM you can download with the Oracle software already installed and configured? Oracle offers downloads of pre-built environments from a basic OEL 5 Linux box all the way through to a downloadable 38GB Oracle EBS R12.1.1 system. I admit, that could be pretty cool. However, it may not be right for your company. My main client never allows consultants to have console type access to our Linux servers. I don’t think my auditors would approve of a pre-built VM for production use, even if it was pre-built by Oracle. As something for quickly throwing up a demo or dev environment, I think it’s fantastic. I hope Oracle continues to do this for more and more of their products. Oracle Enterprise Manager 10gR5 took me roughly 2 weeks to install. Discoverer 11g about a week. Secure Enterprise Search about 2 weeks. It would be great to have a pre-built test system I could reference when building my production systems.

I’ve got numerous ideas for more blog posts with regards to Oracle VM. Feedback directing me to what interests others would be great.

Part 2 coming after the I take the exam later this week. Wish me luck!

Links
Live Migration of EBS Services with Oracle VM
Installing & Configuring OEL 5 with Database 11gR1 as a Paravirtualized Machine (PVM) on an Oracle VM Server
The underground Oracle VM Manual
Official Oracle VM Wiki home page
Oracle VM for x86 Essentials Exam 1Z0-540 Exam Topics Study Guide
Oracle VM 2.2 Documentation Library
Performing Physical to Virtual (P2V) and Virtual to Virtual (V2V) (aka VMware to Oracle) conversions Note the excellent pdf linked from the article too.
Installing, Configuring and Using Oracle VM Server for x86