Oracle Database Appliance Patch Update error when default gateway is not pingable - DCS-10001 and PRGO-1664

When setting up an Oracle Database Appliance (ODA) the configuration of the network with odacli configure-firstnet does check, if the default gateway is pingable. If it is not, the configuration is not possible. 

But one of our customers (or better the network administrators) later configured the environment (router), that pings are not responded, which means, they time out (this is part of the security strategy of this company). The network does work, but the ODA can not ping the default gateway anymore (no one can).

As we tried to patch an ODA from ODA Release 19.21 to 19.24 we therefore run into an ugly error while the Grid Infrastructure was upgraded from 19.21 to 19.24, leaving us with two unzipped home directories:

"DCS-10001:Internal error encountered: Failed to set ping target on public network."

If you take a look into the dcs-agent.log, you can find a more detailled error message:

DEBUG [Server patching : JobId=<something>] [] c.o.d.a.r.s.g.GiOperations: ping failed: unknown gateway <IP-Address of Gateway>

The creation of the prepatch report for the server was fine ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.24.0.0.0). It finished successful without error, so we didn't expect that we run into an error later on. 

But while running the server patch ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-server -v 19.24.0.0.0) we got the above error. 

We asked the network team to fix the ping problem, tested the ping manually and restarted the odacli update-server -v 19.24.0.0.0 a second time. Sometimes, when an ODA update stucks, it is fine to just restart the update-server again, but this time we got another error:

 “DCS-10001:Internal error encountered: Failed to patch GI with RHP : DCS-10001:Internal error encountered: PRGO-1664 : The specified source working copy \"OraGrid192100\" is a software-only working copy...”

Thanks god we did an "odabr backup -snap" before we started the update and by running "odabr restore -snap -force" we were able to restore the "before" image of the operating system stuff. If you start odabr restore without the force option, you will get a warning "Clusterware patchlevel at backup time is not what you have now. ODABR can't be considered a tool to perform Grid Infrastructure downgrade" and the snap will not be restored.

Unfortunately, the grid infrastructure is not part of the odabr backup with newer ODA software releases, which means, we still had two Grid Infrastructure homes and the problem with the working copy as a software only working copy. 

Searching in support.oracle.com does not really help, searching for PRGO-1664 does not result in useful hits. But there is a small section in the note for the odabr tool (2466177.1). "if a GI update was in progress and you have done a restore", it may happen that the executable permissions are set wrong. 

To fix this, set the GRID_HOME to the old grid environment as user root and run [root@oda1]# GRID_HOME/crs/install/rootcrs.pl -lock

This will change the executable permissions - in our case in the 19.21 grid directory back and after this change patching including the odacli update-server -v 19.24.0.0.0 was running fine. 

Depending on the time you did the odabr backup, you need to start with patching the dcs stack or running the prepatch-report creation again. 

For the next ODA software versions, there is an enhancement request placed for the development, so the gateway ping check is then part of the prepatch report and will not lead to a corrupted GI update.

In the meantime it could be helpful to test manually, if the default gateway is pingable. To find out the gateways ip address you can run a "ip route show" command. It outputs a line "default via <IP Address>". This ip address is your gateway - a "ping <IP Address>" will (hopefully) work. If so, you will not run into this problem, we got. We will add the ping test to our manual check list, it is a 10 second work and it will save an hour work later on.

 

Register of a standby database fails (within Oracle GI/repository) on Oracle Database appliance

A couple of days ago I tried to register a standby database on ODA with a newer relase of ODA (in my case 18.8) and Grid Infrastructure. I followed the procedure of the Dbvisit Standby documentation.

In there, it is described that you need to start the database on the standby environment as "read only".

Afterwards, you should be able to register the database in the Grid Infrastructure using odacli.

The procedure is described at this manual in detail and should work with Dbvisit Standby Version 8 and 9.

Unfortunately, if you run the ODACLI to register the database, it fails.

odacli register-database -c OLTP -s odb6 -t SI -sn <mydb.mydbdomain.com> -p <somepassword>

The job at the ODA is started, but it errors out:

Job details
----------------------------------------------------------------
ID: 432c4c54-0d2f-XXXX-XXXX-aa15f27ea3ab
Description: Database service registration with db service name: <mydb.mydbdomain.com>
Status: Failure
Created: May 5, 2020 3:40:06 PM CEST
Message: DCS-10001:Internal error encountered: Failed to archive log all
Session altered.

alter system archive log current
*
ERROR at line 1:
ORA-01109: database not open

This also means, that you see the "failed" configurations in odacli list-databases.

I have created a Service Request with Dbvisit to check the manual/the process of registering the database on ODA. At the meantime, the only solution is to open the database "read/write" to register it at the ODA.

To open the standby read/write it's necessary to "switchover" the primary to the standby you want to register and run odacli register-database on the -now- primary. Afterwards you can switch the roles of the databases again, so your original standby is a standby again at the end.

For your application that means, that it is necessary to have a small maintenance window where the end users can't work.

I will edit this blog post as soon as I do have more information, so come back regularly if you are interested in any news.

Creating KVM network bridge on ODA - Not able to connect to database anymore.

A lot of people are using Oracles KVM solution on ODA (Oracle Database Appliance). My company e.g. runs something like application servers in a Linux VM on the ODA lite models for our own software solutions (like communication server, etc.) if a customer runs ODAs (we call it "Solution-in-a-Box"). But there are also other customers, where we just act as system integrator, who want to use KVM on the ODA.

There is a really nice blog how to enable and use KVM on ODA, the starting point is this blog post by Tammy: kvm-on-oda.

It is straight forward, but one of the things are a little bit crucial. It is how to make the network configuration for the KVM on ODA. The best solution is to BRIDGE the network. Ruggero has written a blog post as part of Tammys blog, how you can enable all the different network types.
Don't use NAT or MacVTap - just follow the configuration steps for "Bridged networking (aka "shared physical device")".

Be sure you have access to the iloms host redirection function, because if there is any misconfiguration with the bridge, you lose the network connectivity und you are not able to connect internal (like you could do with ODA HA).

If you have followed the configuration steps and if you can connect to the ODA host again with the bridge configuration, you are not able to connect to the database(s) on that host anymore. Why? Because one mandatory step is missing at Ruggeros example: The configuration of the clusterware (as grid user)! Some tried to stop and start the listener, but the listener will not be started and errors out.

What you need to do as a last configuration step is to modify your clusterware network configuration. The listener is bound to Network 1, which can easily seen by issueing

$ srvctl config listener
Name: LISTENER
Type: Database Listener
Network: 1, Owner: grid
Home: <CRS home>
End points: TCP:1521
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:

To check the configuration, run the following command:
$ srvctl config network
Network 1 exists
Subnet IPv4: 10.214.0.0/255.255.248.0/btbond1, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:

As you can see, the network is still configured to btbond1 instead of pubbr0.
The syntax (I use the same subnet in my example as Ruggero in his blog) to change this is:
$ srvctl modify network -netnum 1 -subnet 10.214.0.0/255.255.248.0/pubbr0 

Now it is best to either restart the ODA to check if everything is running fine also after an reboot or at least to restart the listener.
$ srvctl stop listener
$ srvctl start listener

Then you are done folks, you can use the bridged device now for your KVM and the databases are reachable again.

DBT-00007 User does not have the appropiate write privilege when starting dbca

At the moment I am preparing some virtualboxes to create the live demos for my presentation about "Single Tenancy is only more complexity!?" at the DOAG Conference 2018.

For this, I have setup a fresh new Linux and installed 18c Grid Infrastructure into it (using role separation, so GI is installed with the grid user, the databases should be installed with the oracle user).

After I have created my 2 ORACLE_HOMEs with the Standard Edition 2 and the Enterprise Edition database (Software only) I wanted to set up the first databases, but unfortunately the dbca (Database configuration assistant) struggled with "a [DBT-00007] User does not have the appropriate write privilege when starting dbca" error. Since I NEVER have seen that error before, I really was wondering, what happens (and I don't think I have missed some steps). And no, I don't want to check this with the Oracle support, like mentioned at the popup window details.

I have setup my ORACLE_BASE and ORACLE_HOME before I have started dbca, so this isn't the cause for this error. As the popup window does not shown anything in addition, I tried to check the log of the dbca at $ORACLE_BASE/cfgtoollogs/dbca - but there wasn't any.
So the problem seemed to by correlated to this directory.

The owner of $ORACLE_BASE/cfgtoollogs was grid (group oinstall) and the rights set were 755 - so only the grid user was allowed to write, group members weren't.

To change this I have first changed the owner of the cfgtoollogs directory:

chown -hR oracle:oinstall /u01/app/oracle/cfgtoollogs
and afterwards I have given write rights to all subdirectories, so also the grid user would still be able to add his stuff to the log directory:

chmod 775 /u01/app/oracle/cfgtoollogs/
chmod 770 /u01/app/oracle/cfgtoollogs/dbca
chmod 770 /u01/app/oracle/cfgtoollogs/asmca
chmod 770 /u01/app/oracle/cfgtoollogs/netca

That's it folks, the dbca started after I have made these changes.

Lost entries at /etc/oratab with Oracle 12.2 / 12c R2 / 18c

I was working at a customer database environment, setting up nearly 30 new databases with oracle 12.1 and 12.2 Standard Edition and Enterprise Edition version. For the datafiles we used ASM with 12.2 Grid Infrastructure for Single Instances (with Restart) and the newest version we got(April 2018).

One of the things I was really going mad was, that some entries at oratab were gone after stopping and/or starting the database with srvctl or by rebooting the nodes. .

First I was thinking about a bug at ASM, because there was also one at 12.1.
But after a while I found a nice note at https://support.oracle.com which states, that this is a new default behavior and not a bug.

In detail, starting with 12.2.0.1.171017 GI RU/PSU (or any version later), the oratab entries are removed automatically, if they have been "added by agent".

The note "Applying 12.2.0.1.171017 RU (patch 26737266) To 12.2 Cluster Removes Oratab Entries (Doc ID 2329359.1)" tells us, that "The agent deletes the oratab entries that have comment "# line added by Agent". For example, when the GI is recycled (including after a node reboot), then the agent will remove the oratab entries that have the comment "# line added by Agent"".

Unfortunately, it was not enough to just remove the comment "# line added by Agent", like it is written at the document - we needed to put own comments "# manually added" at the end of the lines.

Why is Oracle doing this as new default behavior? "With the introduction of the flex ASM (asm may not run on all nodes) and policy managed databases (no instances is tied to a server), the mapping of the asm and RAC database instances to a specific node is no longer guaranteed to be valid, so this is the reason that the oratab entries are being removed."

So if you run into this problem with GI version 12.2 or later (I think this will also happen with Oracle Database Grid Infrastructure 18c) and you want to keep the entries, just add your own comment at the end of every line you want to keep.

If you don't know it, you are going mad, if you know it - life can be easy. 

P.S. If you need to find the names of the databases, this blog entry could be helpful: https://fritshoogland.wordpress.com/tag/recreate-oratab/