Oracle Database Appliance Patch Update error when default gateway is not pingable - DCS-10001 and PRGO-1664

When setting up an Oracle Database Appliance (ODA) the configuration of the network with odacli configure-firstnet does check, if the default gateway is pingable. If it is not, the configuration is not possible. 

But one of our customers (or better the network administrators) later configured the environment (router), that pings are not responded, which means, they time out (this is part of the security strategy of this company). The network does work, but the ODA can not ping the default gateway anymore (no one can).

As we tried to patch an ODA from ODA Release 19.21 to 19.24 we therefore run into an ugly error while the Grid Infrastructure was upgraded from 19.21 to 19.24, leaving us with two unzipped home directories:

"DCS-10001:Internal error encountered: Failed to set ping target on public network."

If you take a look into the dcs-agent.log, you can find a more detailled error message:

DEBUG [Server patching : JobId=<something>] [] c.o.d.a.r.s.g.GiOperations: ping failed: unknown gateway <IP-Address of Gateway>

The creation of the prepatch report for the server was fine ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.24.0.0.0). It finished successful without error, so we didn't expect that we run into an error later on. 

But while running the server patch ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-server -v 19.24.0.0.0) we got the above error. 

We asked the network team to fix the ping problem, tested the ping manually and restarted the odacli update-server -v 19.24.0.0.0 a second time. Sometimes, when an ODA update stucks, it is fine to just restart the update-server again, but this time we got another error:

 “DCS-10001:Internal error encountered: Failed to patch GI with RHP : DCS-10001:Internal error encountered: PRGO-1664 : The specified source working copy \"OraGrid192100\" is a software-only working copy...”

Thanks god we did an "odabr backup -snap" before we started the update and by running "odabr restore -snap -force" we were able to restore the "before" image of the operating system stuff. If you start odabr restore without the force option, you will get a warning "Clusterware patchlevel at backup time is not what you have now. ODABR can't be considered a tool to perform Grid Infrastructure downgrade" and the snap will not be restored.

Unfortunately, the grid infrastructure is not part of the odabr backup with newer ODA software releases, which means, we still had two Grid Infrastructure homes and the problem with the working copy as a software only working copy. 

Searching in support.oracle.com does not really help, searching for PRGO-1664 does not result in useful hits. But there is a small section in the note for the odabr tool (2466177.1). "if a GI update was in progress and you have done a restore", it may happen that the executable permissions are set wrong. 

To fix this, set the GRID_HOME to the old grid environment as user root and run [root@oda1]# GRID_HOME/crs/install/rootcrs.pl -lock

This will change the executable permissions - in our case in the 19.21 grid directory back and after this change patching including the odacli update-server -v 19.24.0.0.0 was running fine. 

Depending on the time you did the odabr backup, you need to start with patching the dcs stack or running the prepatch-report creation again. 

For the next ODA software versions, there is an enhancement request placed for the development, so the gateway ping check is then part of the prepatch report and will not lead to a corrupted GI update.

In the meantime it could be helpful to test manually, if the default gateway is pingable. To find out the gateways ip address you can run a "ip route show" command. It outputs a line "default via <IP Address>". This ip address is your gateway - a "ping <IP Address>" will (hopefully) work. If so, you will not run into this problem, we got. We will add the ping test to our manual check list, it is a 10 second work and it will save an hour work later on.

 

How to start preparing your database for the upgrade to Oracle database 23ai - even without an on-premises released database software!

 As many of you already know, Oracle's autoupgrade tool will be the only supported option for database upgrades in the future. This fact can also be found in the support notes of the 23ai database.

However, the autoupgrade tool - and most people don't know this - already supports the upgrade to Oracle database version 23ai in Release 24 (available since roundabout March of this year). The question therefore arises, how can you TODAY use the autoupgrade tool to optimally prepare your existing databases for the upgrade to 23ai ? Especially now, when the perhaps quiet time of the year is approaching, but the official release date of 23ai has been postponed to “sometime in calendar year 2025” (postponed once again)?

The preparation

First of all, the latest version of the autoupgrade.jar must be downloaded from Oracle Support and can be found at Document ID 2485457.1.
This is currently a build from October 2024 (24.7.241021). The autoupgrade.jar can next be copied to the server and tested for functionality. If no Java is installed on the server, the JDK from the highest Oracle Home release can be used. This has always worked so far.
Oracle's support for direct upgrades to 23ai is limited to 19c and 21c databases, this means older database versions cannot be prepared for a direct upgrade to 23ai using the autoupgrade tool. An intermediate upgrade (to 19c) is therefore necessary beforehand or, for example, data migration with Datapump Export-Import is a possible way to move from older database versions to 23ai. However, no pre-test can be one with datapump.

If the execution of autoupgrade.jar (java -jar autoupgrade.jar -version) works on the server, the next step is to create a config file for the database that is to be prepared for the 23ai upgrade.

The following information is important here:

• the log directories (global and per DB)
• Which SID should be tested/analyzed
  
• The source Oracle Home - this needs to be a 19c or 21c database home
• The target Oracle Home - is set, but this directory does not exist - the database is not yet available on-premises
• Start_time and upgrade_node can be set by default (NOW/localhost)
• Important - the autoupgrade tool normally takes the target_version from the target home. However, as this does not exist, it is important to set 23 fixed as target_version here.

 

 

Config file

 Analyzing the existing database

Once this preparation has been completed, the autoupgrade tool can now be run in Analyze mode against the source database. NO changes are made to the source database, this means this procedure is completely safe.
However, it is important that all pluggable databases (PDB) are open. PDBs that are in mount status are not analyzed.

If all PDBs are open, there is no longer a warning and the corresponding autoupgrade job starts analyzing the existing database.

The “status” command can be used to display the progress of the analysis of all jobs on the Autoupgrade CLI. It shows 4 databases (1 CDB + Seed PDB + 2 user PDBs).

To see the details of the individual databases, just add the job with number to the “status” command. Now one can see the stage progress per container (the 4 databases).

The job should be successfully completed after a few minutes. There are typically no errors in the analysis phase.

The analysis result (in various formats, including JSON, text and HTML) is now available for the analyzed database in the Prechecks subdirectory of the log directories for the job. The next picture does show the HTML.

The first overview already shows that there are components in the 19c database that would be removed during an upgrade to 23ai. These are remnants of Oracle Multimedia - marked yellow.

If one now looks in the detailed areas of the databases (CDB and PDBs), one will find valuable information on how to deal with such changed or removed components. The Autoupgrade tool recommends taking a look at various support notes, for example.

In the database analyzed here (which was already upgraded from a 12c environment), the autoupgrade tool also found a configuration of traditional auditing. Traditional auditing was first supplemented by Oracle some time ago with Unified Auditing and then replaced - or is now officially desupported for the first time, so that Unified Auditing must be switched to.
As the configuration of auditing can be a very time-consuming process, it is advisable to adapt the configuration on the existing database even before the upgrade planning officially starts after 23ai.

The autoupgrade tool may already find things as part of the POST check, things which then need to be  done on the 23ai database, such as changing the LOB type at an audit table. Of course, this cannot yet be prepared in the existing environment, but it does provide an indication of any manual work that may need to be done (and maybe any additional downtime) during a later upgrade.

Incidentally, even though the screenshots are now all from a Linux environment, the analysis method was also tested with a 19.8 database under MS Windows. It should therefore be possible to use this procedure on all supported operating systems.

Now start prepare your database for a successful upgrade to database version 23ai.

Oracle Database Appliance Upgrades from 18.8 to 19.9 - Good to know

 Hi,

I recently have upgraded some ODA lite servers from the appliance software 18.8 to 19.6 and afterwards to 19.9 and, as usual, there are some findings I can and want to share with you, guys.

Typically I do see ODA upgrades by other Oracle consultants running perfectly - or with some small little issues. Luckily, I can learn more if things do fail and one is doing a deep dive into the root causes. And again, we have learned a lot this time - and maybe it can help you preventing to learn things by yourself. 

Let's start with findings for the upgrade from appliance software 18.8 to 19.6. 

1.) ODABR

 ODABR is the Backup and Recovery tool for the Bare Metal ODA and available at myoraclesupport. It is already mentionend in the documentation that it must be installed before the upgrade. Unfortunately, it is not part of the upgrade software itself, so one has to download and install it. It is really useful and you should use odabr backup -snap BEFORE every upgrade! I had to restore the system using odabr at 19.6 to 19.9 server-upgrade. 

For the OS upgrade, odabr is doing the snapshot by itself, so there is no need to do one manually before.

Check for space issues if you have resized the /u01 partition using lvm - you are able to use odabr manually (it is also documented at the patch documentation) with smaller sizes even if the automatic snapshot fails.

2.) ld-linux.so.2 segfault at 0 ip XXXXX errors at the Pre-Check for OS Upgrade  (odacli create-prepatchreport)

As you may know, the ODA is running on Oracle Enterprise Linux 6 with the appliance software 18.8 and with Oracle Enterprise Linux 7 with the appliance software 19.6. The pre-check is originally an upgrade tool from Red Hat and collects a lot of data. 

While the job is running, I have seen different segmentation fault errors - all in ld-linux.so.2 at the console and at the linux log. The job itself is running successful. If you check all other Linux logs (like /var/log/messages, the dcs-agent.log, etc.) this segfault can not be found for any "normal" ODA operation. Just for the precheck itself.

With help of Oracle we have seen that it is safe to proceed to upgrade the OS at this point. All libraries of Oracle Linux 6 will be replaced with Oracle Linux 7 ones. By the way - I have seen some heartbeat messages at the console after the upgrade to 19.6 - but they are gone after the upgrade to 19.9.

3.) Network issues after OS Upgrade (sfpbond vlans down)

Our customer is running with two different vlans using a public bridge (as they do use KVM) over the sfpbond. The upgrade itself was fine. The sfpbond1.200 and sfpbond1.3001 vlan were down after the upgrade. They were missing an "ONPARENT=yes" at the ifcfg-scripts. After adding this parameter, the bond was working fine with the vlans. OEL 6 didn't need these to run, but this was a misconfiguration made by ourselves (as the deployment with 12.1 wasn't supported with vlan/kvm by odacli).

4.) After the OS Upgrade, do a new ODABR backup - I didn't needed it, but you should have it. 

Next to the 19.6 to 19.9 upgrade.

1.) It is essential that you DO a ODABR backup before you start with the upgrade to ODA appliance software 19.9. The prepatch-report WILL fail (at least at my ODAs it fails at a 100% rate), so don't get frightened about it. There are two different things you also can see here at the screenshot of my describe-prepatchreport output:

The first is that the validate clones location exist check fails due to a grid home bundle patch not found. /opt/oracle/oak/pkgrepos/orapkgs/clones/grid19.tar.gz. This error can be safely ignored as this package is used for the upgrade to 19.6 only and you may/surely have made an odacli cleanup-patchrepo before ;-).

The second thing is that the orachk validation failed and as a consequence also the Software home check failed. This is something already known. You have to use the "-sko" (which means skip orachk) parameter for your odacli update-server run. 

2.) Another issue raised up at the upgrade server state. The ILOM patch was not successful. I had to revert to the last ODABR snapshot, to reboot the server, to create a new snapshot backup with ODABR and to patch the ilom manually with the right ilom patch. The reason why this happens is that Oracle has changed something at the deployment of the ilom patch. You now need to have port 623 open (there is a support.oracle.com note for this) between the public ODA interface and your Ilom IP addresses for IPMI over UDP. 

I have checked that with the customer - they say the port is open and nc - well, see yourself:

[root@YYY ~]# nc -z -v -u 172.17.X.XX 623

Connection to 172.17.X.XX 623 port [udp/asf-rmcp] succeeded!

But the ILOM upgrade still fails. The root cause for this is unknown as the appliance should try two different ways. The first is using IPMI over UDP, because it is faster, if this fails the second try should use the internal bus, but it stops before the second way is tried.

3.) KVM machines are not running 

Yes, after you have upgraded to 19.6 or 19.9 your 18.X VMs will not run if you use 

virtsh start edm38 (where edm38 is my machine name)

=> error: Failed to start domain edm38

=> error: Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory

The reason for this problem is clear - qemu has changed with Oracle Enterprise Linux 7. The old qemu-kvm was replaced by qemu-system-x86_64. Also the machine types have changed (e.g. rhel6.6.0 does not exists anymore).

You need to edit the virsh configuration of your machines. Use virsh edit as vi does not work with the xml description files. 

First select the machine type you want to use by running qemu-system-x86_64 -M ?

Personally I have picked just "pc" as it is an alias of the newest pc-i440fx-3.1 machine type. 

Second find all your machine names with virsh list --all (virsh list does show only running VMs):

Third run virsh edit MachineName and change the following lines:

FROM <type arch='x86_64' machine='rhel6.6.0'>hvm</type>

TO  <type arch='x86_64' machine='<YourMachineType'>hvm</type>

In my case I changed it to:

<type arch='x86_64' machine='pc'>hvm</type>

And also change the line  

FROM <emulator>/usr/libexec/qemu-kvm</emulator>

TO  <emulator>/usr/bin/qemu-system-x86_64</emulator>

Don't forget to change the folder to /usr/bin as qemu-system-x86_64 is not in /usr/libexec anymore.

Save the file and exit the editor - now you should be able to start the vm with virsh start MachineName.

4.) Creating a new database fails in the step of configuring the network (the database itself is created and configured, but the odacli create-database job failed with "Failed to associate network").
As I have only vlans on these machines, I think it will have to do something with that and that I have upgraded the servers from a 12.1 deployed more than 3 years ago. Maybe there isn't something configured like it is done with newer deployments using the VLAN commands provided these days.

I wasn't investigating that much, what the root cause of this error is. The database wasn't registered with the listener (and I wasn't also not able to add the database configuration and add a network with odacli), so I changed it manually

What was doing the trick on my database was to add the right "listener_networks" entry: 

alter system set listener_networks='((NAME=net1)(LOCAL_LISTENER=(ADDRESS=(PROTOCOL=TCP)(HOST=172.16.XXX.XXX)(PORT=1521))))' scope=both; 

These were my findings, I hope you don't get them. But if you get them, my little post will maybe help you out.